Do Not Crawl External Links

This topic is: resolved

 

Thank you for contacting me. Please note that I live in the GMT+3 time zone - responses might be delayed by this.

Viewing 1 reply thread
  • Author
    Posts

    • November 7, 2019 at 11:20 pm #697


      Omini
      Participant
      Post count: 21

      <table class=”responsive cr_main_table_nowr”>
      <tbody>
      <tr>
      <td>
      <div> <b>Do Not Crawl External Links:</b></div></td>
      <td></td>
      </tr>
      </tbody>
      </table>
      Hello,
      This option is really ambiguous I really don’t know what it does exactly!

      http://prntscr.com/ptyyfo
      Thanks

    • November 8, 2019 at 5:21 am #698


      Szabi – CodeRevolution
      Keymaster
      Post count: 4182

      Hello,

      This feature is useful if you are crawling websites that have mixed links in their scraped content (internal and external).

      If you check this, the plugin will not follow links that are not from the crawled websites (external links) – it will crawl only links that belong to the website where you started scraping.

      If you uncheck this, you might get unexpected results from scraping, because if the plugin finds an external link, it might leave the website where you started crawling, and might start importing from a different website.

      However, in some use cases, disabling this comes useful.

      Regards,

      Szabi – CodeRevolution.

  • Author
    Posts
Viewing 1 reply thread

The topic ‘ Do Not Crawl External Links’ is closed to new replies.