Page numbers to crawl

This topic is: resolved

 

Thank you for contacting me. Please note that I live in the GMT+3 time zone - responses might be delayed by this.

Viewing 1 reply thread
  • Author
    Posts

    • March 29, 2024 at 9:10 pm #10327


      [email protected]
      Participant
      Post count: 12

      I’m crawling a url with infinite scroll. I am able to find a link at the bottom of the page that shows a filter tag at the end of the url. ?page=2 ?page=3 etc. Each page holds links to 20 product pages, so I am able to crawl and scrape 20 products at a time.

      My question is this:

      Is there a way to set up the seed url to automatically add the filter tag so that I can set it up to append the ?page=n+1 for x number of pages to the url so that I don’t have to do it manually?

       

      Thanks

    • March 30, 2024 at 6:03 am #10328


      Szabi – CodeRevolution
      Keymaster
      Post count: 4195

      Hello,

      First of all, thank you for your purchase.

      Yes, this should be possible. Please use in the ‘Scraper Start (Seed) URL’ parameter, the following shortcode structure: %%counter_START_STOP_INCREMENT%%

      Example: %%counter_1_2_1%% will be replaced by values 1 and 2 (start 1, stop 2, with increment)

      Result: https://example.com/page/%%counter_1_2_1%%/ – will import https://example.com/page/1/ and https://example.com/page/2/

      Regards, Szabi – CodeRevolution.

  • Author
    Posts
Viewing 1 reply thread

The topic ‘Page numbers to crawl’ is closed to new replies.