Thank you for contacting me. Please note that I live in the GMT+3 time zone - responses might be delayed by this.
-
Posts
-
-
I’m crawling a url with infinite scroll. I am able to find a link at the bottom of the page that shows a filter tag at the end of the url. ?page=2 ?page=3 etc. Each page holds links to 20 product pages, so I am able to crawl and scrape 20 products at a time.
My question is this:
Is there a way to set up the seed url to automatically add the filter tag so that I can set it up to append the ?page=n+1 for x number of pages to the url so that I don’t have to do it manually?
Thanks
-
Hello,
First of all, thank you for your purchase.
Yes, this should be possible. Please use in the ‘Scraper Start (Seed) URL’ parameter, the following shortcode structure: %%counter_START_STOP_INCREMENT%%
Example: %%counter_1_2_1%% will be replaced by values 1 and 2 (start 1, stop 2, with increment)
Result: https://example.com/page/%%counter_1_2_1%%/ – will import https://example.com/page/1/ and https://example.com/page/2/
Regards, Szabi – CodeRevolution.
-
The topic ‘Page numbers to crawl’ is closed to new replies.