Site icon CodeRevolution Support

Page numbers to crawl

I'm crawling a url with infinite scroll. I am able to find a link at the bottom of the page that shows a filter tag at the end of the url. ?page=2 ?page=3 etc. Each page holds links to 20 product pages, so I am able to crawl and scrape 20 products at a time. My question is this: Is there a way to set up the seed url to automatically add the filter tag so that I can set it up to append the ?page=n+1 for x number of pages to the url so that I don't have to do it manually?   Thanks
Exit mobile version