Crawling a page with infinite scroll

This topic is: resolved

 

Thank you for contacting me. Please note that I live in the GMT+3 time zone - responses might be delayed by this.

Tagged: 

This topic has 3 replies, 2 voices, and was last updated 1 year, 3 months ago by Szabi – CodeRevolution.

Viewing 3 reply threads
  • Author
    Posts

    • September 12, 2023 at 10:25 pm #8622


      steveg
      Participant
      Post count: 4

      Hi, just a quick question – is the Crawlomatic plugin able to scrape a page with infinite scrolling? I was doing some testing and got stuck on a page with infinite scroll. I did try setting ‘<b>Seed Page Pagination Query Type</b>’ to ‘<b>auto detect</b>’. But it ends up trying to read other content on the page, in the footer. Just wondering if I missed a certain setting that might make it possible to continue scrolling.

    • September 13, 2023 at 6:50 am #8633


      Szabi – CodeRevolution
      Keymaster
      Post count: 4620

      Hello,

      First of all, thank you for your purchase.

      Loading posts listed in infinite scroll type pages is currently not possible, however, you can send me the URL you are trying to scrape and I will check on alternatives to scrape more posts from it. You can send the URL to my email address: kisded@yahoo.com – I will try to help.

      Regards, Szabi – CodeRevolution.

    • September 13, 2023 at 1:13 pm #8638


      steveg
      Participant
      Post count: 4

      Thanks for the info. It’s no problem. I use webscraper.io for that project and import results into WordPress. Just thought maybe I missed a setting in Crawlomatic when I tried it. I use Crawlomatic for other projects with no problems. Love it!

    • September 13, 2023 at 1:25 pm #8639


      Szabi – CodeRevolution
      Keymaster
      Post count: 4620

      I suggest you go for sitemap scraping as an alternative, if this is possible, as shown here: https://www.youtube.com/watch?v=xi1S1093ubo

      Regards.

  • Author
    Posts
Viewing 3 reply threads

The topic ‘Crawling a page with infinite scroll’ is closed to new replies.