Scraping works on some sites but not others

This topic is: resolved

Thank you for contacting me. Please note that I live in the GMT+3 time zone - responses might be delayed by this.

This topic has 1 reply, 2 voices, and was last updated 1 year, 8 months ago by Szabi – CodeRevolution.

Viewing 1 reply thread

Author

Posts
- March 28, 2023 at 2:47 am #7211
  
  dmx
  Participant
  
  Post count: 21
  
  Scraping works on some sites but not others. Here’s an example of one that doesn’t seem to work. Any thoughts?
  
  https://www.dailykos.com/stories/2023/3/27/2160521/-The-Red-State-brain-drain-accelerates-Liberal-students-refuse-to-attend-college-in-a-Red-state
  
  Add New Note to this Reply
- March 28, 2023 at 5:15 am #7212
  
  Szabi – CodeRevolution
  Keymaster
  
  Post count: 4577
  
  Hello,
  
  First of all, thank you for your purchase.
  
  The URL you mentioned is using JavaScript loading of content and text, this is why the plugin is not able to extract the article.
  
  The only method to scrape this and also similar articles is to use a headless browser to render the page before passing it to the plugin for scraping. This can be done, if you install Puppeteer (a headless browser) on your server and set up the plugin to use it for scraping. Please check this tutorial video for help on installing Puppeteer: https://www.youtube.com/watch?v=pRUDcSOe724
  
  Also, this tutorial for use cases and examples: https://www.youtube.com/watch?v=g99IlDkt_SY
  
  As an alternative, the plugin also supports HeadlessBrowserAPI, which is a cloud based services, it is also able to render JavaScript, please check details: https://www.youtube.com/watch?v=rj-LOI-sc14
  
  Regards, Szabi – CodeRevolution.
  
  Add New Note to this Reply
Author

Posts

Viewing 1 reply thread

The topic ‘Scraping works on some sites but not others’ is closed to new replies.