Plugin not scraping properly

This topic is: resolved

 

Thank you for contacting me. Please note that I live in the GMT+3 time zone - responses might be delayed by this.

This topic has 1 reply, 2 voices, and was last updated 2 years, 4 months ago by Szabi – CodeRevolution.

Viewing 1 reply thread
  • Author
    Posts

    • July 28, 2022 at 3:14 am #5605


      shynes
      Participant

      We Tried to get plugin to scrape data, but it only does it periodically, and often stops after 8 records. Please help.  We got it to work a few times, the HTML does not change on the page where it should break. Don’t think such a small site has a “blocker” after 8 slow scrapes

      Edit Miners ‹ Junior Miners — WordPress - Google Chrome 2022-07-27 at 11.13.06 PM
      Scraping: https://ww3.northernminer.com/stocktables/byCompany.aspx?filter=w&sort=name

      See attachment

       

      Attachments:
      You must be logged in to view attached files.

    • July 28, 2022 at 6:56 am #5609


      Szabi – CodeRevolution
      Keymaster
      Post count: 4622

      Hello,

      First of all, thank you for your purchase.

      I checked the link you mentioned and the first think I want to point out is that it contains (in its listings) multiple links which don’t work. Example, in the listing: https://ww3.northernminer.com/stocktables/byCompany.aspx?filter=w&sort=name

      You will find multiple not working links, example: https://ww3.northernminer.com/esource/profile.aspx?company_id=083357410

      With this said, now lets cover the plugin settings which are needed to scrape this site:

      I used the below settings and I was able to grab multiple links from the source page (for me, grabbing 10 links in one go worked well):

       

       

      Scraper Start (Seed) URL / Keywords
      https://ww3.northernminer.com/stocktables/byCompany.aspx?filter=w&sort=name

      Do Not Scrape Seed URL:
      Checked

      Seed Page Crawling Query Type:
      Visual Selector

      Seed Page Crawling Query String:
      //*[@id=’ctl00_leftwideContent_results’]

      Crawled Pages Crawling Query Type:
      Disabled

      Content Query Type:
      ID

      Content Query String:
      ctl00_websiteContent_upnlProfileContent

       

      If the above settings are not helping, please send me the logs that appear in the plugin’s ‘Activity and Logging’ menu and I will check on them, there we might see more details why this is not working for you.

      Regards,

      Szabi – CodeRevolution.

       

       

       

      Regards, Szabi – CodeRevolution.

  • Author
    Posts
Viewing 1 reply thread

The topic ‘Plugin not scraping properly’ is closed to new replies.