Google Search Result Crawling Not working

This topic is: resolved

 

Thank you for contacting me. Please note that I live in the GMT+3 time zone - responses might be delayed by this.

Tagged: 

This topic has 4 replies, 2 voices, and was last updated 4 years, 2 months ago by Szabi – CodeRevolution.

Viewing 4 reply threads
  • Author
    Posts

    • September 24, 2020 at 1:20 am #2005


      juntogroup
      Participant
      Post count: 2

      I have tried using class id “g” , “r” and “rc” for obtaining google search results with no luck. Please help. I was not able to accomplish this on the demo blog either but thought maybe it was a glitch that would be fixed when I installed it on my website.

       

      I followed the guide here to no avail: https://www.youtube.com/watch?v=l9sUu8rcwoc&ab_channel=CodeRevolutionTV

       

      Thank you for the help

       

       

    • September 24, 2020 at 4:47 am #2009


      Szabi – CodeRevolution
      Keymaster
      Post count: 4620

      Hello,

      First of all, thank you for your purchase.

      Unfortunately, Google search results page keeps changing and they keep updating the way results are provided. Also, they are trying to secure it better and better (maybe they also watch my videos, haha).

      Today, you can make this work only if you use phantomjs or puppeteer together with the plugin – these are headless browsers that will also be able to execute JavaScript on pages and get results from pages exactly like a real human would get (JavaScript is not executable by PHP coded plugins – this includes practically all WordPress plugins). Please check details about this in below videos:

      https://www.youtube.com/watch?v=g99IlDkt_SY

      https://www.youtube.com/watch?v=wWuI1mdIHwA

      https://www.youtube.com/watch?v=YzsJKJlP5gM

      https://www.youtube.com/watch?v=hnEPlQSeAZE

      Regards, Szabi – CodeRevolution.

    • September 25, 2020 at 9:11 pm #2017


      juntogroup
      Participant
      Post count: 2

      This didn’t work for me. Tried using both phantomjs and puppeteer but I am still stuck on the yellow triangle of death.

    • September 26, 2020 at 6:34 am #2018


      Szabi – CodeRevolution
      Keymaster
      Post count: 4620

      Well, this is interesting.

      This works on my localhost test site, however, on any of of my live websites, it is not working, however, I have the same settings and same plugin version.

      I assume that they are doing some kind of detection based on IP address? Really not sure what is happening here, I tried different user-agents, setting cookies also changing IP, but nothing worked on live servers. It works only on localhost, but on this IP I am also regularly browsing Google from my real browsers, so this might be why they are accepting connection from here.

      I will be looking more into this, if I find something out, I let you know.

      Regards.

       

    • September 26, 2020 at 6:44 am #2019


      Szabi – CodeRevolution
      Keymaster
      Post count: 4620

      Also, please note that you must uncheck the ‘Do Not Crawl External Links‘ checkbox in importing rule settings.

      Also, URL that you scrape should be in this format:

      https://www.google.com/search?q=mailer&oq=mailer&aqs=chrome..69i57j0l7.6496j0j7&sourceid=chrome&ie=UTF-8

  • Author
    Posts
Viewing 4 reply threads

The topic ‘Google Search Result Crawling Not working’ is closed to new replies.