Crawlomatic Not able to scrapping!

This topic is: resolved

 

Thank you for contacting me. Please note that I live in the GMT+3 time zone - responses might be delayed by this.

Tagged: 

This topic has 20 replies, 2 voices, and was last updated 5 years, 1 month ago by Omini.

Viewing 20 reply threads
  • Author
    Posts

    • November 5, 2019 at 2:52 pm #650


      Omini
      Participant
      Post count: 21


      Hello,
      I don’t know why the Crawlomatic not able to scraping this site! even im sure 100% my XPath i defined works perfectly!

    • November 5, 2019 at 2:53 pm #651


      Omini
      Participant
      Post count: 21

      error https://prnt.sc/pssgi4

    • November 5, 2019 at 4:44 pm #652


      Omini
      Participant
      Post count: 21

      the problem was when I add extra

      /

      Anyway now the problem with images http://prntscr.com/psu82a
      it’s not shown when I edit the dom manually like this it’s shown

      http://prntscr.com/psu8x6

      I already sent to your email my WordPress admin panel

    • November 5, 2019 at 6:12 pm #653


      Omini
      Participant
      Post count: 21

      Also, tags and categories not working I defined regex as expected but it’s not using any tags on the posts!

    • November 6, 2019 at 7:12 am #660


      Szabi – CodeRevolution
      Keymaster
      Post count: 4622

      Hello,

      First of all, thank you for your purchase.

      I checked your website and the issue was caused because the source site uses lazy loading for images.

      The plugin can handle also lazy loaded images, you have to use the ‘Lazy Loading Images HTML Tag’ settings field, in importing rule settings. I set this to ‘data-lazy-src’ for you, and now images are showing correctly. Please check this tutorial video for more details on this: https://www.youtube.com/watch?v=BMzJWZdodlo

      Regarding the issues with tags and categories, you also have to input a Regex delimiter for it to work.

      Example: instead of:

      <meta property=”article:tag” content=”(?<tag>.*?)”

      you should have:

      #<meta property=”article:tag” content=”(?<tag>.*?)”#

      I fixed this also, now tags and categories are importing correctly.

      Also, to enable category importing, you have to check also the ‘Add Post Categories From Query Strings (Below)’ checkbox from importing rule settings (checked it for you).

      Regards,
      Szabi – CodeRevolution.

    • November 6, 2019 at 7:13 am #661


      Szabi – CodeRevolution
      Keymaster
      Post count: 4622

      Also, if you haven’t already, I would recommend that you check also the plugin’s tutorial videos, here: https://www.youtube.com/watch?v=7BYDa72zY7c&list=PLEiGTaa0iBIgcqNzVBaoTCS4ws47vNMuQ

      Regards.

    • November 6, 2019 at 11:59 am #662


      Omini
      Participant
      Post count: 21

      Thank you.
      Sure but can u make more tutorials about crawlomatic plugin because it’s a very important plugin and it has many options we still not using it because we don’t know what it really does!

      Thank you for the amazing plugins.

       

    • November 6, 2019 at 12:09 pm #663


      Szabi – CodeRevolution
      Keymaster
      Post count: 4622

      Hello,

      Yes, I am working on more tutorial videos for the plugin, because it is the most complex plugin I built so far, and indeed, it needs more tutorials than it has right now.

      Please follow my YouTube channel for new videos.

      Regards.

    • November 6, 2019 at 12:42 pm #664


      Omini
      Participant
      Post count: 21

      Sure.
      I’m waiting for that 🙂

      Regards

    • November 6, 2019 at 1:58 pm #665


      Omini
      Participant
      Post count: 21

      And it’s much easier to start settings with the custom shortcode creator because we will start the crawling after we’re sure we defined custom shortcode creator correctly.

      I think the important options should move to the first options instead of what the order has now.

      What do you think?

    • November 6, 2019 at 2:04 pm #666


      Omini
      Participant
      Post count: 21

      And about the strip options, it’s better to have a new category for Stripping options

      instead of Other Post Customizations:

      http://prntscr.com/pt9ftq

       

      and for images options, it’s better to have all image options on a specific category instead on search on many places to find images options to edit

      http://prntscr.com/pt9he3

    • November 6, 2019 at 2:24 pm #667


      Szabi – CodeRevolution
      Keymaster
      Post count: 4622

      Great ideas! Thank you.

      I will make these changes in the next plugin update. 🙂

    • November 6, 2019 at 5:31 pm #668


      Omini
      Participant
      Post count: 21

      Thanks.
      Also about Do Not Scrape Start Url should be also one of the first options.

      Regards

    • November 6, 2019 at 5:39 pm #669


      Omini
      Participant
      Post count: 21

      About the Schedule can you change it to mins instead of hours?
      Sometimes we need to keep somethings updated every second or minute so hours its too much sometimes for content like for example BTC price it’s changed every second.

      Thank you

    • November 6, 2019 at 6:06 pm #670


      Szabi – CodeRevolution
      Keymaster
      Post count: 4622
      This reply has been marked as private.

    • November 6, 2019 at 6:15 pm #671


      Omini
      Participant
      Post count: 21

      Okay, just as a question, for now, every task is finished running the variables and the browser will be closed automatically from the server process?

      Like if the server using PhantomJs when the crawling is finished it will release the PhantomJs resources from the task manager(CPU)?

      Thanks

    • November 6, 2019 at 7:37 pm #672


      Szabi – CodeRevolution
      Keymaster
      Post count: 4622

      Yes, all resources are released when importing finishes in the plugin.

      Regards.

    • November 6, 2019 at 10:10 pm #673


      Omini
      Participant
      Post count: 21

      One problem I faced when trying to extract the TAG when using Regex-All Match that will give me the tags like this

      Tag1 Tag2 Tag3

      instead of
      Tag1, Tag2, Tag3

       

      Screenshot: http://prntscr.com/ptgen1
      The Tags Android, APPs I added it manually.

      May you try it your self and tell me if the wrong from the plugin or not?
      Thanks

    • November 6, 2019 at 11:08 pm #674


      Omini
      Participant
      Post count: 21

      Email sent.
      Thanks

    • November 7, 2019 at 8:52 am #684


      Szabi – CodeRevolution
      Keymaster
      Post count: 4622

      Hello,

      I tried to check on the tag related issue on your website, but I get this error when I try to log in:

      403 Forbidden

      Request forbidden by administrative rules.

       

      Regards.

    • November 7, 2019 at 9:00 am #685


      Omini
      Participant
      Post count: 21

      sent new details.
      Then I moved to a new domain that’s why you got this error

  • Author
    Posts
Viewing 20 reply threads

The topic ‘Crawlomatic Not able to scrapping!’ is closed to new replies.