Remembering last link crawled in TXT file

This topic is: resolved

 

Thank you for contacting me. Please note that I live in the GMT+3 time zone - responses might be delayed by this.

Tagged: 

Viewing 5 reply threads
  • Author
    Posts

    • January 6, 2023 at 5:29 am #6573


      steveg
      Participant
      Post count: 4

      Hello,

      I am testing out the feature to crawl a TXT file. I have a huge list of links set up to crawl for updating data. I set the rule active to test only 10 posts per hour. The rule works as expected, but every time the rule runs, it starts over from the first link in the list. I have set ‘Remember Last Paged URL‘ on, but it doesn’t start where it left off. Any suggestions?

    • January 6, 2023 at 5:54 am #6575


      Szabi – CodeRevolution
      Keymaster
      Post count: 4559

      Hello,

      First of all, thank you for your purchase.

      Please go to the plugin’s ‘Main Settings’ and check the ‘Remember Imported Links And Do Not Crawl Them Twice’ checkbox -> save settings.

      Let me know if this helped.

      Regards, Szabi – CodeRevolution.

    • January 6, 2023 at 4:43 pm #6576


      steveg
      Participant
      Post count: 4

      From what I can tell, this isn’t working either. It seems to be skipping everything that’s already posted and creating any new products that aren’t found. It won’t fetch the ones I want updated. I have ‘<b>Update Post If It Is Already Posted</b>’ active for the rule.

    • January 6, 2023 at 8:45 pm #6578


      Szabi – CodeRevolution
      Keymaster
      Post count: 4559

      If you want to update products, I suggest you use instead the automatic updating feature of the plugin, please check this tutorial video for details on this feature: https://www.youtube.com/watch?v=X4VsajtnkBQ

    • January 6, 2023 at 9:10 pm #6579


      steveg
      Participant
      Post count: 4

      This is very close to what I need. But I have precise data targeted using XPath that I need updated without overwriting anything else. Would be cool to see this feature per rule, when active. I’ll keep working to try and find a workaround of some sort.

    • January 7, 2023 at 8:15 am #6583


      Szabi – CodeRevolution
      Keymaster
      Post count: 4559

      I understand. Suggestion noted.

  • Author
    Posts
Viewing 5 reply threads

The topic ‘Remembering last link crawled in TXT file’ is closed to new replies.