Thank you for contacting me. Please note that I live in the GMT+3 time zone - responses might be delayed by this.
-
Posts
-
-
“Hello, I am KO.
I have two questions. I’ve purchased both Crawlomatic and Aiomatic this month, and I’m currently using them for blog testing. However, I’ve encountered an issue while attempting to crawl travel-related data.
The website I’m trying to crawl is called ‘getyourguide.’ I want to crawl the specific products from this website:
- https://www.getyourguide.com/sydney-l200/
- https://www.getyourguide.com/sydney-l200/from-sydney-blue-mountains-tour-with-waterfall-walk-lunch-t483885/?ranking_uuid=d3917904-8a66-434e-9afe-4269889d6911
The first URL serves as the seed URL, and the second one is intended for scraping information about a single product. However, even when I configure the ‘Content Query type’ to ‘visual selector’ in ‘Advanced Settings’ -> ‘Content scraping Customizations,’ the webpage displays only a white screen. I’ve also attempted to use ‘visual selector’ in ‘Crawling Restrictions,’ but it still results in a white screen. I’m perplexed by this issue.
Interestingly, it works perfectly fine on other websites. Despite watching 99 videos related to Crawlomatic that you’ve posted, I still can’t pinpoint what’s causing this problem. Please assist me.
Additionally, when I use Aiomatic for writing, automatic comments are being generated, and I can’t find an option to disable this feature in the settings. I’ve searched thoroughly but haven’t been able to locate it. Could you please provide guidance on how to disable this feature?”
-
Hello,
First of all, thank you for your purchase.
I checked and the specific site you are trying to scrape uses JavaScript to render its content. This makes regular scrapers not able to display or scrape their content.
However, Crawlomatic is able to do this, if you combine it with a headless browser, like Puppeteer, which will scrape the site content and render also JavaScript on it.
Tutorial video on how to install Puppeteer on your server: https://www.youtube.com/watch?v=pRUDcSOe724
Tutorial on usage with Crawlomatic: https://www.youtube.com/watch?v=ZljpMpmi_dU
or: https://www.youtube.com/watch?v=TXAdvsVCuy8
As an alternative, you can also use HeadlessBrowserAPI: https://headlessbrowserapi.com/
Regarding Aiomatic, please go to its ‘AI Content Editor’ menu -> ‘Editing Templates and Options’ tab -> Disable the ‘Automatically Add Comments To Posts’ settings field.
Regards, Szabi – CodeRevolution.
-
The topic ‘about crawlomatic assistance’ is closed to new replies.