Site icon CodeRevolution Support

Issue scraping sites where links appear abbreviated in code

Hi,   Thanks for your amazing plugin. One issue I keep running into is certain websites don't have links that contain the entire url (i.e. "www.example.com/example-article/"), and instead contain abbreviated links (i.e. just "/example-article"). The crawler always fails on these sites. Is there a way to prepend the URL of the website to these crawled links so they don't fail? An example site that the plugin struggles with is: https://www.iflscience.com/space-and-physics Sorry if this has already been answered. I tried searching but couldn't find anything! Thanks so much.  
Exit mobile version