“Crawlomatic Multisite Scraper Post Generator for WordPress” Help by CodeRevolution

---

“Crawlomatic Multisite Scraper Post Generator for WordPress Help”

---

Table of Contents

1. About the plugin
2. Wordpress installation
3. Plugin Settings
4. Summary
5. Sources and Credits

---

About this plugin - top

This plugin will allow you to build from scratch an autoblogging system, based on website crawling and scraping you define. This plugin provides one solution to the ongoing search for new blog content. It not only provides a steady stream of rich, searchable content, it also helps to keep your website up to date with the latest news. You can build rules that will generate posts that are near to unique in website crawlers's eyes. They require none to very little customization.
The goal is to achieve the following:
Crawlomatic Multisite Scraper Post Generator will import items from any websites using crawling and scraping. Resulting blog posts are fully customizable and rich in content. List of customizable stuff for each post:

• post author user name
• post status (published, draft, pending, private, trash)
• automatically generate post categories or tags from marketplace items
• manually add post categories or tags to items
• generate post or page
• automatically generate a featured image for the post
• enable/disable comments for the generated post
• enable/disable pingbacks/trackbacks for the generated post
• crawl post url and grab post full content
• customize post title and content (with the included wide variaty of relevant post shortcodes)

Other plugin features:

• Google Translate support ' select the language in which you want to post your articles
• Text Spinner support ' automatically modify generated text, changing words with their synonyms ' great SEO value!
• Supports all website crawling and scraping types and also Atom feed type
• Using SimplePie
• Keyword Replacer (replace relevant keywords with your links)
• Random sentence generator (relevant sentences - as you define them)
• detailed plugin activity logging
• scheduled rule runs

Plugin requirements:

PHP DOM -> how to install it (if you don't have it, but probably you already have it): http://php.net/manual/en/dom.setup.php

Wordpress installation - top

Youtube video tutorial:

I also provided a Quick Install Guide to feature an easy plugin installation for everyone.

To make this plugin work, first, you'll need to install the plugin. The easiest method is to take the .zip file you've downloaded and upload it via Plugins > Add New > Upload Plugin in the WordPress Dashboard. Once the plugin is installed, be sure to Activate it.

Now that you've installed and activated the plugin, you'll see a new menu item created inside WordPress called 'Crawlomatic Multisite Scraper Post Generator'. First thing first, let's head over to Settings > Crawlomatic Multisite Scraper Post Generator and take a look at what options are available.

Plugin Settings - top

Refreshingly, Crawlomatic Multisite Scraper Post Generator has a super-simple settings screen. Let's look at first at the settings panel:

Here you can find the steps needed in configuring your plugin even if you have no HTML knowledge at all. You can find options for:

HINT! Don't forget to click the Save button every time you modified your settings, otherwise the modifications will be lost!

Main Settings:

• Crawlomatic Multisite Scraper Post Generator Main Switch: Enable or disable the Crawlomatic Multisite Scraper Post Generator. This acts like a main switch.
• Enable Post Auto Deletion Feature: Click this option to enable the post auto deletion feature after a period of time (defined by the 'Automatically Delete Post' settings for each rule).
• Clear Curl Decoding Value: Click this option if you are experiencing malformed content in your post generation. Enabling this value may resolve such issues. If you do not have such issues with generated post content, please leave this checkbox disabled.
• Do Not Check For Duplicate Posts: Choose if you want to skip checking for duplicate posts when publishing new posts (check this if you have 10000+ posts on your blog and you are experiencing slowdows when the plugin is running. If you check this, duplicate posts will be posted! So use it only when it is necesarry.
• Update Post If It Is Already Posted: Select if you want to update the post when it is found as it is already posted (by title). If you uncheck this value, if duplicate post found, no action will be taken.
• Timeout for One Request (seconds): Set the timeout (in seconds) for every rule running. I recommend that you leave this field at it's default value (3600).
• Delay Between Request (milliseconds): Set the timeout (in milliseconds) between each subsequest call to the crawled website. Increase this value if the website has a anti-crawling mechanism active. Here you can also input a pair of values, separrated by comma (ex: 300,500). In this case, a random timeout will be selected, between the two values you specified.
• Delay Between Rule Running: Define a number of seconds the plugin should wait between the rule running. Use this to not decrease the use of your server's resources. Leave blank to disable.
• Iframe Resize Width: Resize the iframes that were imported from the custom content. If you want to disable this feature, leave this field blank.
• Iframe Resize Height: Resize the iframes that were imported from the custom content. If you want to disable this feature, leave this field blank.
• Disable Automatic Excerpt Generation: Disable excerpt automatic generation for resulting blog posts.
• Crawled Category Separator: Input the separator character for category assignation from crawled content.
• Add Attributes to Internal Links: Input the attributes you want to set for each internal link from content.
• Add Attributes to External Links: Input the attributes you want to set for each external link from content.
• Replace All URLs from Content With This URL: Do you want to replace all URLs from generated posts content with this predefined URL?
• 'Read More' Button Text: Insert the desired text to be show for the 'Read More' buttons. Exemple: for the %%item_read_more_button%% shortcode or for the excerpt.
• Strip Links From Generated Post Content: Choose if you want to strip links from the generated post content.
• Show Extended Item Information Metabox in Post: Choose if you want to show an extended information metabox under every plugin generated post.
• Enable Logging for Rules: Do you want to enable logging for rules?
• Enable Detailed Logging for Rules: Do you want to enable detailed logging for rules? Note that this will dramatically increase the size of the log this plugin generates.
• Automatically Clear Logs After: Choose if you want to automatically clear logs after a period of time.
• Timeout for Rule Running (seconds): Set the timeout (in seconds) for every rule running. I recommend that you leave this field at it's default value (3600).
• Send Rule Running Summary in Email: Choose if you want to receive a summary of the rule running in an email.
• Email Address: Input the email adress where you want to send the report. You can input more email addresses, separated by commas.
• Minimum Title Word Count: Set the minimum word count for post titles. Items that have less than this count will not be published. To disable this feature, leave this field blank.
• Maximum Title Word Count: Set the maximum word count for post titles. Items that have more than this count will not be published. To disable this feature, leave this field blank.
• Minimum Content Word Count: Set the minimum word count for post content. Items that have less than this count will not be published. To disable this feature, leave this field blank.
• Maximum Content Word Count: Set the maximum word count for post content. Items that have more than this count will not be published. To disable this feature, leave this field blank.
• Banned Words List: Do not include posts that's title or content contains at least one of these words. Separate words by comma. To disable this feature, leave this field blank.
• Required Words List: Do not include posts that's title or content does not contain at least one of these words. Separate words by comma. To disable this feature, leave this field blank.
• Strip HTML Elements from Final Content by ID: Strip HTML elements from final content that have this IDs. You can insert more IDs, separeted by comma. To disable this feature, leave this field blank.
• Strip HTML Elements from Final Content by Class: Strip HTML elements from final content that have this class. You can insert more classes, separeted by comma. To disable this feature, leave this field blank.
• Automatically Translate Content To: Do you want to automatically translate generated content using Google Translate?
• Skip Posts That Do Not Have Images: Choose if you want to skip posts that do not have images.
• Skip Posts Older Than a Selected Date: Choose if you want to skip posts that are older than a selected date.
• Choose if you want to skip posts that are older than a selected date. Select the date prior which you want to skip posts.
• Spin Text Using Word Synonyms: Do you want to randomize text by changing words of a text with synonyms using one of the listed methods? Note that this is an experimental feature and can in some instances drastically increase the rule running time!
• First List of Possible Sentences (%%random_sentence%%): Insert some sentences from which you want to get one at random. You can also use variables defined below. %something ==> is a variable. Each sentence must be sepparated by a new line.
• Second List of Possible Sentences (%%random_sentence2%%): Insert some sentences from which you want to get one at random. You can also use variables defined below. %something ==> is a variable. Each sentence must be sepparated by a new line.
• List of Possible Variables: Insert some variables you wish to be exchanged for different instances of one sentence. Please format this list as follows:Variablename => Variables (seperated by semicolon)Example:adjective => clever;interesting;smart;huge;astonishing;unbelievable;nice;adorable;beautiful;elegant;fancy;glamorous;magnificent;helpful;awesome
• Custom HTML Code #1: Insert a custom HTML code that will replace the %%custom_html%% variable. This can be anything, even an Ad code.
• Custom HTML Code #2: Insert a custom HTML code that will replace the %%custom_html2%% variable. This can be anything, even an Ad code.
• ID (Affiliate Keyword Replacer Tool Settings:) This is the ID of the rule.
• DEL (Affiliate Keyword Replacer Tool Settings:) Do you want to delete this rule?
• Search Keyword (Affiliate Keyword Replacer Tool Settings:) This keyword will be replaced with a link you define.
• Replacement Keyword (Affiliate Keyword Replacer Tool Settings:) This keyword will replace the search keyword you define. Leave this field blank if you only want to add an URL to the specified keyword.
• Link To Add (Affiliate Keyword Replacer Tool Settings:) Define the link you want to appear the defined keyword. Leave this field blank if you only want to replace the specified keyword without linking from it.

Rules List Settings Screen:

• ID: This is the ID of the rule.
• Scraper Start URL Input the URL which you want to use as a crawling starting point. This URL will also be scraped.
• Schedule (Hours) Select the interval in hours after which you want this rule to run.
• Max Posts At A Time Select the maximum number of posts that this rule can create at once. 0-50 interval allowed..
• Del: Do you want to delete this rule?
• Post Status: Select the status that you want for the automatically generated posts to have.
• Item Type : Select the type (post/page) for your automatically generated item.
• Post Author :  Select the author that you want to assign for the automatically generated posts.
• Active: Do you want to enable this rule? You can deactivate any rule (you don't have to delete them to deactivate them).
• Info: The number of items (posts, pages) this rule has generated so far.
• Actions: Do you want to run this rule now? Note that only one instance of a rule is allowed at once.

Advanced Plugin Settings Dropdown Screen:

• Generated Post Title: Set the title of the generated posts for user rules. You can use the following shortcodes: %%random_sentence%%, %%random_sentence2%%, %%item_title%%, %%item_description%%, %%item_content%%, %%item_cat%%, %%item_tags%%
• Generated Post Content: Set the content of the generated posts for user rules. You can use the following shortcodes: %%custom_html%%, %%custom_html2%%, %%random_sentence%%, %%random_sentence2%%, %%item_title%%, %%item_description%%, %%item_content%%, %%item_content_plain_text%%, %%item_img_url%%, %%item_url%%, %%item_cat%%, %%item_tags%%
• Generated Post Format: If your template supports "Post Formats", than you can select one here. If not, leave this at it\'s default value.
• Additional Post Category: Select the post category that you want for the automatically generated posts to have.
• Categories To Remove: Input a list of categories that you want to remove from the list of automatic categories. Input more categories, separated by comma. Ex: remove1, remove2, remove3
• Skip Checking OG:Image Meta: Do you want to skip checking the og image meta tag when searching for a featured image for the generated article?.
• Skip Checking Post Content: Do you want to skip checking the post content when searching for a featured image for the generated article?.
• Banned Words List: Do not include posts that's title or content contains at least one of these words. Separate words by comma. To disable this feature, leave this field blank.
• Required Words List: Do not include posts that's title or content does not contain at least one of these words. Separate words by comma. To disable this feature, leave this field blank.
• Run Regex On Content: Run regex on post content. To disable this feature, leave this field blank.
• Replace Matches From Regex: Replace the above regex matches with this regex expression. If you want to strip matched content, leave this field blank.
• Strip HTML From Full Content: Do you want to strip HTML from Full Article Content?
• Strip HTML Elements by ID: Strip HTML elements from final content that have this IDs. You can insert more IDs, separeted by comma. To disable this feature, leave this field blank.
• Strip HTML Elements by Class: Strip HTML elements from final content that have this class. You can insert more classes, separeted by comma. To disable this feature, leave this field blank.
• Limit Article Word Count: Set the maximum word count for the article. If you set this value, a "Read More" link will be automatically appended to the end of the article for articles that where shortened. To disable this feature, leave this field blank.
• Automatically Translate Content To: Do you want to automatically translate generated content using Google Translate?
• Source Articles Language: If you use the above translate feature and want to translate not english articles to other languages, please select their original language here.
• Hide Google Translate Popup: Do you want to hide the Google Translate Popup that shows when hovering on the resulting translated text, containing the original text before translation?
• Automatically Delete Post: Do you want to automatically delete generated posts after a period of time? To disable this feature, leave this field blank.
• Percentage Of The Content To Keep: Select a percentage of the grabbed content that will be included in the article. This is useful when the author states that you can legally get up to 25% of the full article. To disable this feature, leave this field blank.
• Post Custom Fields: Set the custom fields that will be set for generated posts. The syntax for this field is the following: custom_field_name1 => custom_field_value1, custom_field_name2 => custom_field_value2, ... . In custom_field_valueX, you can use shortcodes, same like in post content. Example (without quotes): 'title_custom_field => %%item_title%%'. You can use the following shortcodes: %%custom_html%%, %%custom_html2%%, %%random_sentence%%, %%random_sentence2%%, %%item_title%%, %%item_content%%, %%item_url%%, %%item_cat%%, %%item_tags%%, %%item_content_plain_text%%, %%item_read_more_button%%, %%item_show_image%%, %%item_image_URL%%, %%item_description%%, %%item_pub_date%%
• Send Cookies With Request: Select the cookie values you want to send with each request. The syntax for this field is the following: cookie_key1=cookie_value1; cookie_key2=cookie_value2 . To disable this feature, leave this field blank.
• Do Not Crawl External Links: Choose if you want to allow crawling of external links or not. In most cases, unchecking this will result in unpredicted posts appearing.
• Maximum Link Crawling Depth: Choose the maximum depth when crawling the selected link. The default value is 2. Choosing a greater value may cause unexpected posts appearing on your website.
• Content Query Type (Optional): Select the query type you want to search for the article full content.
• Content Query String (Optional): Input the search query for full content searching. This can be the ID or class name you want to search, the regex expression you want to apply for or the XPath Expression you want to query for. More info/tutorial. If you leave this field blank, content will be automatically detected for you.
• Title Query Type (Optional): Select the query type you want to search for the article title.
• Title Query String (Optional): Input the search query for title searching. This can be the ID or class name you want to search, the regex expression you want to apply for or the XPath Expression you want to query for. More info/tutorial. If you leave this field blank, the title will be automatically detected for you.
• Featured Image Query Type (Optional): Select the query type you want to search for the article image.
• Featured Image Query String (Optional): Input the search query for image searching. This can be the ID or class name you want to search, the regex expression you want to apply for or the XPath Expression you want to query for. More info/tutorial. If you leave this field blank, the featured image will be automatically detected for you.
• Date Query Type (Optional): Select the query type you want to search for the article date.
• Date Query String (Optional): Input the search query for date searching. This can be the ID or class name you want to search, the regex expression you want to apply for or the XPath Expression you want to query for. More info/tutorial. If you leave this field blank, the current date will be automatically detected for you.
• Category Query Type (Optional): Select the query type you want to search for the article category.
• Category Query String (Optional): Input the search query for category searching. This can be the ID or class name you want to search, the regex expression you want to apply for or the XPath Expression you want to query for. More info/tutorial. If you leave this field blank, the current date will be automatically detected for you.
• Also Get Found CSS Files Content: Do you want to also get any CSS file content that was included in the scaped URL?
• Source Article Encoding: Select the source page encoding
• Automatically Add Post Categories From Items: Do you want to automatically add post categories from the feed items?
• Automatically Add Post Tags From Items: Do you want to automatically add post tags from the feed items?
• Additional Post Tags: Select the post tags that you want for the automatically generated posts to have.
• Enable Comments For Generated Posts: Do you want to enable comments for the generated posts?
• Enable Pingbacks/Trackbacks For Generated Posts: Do you want to enable pingbacks and trackbacks for the generated posts?
• Automatically Get Featured Image for Generated Post: Do you want to set featured image for generated post (to the first image that was found in the post)?
• Default Featured Image if No Image Available for Item: Insert a link to a valid image that will be set for the featured image for the posts that do not have a valid image attached or if you disabled automatical featured image generation. To disable this feature, leave this field blank.
• Try to Get Full Article Content: Do you want to try to get full article content from the linked URL? This will replace the content of the %%item_content%% shortcode.
• Full Content Query Type: Select the query type you want to search for the article full content. XPath: More info/tutorial.
• Search Query String: Input the search query for full content searching. This can be the ID or class name you want to search for, the regex expression you want to apply or the XPath Expression you want to query for. More info/tutorial.
• Strip HTML from Full Article Content: Do you want to strip HTML from Full Article Content?
• Get Only The First Found Element's Content: Do you want to get only one element or all elements that fit the defined rule?
• Get Only The Inner Div Content: Do you want to get only the inner content of the div or the full div (with tags you searched for)?
• Also Get Found CSS Files Content: Do you want to also get any CSS file content that was included in the scaped URL?

Activity and Loggind Screen:

• Rules Currently Running: These rules are currently running on your server.
• Restore Plugin Default Settings: Hit this button and the plugin settings will be restored to their default values. Warning! All settings will be lost!
• Delete All Posts Generated by this Plugin: Hit this button and all posts generated by this plugin will be deleted!
• Activity Log: This is the main log of your plugin. Here will be listed every single instance of the rules you run or are automatically run by schedule jobs (if you enable logging, in the plugin configuration).

How to customize resulting posts with built in shortcodes?

You can use the following shortcodes in post title and content editing:

• %%random_sentence%% - generates a random sentence (as you have defined them in plugin config.) More on how to configure the random sentence generator, check plugin documentation (down)
• %%random_sentence2%% - generates a random sentence (from the second geenrator) (as you have defined them in plugin config.) More on how to configure the random sentence generator, check plugin documentation (down)
• %%custom_html%% - includes the custom HTML you defined in the Main Settings interface (Custom HTML Code #1:)
• %%custom_html2%% - includes the custom HTML you defined in the Main Settings interface (Custom HTML Code #2:)
• %%item_title%% - includes the item title
• %%item_description%% - includes a short description of the article
• %%item_content%% - includes the item full HTML content
• %%item_content_plain_text%% - includes the items plain text content
• %%item_img_url%% - includes the article's image URL (if it exists)
• %%item_url%% - includes the article's URL
• %%item_cat%% - includes the article's category (if it exists)
• %%item_tags%% - includes the article's tag (if it exists)

How to work with 'Random Sentence Generator'?

You can use the Random Sentence Generator by inserting list of sentences to the 'First List of Possible Sentences (%%random_sentence%%)' text field. Here, you can add full sentences, or sentences with variables. Variables are structured as: %something. A possible list of sentences can be:

• Meet this %adjective %noun %sentence_ending
• This is the %adjective %noun ever%sentence_ending
• I %love_it %nouns , because they are the %adjective %sentence_ending
• My %family says this plugin is very %adjective %sentence_ending
• These %nouns are quite %adjective %sentence_ending

You can define variables in the 'List of Possible Variables:' textfield. Here you can define values to variables, values which will be used in the random sentence forming. Variables can be defined as follows:

variable_name1 => variable_value1;variable_value2;variable_value3

variable_name2 => variable2_value1;variable2_value2;variable2_value3

You can also use variables in variable definition. Example:

noun_with_adjective => %noun;%adjective %noun

An example of variable definition:

adjective_very => %adjective;very %adjective;

adjective => clever;interesting;smart;huge;astonishing;unbelievable;nice;adorable;beautiful;elegant;fancy;glamorous;magnificent;helpful;awesome

noun_with_adjective => %noun;%adjective %noun

noun => plugin;WordPress plugin;item;ingredient;component;constituent;module;add-on;plug-in;addon;extension

nouns => plugins;WordPress plugins;items;ingredients;components;constituents;modules;add-ons;plug-ins;addons;extensions

love_it => love;adore;like;be mad for;be wild about;be nuts about;be crazy about

family => %adjective %family_members;%family_members

family_members => grandpa;brother;sister;mom;dad;grandma

sentence_ending => .;!;!!

These random sentences can be used in post titles or post content, by adding the %%random_sentence%% or %%random_sentence2%% shortcode.

Available Shortcodes - top

Available shortcodes:

• [crawlomatic-scraper] to dynamically scrape content from any site - documentation here
• [crawlomatic-list-posts] to include a list that contains only posts imported by this plugin
• [crawlomatic-display-posts] to include a WordPress like post listing. Usage: [crawlomatic-list-posts type='any/post/page/...', order='ASC/DESC', orderby='title/ID/author/name/date/rand/comment_count', posts=number_of_posts_to_show, category='posts_category', ruleid='ID_of_crawlomatic_rule']

Example: [crawlomatic-list-posts type='any', order='ASC', orderby='date', posts=50, category= '', ruleid='0']
Example 2: [crawlomatic-display-posts include_excerpt='true' image_size='thumbnail' wrapper='div']

Available parameters for the [crawlomatic-display-posts] shortcode:

Summary - top

Crawlomatic Multisite Scraper Post Generator is a simple, yet powerful tool you can use to create a profitable affiliate marketing webpage. The setup and settings of the plugin couldn't have been easier. Now, let's go and enjoy the results of this great plugin! Have fun using it!

Sources and Credits - top

This component was made by Szabi CodeRevolution, for more information and support contact us at support@coderevolution.ro

Once again, thank you so much for purchasing this item. As I said at the beginning, I'd be glad to help you if you have any questions regarding this plugin and I'll do my best to assist.

CodeRevolution

---