WP Content Crawler – Get content from almost any site, automatically!


Get content material from nearly any web site to your WordPress weblog, routinely!


  • Create a private web site which collects information, posts, and so on. out of your favourite websites to see them in a single place
  • Use it with WooCommerce to gather merchandise from purchasing websites
  • Accumulate merchandise from affiliate applications to generate income
  • Accumulate posts to create a check atmosphere on your plugin/theme
  • Accumulate plugins, themes, apps, photographs from different websites to create a set of them
  • Preserve monitor of rivals
  • You may think about something. The web is filled with contents :)

Before you purchase, ensure you do the next:

  • Watch the quick start video and use the plugin within the
    demo. You may also watch the opposite
    video tutorials to discover ways to use
    the plugin. There are additionally many guides
    explaining do sure issues with the plugin.
  • Make certain the plugin can retrieve the info from the location you wish to crawl by following the directions in

    Can I get content from X site?
  • If you’re nonetheless unsure if the plugin can retrieve content material from a particular web site, ask us within the feedback
  • You may verify the FAQs
    when you’ve got any questions. If the reply to your query just isn’t there, you possibly can all the time ask us within the feedback


WP Content Crawler - Get content from almost any site, automatically! - 1

WP Content Crawler - Get content from almost any site, automatically! - 2


It’s all about CSS selectors and you’ll discover ways to use them in minutes by watching the introduction tutorial. The plugin’s Visible Inspector software additionally helps you discover CSS selectors simply by clicking onto the weather within the goal websites. Right here is the gist of it:

WP Content Crawler - Get content from almost any site, automatically! - 3

WP Content Crawler - Get content from almost any site, automatically! - 4


Right here is the checklist of some options of WP Content material Crawler. To find out about the entire options, please see the options desk under.

WP Content Crawler - Get content from almost any site, automatically! - 5
WP Content Crawler - Get content from almost any site, automatically! - 6

WP Content Crawler - Get content from almost any site, automatically! - 7

WP Content Crawler - Get content from almost any site, automatically! - 8

WP Content Crawler - Get content from almost any site, automatically! - 9


WP Content Crawler - Get content from almost any site, automatically! - 10
WP Content material Crawler introduction video (English)

WP Content Crawler - Get content from almost any site, automatically! - 11
WP Content material Crawler introduction video (Turkish)


WP Content Crawler - Get content from almost any site, automatically! - 12
Fast Begin Information

WP Content Crawler - Get content from almost any site, automatically! - 13
Utilizing CSS Selectors in WP Content material Crawler

WP Content Crawler - Get content from almost any site, automatically! - 14
HTML & CSS Selectors

WP Content Crawler - Get content from almost any site, automatically! - 15
Utilizing brief codes to put any information wherever within the submit

WP Content Crawler - Get content from almost any site, automatically! - 16
Save photographs as WooCommerce product gallery

WP Content Crawler - Get content from almost any site, automatically! - 17
Save arcade video games

WP Content Crawler - Get content from almost any site, automatically! - 18WP Content Crawler - Get content from almost any site, automatically! - 19


Save each submit element
Title, excerpt, content material, tags, classes, slug, date, customized meta, taxonomies, meta key phrases, meta description, featured picture, submit photographs, standing… Simply every part.
Visible Inspector
Simply click on to a component to seek out its CSS selector. You may also get various CSS selectors that you just is perhaps thinking about. There isn’t a want to depart your admin panel anymore.
Crawl (scrape, seize, save) posts
After the settings are configured, the plugin finds URLs of the posts and crawls them routinely within the background.
Recrawl (replace) posts
Recrawl posts routinely to maintain them up to date on a regular basis. You may restrict what number of occasions a submit will be up to date, set replace interval, and ignore outdated posts.
Delete posts
You wish to delete outdated crawled posts? The plugin can delete them routinely.

Management scheduling
You may set what number of occasions URL assortment and submit crawling occasions ought to run every time for a web site. As an illustration, it can save you 3 posts each minute, or run URL assortment 5 occasions each 2 minutes.

Save classes
The goal class doesn’t exist in your web site? No drawback. The plugin can create the goal classes for you. Simply outline the CSS selectors that discover class names. They will even be created as subcategories.
Save slugs (permalink)
You may outline the permalink of the posts. You will get the permalink from the goal web site, enter customized textual content, and even create templates for the slugs by utilizing brief codes.
Save taxonomies
Save taxonomy values by retrieving them from the goal web site or coming into manually. Saving particulars of customized submit sorts is less complicated than ever.

Save posts into customized classes
A customized submit kind has customized classes? No drawback. You may outline customized class taxonomies utilized by the customized submit kind and choose these classes when defining the classes of the submit. The plugin may create customized classes for you.

Customized submit meta
Save something as customized submit meta. You need to use a CSS selector or simply kind the worth.

Content material templates
Put together submit content material, title, excerpt, checklist merchandise and gallery merchandise templates utilizing brief codes. Furthermore, you possibly can outline templates for values of every CSS selector utilizing the choices field.

Different selectors
You may write various selectors to get the info even when the goal web site has submit pages designed otherwise from one another.

Discover and exchange something
You need to use plain textual content or common expressions to seek out and exchange something. You may even modify the HTML of the web page, create your personal HTML components and write selectors to make use of them. You may even change picture URLs. You might have the facility.

Paginated posts
Goal submit has a couple of web page? No worries. It can save you paginated posts as properly.
Record kind posts
Some websites create posts with a listing inside. You may extract the checklist from the submit, create a template that ought to be utilized to every checklist merchandise and even reverse the checklist.
Take away pointless components
Typically it is advisable to eliminate some components, similar to commercials, feedback, you identify it. Simply write its CSS selector and it’s eliminated.
Mechanically insert class URLs
Goal web site has tons of of classes? Piece of cake. Simply write the CSS selector and the plugin will insert them for you.
Submit sorts
Set submit kind. It may be a submit, a web page, a product, or every other submit kind obtainable in your WordPress set up.
Take away hyperlinks
You may take away hyperlinks from the submit. Simply verify the checkbox and the hyperlinks are gone. That simple.
Password safety
You may set a password for the posts to point out them solely to the customers who’ve the password.
You may add notes for your self to remind you issues in regards to the web site. CSS selectors, TODO checklist, something.

Check every part on the fly
Check submit crawling, URL assortment, CSS selectors, common expressions, discover and exchange choices and proxies on the fly. You may also allow caching to carry out the exams a lot quicker and cut back the requests despatched to the goal web site.

Check all of the settings of a web site directly
Utilizing the tester, you possibly can check all choices you configured within the web site settings to verify every part works as you need earlier than enabling computerized crawling.
Utilizing the instruments, it can save you posts manually with their URL, recrawl posts with their ID or delete already-saved URLs.
Customized common settings for every web site
You may present customized common settings for every submit to override them and make them appropriate for a web site.
Submit standing
You may immediately publish the saved posts or maintain them as draft to verify them earlier than publishing.
Save all photographs in submit content material
Saving all photographs within the content material of the submit is as simple as checking a single checkbox.

Save photographs as gallery
It can save you the photographs within the goal web page as gallery and supply a template for every picture to make it appropriate for the gallery library that you just use on frontend. You may also save the photographs as WooCommerce gallery by simply checking one checkbox.

Any information as brief code
Get something from goal web page as a brief code and use the brief codes within the plugin’s templates to put any information wherever you need.
Use a proxy or proxies to get content material from the websites to which your IP doesn’t have entry.
Connect cookies, similar to session cookies, to every request. By this manner, for instance, you possibly can crawl the goal web site as if you’re logged in.
Crawl as many posts as you need
You may set what number of occasions submit crawling or URL assortment CRON occasions ought to run. By this manner, you possibly can, e.g., save 100 posts each minute. Simply watch out and think about your server’s capability.
E-mail notifications
Set CSS selectors whose values shouldn’t be empty for class and submit pages. When an empty worth is discovered utilizing these selectors, you may get an electronic mail notification.
Get information from JSON
Once you allow JSON parsing for a CSS selector, you may get the values from the JSON simply.

Superior HTML manipulations
Discover-replace in response HTML, discover and exchange in factor attributes, trade factor attributes, take away factor attributes, manipulate HTML of a component, take away HTML components…


Computerized translation
Use the synthetic intelligence of Google Cloud Translation API, Microsoft Translator Textual content API, Yandex Translate API or Amazon Translate API to routinely translate the posts. Notice that these are paid providers. They typically provide the service at no cost for a restricted period of time. You may see their pricing pages to study extra.

Computerized spinning
Use spinning to routinely rewrite crawled posts’ contents to enhance search engine marketing. The plugin at the moment implements Spin Rewriter API and Turkce Spin API, that are paid providers. You may go to their web site to study the pricing particulars.

Duplicate submit verify
Examine duplicate posts by URL, submit title and/or submit content material. If you’re utilizing WooCommerce, merchandise whose SKU already exists are thought-about as duplicate and they won’t be added to your web site.

Scheduled posts
You may add/take away minutes to/from the submit date. By this manner, you possibly can schedule submit publishing.
Save WooCommerce merchandise
Save worth, stock, transport, attributes, and superior choices. It can save you the product as a easy or an exterior product. You may also set downloadable file choices and outline the product as digital. The choices can be found for WooCommerce variations larger than or equal to three.3.
Choices field
You might have the management! Outline many choices for the values discovered by a CSS selector. The choices embody find-replace, calculation, template, and JSON parsing settings. You may simply import/export the choices outlined within the choices bins as properly.
Deal with information like a professional
Rename, copy, and transfer saved information simply. You may also outline title, description, caption, and alt texts for the saved media information utilizing templates through which you should use any brief code. Additionally it is attainable to provide random names to the saved information.
Deal with iframes and scripts like a professional
WordPress doesn’t enable displaying iframes and scripts since they pose a safety danger. You may flip iframe and script HTML components into brief codes by simply checking a checkbox. The brief code will present iframes and scripts from the allowed supply domains outlined by you.
Fast save
With fast save button, it can save you the settings way more rapidly. No want to attend for web page to reload.
Common expressions
Outline common expressions in find-replace choices to find-replace something. You may also use delimiters and modifiers to match extra exactly.
Save “srcset” attributes
When various sizes of the saved photographs can be found, the plugin assigns them into srcset attribute of img components in order that your pages will load quicker in several display screen sizes.

Save “alt” and “title” attributes
Once you save photographs, their “alt” and “title” attributes are routinely retrieved from the goal web site and assigned to the saved media. You may also outline templates for them to use your search engine optimization methods.

Study when there’s a drawback. The plugin will present you the main points of the error so to repair it straight away.

Deal with character encoding issues
The plugin is ready to deal with completely different character encodings, even when the goal web site incorporates blended encodings. You may convert the encoding by checking a single checkbox.

Navigate between settings simply
Repair navigation to the highest! The plugin shops the place you have been earlier than switching to a brand new tab and restores your earlier location if you activate that tab once more. No extra getting misplaced among the many settings.
Handbook crawling software
With handbook crawling software, save a number of posts by coming into their URLs. You may also enter class URLs in order that the software can get submit URLs from there. Furthermore, you possibly can set it to crawl a number of posts on the identical time.

Add URLs to the database
The plugin collects URLs routinely. Nonetheless, if you’d like it to crawl solely sure URLs, you possibly can add them to the database manually utilizing the handbook crawling software. By this manner, the required URLs shall be crawled utilizing your scheduling choices, routinely.

Allow/disable computerized crawling for a particular web site
You may allow or disable computerized crawling for every web site individually.
You may import and export web site settings simply. Simply copy and paste the code created by the plugin.
Add limitless websites to the plugin and activate what number of of them you need.

Detailed dashboard
See what’s occurring within the background. Lively websites, variety of posts crawled, variety of posts up to date, final crawled and up to date posts, final added URLs, final and subsequent run of CRON occasions, at the moment being saved posts and URLs…

Get updates out of your admin panel
You may replace the plugin with only one click on at any time when an replace is prepared. Simply go to your updates web page in your admin panel.
Use probably the most safe PHP
The plugin helps the newest variations of PHP.
Use probably the most trendy browsers
The plugin helps Chrome, Firefox, Safari, Opera, and Edge.
Interactive guides
Interactive guides present you configure settings to attain sure issues, step-by-step, like a dwelling documentation. You can begin these guides everytime you need. You may even begin them from a particular step.
On-line documentation
You may verify the web documentation everytime you really feel a necessity.
Fast guides proper subsequent to the settings
Every setting within the plugin has a fast information that may make it easier to perceive what every setting does.
Video tutorials
Watch video tutorials to simply discover ways to use the plugin.
Able to translate
You may translate the plugin into your personal language utilizing Poedit.

With filters, you are able to do issues conditionally. For instance, you possibly can enhance the value of a product if one
of its attribute values incorporates a particular phrase. Filters comprise many motion instructions. See
the commands
within the documentation.

Necessities PHP >= 7.3, json, mbstring, curl, dom, fileinfo, WP-Cron. These are already obtainable in most hosts. Even when the extensions will not be already lively, most internet hosting websites allow you to allow these from their management panel. See the documentation for extra info.
Examined with WP variations 6.0, 5.9, 5.8, 5.7, 5.6, 5.5, 5.4, 5.3, 5.2, 5.1, 5.0
Examined with WooCommerce variations 6.5, 6.4, 6.3, 6.2, 6.0, 5.7, 5.5, 5.0, 4.9, 4.5
Languages English, Türkçe
Shortcomings The plugin can not retrieve content material that’s created by utilizing JavaScript. For extra info, please see Can I get content from X site?.


WP Content Crawler - Get content from almost any site, automatically! - 20

WP Content Crawler - Get content from almost any site, automatically! - 21WP Content Crawler - Get content from almost any site, automatically! - 22


Issues with crawling an internet site

  • Not a straightforward job, requires superior programming expertise
  • Each web site is completely different and wishes tailor-made crawling implementation
  • Not simply each web site is completely different, but additionally pages of a single web site can differ
  • Pages and their supply codes have to be investigated intensively to give you a crawling plan
  • Figuring out save sure info in a particular place in WordPress requires information in regards to the inner construction of WordPress and the way WordPress works
  • If sure info ought to be saved into a particular discipline outlined by a third-party plugin, one ought to modify the crawling implementation after researching for hours about save that info
  • One ought to learn about how HTML works and extract sure elements from HTML code
  • One ought to deal with all attainable inconsistencies that is perhaps within the supply codes of internet sites to supply a strong answer that may maintain working
  • What if the posts have to be shared in common time intervals?
  • What if you wish to crawl new posts added to an internet site after a while?
  • What about translating the posts from one language to a different?
  • What if the posts must be paraphrased to supply a greater search engine marketing for the web site?
  • What if some info shouldn’t be retrieved?
  • What if sure info ought to be modified to make it appropriate on your web site?
  • What if one other web site must be crawled, not only one?
  • What if that different web site wants a distinct crawling plan?
  • What if it is advisable to login to the web site to crawl it?
  • What if the web site modifications its supply code?
  • What if you wish to replace the crawled posts by recrawling them from the unique web site?
  • What if you wish to make sure that if the data is retrieved precisely as you need it earlier than routinely posting the posts to your web site?
  • What if you wish to guarantee your web site’s safety by ensuring no malicious-code-executing code leads to your web site?
  • And plenty of extra what-ifs that you just may not even think about until you come throughout them

Our imaginative and prescient and mission

We consider that sturdy, dependable, and automated crawling capabilities ought to be obtainable for anybody. We wish to democratize this discipline by letting anybody have these capabilities, not simply builders. With this function, we goal to supply a plugin that you’ll fall in love with and really feel at house when utilizing it. To let it accessible by anybody, we make the plugin low-cost and easy-to-use. We don’t implement the options simply to make gross sales. We plan and execute for the long run. We all the time take heed to your suggestions and make required modifications accordingly. We expect that WordPress plugins ought to be developed with enterprise-level care. So, we intensively check the plugin earlier than every launch with automated end-to-end UI exams, at the moment over 1700 exams, that run in many various environments within the cloud for a complete of over 40 hours to make sure the plugin is appropriate along with your server and WordPress environments and also you, our useful prospects, get the standard and reliability you deserve.

How we resolve these issues

We’ve got been creating WP Content material Crawler for nearly 4 years such that we’ve got come throughout nearly all of the what-ifs. Working with our prospects and listening to their wants, we offer sturdy and dependable options to those issues. We consider that one ought to simply present from which web site the data ought to be retrieved and what info ought to be retrieved from that web site after which begin crawling that web site, with out worrying in regards to the complicated behind-the-scenes operations.

To make it obtainable to anybody, we offer an in depth on-line documentation that incorporates not simply the outline of the settings however use the settings to attain your objectives. Typically you may not really feel like studying the documentation. We additionally present interactive step-by-step guides which are obtainable within the plugin, only one click on away. You can begin the interactive guides displaying you step-by-step how you are able to do sure issues any time and from any step you need.

One of the crucial distinctive options of WP Content material Crawler is the flexibility to check nearly any configuration. By this manner, you’ll not come throughout any surprises after you allow computerized crawling. When testing, the errors associated to your settings are proven so to repair them earlier than they trigger any issues.

WP Content material Crawler has so many options that even we have no idea what number of of them are there. You may routinely crawl, replace, and delete the posts, you possibly can translate posts, spin posts, you possibly can even outline what fields have to be translated or spun if you do not need all of them modified. You could find-replace nearly something. You may assign some info from the goal submit to a brief code and place that info wherever within the submit. It can save you WooCommerce merchandise. It can save you particulars for third-party plugins that we don’t even know they exist. The options of the plugin are designed such that you just really feel that you’re in management if you use them. We make them as versatile as attainable to make them suit your wants. When designing new options, we all the time needless to say you may want a extra superior model of that characteristic and we design the options accordingly. We be sure that the options and your entire code of the plugin are maintainable and extendable in order that we are able to all the time enhance the plugin.


Changelog is kept in the documentation site. Click here to see the changelog.


Related Posts