Get content material from nearly any web site to your WordPress weblog, routinely!
FOR WHAT IT CAN BE USED
- Create a private web site which collects information, posts, and so on. out of your favourite websites to see them in a single place
- Use it with WooCommerce to gather merchandise from purchasing websites
- Accumulate merchandise from affiliate applications to generate income
- Accumulate posts to create a check atmosphere on your plugin/theme
- Accumulate plugins, themes, apps, photographs from different websites to create a set of them
- Preserve monitor of rivals
- You may think about something. The web is filled with contents
Before you purchase, ensure you do the next:
-
Watch the quick start video and use the plugin within the
demo. You may also watch the opposite
video tutorials to discover ways to use
the plugin. There are additionally many guides
explaining do sure issues with the plugin. -
Make certain the plugin can retrieve the info from the location you wish to crawl by following the directions in
Can I get content from X site? FAQ. -
If you’re nonetheless unsure if the plugin can retrieve content material from a particular web site, ask us within the feedback
part. -
You may verify the FAQs
when you’ve got any questions. If the reply to your query just isn’t there, you possibly can all the time ask us within the feedback
part.
QUICK START
HOW IT WORKS
It’s all about CSS selectors and you’ll discover ways to use them in minutes by watching the introduction tutorial. The plugin’s Visible Inspector software additionally helps you discover CSS selectors simply by clicking onto the weather within the goal websites. Right here is the gist of it:
WHAT WP CONTENT CRAWLER CAN DO
Right here is the checklist of some options of WP Content material Crawler. To find out about the entire options, please see the options desk under.
SEE IT IN ACTION, LEARN IN MINUTES
 |
WP Content material Crawler introduction video (English) |
 |
WP Content material Crawler introduction video (Turkish) |
VIDEO TUTORIALS
 |
Fast Begin Information |
 |
Utilizing CSS Selectors in WP Content material Crawler |
 |
HTML & CSS Selectors |
 |
Utilizing brief codes to put any information wherever within the submit |
 |
Save photographs as WooCommerce product gallery |
 |
Save arcade video games |
MAIN FEATURES
| Save each submit element Title, excerpt, content material, tags, classes, slug, date, customized meta, taxonomies, meta key phrases, meta description, featured picture, submit photographs, standing… Simply every part. |
Visible Inspector Simply click on to a component to seek out its CSS selector. You may also get various CSS selectors that you just is perhaps thinking about. There isn’t a want to depart your admin panel anymore. |
|
| Crawl (scrape, seize, save) posts After the settings are configured, the plugin finds URLs of the posts and crawls them routinely within the background. |
Recrawl (replace) posts Recrawl posts routinely to maintain them up to date on a regular basis. You may restrict what number of occasions a submit will be up to date, set replace interval, and ignore outdated posts. |
|
| Delete posts You wish to delete outdated crawled posts? The plugin can delete them routinely. |
Management scheduling |
|
| Save classes The goal class doesn’t exist in your web site? No drawback. The plugin can create the goal classes for you. Simply outline the CSS selectors that discover class names. They will even be created as subcategories. |
Save slugs (permalink) You may outline the permalink of the posts. You will get the permalink from the goal web site, enter customized textual content, and even create templates for the slugs by utilizing brief codes. |
|
| Save taxonomies Save taxonomy values by retrieving them from the goal web site or coming into manually. Saving particulars of customized submit sorts is less complicated than ever. |
Save posts into customized classes |
|
| Customized submit meta Save something as customized submit meta. You need to use a CSS selector or simply kind the worth. |
Content material templates |
|
| Different selectors You may write various selectors to get the info even when the goal web site has submit pages designed otherwise from one another. |
Discover and exchange something |
|
| Paginated posts Goal submit has a couple of web page? No worries. It can save you paginated posts as properly. |
Record kind posts Some websites create posts with a listing inside. You may extract the checklist from the submit, create a template that ought to be utilized to every checklist merchandise and even reverse the checklist. |
|
| Take away pointless components Typically it is advisable to eliminate some components, similar to commercials, feedback, you identify it. Simply write its CSS selector and it’s eliminated. |
Mechanically insert class URLs Goal web site has tons of of classes? Piece of cake. Simply write the CSS selector and the plugin will insert them for you. |
|
| Submit sorts Set submit kind. It may be a submit, a web page, a product, or every other submit kind obtainable in your WordPress set up. |
Take away hyperlinks You may take away hyperlinks from the submit. Simply verify the checkbox and the hyperlinks are gone. That simple. |
|
| Password safety You may set a password for the posts to point out them solely to the customers who’ve the password. |
Notes You may add notes for your self to remind you issues in regards to the web site. CSS selectors, TODO checklist, something. |
|
|
Check every part on the fly |
Check all of the settings of a web site directly Utilizing the tester, you possibly can check all choices you configured within the web site settings to verify every part works as you need earlier than enabling computerized crawling. |
|
| Instruments Utilizing the instruments, it can save you posts manually with their URL, recrawl posts with their ID or delete already-saved URLs. |
Customized common settings for every web site You may present customized common settings for every submit to override them and make them appropriate for a web site. |
|
| Submit standing You may immediately publish the saved posts or maintain them as draft to verify them earlier than publishing. |
Save all photographs in submit content material Saving all photographs within the content material of the submit is as simple as checking a single checkbox. |
|
|
Save photographs as gallery |
Any information as brief code Get something from goal web page as a brief code and use the brief codes within the plugin’s templates to put any information wherever you need. |
|
| Proxy Use a proxy or proxies to get content material from the websites to which your IP doesn’t have entry. |
Cookies Connect cookies, similar to session cookies, to every request. By this manner, for instance, you possibly can crawl the goal web site as if you’re logged in. |
|
| Crawl as many posts as you need You may set what number of occasions submit crawling or URL assortment CRON occasions ought to run. By this manner, you possibly can, e.g., save 100 posts each minute. Simply watch out and think about your server’s capability. |
E-mail notifications Set CSS selectors whose values shouldn’t be empty for class and submit pages. When an empty worth is discovered utilizing these selectors, you may get an electronic mail notification. |
|
| Get information from JSON Once you allow JSON parsing for a CSS selector, you may get the values from the JSON simply. |
Superior HTML manipulations |
|
|
Computerized translation |
Computerized spinning Use spinning to routinely rewrite crawled posts’ contents to enhance search engine marketing. The plugin at the moment implements Spin Rewriter API and Turkce Spin API, that are paid providers. You may go to their web site to study the pricing particulars. |
|
|
Duplicate submit verify |
Scheduled posts You may add/take away minutes to/from the submit date. By this manner, you possibly can schedule submit publishing. |
|
| Save WooCommerce merchandise Save worth, stock, transport, attributes, and superior choices. It can save you the product as a easy or an exterior product. You may also set downloadable file choices and outline the product as digital. The choices can be found for WooCommerce variations larger than or equal to three.3. |
Choices field You might have the management! Outline many choices for the values discovered by a CSS selector. The choices embody find-replace, calculation, template, and JSON parsing settings. You may simply import/export the choices outlined within the choices bins as properly. |
|
| Deal with information like a professional Rename, copy, and transfer saved information simply. You may also outline title, description, caption, and alt texts for the saved media information utilizing templates through which you should use any brief code. Additionally it is attainable to provide random names to the saved information. |
Deal with iframes and scripts like a professional WordPress doesn’t enable displaying iframes and scripts since they pose a safety danger. You may flip iframe and script HTML components into brief codes by simply checking a checkbox. The brief code will present iframes and scripts from the allowed supply domains outlined by you. |
|
| Fast save With fast save button, it can save you the settings way more rapidly. No want to attend for web page to reload. |
Common expressions Outline common expressions in find-replace choices to find-replace something. You may also use delimiters and modifiers to match extra exactly. |
|
| Save “srcset” attributes When various sizes of the saved photographs can be found, the plugin assigns them into srcset attribute of img components in order that your pages will load quicker in several display screen sizes. |
Save “alt” and “title” attributes |
|
| Warnings Study when there’s a drawback. The plugin will present you the main points of the error so to repair it straight away. |
Deal with character encoding issues |
|
| Navigate between settings simply Repair navigation to the highest! The plugin shops the place you have been earlier than switching to a brand new tab and restores your earlier location if you activate that tab once more. No extra getting misplaced among the many settings. |
Handbook crawling software With handbook crawling software, save a number of posts by coming into their URLs. You may also enter class URLs in order that the software can get submit URLs from there. Furthermore, you possibly can set it to crawl a number of posts on the identical time. |
|
|
Add URLs to the database |
Allow/disable computerized crawling for a particular web site You may allow or disable computerized crawling for every web site individually. |
|
| Import/export You may import and export web site settings simply. Simply copy and paste the code created by the plugin. |
Limitless Add limitless websites to the plugin and activate what number of of them you need. |
|
|
Detailed dashboard |
Get updates out of your admin panel You may replace the plugin with only one click on at any time when an replace is prepared. Simply go to your updates web page in your admin panel. |
|
| Use probably the most safe PHP The plugin helps the newest variations of PHP. |
Use probably the most trendy browsers The plugin helps Chrome, Firefox, Safari, Opera, and Edge. |
|
| Interactive guides Interactive guides present you configure settings to attain sure issues, step-by-step, like a dwelling documentation. You can begin these guides everytime you need. You may even begin them from a particular step. |
On-line documentation You may verify the web documentation everytime you really feel a necessity. |
|
| Fast guides proper subsequent to the settings Every setting within the plugin has a fast information that may make it easier to perceive what every setting does. |
Video tutorials Watch video tutorials to simply discover ways to use the plugin. |
|
| Able to translate You may translate the plugin into your personal language utilizing Poedit. |
Filters |
| Necessities | PHP >= 7.3, json, mbstring, curl, dom, fileinfo, WP-Cron. These are already obtainable in most hosts. Even when the extensions will not be already lively, most internet hosting websites allow you to allow these from their management panel. See the documentation for extra info. |
| Examined with WP variations | 6.0, 5.9, 5.8, 5.7, 5.6, 5.5, 5.4, 5.3, 5.2, 5.1, 5.0 |
| Examined with WooCommerce variations | 6.5, 6.4, 6.3, 6.2, 6.0, 5.7, 5.5, 5.0, 4.9, 4.5 |
| Languages | English, Türkçe |
| Shortcomings | The plugin can not retrieve content material that’s created by utilizing JavaScript. For extra info, please see Can I get content from X site?. |
HAPPY CUSTOMERS
WHY WP CONTENT CRAWLER
Issues with crawling an internet site
- Not a straightforward job, requires superior programming expertise
- Each web site is completely different and wishes tailor-made crawling implementation
- Not simply each web site is completely different, but additionally pages of a single web site can differ
- Pages and their supply codes have to be investigated intensively to give you a crawling plan
- Figuring out save sure info in a particular place in WordPress requires information in regards to the inner construction of WordPress and the way WordPress works
- If sure info ought to be saved into a particular discipline outlined by a third-party plugin, one ought to modify the crawling implementation after researching for hours about save that info
- One ought to learn about how HTML works and extract sure elements from HTML code
- One ought to deal with all attainable inconsistencies that is perhaps within the supply codes of internet sites to supply a strong answer that may maintain working
- What if the posts have to be shared in common time intervals?
- What if you wish to crawl new posts added to an internet site after a while?
- What about translating the posts from one language to a different?
- What if the posts must be paraphrased to supply a greater search engine marketing for the web site?
- What if some info shouldn’t be retrieved?
- What if sure info ought to be modified to make it appropriate on your web site?
- What if one other web site must be crawled, not only one?
- What if that different web site wants a distinct crawling plan?
- What if it is advisable to login to the web site to crawl it?
- What if the web site modifications its supply code?
- What if you wish to replace the crawled posts by recrawling them from the unique web site?
- What if you wish to make sure that if the data is retrieved precisely as you need it earlier than routinely posting the posts to your web site?
- What if you wish to guarantee your web site’s safety by ensuring no malicious-code-executing code leads to your web site?
- And plenty of extra what-ifs that you just may not even think about until you come throughout them
Our imaginative and prescient and mission
We consider that sturdy, dependable, and automated crawling capabilities ought to be obtainable for anybody. We wish to democratize this discipline by letting anybody have these capabilities, not simply builders. With this function, we goal to supply a plugin that you’ll fall in love with and really feel at house when utilizing it. To let it accessible by anybody, we make the plugin low-cost and easy-to-use. We don’t implement the options simply to make gross sales. We plan and execute for the long run. We all the time take heed to your suggestions and make required modifications accordingly. We expect that WordPress plugins ought to be developed with enterprise-level care. So, we intensively check the plugin earlier than every launch with automated end-to-end UI exams, at the moment over 1700 exams, that run in many various environments within the cloud for a complete of over 40 hours to make sure the plugin is appropriate along with your server and WordPress environments and also you, our useful prospects, get the standard and reliability you deserve.
How we resolve these issues
We’ve got been creating WP Content material Crawler for nearly 4 years such that we’ve got come throughout nearly all of the what-ifs. Working with our prospects and listening to their wants, we offer sturdy and dependable options to those issues. We consider that one ought to simply present from which web site the data ought to be retrieved and what info ought to be retrieved from that web site after which begin crawling that web site, with out worrying in regards to the complicated behind-the-scenes operations.
To make it obtainable to anybody, we offer an in depth on-line documentation that incorporates not simply the outline of the settings however use the settings to attain your objectives. Typically you may not really feel like studying the documentation. We additionally present interactive step-by-step guides which are obtainable within the plugin, only one click on away. You can begin the interactive guides displaying you step-by-step how you are able to do sure issues any time and from any step you need.
One of the crucial distinctive options of WP Content material Crawler is the flexibility to check nearly any configuration. By this manner, you’ll not come throughout any surprises after you allow computerized crawling. When testing, the errors associated to your settings are proven so to repair them earlier than they trigger any issues.
WP Content material Crawler has so many options that even we have no idea what number of of them are there. You may routinely crawl, replace, and delete the posts, you possibly can translate posts, spin posts, you possibly can even outline what fields have to be translated or spun if you do not need all of them modified. You could find-replace nearly something. You may assign some info from the goal submit to a brief code and place that info wherever within the submit. It can save you WooCommerce merchandise. It can save you particulars for third-party plugins that we don’t even know they exist. The options of the plugin are designed such that you just really feel that you’re in management if you use them. We make them as versatile as attainable to make them suit your wants. When designing new options, we all the time needless to say you may want a extra superior model of that characteristic and we design the options accordingly. We be sure that the options and your entire code of the plugin are maintainable and extendable in order that we are able to all the time enhance the plugin.
CHANGELOG
Changelog is kept in the documentation site. Click here to see the changelog.
