How to optimize your Google Crawl Budget using SEM Rush Log File Analyzer and Rank Math SEO

Today I ran the SEM Rush Log File Analyzer tool, and it showed some interesting things that are a complete waste of our Google Crawl Budget.

Specifically, I learned that Google is wasting some of our crawl budget on the /wp-includes/ directory, certain plugin directories, and cache directories. I also noticed random files being crawled that don’t need to be such as “ads.txt”.

Since we use Rank Math SEO plugin on our WordPress site, I updated the robots.txt file that is dynamically-generated by that plugin as follows:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/uploads/
Disallow: ads.txt
Allow: /wp-admin/admin-ajax.php
Sitemap: https://themightymo.com/sitemap_index.xml
How to optimize your Google Crawl Budget using SEM Rush and Rank Math SEO
How to optimize your Google Crawl Budget using SEM Rush Log File Analyzer and Rank Math SEO 2

Conclusion

The Log File Analyzer in SEMrush reveals your current Google Crawl Budget. Check it regularly to monitor changes. If it’s low, use Rank Math SEO to optimize site speed and fix technical issues like 404 errors. Reduce low-quality content bloat. Ensure XML sitemaps and robots.txt are configured properly in Rank Math. Add more unique, relevant content that Google wants to crawl and index. Rank Math’s SEO analysis can identify issues hurting your budget. Keep optimizing with both tools to maintain a healthy Google Crawl Budget, leading to better indexing and rankings.

Posted in

Toby Cryns

Toby Cryns is a freelance CTO, expert WordPress consultant, and teacher.

He offers free advice to improve your freelance biz.

He also publishes small droppings every now and then to twitter.com/tobycryns and twitter.com/themightymo

Follow Toby's contributions on Github and WP.org.