So I recently fired Godaddy as my host (yeah, yeah… about time, I know). I was forced to because a recent WordPress update completely blew up my site and despite my best efforts, a FULL restore, and consultation with one of my friends who is a WordPress expert, the site could not be fixed and it looked like Godaddy’s fault from permission problems. The hunt for a new host began. I checked out WPEngine first and although they are a great host they are far too expensive for what they offer. There are lots of other WordPress hosts out there with comparable features. I contacted several more and finally settled on Siteground. I didn’t want to bring my crap site over, so I started fresh with a new install and a brand new theme (as if you couldn’t tell already). I was going through my post-launch checklist and when I got to “post robots.txt” I encountered a problem. I checked my robots.txt file and noticed a crawl delay of 10 seconds. What? Where did that come from?? I checked my theme and all of my plugins, nothing. I re-uploaded my file and the crawl delay was still there. I checked the file using Yoast and no crawl delay there either. What the heck?? That’s when I contacted my host about it. They told me that they inject that into every robots.txt file on shared hosting because their tests show that their servers handle traffic better if they do it. (Gosh, you don’t say?! Duh!) So, they took it upon themselves to add this crawl delay to every site on their shared system. The ONLY way to get rid of this delay is to upgrade to their cloud hosting services – for $80 a month. I had no choice but to do so until I can find a new host without this needless and draconian rule.
I believe their actions are pretty underhanded and expose the HUGE underlying weakness in their systems. I get it – a shared environment is limited, but wow – fix it with better infrastructure or tighter bandwidth restrictions, not with a sneaky back-door trick that you don’t inform your users of! I’ll bet 99% of their clients have no idea they do that. Sheesh.
Update: As of 9/9/17, the default crawl delay has been removed. I chatted with Siteground’s tech support who confirmed the news. Fantastic. I’m 100% sure that this was removed due to my raising hell with them over it. I’m VERY glad they were listening. I now recommend them to all of my clients.
