More Bad News for Cuil

Cuil, the recently launched search engine and purported ‘Google Killer,’ is in the news again. And unfortunately, its for another round of bad press. Apparently, their robot, Twiceler, greedily consumes more than its share of bandwidth when crawling the web, causing many websites to go down.

Tech Crunch, which aired this story yesterday, was tipped off by a webmaster whose sites had fallen victim. Tech Crunch quotes the tipster:

“I don’t know what spawned it, but when Cuil attempts to index a site, it does so by completely hammering it with traffic. So much, that it completely brings the site down.”

And this webmaster is not alone. Apparently, many other websites have been hit hard as well, suffering unusually high loads and downtime as a result.

With so many robots, web crawlers, and spiders around the internet, you might be wondering why Cuil’s bot alone has caused so much damage. This is because of the reckless methods that it uses to find pages. Tech Crunch summarizes the prevailing opinion:

Website owners are also saying that the way Cuil indexes sites isn’t scientific in any way and is actually quite “amateurish.” According to those who experienced the Twiceler onslaught, the bot seems to “randomly hit a site and continue to guess and generate pseudo-random URLs in an attempt to find pages that aren’t accessible by links. And by doing this, they completely bring a site down to where it’s not functional.”

Have You Been A Victim?

If you have experienced some problems with your website, you might want to look at your log files to see if Cuil was the culprit. According to Cuil, Twiceler uses the following IP addresses:

• 38.99.13.121 to 38.99.13.126
• 38.99.44.101 to 38.99.44.106
• 64.1.215.162 to 64.1.215.166
• 208.36.144.6 to 208.36.144.10

And if you find that Cuil has indeed been repeatedly accessing your website and generating 404 Errors, it might be necessary to block it. Click here to get info from Cuil about restricting Twiceler.

Surely, Cuil has a lot of improvements to make before it becomes a viable search engine. And if its current problems are not bad enough, there might be some more trouble looming on the horizon. For example, some webmasters who have suffered downtime and lost revenue want to fight back. In fact, there are rumors — as well as some unsubstantiated claims — that a class action lawsuit may be filed against Cuil this week. It will be interesting to see how this story develops.