I have a directory on my server that is literally jam packed full of neat softwares and other cool stuff (not exactly legal). A few weeks ago I notice _huge_ amounts of bandwidth being used for long periods at a time. First reaction was "Yay! Hits!" But then after a few days I started to get worreid as the bandwidth never slowed - it stayed 100% outgoing...
So I checked the logs and found out that Google has crawled my entire _HIDDEN_ directory with all my not-exactly-legal stuff in it. What fun!!!! :O Now, Google was done, but now come the visitors, downloading all this stuff from my server!! CRAP!!!
So, I took down the music symlink, and password protected the downloads directory... Yay! bandwidth problem fixed! Well, ... here's the problem... Google still has all 17,000+ tracks of music in the cache! So, I'm still getting visitors wanting to get this from me, but they are getting a 404. I am getting a SH1T TON of 404 errors :/ yay!
I am still getting a higher volumn of bw useage for my domain, which I cannot locate anywhere as to whats using that much bw. So, I can only assume its the bw for all these 404 errors (I get more 404 errors than I do views to all pages combined
).
Now, what I would like to do is redirect all these 404's from the music directory to a smaller custom 404 page (to cut down on bw usage even more). But this 404 would have to cover all sub directories too (sometimes 5+ deep).
Or maybe someone has a better idea on what I can do here?
