[littlepaws]

How can i block all Bad Bot, spam Bot / spdiers from crawlling my website.

I only want msn, gooogle, ask and msn

[Advertisements]

Use your .htaccess file: Blocking Bad Robots and Web Scrapers with RewriteRules

Related:

How to build a Bot Trap and keep bad bots away from a web site
User agent strings to block

Edited by Major Payne, 23 March 2009 - 03:57 PM.

[Major Payne]

Use your .htaccess file: Blocking Bad Robots and Web Scrapers with RewriteRules

Related:

How to build a Bot Trap and keep bad bots away from a web site
User agent strings to block

Edited by Major Payne, 23 March 2009 - 03:57 PM.

[littlepaws]

hi built a bottrap also added robots.txt

User-agent: Googlebot
Disallow:

User-agent: MSNBot
Disallow:

User-agent: Slurp
Disallow:

User-agent: Teoma
Disallow:

User-agent: Robozilla
Disallow:

User-agent: *
Disallow: /bot-trap/

Crawl-delay: 10
Disallow: /cgi-bin/
Disallow: /images/

Im gusing all bots should goto my bot-trap apart from the ones listed.

Have i missed anything because some bots are still getting through?

[Major Payne]

Technically, bots that do not play by the rules will not be kept out anyway. Might look into these:

Project Honeypot
Honeypots, Honeynets
Building a Honeypot

[littlepaws]

Im using this one http://danielwebb.us...tware/bot-trap/

I have created robots.txt should stop all apart from the one's shown.
Banned some heavy usage IP address's via cpanel
And Created a bot-trap

Im trying to decrease the CPU usage on my shared hosting.
I think bots are one of the cause, do you know of any other techniques.

[littlepaws]

I want to use cpanel to ban a bot found IP address 91.204.61.53

Used http://cqcounter.com/whois/ to look it up and it comes from Ukraine.

looking down the information provided:

% Information related to '91.204.61.0/24AS47359'

route: 91.204.61.0/24
descr: CityNet Network
origin: AS47359
mnt-by: CITYNET-ZP-MNT
source: RIPE # Filtered

% Information related to '91.204.60.0/22AS47359'

route: 91.204.60.0/22
descr: CityNet Network
origin: AS47359
mnt-by: CITYNET-ZP-MNT
source: RIPE # Filtered


% Information related to '91.204.60.0 - 91.204.63.255'

CAn I just place with cpanel

91.204.60.0/22
91.204.61.0/24

To block this bot?

[littlepaws]

Found 2 more:

96.249.40.91

NetRange: 96.224.0.0 - 96.255.255.255
CIDR: 96.224.0.0/11


217.212.224.182

route: 217.208.0.0/13
route: 217.212.224.0/19
Range
217.212.224.128 - 217.212.224.255

I want to ban these via cpanel.

What ip information do I use the single IP or a range or CIDR Format

Edited by littlepaws, 25 March 2009 - 04:38 AM.

[Major Payne]

If you enter a complete IP address (242.21.11.7), only that IP will be banned. If you enter a partial IP (243.21.11. or 243.21.11), any IPs that begin with the partial IP will be banned. For example, banning 243.21.11 will prevent 243.21.11.7 from accessing your board. However, 243.21.115.7 would still be able to access your board.

You may also use an '*' as a wildcard for increased flexibility. For example, if you enter 243.21.11*, many IPs will be banned including: 243.21.11.7, 243.21.115.7, 243.21.119.225. Place a space or a line break between each IP address.

See IP blocking.

If you seem to be under attack by any of these, you might consider reporting it to your hosting service via support. A friend of mine had this problem and when it was reported, it went all the way to Washington, D.C. . Next thing he knew he had Homeland Security knocking on his door. They eventually shut down several rings in Russia and China which involved getting those country's ambassadors involved. Scared him half to death. lol.