Jump to content

Welcome to Geeks to Go - Register now for FREE

Geeks To Go is a helpful hub, where thousands of volunteer geeks quickly serve friendly answers and support. Check out the forums and get free advice from the experts. Register now to gain access to all of our features, it's FREE and only takes one minute. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more.

Create Account How it Works
Photo

Bad Bot, spam Bot / spdiers


  • Please log in to reply

#1
littlepaws

littlepaws

    Member

  • Member
  • PipPip
  • 89 posts
How can i block all Bad Bot, spam Bot / spdiers from crawlling my website.

I only want msn, gooogle, ask and msn
  • 0

Advertisements


#2
Major Payne

Major Payne

    Retired Staff

  • Retired Staff
  • 5,307 posts
Use your .htaccess file: Blocking Bad Robots and Web Scrapers with RewriteRules

Related:

How to build a Bot Trap and keep bad bots away from a web site
User agent strings to block

Edited by Major Payne, 23 March 2009 - 03:57 PM.

  • 0

#3
littlepaws

littlepaws

    Member

  • Topic Starter
  • Member
  • PipPip
  • 89 posts
hi built a bottrap also added robots.txt

User-agent: Googlebot
Disallow:

User-agent: MSNBot
Disallow:

User-agent: Slurp
Disallow:

User-agent: Teoma
Disallow:

User-agent: Robozilla
Disallow:

User-agent: *
Disallow: /bot-trap/

Crawl-delay: 10
Disallow: /cgi-bin/
Disallow: /images/

Im gusing all bots should goto my bot-trap apart from the ones listed.

Have i missed anything because some bots are still getting through?
  • 0

#4
Major Payne

Major Payne

    Retired Staff

  • Retired Staff
  • 5,307 posts
Technically, bots that do not play by the rules will not be kept out anyway. Might look into these:

Project Honeypot
Honeypots, Honeynets
Building a Honeypot
  • 0

#5
littlepaws

littlepaws

    Member

  • Topic Starter
  • Member
  • PipPip
  • 89 posts
Im using this one http://danielwebb.us...tware/bot-trap/

I have created robots.txt should stop all apart from the one's shown.
Banned some heavy usage IP address's via cpanel
And Created a bot-trap

Im trying to decrease the CPU usage on my shared hosting.
I think bots are one of the cause, do you know of any other techniques.
  • 0

#6
littlepaws

littlepaws

    Member

  • Topic Starter
  • Member
  • PipPip
  • 89 posts
I want to use cpanel to ban a bot found IP address 91.204.61.53

Used http://cqcounter.com/whois/ to look it up and it comes from Ukraine.

looking down the information provided:

% Information related to '91.204.61.0/24AS47359'

route: 91.204.61.0/24
descr: CityNet Network
origin: AS47359
mnt-by: CITYNET-ZP-MNT
source: RIPE # Filtered

% Information related to '91.204.60.0/22AS47359'

route: 91.204.60.0/22
descr: CityNet Network
origin: AS47359
mnt-by: CITYNET-ZP-MNT
source: RIPE # Filtered


% Information related to '91.204.60.0 - 91.204.63.255'

CAn I just place with cpanel

91.204.60.0/22
91.204.61.0/24

To block this bot?
  • 0

#7
littlepaws

littlepaws

    Member

  • Topic Starter
  • Member
  • PipPip
  • 89 posts
Found 2 more:

96.249.40.91

NetRange: 96.224.0.0 - 96.255.255.255
CIDR: 96.224.0.0/11


217.212.224.182

route: 217.208.0.0/13
route: 217.212.224.0/19
Range
217.212.224.128 - 217.212.224.255

I want to ban these via cpanel.

What ip information do I use the single IP or a range or CIDR Format

Edited by littlepaws, 25 March 2009 - 04:38 AM.

  • 0

#8
Major Payne

Major Payne

    Retired Staff

  • Retired Staff
  • 5,307 posts
If you enter a complete IP address (242.21.11.7), only that IP will be banned. If you enter a partial IP (243.21.11. or 243.21.11), any IPs that begin with the partial IP will be banned. For example, banning 243.21.11 will prevent 243.21.11.7 from accessing your board. However, 243.21.115.7 would still be able to access your board.

You may also use an '*' as a wildcard for increased flexibility. For example, if you enter 243.21.11*, many IPs will be banned including: 243.21.11.7, 243.21.115.7, 243.21.119.225. Place a space or a line break between each IP address.

See IP blocking.

If you seem to be under attack by any of these, you might consider reporting it to your hosting service via support. A friend of mine had this problem and when it was reported, it went all the way to Washington, D.C. . Next thing he knew he had Homeland Security knocking on his door. They eventually shut down several rings in Russia and China which involved getting those country's ambassadors involved. Scared him half to death. lol.
  • 0






Similar Topics

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

As Featured On:

Microsoft Yahoo BBC MSN PC Magazine Washington Post HP