Jump to content

Welcome to Geeks to Go - Register now for FREE

Geeks To Go is a helpful hub, where thousands of volunteer geeks quickly serve friendly answers and support. Check out the forums and get free advice from the experts. Register now to gain access to all of our features, it's FREE and only takes one minute. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more.

Create Account How it Works
Photo

Can You Surf An Entire Website Stored In The Wayback Machine Or Just T

- - - - -

  • Please log in to reply

#1
Shah Saeed

Shah Saeed

    Member

  • Member
  • PipPip
  • 20 posts

I Have Tried Using Wayback Machine To Browse Old Websites Just For Fun.But The Thing Is That If I Click Search A Term on Youtube Or Google This Message Comes Up

 

Page cannot be crawled or displayed due to robots.txt.

 
 
Is There Any Way To Search An Entire Website On The Wayback Machine Or Does It Only Display Snapshots?????
 
Also Does robots.txt Completely Remove A Link Or Page Or Does It Just Make It Inaccessible????????And Is There Anyway To Bypass It?????

Edited by Shah Saeed, 19 January 2016 - 10:44 AM.

  • 0

Advertisements


#2
paws

paws

    WTT Tech Teacher

  • Tech Academy Moderator
  • 917 posts

Lots of information here:

https://archive.org/...on-to-robotstxt

 

Its not looking good for you... :no:

Regards

paws


  • 0






Similar Topics

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

As Featured On:

Microsoft Yahoo BBC MSN PC Magazine Washington Post HP