Jump to content

Welcome to Geeks to Go - Register now for FREE

Geeks To Go is a helpful hub, where thousands of volunteer geeks quickly serve friendly answers and support. Check out the forums and get free advice from the experts. Register now to gain access to all of our features, it's FREE and only takes one minute. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more.

Create Account How it Works
Photo

Python Web Scraping


  • Please log in to reply

#1
Camarov

Camarov

    New Member

  • Member
  • Pip
  • 1 posts

Can anyone please help me web scrape in python, I have used BeautifulSoup but when the code prints it looks like "b'code'". But because I need to add it to another URL it adds the b'' in which I can't have. What can you say about this https://mydataprovider.com/sites/ .Please help.


Edited by Camarov, 02 March 2020 - 01:58 AM.

  • 0

Advertisements


#2
martinsmith123

martinsmith123

    Member

  • Member
  • PipPip
  • 43 posts
Scrapy is a Python framework for large scale web scraping It gives you all the tools you need to efficiently extract data from websites, process them as you want, and store them in your preferred structure and format if you start writing code for every little task you perform, you will eventually end up creating your own scraping framework. Scrapy is that framework.
Writing Custom Spiders
 a spider is a program that downloads content from web sites or a given URL. When extracting data on a larger scale, you would need to write custom spiders for different websites since there is no “one size fits all” approach in web scraping owing to diversity in website designs. You also would need to write code to convert the extracted data to a structured format and store it in a reusable format like CSV, JSON, excel That’s a lot of code to write, luckily scrapy comes with most of these functionality built in.

  • 0






Similar Topics

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

As Featured On:

Microsoft Yahoo BBC MSN PC Magazine Washington Post HP