10 个 Python 爬虫入门实战示例

环境准备

安装 requests 库：

pip install requests

1. 爬取百度页面

import requests
response = requests.get("http://www.baidu.com")
response.encoding = response.apparent_encoding
print("状态码:" + str(response.status_code))
print(response.text)

2. GET 方法实例

import requests
response = requests.get("http://httpbin.org/get")
print(response.status_code)
print(response.text)

3. POST 方法实例

import requests
response = requests.post("http://httpbin.org/post")
print(response.status_code)
print(response.text)

4. PUT 方法实例

import requests
response = requests.put("http://httpbin.org/put")
print(response.status_code)
print(response.text)

5. GET 方法传参 (1)

import requests
response = requests.get("http://httpbin.org/get?name=hezhi&age=20")
print(response.status_code)
print(response.text)

6. GET 方法传参 (2)

import requests
data = {
    "name": "hezhi",
    "age": 20
}
response = requests.get("http://httpbin.org/get", params=data)
print(response.status_code)
print(response.text)

7. POST 方法传参

import requests
data = {
    "name": "hezhi",
    "age": 20
}
response = requests.post("http://httpbin.org/post", params=data)
print(response.status_code)
print(response.text)

8. 绕过反爬机制 (设置 User-Agent)

import requests
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36"
}
response = requests.get("http://www.zhihu.com", headers=headers)
print(response.status_code)
print(response.text)

9. 爬取信息并保存到本地

import requests
url = "http://www.baidu.com"
response = requests.get(url)
response.encoding = "utf-8"
print("状态码是:" + str(response.status_code))
print("响应内容:")
print(response.text)
file = open("./baidu.html", "w", encoding="utf-8")
file.write(response.text)
file.close()

10. 爬取图片，保存到本地

import requests
response = requests.get("https://www.baidu.com/img/baidu_jgylogo3.gif")
file = open("./baidu_logo.gif", "wb")
file.write(response.content)
file.close()