Python 爬虫快速入门

Python 爬虫快速入门 | 极客日志

引入模块
```
from urllib import request
```
定义 URL 路径
```
url = "http://www.baidu.com"
```
发送请求 使用 urlopen 发送请求，返回一个响应对象 response。
```
response = request.urlopen(url)
```
读取响应体 以字节形式获取网页源码。
```
raw_data = response.read()
```
编码转换 网络传输通常为字节流，需解码为文本。
- 编码：文本 -> byte (encode)
- 解码：byte -> 文本 (decode)
```
# 将字节转为 UTF-8 文本
text = response.read().decode("utf-8")
print(text)
```

写入文件 将爬取内容保存至本地。

with open('example.html', 'w', encoding='utf-8') as fp:
    fp.write(text)

常用属性与方法
- 读取状态码：response.getcode()
- 读取请求 URL：response.geturl()
- 读取响应头：response.getheaders()

引入模块
```
from urllib import parse
```

单个参数编码/解码

# 编码
result = parse.quote("小鲁班")

# 解码
result1 = parse.unquote("%E9%E1%3E")

多个参数编码方式 1 (format)

url2 = "http://www.baidu.com?name={}&pwd={}"
url1 = url2.format(parse.quote('用户名'), parse.quote('密码'))

多个参数编码方式 2 (urlencode)

obj = {
    "name": "肖",
    "password": "213123 丽华",
    "age": "五十岁"
}
base_url = "http://www.baidu.com/s?"
params = parse.urlencode(obj)
url = base_url + params

request.urlretrieve('图片路径', '保存路径')

url = "http://www.baidu.com/"
headers = {
    'User-Agent': "Mozilla/5.0 (Linux; Android 11; Pixel 5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.91 Mobile Safari/537.36"
}
req = request.Request(url=url, headers=headers)
response = request.urlopen(req)

找到 URL

post_url = "https://fanyi.baidu.com/sug"

准备表单参数
```
data_dict = {"kw": "sogreat"}
```

URL 编码与转 Bytes

data = parse.urlencode(data_dict)
data = data.encode("utf-8")

封装请求并发送

rq = request.Request(url=post_url, headers={}, data=data)
response = request.urlopen(rq)
text = response.read().decode('utf-8')

解析 JSON 数据

import json
json_obj = json.loads(text, encoding='utf-8')
for s in json_obj["data"]:
    print(s)

import json
from urllib import request, parse

headers1 = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36 Edg/100.0.1185.44"
}
newaddr = input("请输入地址:")
KFCurl = "http://www.kfc.com.cn/kfccda/ashx/GetStoreList.ashx?op=keyword"
pageindex = 1

def prepare_url(url, headers, datas):
    params = parse.urlencode(datas)
    data = params.encode(encoding="utf-8")
    req = request.Request(url=url, headers=headers, data=data)
    return req

def request_with_url(reqs):
    response1 = request.urlopen(reqs)
    address = response1.read().decode('utf-8')
    return address

def parse_data(text):
    try:
        address1 = json.loads(text)
        if len(address1['Table1']) == 0:
            return 'null'
        for addr in address1['Table1']:
            print(addr)
    except Exception as e:
        print(f"解析错误：{e}")
        return 'error'

while True:
    datas = {
        "cname": "",
        "pid": "",
        "pageIndex": pageindex,
        "pageSize": 10,
        "keyword": newaddr
    }
    reqs = prepare_url(KFCurl, headers1, datas)
    text = request_with_url(reqs)
    if parse_data(text) == "null":
        break
    pageindex += 1

import urllib.request

headers1 = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36 Edg/100.0.1185.44"
}
input_name = "%E6%96%97%E7%BD%97%E5%A4%A7%E9%99%86"  # 斗罗大陆
start_page = 1
end_page = 5

for page in range(start_page, end_page + 1):
    url = "https://tieba.baidu.com/f?kw{}=&ie=utf-8&pn={}".format(input_name, (page - 1) * 50)
    request_obj = urllib.request.Request(url=url, headers=headers1)
    response = urllib.request.urlopen(request_obj)
    text = response.read().decode('utf-8')
    
    file_path = "page{}.html".format(page)
    with open(file_path, 'w', encoding="utf-8") as fp:
        fp.write(text)
    print(file_path, "下载完毕")
print("所有内容下载完毕")

import json
from urllib import request

headers = {
    'User-Agent': "Mozilla/5.0 (Linux; Android 11; Pixel 5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.91 Mobile Safari/537.36"
}

base_url = "https://movie.douban.com/j/new_search_subjects?sort=U&range=0,10&tags=&start={}&genres=%E5%8A%A8%E7%94%BB"

for p in range(0, 4):
    total = p * 20
    url = base_url.format(total)
    req = request.Request(url=url, headers=headers)
    response = request.urlopen(req)
    text = response.read().decode('utf-8')
    json_obj = json.loads(text)
    for val in json_obj['data']:
        title = val['title']
        rate = val['rate']
        print("title:{},rate:{}".format(title, rate))

try:
    response = request.urlopen(req, timeout=10)
except Exception as e:
    print(f"请求失败：{e}")

import time
import random
time.sleep(random.uniform(1, 3))

Python 爬虫快速入门

Python 爬虫快速入门

1. 基本类库

1.1 urllib.request (请求)

1.2 urllib.parse (URL 编码)

1.3 urlretrieve (便捷下载)

1.4 伪装浏览器访问

2. 处理常见 URL 请求

2.1 定位 URL

2.2 处理 POST 请求

案例：抓取百度翻译

案例：抓取 KFC 餐厅地址信息

2.3 处理 GET 请求

案例：抓取百度贴吧信息

2.4 处理 Ajax 请求

案例：爬取豆瓣电影动画排行榜

3. 异常处理与最佳实践

3.1 异常捕获

3.2 遵守 robots.txt

3.3 设置延时

3.4 法律合规

更多推荐文章

相关免费在线工具

Python 爬虫快速入门

Python 爬虫快速入门

1. 基本类库

1.1 urllib.request (请求)

1.2 urllib.parse (URL 编码)

1.3 urlretrieve (便捷下载)

1.4 伪装浏览器访问

2. 处理常见 URL 请求

2.1 定位 URL

2.2 处理 POST 请求

案例：抓取百度翻译

案例：抓取 KFC 餐厅地址信息

2.3 处理 GET 请求

案例：抓取百度贴吧信息

2.4 处理 Ajax 请求

案例：爬取豆瓣电影动画排行榜

3. 异常处理与最佳实践

3.1 异常捕获

3.2 遵守 robots.txt

3.3 设置延时

3.4 法律合规

微信扫一扫，关注极客日志

更多推荐文章

相关免费在线工具