利用 Web Unlocker API 实现亚马逊数据的高效采集

（2）编写代码

将上一步获取的凭证信息填入代码中。注意代理认证格式需符合平台规范。

import requests
from bs4 import BeautifulSoup
import pandas as pd
import warnings

# 忽略 SSL 警告
warnings.filterwarnings('ignore', message='Unverified HTTPS request')

# 您的 Bright Data 凭证
customer_id = "brd-customer-hl_da15f828"
zone_name = "web_unlocker3"
zone_password = "q9crj4rw9004"

# 代理设置
proxy_url = "brd.superproxy.io:33335"
proxy_auth = f"brd-customer-{customer_id}-zone-{zone_name}:{zone_password}"
proxies = {
    "http": f"http://{proxy_auth}@{proxy_url}",
    "https": f"http://{proxy_auth}@{proxy_url}"
}

# 目标亚马逊搜索 URL
target_url = "https://www.amazon.com/s?k=gaming&language=zh&_encoding=UTF8"

# 模拟真实浏览器请求头
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "Referer": "https://www.amazon.com/"
}

try:
    print("正在发送请求...")
    response = requests.get(target_url, proxies=proxies, headers=headers, verify=False)
    print(f"请求状态码：{response.status_code}")

    # 保存原始 HTML
    with open("amazon_gaming_search.html", "w", encoding="utf-8") as file:
        file.write(response.text)
    print("HTML 已保存到 amazon_gaming_search.html")

    # 解析结果
    soup = BeautifulSoup(response.text, "html.parser")
    search_results = []

    # 选择器定位产品卡片
    product_cards = soup.select(".s-result-item[data-asin]:not([data-asin=''])")
    print(f"找到 {len(product_cards)} 个产品")

    for card in product_cards:
        asin = card.get("data-asin")
        try:
            title_element = card.select_one("h2 a span")
            title = title_element.text.strip() if title_element else "N/A"
            
            price_element = card.select_one(".a-price .a-offscreen")
            price = price_element.text.strip() if price_element else "N/A"
            
            rating_element = card.select_one(".a-icon-star-small")
            rating = rating_element.text.strip() if rating_element else "N/A"
            
            reviews_element = card.select_one("span.a-size-base.s-underline-text")
            reviews = reviews_element.text.strip() if reviews_element else "N/A"

            search_results.append({
                "asin": asin,
                "title": title,
                "price": price,
                "rating": rating,
                "reviews": reviews,
                "url": f"https://www.amazon.com/dp/{asin}"
            })
            print(f"已解析：{title[:30]}...")
        except Exception as e:
            print(f"解析产品 {asin} 时出错：{str(e)}")

    # 导出 CSV
    if search_results:
        df = pd.DataFrame(search_results)
        df.to_csv("amazon_gaming_search_results.csv", index=False, encoding="utf-8-sig")
        print(f"成功抓取 {len(search_results)} 条数据，已保存至 CSV")
        print(df.head())
    else:
        print("未找到有效搜索结果")

except Exception as e:
    print(f"请求失败：{str(e)}")

6. 运行结果

脚本执行成功后，会生成 HTML 备份文件以及包含结构化数据的 CSV 文件，方便后续分析。

三、Web Scraper 方案

除了 API 方式，Web Scraper 提供了更灵活的 GUI 操作体验。它内置了网站解锁功能，可自动处理 IP 轮换和验证码，无需维护底层基础设施。

1. 快速上手

通过图形化界面配置抓取规则，支持提取标题、卖家、品牌、价格等字段，输出格式可选 JSON、NDJSON 或 CSV。

2. 代码集成示例

若需结合 Selenium 进行深度交互，可使用如下逻辑定位元素并写入文件。

# 获取商品信息
product_elements = driver.find_elements(By.CSS_SELECTOR, ".s-main-slot .s-result-item")

with open('amazon_products.csv', 'w', newline='', encoding='gbk') as csvfile:
    fieldnames = ['Title', 'Price', 'Image URL']
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    
    for index, product in enumerate(product_elements):
        try:
            title = product.find_element(By.CSS_SELECTOR, ".a-text-normal").text
            price = product.find_element(By.CSS_SELECTOR, ".a-price-whole").text
            image_url = product.find_element(By.CSS_SELECTOR, "img.s-image").get_attribute("src")
            
            writer.writerow({'Title': title, 'Price': price, 'Image URL': image_url})
        except Exception as e:
            print(f"跳过第 {index + 1} 个商品：{str(e)}")
        time.sleep(2)

driver.quit()

四、SERP API

SERP API 专注于搜索引擎结果页的数据收集。它模拟真实浏览器行为并提供完整的 JavaScript 支持，能有效绕过搜索引擎的访问限制，实时提供结构化的搜索数据。

五、总结

Web Unlocker API、Web Scraper 及 SERP API 共同构成了完整的数据采集解决方案。Web Unlocker 侧重于自动化解锁与指纹伪装，适合高难度目标；Web Scraper 提供可视化配置，降低技术门槛；SERP API 则扩展了搜索场景的数据获取能力。合理组合这些工具，可在无需深厚编程背景的情况下，实现专业级的数据采集工作。