基于 Rust 实现 GitHub Trending 热门仓库爬虫

核心逻辑实现

1. 导入模块与初始化

我们需要引入必要的库来管理日志、HTTP 请求、HTML 解析以及数据序列化。初始化日志有助于在运行过程中观察程序状态。

use anyhow::{Context, Result};
use log::info;
use reqwest::Client;
use scraper::{Html, Selector};
use serde::Serialize;
use std::fs::File;
use std::path::Path;

fn init_logger() {
    env_logger::Builder::from_env(env_logger::Env::default().default_filter_or("info")).init();
}

2. 定义数据结构

为了将网页数据转换为 JSON，我们需要定义一个结构体来映射 GitHub 页面的字段。使用 #[derive(Serialize)] 可以自动生成序列化逻辑。

#[derive(Debug, Serialize)]
struct GithubRepo {
    author: String,
    name: String,
    description: Option<String>,
    stars: String,
    forks: String,
    today_stars: String,
    url: String,
}

3. 获取页面内容

发送 GET 请求时，必须设置 User-Agent 模拟浏览器行为，否则容易被 GitHub 拦截。同时设置超时时间，防止网络异常导致程序挂起。

async fn fetch_trending_page(client: &Client) -> Result<String> {
    let url = "https://github.com/trending/rust?since=daily";
    info!("Fetching page: {}", url);

    let response = client
        .get(url)
        .header(
            "User-Agent",
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
        )
        .send()
        .await
        .context(format!("Failed to request URL: {}", url))?;

    if !response.status().is_success() {
        return Err(anyhow::anyhow!("Request failed with status: {}", response.status()));
    }

    let html = response.text().await.context("Failed to read response body")?;
    info!("Successfully fetched page (size: {} bytes)", html.len());
    Ok(html)
}

4. 解析 HTML 提取数据

这是最关键的一步。为了提高鲁棒性，我们尽量使用语义化的属性（如 href 后缀、data 属性）而非易变的类名。对于缺失的数据，使用默认值避免程序崩溃。

fn parse_repos(html: &str) -> Result<Vec<GithubRepo>> {
    info!("Starting to parse repositories...");
    let document = Html::parse_document(html);

    // 定义 CSS 选择器
    let repo_selector = Selector::parse("article.Box-row")
        .map_err(|e| anyhow::anyhow!("Failed to parse repo selector: {}", e))?;
    let author_name_selector = Selector::parse("h2 a")
        .map_err(|e| anyhow::anyhow!("Failed to parse author-name selector: {}", e))?;
    let desc_selector = Selector::parse("p")
        .map_err(|e| anyhow::anyhow!("Failed to parse description selector: {}", e))?;
    let stars_selector = Selector::parse("a[href$='/stargazers']")
        .map_err(|e| anyhow::anyhow!("Failed to parse stars selector: {}", e))?;
    let forks_selector = Selector::parse("a[href$='/forks']")
        .map_err(|e| anyhow::anyhow!("Failed to parse forks selector: {}", e))?;
    let today_stars_selector = Selector::parse("span[data-menu-button-text]")
        .map_err(|e| anyhow::anyhow!("Failed to parse today-stars selector: {}", e))?;

    let mut repos = Vec::new();

    for repo_node in document.select(&repo_selector) {
        // 提取作者和仓库名
        let author_name_element = repo_node
            .select(&author_name_selector)
            .next()
            .context("Missing author/name element")?;

        let author_name_text = author_name_element
            .text()
            .collect::<String>()
            .trim()
            .to_string();

        let (author, name) = author_name_text
            .split_once('/')
            .context(format!("Invalid author/name format: '{}'", author_name_text))?;

        let author = author.trim().to_string();
        let name = name.trim().to_string();

        // 拼接完整链接
        let url = author_name_element
            .value()
            .attr("href")
            .context("Missing href attribute")?
            .to_string();
        let url = format!("https://github.com{}", url);

        // 提取描述（可选）
        let description = repo_node
            .select(&desc_selector)
            .next()
            .map(|elem| elem.text().collect::<String>().trim().to_string());

        // 提取统计信息，缺失则默认为 "0"
        let stars = repo_node
            .select(&stars_selector)
            .next()
            .map(|elem| elem.text().collect::<String>().trim().to_string())
            .unwrap_or_else(|| "0".to_string());

        let forks = repo_node
            .select(&forks_selector)
            .next()
            .map(|elem| elem.text().collect::<String>().trim().to_string())
            .unwrap_or_else(|| "0".to_string());

        let today_stars = repo_node
            .select(&today_stars_selector)
            .next()
            .map(|elem| elem.text().collect::<String>().trim().to_string())
            .unwrap_or_else(|| "0".to_string());

        repos.push(GithubRepo {
            author,
            name,
            description,
            stars,
            forks,
            today_stars,
            url,
        });
    }

    info!("Successfully parsed {} repositories", repos.len());
    Ok(repos)
}

5. 保存结果

将解析后的列表序列化为格式化的 JSON 文件，方便后续查看或处理。

fn save_repos_to_json(repos: &[GithubRepo], path: &str) -> Result<()> {
    info!("Saving repositories to JSON file: {}", path);
    let file = File::create(Path::new(path))
        .context(format!("Failed to create file: {}", path))?;

    serde_json::to_writer_pretty(file, repos)
        .context("Failed to serialize repos to JSON")?;

    info!("Successfully saved {} repos to {}", repos.len(), path);
    Ok(())
}

6. 主函数入口

使用 tokio 启动异步运行时，串联起请求、解析和保存的流程。

#[tokio::main]
async fn main() -> Result<()> {
    init_logger();
    info!("Starting GitHub Trending Rust Crawler...");

    let client = Client::builder()
        .connect_timeout(std::time::Duration::from_secs(10))
        .timeout(std::time::Duration::from_secs(15))
        .build()
        .context("Failed to create HTTP client")?;

    let html = fetch_trending_page(&client).await?;
    let repos = parse_repos(&html)?;
    save_repos_to_json(&repos, "trending_repos.json")?;

    info!("Crawler finished successfully! Check 'trending_repos.json' for results.");
    Ok(())
}

运行与验证

直接执行以下命令即可运行爬虫：

cargo run

若需查看详细调试日志，可设置环境变量：

RUST_LOG=debug cargo run

运行成功后，当前目录下会生成 trending_repos.json 文件，其中包含所有抓取到的仓库信息。示例数据如下：

[
  {
    "author": "YaLTeR",
    "name": "niri",
    "description": "A scrollable-tiling Wayland compositor.",
    "stars": "14,823",
    "forks": "523",
    "today_stars": "0",
    "url": "https://github.com/YaLTeR/niri"
  },
  {
    "author": "librespot-org",
    "name": "librespot",
    "description": "Open Source Spotify client library",
    "stars": "6,131",
    "forks": "773",
    "today_stars": "0",
    "url": "https://github.com/librespot-org/librespot"
  }
]

总结

本项目利用 Rust 生态中的成熟库构建了高效的异步爬虫。通过 reqwest 处理网络请求，scraper 解析 HTML，配合 tokio 实现高并发能力。代码特别强化了 CSS 选择器的语义化匹配与异常容错处理，降低了因目标网站改版导致的维护成本。最终输出的结构化 JSON 数据可直接用于数据分析或二次开发。