基于 Rust 实现 GitHub Trending 热门仓库爬取

核心逻辑实现

1. 导入依赖与日志初始化

在 src/main.rs 中引入必要的模块。这里我们使用 anyhow::Result 统一返回类型，减少样板代码。

use anyhow::{Context, Result};
use log::info;
use reqwest::Client;
use scraper::{Html, Selector};
use serde::Serialize;
use std::fs::File;
use std::path::Path;

// 初始化日志，默认过滤级别设为 info
fn init_logger() {
    env_logger::Builder::from_env(
        env_logger::Env::default().default_filter_or("info")
    ).init();
}

2. 定义数据结构

我们需要一个结构体来存储仓库信息，并实现 Serialize trait 以便输出 JSON。字段设计尽量贴合 GitHub 页面的实际数据。

#[derive(Debug, Serialize)]
struct GithubRepo {
    author: String,
    name: String,
    description: Option<String>,
    stars: String,
    forks: String,
    today_stars: String,
    url: String,
}

3. 构建 HTTP 客户端

发送请求时，伪装 User-Agent 是防止被 GitHub 拦截的关键。同时设置合理的超时时间，避免网络波动导致程序挂起。

async fn fetch_trending_page(client: &Client) -> Result<String> {
    let url = "https://github.com/trending/rust?since=daily";
    info!("Fetching page: {}", url);

    let response = client
        .get(url)
        .header(
            "User-Agent",
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
        )
        .send()
        .await
        .context(format!("Failed to request URL: {}", url))?;

    if !response.status().is_success() {
        return Err(anyhow::anyhow!("Request failed with status: {}", response.status()));
    }

    let html = response.text().await.context("Failed to read response body")?;
    info!("Successfully fetched page (size: {} bytes)", html.len());
    Ok(html)
}

4. 解析 HTML 提取信息

这是最关键的一步。为了提高鲁棒性，我们不再依赖容易变动的类名（class），而是优先使用语义化属性（如 href 后缀或 data-* 属性）。对于缺失的数据，使用 unwrap_or_else 提供默认值，防止程序崩溃。

fn parse_repos(html: &str) -> Result<Vec<GithubRepo>> {
    info!("Starting to parse repositories...");
    let document = Html::parse_document(html);

    // 定义 CSS 选择器，优先使用语义化属性
    let repo_selector = Selector::parse("article.Box-row")
        .map_err(|e| anyhow::anyhow!("Failed to parse repo selector: {}", e))?;

    let author_name_selector = Selector::parse("h2 a")
        .map_err(|e| anyhow::anyhow!("Failed to parse author-name selector: {}", e))?;

    let desc_selector = Selector::parse("p")
        .map_err(|e| anyhow::anyhow!("Failed to parse description selector: {}", e))?;

    let stars_selector = Selector::parse("a[href$='/stargazers']")
        .map_err(|e| anyhow::anyhow!("Failed to parse stars selector: {}", e))?;

    let forks_selector = Selector::parse("a[href$='/forks']")
        .map_err(|e| anyhow::anyhow!("Failed to parse forks selector: {}", e))?;

    let today_stars_selector = Selector::parse("span[data-menu-button-text]")
        .map_err(|e| anyhow::anyhow!("Failed to parse today-stars selector: {}", e))?;

    let mut repos = Vec::new();

    for repo_node in document.select(&repo_selector) {
        // 提取作者和仓库名
        let author_name_element = repo_node
            .select(&author_name_selector)
            .next()
            .context("Missing author/name element")?;

        let author_name_text = author_name_element
            .text()
            .collect::<String>()
            .trim()
            .to_string();

        let (author, name) = author_name_text
            .split_once('/')
            .context(format!("Invalid author/name format: '{}'", author_name_text))?;

        let author = author.trim().to_string();
        let name = name.trim().to_string();

        // 拼接完整 URL
        let url = author_name_element
            .value()
            .attr("href")
            .context("Missing href attribute")?
            .to_string();
        let url = format!("https://github.com{}", url);

        // 提取描述（可选）
        let description = repo_node
            .select(&desc_selector)
            .next()
            .map(|elem| elem.text().collect::<String>().trim().to_string());

        // 提取星标、分支、今日新增星标（带容错）
        let stars = repo_node
            .select(&stars_selector)
            .next()
            .map(|elem| elem.text().collect::<String>().trim().to_string())
            .unwrap_or_else(|| "0".to_string());

        let forks = repo_node
            .select(&forks_selector)
            .next()
            .map(|elem| elem.text().collect::<String>().trim().to_string())
            .unwrap_or_else(|| "0".to_string());

        let today_stars = repo_node
            .select(&today_stars_selector)
            .next()
            .map(|elem| elem.text().collect::<String>().trim().to_string())
            .unwrap_or_else(|| "0".to_string());

        repos.push(GithubRepo {
            author,
            name,
            description,
            stars,
            forks,
            today_stars,
            url,
        });
    }

    info!("Successfully parsed {} repositories", repos.len());
    Ok(repos)
}

5. 保存结果到 JSON

使用 serde_json 将数据序列化为格式化 JSON，方便后续查看或处理。

fn save_repos_to_json(repos: &[GithubRepo], path: &str) -> Result<()> {
    info!("Saving repositories to JSON file: {}", path);
    let file = File::create(Path::new(path))
        .context(format!("Failed to create file: {}", path))?;

    serde_json::to_writer_pretty(file, repos)
        .context("Failed to serialize repos to JSON")?;

    info!("Successfully saved {} repos to {}", repos.len(), path);
    Ok(())
}

6. 主函数入口

整合上述流程，使用 tokio 启动异步运行时。

#[tokio::main]
async fn main() -> Result<()> {
    init_logger();
    info!("Starting GitHub Trending Rust Crawler...");

    let client = Client::builder()
        .connect_timeout(std::time::Duration::from_secs(10))
        .timeout(std::time::Duration::from_secs(15))
        .build()
        .context("Failed to create HTTP client")?;

    let html = fetch_trending_page(&client).await?;
    let repos = parse_repos(&html)?;
    save_repos_to_json(&repos, "trending_repos.json")?;

    info!("Crawler finished successfully! Check 'trending_repos.json' for results.");
    Ok(())
}

运行与验证

直接执行编译后的程序即可：

cargo run

如果需要更详细的调试信息，可以设置环境变量：

RUST_LOG=debug cargo run

运行成功后，项目根目录下会生成 trending_repos.json 文件。示例数据如下：

[
  {
    "author": "YaLTeR",
    "name": "niri",
    "description": "A scrollable-tiling Wayland compositor.",
    "stars": "14,823",
    "forks": "523",
    "today_stars": "0",
    "url": "https://github.com/YaLTeR/niri"
  },
  {
    "author": "librespot-org",
    "name": "librespot",
    "description": "Open Source Spotify client library",
    "stars": "6,131",
    "forks": "773",
    "today_stars": "0",
    "url": "https://github.com/librespot-org/librespot"
  }
]

总结

本项目利用 Rust 的异步特性与强大的类型系统，构建了一个健壮的 GitHub Trending 爬虫。通过 reqwest 处理网络请求，scraper 进行 DOM 解析，配合 serde 完成数据持久化。相较于传统脚本语言，Rust 在内存安全与并发性能上表现优异。优化后的代码采用了语义化 CSS 选择器与完善的错误处理机制，有效降低了因目标网站改版导致的维护成本。最终输出的 JSON 格式清晰，便于集成到自动化工作流中。