FineWeb动手编码教程：流式处理、过滤、去重、分词与大规模网页语料分析
英文摘要
A hands-on tutorial streams 3,000 documents from the FineWeb sample-10BT subset without downloading the full multi-terabyte corpus. It reproduces quality filters (Gopher, C4, custom), finding most already-passed due to pre-filtering. MinHash-based deduplication with 128 permutations and 0.7 threshold identifies few near-duplicate pairs, consistent with per-crawl deduplication. GPT-2 token counts are verified against the stored field, showing near-perfect match (mean absolute difference ~0). Analytics cover token distribution, language scores, characters per token, and top domains, providing practical insights for scaling corpus preprocessing pipelines.
中文摘要
本教程流式处理了FineWeb sample-10BT的3,000篇文档，无需下载完整的多TB语料。复现了Gopher、C4及自定义质量过滤器，由于数据已预过滤，大部分文档通过检测。使用128个哈希排列和0.7阈值的MinHash去重仅发现极少数近似重复对，证实了每次爬取已去重。通过与存储字段对比验证GPT-2分词数，平均绝对差接近0，高度一致。分析涵盖token分布、语言得分、每字符token数和顶级域名，为大规模语料预处理管线提供了可操作的参考。
关键要点
Streamed 3,000 docs from FineWeb sample-10BT without downloading the full corpus, using HuggingFace datasets streaming.
流式加载FineWeb sample-10BT的3,000篇文档，无需下载完整数据集。
Reproduced Gopher, C4, and custom quality filters; most documents passed because FineWeb is pre-filtered, with a few flagged for issues like word count or boilerplate.
复现Gopher、C4和自定义质量过滤器，大部分文档通过，少数因字数、模板等问题被标记。
MinHash deduplication with 128 perm and 0.7 Jaccard threshold found very few near-duplicate pairs, aligning with per-crawl deduplication.
使用128排列、0.7阈值的MinHash去重仅发现极少量近似重复对，与每次爬取已去重吻合。
GPT-2 token count verification via tiktoken showed mean absolute difference ~0 and near 100% exact match, confirming metadata accuracy.
用tiktoken验证GPT-2分词数，平均绝对差接近0、近乎100%精确匹配，确认元数据准确。
Generated analytics: token count distribution, language score histogram with 0.65 cutoff, characters per token compression metric, and top-15 domains.
产出分析：token计数分布、0.65阈值语言得分直方图、每字符token压缩率、前15域名。