机器学习与数据科学博士生系列论坛（第一百零四期）—— Pretraining Data Strategy for Large Language Models
Holder： Yuyang Sun（Peking University）
Time：2026-06-25 16:00-17:00
Location：Tencent Conference 928-6293-8217
Abstract:
Pretraining data strategy has become a central problem in large language model development, where data quality is not only about filtering noisy documents, but also about allocating a finite training budget across topics, sources, quality levels, repetitions, and training stages.
This talk reviews how pretraining data can be organized and optimized: from topic-based web mixing and quality-score fusion, to multi-source mixture search and repetition-aware modeling. We then discuss a stage-wise view of pretraining data design, where the key unit shifts from a static corpus to an executable training recipe. In this setting, high-quality data should be informative, diverse, capacity-aware, and aligned with the objectives of each training stage.
About the Speaker:
该线上论坛每两周主办一次（除了公共假期）。论坛每次邀请一位博士生就某个前沿课题做较为系统深入的介绍，主题包括但不限于机器学习、高维统计学、运筹优化和理论计算机科学。
Your participation is warmly welcomed!
欢迎扫码关注北大统计科学中心公众号，了解更多讲座信息!
