Hugging Face's logo Hugging Face
  • Models
  • Datasets
  • Spaces
  • Buckets new
  • Docs
  • Enterprise
  • Pricing
    • Website
      • Tasks
      • HuggingChat
      • Collections
      • Languages
      • Organizations
    • Community
      • Blog
      • Posts
      • Daily Papers
      • Hardware
      • Learn
      • Discord
      • Forum
      • GitHub
    • Solutions
      • Team & Enterprise
      • Hugging Face PRO
      • Enterprise Support
      • Inference Providers
      • Inference Endpoints
      • Storage Buckets

  • Log In
  • Sign Up
pietrolesci 's Collections
UnimixLM
Interesting Pre-Training Datasets
The Pile Companion
Generalisation-Profiles
Machine Translation Datasets
Text Classification Datasets
Dialogue State Tracking Datasets
NLI Eval Datasets
AnchorAL
Memorisation-Profiles
Tokenisation-Bias

Interesting Pre-Training Datasets

updated Sep 19, 2025
Upvote
-

  • Zyphra/Zyda-2

    Preview • Updated Aug 6, 2025 • 48.4k • 101

    Note Look at the preprocessing code: https://github.com/Zyphra/Zyda_processing


  • HuggingFaceTB/dclm-edu

    Viewer • Updated Mar 7, 2025 • 1B • 3.28k • 43

  • HuggingFaceFW/fineweb-edu

    Viewer • Updated Jul 11, 2025 • 3.5B • 431k • 1.31k

  • HuggingFaceTB/stack-edu

    Viewer • Updated Mar 20, 2025 • 167M • 7.41k • 81

  • HuggingFaceTB/finemath

    Viewer • Updated Feb 6, 2025 • 48.3M • 38.6k • 383

  • bigcode/the-stack

    Viewer • Updated Aug 3 • 471M • 10.5k • 1.08k

  • bigcode/the-stack-v2

    Updated Aug 3 • 6.15k • 635

  • HuggingFaceTB/smollm-corpus

    Viewer • Updated Sep 6, 2024 • 237M • 53.7k • 486

  • mlfoundations/dclm-baseline-1.0

    Viewer • Updated Jul 22, 2024 • 3.02B • 651k • 315

  • HuggingFaceFW/fineweb-2

    Viewer • Updated Oct 27, 2025 • 4.48B • 99.7k • 889
Upvote
-
  • Collection guide
  • Browse collections
Company
TOS Privacy About Careers
Website
Models Datasets Spaces Pricing Docs