Skip to content

About

Pretraining data reconstruction scripts for Apertus

Resources

Stars

170 stars

Watchers

5 watching

Forks

Latest commit

 

History

4 Commits

Folders and files

Repository files navigation

This repository contains the pipelines generating the pretraining data used to train Apertus v1 and v1.5. These pipelines are built on top of datatrove, a framework for large scale data processing. This project extends datatrove with custom modules and Apertus pretraining data pipeline. Note that the pipelines are created to work in the CSCS infrastructure on Alps and needs to be adjusted to specific environment to before run out of the box.

Project Structure 📂

The repository is organized to separate the core code from the examples and documentation.

  • src/data_pipeline_pretrain: This directory contains all the custom datatrove extensions, including new annotators, filters, formatters and tokenizers that are not part of the standard datatrove library.
  • pipelines/: This folder holds the main pipeline definitions. Each subdirectory within pipelines corresponds to a specific pretraining data. For details refer to next Section Pretraining Data.
  • examples/: A collection of example scripts demonstrating how to annotate the FineWeb samples with embeddings, toxicity score, code quality or how to tokenize the data to be used in Megatron.

How to Run a Pipeline 🏃‍♂️

Important

Due to the infrastructure-specific nature of these pipelines (designed for CSCS on Alps), they are not directly runnable out of the box. You will need to adjust the scripts and configurations to your specific computing environment.

General Workflow

  1. Clone the repository: First, ensure you have the project cloned to your local environment.
  2. Set up the environment: The project's dependencies are defined in the Dockerfile. To set up the environment, either build the Docker container as specified or replicate the environment manually based on the Dockerfile contents.
  3. Select a pipeline and review its configuration: Before running the pipelines, all necessary datasets need to be downloaded first. Then, Navigate to the pipeline you want to run, for example, pipelines/fineweb-2/main.py. The paths and configurations are defined within this file. Adjust them according to your environment, including any SLURM configurations or by switching to the LocalPipelineExecutor for local execution.
  4. Execute the pipeline: The pipelines are designed to be executed directly as Python scripts. You can run them in your environment using commands like python pipelines/euroblocks/main.py or with specific arguments, such as python pipelines/fineweb-2/main.py quality_10-filterrobots.

Pretraining Data ✨

This section lists the specific datasets and the corresponding scripts to process them. If you want to apply robots.txt filtering, first download the respective metadata from swiss-ai/datasets.

  • FineWeb-HQ: First Download FineWeb (v1.3.0) and the English HQ classifier, then adjust and run python pipelines/fineweb/main.py quality_33-filterrobots or python pipelines/fineweb/main.py quality_33-filterrobots.

  • FineWeb2-HQ: First Download FineWeb-2 (v1.0.2) and the multilingual HQ classifiers, then adjust and run python pipelines/fineweb/main.py quality_33-filterrobots or python pipelines/fineweb/main.py quality_33-filterrobots.

  • DCLM-Edu: First Download DCLM-Edu and run python pipelines/dclm-edu/main.py quality_33-filterrobots or python pipelines/fineweb/main.py filterrobots_fine.

  • FineWeb-Edu: First Download FineWeb-Edu-Score-2 or FineWeb-Edu, then adjust and run python pipelines/fineweb/main.py filterrobots or python pipelines/fineweb/main.py filterrobots .

  • Code: Datasets consists of two original datasets:

    1. StarCoderData
    2. CommonPile/Stackv2

    To download the datasets we can utilize downloader.py at the examples/code_pipeline/. CommonPile/Stackv2 is a open-licensed filtered version of The Stackv2 annotated with the educational value ranging from 0-4. In order to follow similar format we decided to create the following pipeline at: examples/code_pipeline/, consisting of following steps:

    1. Annotation annotator.py: Annotation of some of the samples from per language using Qwen/Qwen2.5-Coder-32B-Instruct
    2. Processor postprocessor.py: Processing and formatting of the annotations in order to train the classifiers for each language and unify metrics from annotations.
    3. Training classifiers: Training of the classifiers, based on the microsoft/codebert-base.
    4. Classifying classify.py: Finally, classification of all samples from each of the languages and finally utilizing postprocessor in order to simplify the metrics. This results in the final dataset with code quality values from 0-4, that we be obtained by applying classifiers to the individual languages in the dataset.
  • Math: First Download FineMath or MegaMath, then adjust and run python pipelines/finemath/main.py filterrobots_fine or python pipelines/megamath/main.py filterrobots.

  • Cooldown Download Euroblocks or Europarl and run python pipelines/fineweb/main_bidirectional.py, python pipelines/euroblocks/main.py, respectively. For Paradocs you need to adjust and run python pretrain-data/pipelines/paradocs/preprocessing/run.py, then pretrain-data/pipelines/paradocs/main.py.

Finally to tokenize the data to be used by megatron run:

python3 examples/preprocess_megatron.py \
--tokenizer-name-or-path swiss-ai/Apertus-70B-Instruct-2509 \
--output-folder $output_folder \
--logging-dir $logging_dir \
--n-tasks $number_of_tasks \
--dataset $input_folder \
--column $COLUMN_KEY

About

Pretraining data reconstruction scripts for Apertus

Resources

Stars

170 stars

Watchers

5 watching

Forks

Releases

Packages

Contributors

Languages