| --- |
| library_name: transformers |
| license: apache-2.0 |
| datasets: |
| - allenai/tulu-3-sft-mixture |
| - lamm-mit/bio-silk-mech-mix-q-a-35K-messages-only |
| - lamm-mit/graph_reasoning_v3_messages |
| base_model: |
| - GSAI-ML/LLaDA-8B-Instruct |
| --- |
| |
| # Edit Flows base model |
|
|
|
|
| ## Training |
|
|
| Trained on a single 8xH100 node: |
| ``` |
| accelerate launch \ |
| --config_file scripts/accelerate_configs/zero2.yaml \ |
| examples/editflow/llada/adapt.py --model_name_or_path "GSAI-ML/LLaDA-8B-Instruct" \ |
| --lm_head_key "model.transformer.ff_out" \ |
| --init_editflow_from_src True \ |
| --per_device_train_batch_size 1 \ |
| --per_device_eval_batch_size 1 \ |
| --gradient_accumulation_steps 4 \ |
| --dataset_args "allenai/tulu-3-sft-mixture[train:500000]|lamm-mit/bio-silk-mech-mix-q-a-35K-messages-only|lamm-mit/graph_reasoning_v3_messages" \ |
| --output_dir "models/LlaDA-8B-EditFlow-instruct-v500" \ |
| --x0_sampler "masks[length:128]" --max_length 1500 \ |
| --num_train_epochs 4 \ |
| --learning_rate 1e-5 \ |
| --push_to_hub True --save_strategy "steps" --save_steps 1000 \ |
| --hub_model_id lamm-mit/LlaDA-8B-EditFlow-instruct-v500 \ |
| --hub_private_repo True --eval_strategy "no" \ |
| --warmup_steps 50 |
| ``` |
|
|
| ## Sampling |
| ``` |
| python examples/editflow/sample.py \ |
| --model_name_or_path "odels/LlaDA-8B-EditFlow-instruct-v500" \ |
| --mask_length 128 --seed 7070 \ |
| --prompt "Define materiomics." |
| ``` |
