Example Run

This page records a concrete run of the profile in subblock/trainer/config.yaml. It shows the commands, effective training settings, elapsed time, curves, and files produced so you can estimate the cost of a similar job.

Reference, not a benchmark

The run used eight GPUs, but the GPU model is not recorded in the artifacts. Sequence lengths, GPU type, storage throughput, cache state, and network speed can all change the runtime substantially.

Trainer succeeded; the wrapper did not

The LLaMA-Factory trainer completed all 24 steps and wrote the model and metrics shown below, but the enclosing pipeline later exited with code 2 during post-training output handling. Treat this as measured trainer provenance, not as evidence that the historical end-to-end wrapper completed successfully.

Run at a glance

Item	Measured value
Base model	`Qwen/Qwen3.5-35B-A3B-Base`
Dataset	512 ShareGPT samples from `SWE-Lego/samples_for_llama_factory_sft`
Fine-tuning	Full SFT, 34.66B trainable parameters (98.73%)
Devices	8
Effective batch	64 (`1 per device × 8 devices × 8 accumulation`)
Schedule	3 epochs, 24 optimizer steps, cosine LR with 10% warmup
Trainer runtime	2h 6m 57s
End-to-end wall time	2h 15m 17s
Mean training loss	0.3424
Output directory size	131 GiB as reported by `du -sh`

1. Use the example profile

The relevant parts of the checked-in config.yaml are:

runtime_info:
  input:
    source:
      type: hf_lf
      hf_hub_url: SWE-Lego/samples_for_llama_factory_sft
      hf_file_name: jierun_glm52_openswe_mirror_score4_proto_oh_sdk_512_for_qwen3_5.json
    conversion:
      max_instances: 0
      data_name: glm52_openswe_mirror_score4_proto_oh_sdk_512_qwen3_5
    model:
      model_name_or_path: Qwen/Qwen3.5-35B-A3B-Base
    training:
      finetuning_type: full
      deepspeed: artifacts/training_config/deepspeed/ds_z3_config.json
      template: qwen3_5
      cutoff_len: 131072
      rope_scaling: null
      per_device_train_batch_size: 1
      gradient_accumulation_steps: 8
      learning_rate: 5.0e-5
      num_train_epochs: 3.0
      lr_scheduler_type: cosine
      warmup_ratio: 0.1
      bf16: true
      flash_attn: fa2
    infrastructure:
      n_gpus_per_node: 8

source.type: hf_lf downloads the selected ready-made LF file and skips trajectory conversion. The generated LLaMA-Factory config is saved under artifacts/training_config/ before training starts.

See Configuration for every field and Data Pipeline for other source types.

2. Launch the run

From subblock/trainer:

# First-time environment setup
bash scripts/install_env.sh

# Validate repos, environment, source, model, output, and GPU count
bash scripts/dryrun.sh

# Run preflight, data registration, training, and archival
bash scripts/start.sh

The training stage expands to an eight-process torchrun job with DeepSpeed ZeRO-3. In this example, LLaMA-Factory reported 512 examples, an effective batch of 64, and 24 optimizer steps:

steps per epoch = 512 / 64 = 8
total steps     = 8 × 3 epochs = 24

3. Time budget

All timestamps below are UTC+8.

Stage	Timestamp / duration
Pipeline started	21:29:13
Trainer loop started	21:34:47
24/24 steps and checkpoint save completed	23:41:44
Final model copy and plots completed	23:43:14
Run archive completed	23:44:30
LLaMA-Factory `train_runtime`	7,617.15 s = 2h 6m 57s
Whole `start.sh` window	8,117 s = 2h 15m 17s

The measured throughput was 0.202 samples/s and 0.003 optimizer steps/s. Environment installation is not included in this run window; the artifacts do not record whether the base model was already cached.

4. Training curves

Step loss and learning-rate curves over 24 optimizer steps

The chart is generated from trainer_log.jsonl. Loss generally decreased across the three epochs; the final step bounced slightly after the minimum at step 23. Learning rate warmed up to 5e-5 at step 4 and then followed cosine decay.

Step	Epoch	Step loss	Learning rate
1	0.125	0.5097	0
4	0.500	0.5361	5.00e-5
8	1.000	0.3792	4.57e-5
16	2.000	0.2873	1.94e-5
23	2.875	0.2068	1.11e-6
24	3.000	0.2313	2.79e-7

How to read the loss values

The train_loss value of 0.3424 in train_results.json is the mean over the run. It is not the last logged step loss (0.2313). No evaluation dataset was configured, so this run has no eval_loss or accuracy curve.

5. Generated artifacts

The run name comes from training.output_dir:

qwen3_5_35b_a3b_base_glm52_openswe_mirror_score4_proto_oh_sdk_512_gbs64pbs1acc8_lr5e-5_epo3

The main outputs are:

artifacts/
├── model/<run>/                         # 131 GiB total
│   ├── model-00001-of-00016.safetensors # final model, 16 shards / ~66 GiB
│   ├── ...
│   ├── model-00016-of-00016.safetensors
│   ├── checkpoint-24/                   # final-step checkpoint / ~66 GiB
│   ├── trainer_log.jsonl                # per-step loss and learning rate
│   ├── trainer_state.json               # full Trainer history
│   ├── train_results.json               # final metrics and runtime
│   ├── all_results.json                 # throughput and FLOPs
│   └── training_loss.png                # LLaMA-Factory loss plot
├── logs/<run>_20260716_212943.log        # full console log / 2.2 MiB
├── training_config/<run>.yaml           # generated LF config
├── wandb/offline-run-...-tgmzzhok/      # offline WandB run / 4.1 MiB
└── data/hf_data/...json                 # downloaded dataset / 121 MiB

The model index reports 70,214,363,872 bytes of weights (about 65.4 GiB). The output directory holds both checkpoint-24/ and a final root-level model copy, which is why it occupies about 131 GiB. Budget additional space for the base-model cache, environment, logs, and downloaded data.

See Results & Artifacts for the general artifact contract.

The values on this page can be verified from:

artifacts/model/<run>/train_results.json — mean loss, runtime, and throughput
artifacts/model/<run>/trainer_log.jsonl — step loss and learning rate
artifacts/model/<run>/trainer_state.json — complete Trainer state
artifacts/logs/<run>_20260716_212943.log — dataset size, batch size, parameter count, and training lifecycle