Conversion Pipeline

STEP 0 of scripts/train.sh obtains a training-ready LLaMA-Factory dataset: it converts raw trajectories, downloads one exact Hub file, or defers to an already-ready Hub/local source. scripts/dataprep.sh exposes only the Harbor conversion path.

Input sources

config.yaml → runtime_info.input.source.type selects where the LF dataset comes from. Trajectory conversion (the flow below) only runs for harbor_job.

`source.type`	Input	Conversion	Required fields
`harbor_job` (default)	Raw Harbor trajectories	Runs STEP 0	`source.scaffold`, `source.job_dir`
`hf_lf`	An LF/ShareGPT dataset on the HuggingFace Hub	Exact file download or train-time Hub load	`source.hf_hub_url` (+ optional `hf_file_name`, `hf_subset`, `hf_split`)
`local_lf`	An existing local LF/ShareGPT json	Skipped	`source.lf_path`

For hf_lf, a non-empty source.hf_file_name downloads only that repository file under artifacts/data/hf_data/ and registers it as a local file_name. If it is empty, LLaMA-Factory registers hf_hub_url and loads the optional subset/split at train time. hf_subset is ignored in exact-file mode. Export HF_TOKEN for private datasets; do not commit it in config.yaml. For local_lf, the JSON at source.lf_path is registered as-is. Ready-made sources still use conversion.data_name as the registered key, and conversion.max_instances (>0) becomes num_samples.

Scoring and repo filtering only apply to harbor_job

The quality scoring and eval-repo exclusion happen during conversion. A hf_lf / local_lf dataset is consumed as-is, so make sure it is already clean (no SWE-bench benchmark repos) and pre-filtered upstream.

The flow

Harbor job trajectories (job_dir, per-scaffold)
    └─▶  python -m swe_data_process.<subpackage>.convert_*_to_im
              └─▶  Intermediate "IM" format (PangUML v2 JSONL, score in meta_info.unique_info)
                        └─▶  rule_score.py (auto-invoked) + optional llm_score.py
                                  └─▶  LLaMA-Factory "LF" format (ShareGPT JSON)
                                            └─▶  artifacts/data/lf_data/<dataset>.json

Conversion is invoked as python -m swe_data_process.<subpackage>... with PYTHONPATH=repos/swe_data_process/src, using the uv environment at meta_info.environment.sft_uv. The converter is selected from source.scaffold — see Scaffolds.

Run it standalone

scripts/dataprep.sh runs conversion only — no dataset registration, no training — so you can inspect and validate the data before committing GPUs to a run:

bash scripts/dataprep.sh

It reads runtime_info.input.source and runtime_info.input.conversion from config.yaml and writes the IM JSONL and LF JSON under artifacts/data/. For hf_lf or local_lf there is nothing to convert, so dataprep.sh exits early. Run scripts/train.sh to perform an exact-Hub-file download when configured, register the source, and train.

Conversion settings

From config.yaml → runtime_info.input.conversion:

Field	Meaning
`max_instances`	Harbor conversion cap; for ready-made sources, registered `num_samples`
`exclude_repos_file`	Repo exclusion list (default `artifacts/data/excluded_repos.txt`)
`data_name`	Base name for the LF file and the registered dataset key

Output layout

artifacts/data/
├── excluded_repos.txt          # eval-repo filter (owner/repo per line)
├── hf_data/                    # exact files downloaded from Hugging Face
└── lf_data/
    ├── <dataset>.json          # LF / ShareGPT training data
    └── dataset_info.json       # LLaMA-Factory dataset registration

Both the LF file and dataset_info.json live in artifacts/data/lf_data/; the generated training YAML sets dataset_dir to point there. See Dataset registration for how the data is wired into training.

Partial outputs

If a converter is interrupted, a partial IM-only or LF-only file may remain. Delete the partial file (or restore the missing pair) before rerunning, or the converter may skip work it thinks is already done.