Experiment tracking¶
Training metrics flow through a Reporter attached to the pipeline as a ReportCallback. See the report API. Examples do not call wandb.log directly from the pipeline; configuration lives in Hydra under reporter, mlflow, and wandb.
For the Hydra entry and teardown hooks, see Running an experiment. For log intervals and halt behavior inside the loop, see Training loop.
Choosing a backend¶
Set reporter in config (string, case-insensitive):
| Value | Implementation | Notes |
|---|---|---|
none |
NullReporter |
No tracking run; metrics still go to the file logger when configured |
mlflow |
MLflowReporter |
Default in most examples |
wandb |
WandbReporter |
Uses cfg.wandb |
get_reporter builds the backend. The run is not active until open().
MLflow fields (cfg.mlflow)¶
| Field | Purpose |
|---|---|
tracking_uri |
e.g. sqlite:///artifacts/mlflow.db for local SQLite |
artifact_location |
e.g. artifacts/mlruns |
experiment |
MLflow experiment name |
WandB fields (cfg.wandb)¶
| Field | Purpose |
|---|---|
project |
WandB project |
entity |
Team or user |
group |
Run grouping |
mode |
e.g. offline |
reinit |
Whether to start a fresh run (default true in examples) |
Example defaults (example/butterfly_edm/config.yaml):
reporter: mlflow
mlflow:
tracking_uri: sqlite:///artifacts/mlflow.db
artifact_location: artifacts/mlruns
experiment: test
wandb:
reinit: true
mode: offline
project: test
group: default
entity: superposed-tree
Lifecycle¶
On the main process only, run_experiment:
- Asks the task for
run_nameand resolves the Hydra config. - Asks the task for
resume_run_id.TrainSpecreads it from the latest checkpoint directory when continuing a run;InferSpecalways opens a new run named<training run>-infer. - Enters
ReporterContext, which callsopen()on enter andclose(success=...)on exit.
After the LR scheduler is built, TrainSpec calls reporter.store_config with the post-scheduler config so the tracked run matches what training actually uses. Task hooks reach the same run through run.reporter, which is None off the main process.
Pipeline connection¶
reporter.as_callback() registers a ReportCallback on the pipeline:
| Event | Action |
|---|---|
LOG |
reporter.log(metrics, step=...) |
PRE_FINALIZE_SAVE |
Writes report_run.json into the checkpoint directory about to be published |
Other callback events are ignored by the report callback.
Resume with report_run.json¶
When a checkpoint is published, the main process may write report_run.json:
{"backend": "mlflow", "run_id": "<id>"}
On the next launch, resume_run_id(latest_checkpoint_dir, reporter_kind) returns that id only when the file’s backend matches the configured reporter. The reporter reopens or attaches to the same run.
Checkpoint layout and distributed save rules are described under Checkpoints in the training guide.
Distributed failures¶
Reporter open on main. If opening the reporter or logger fails on the main process, that rank sets Accelerate’s trigger; every rank observes the failure, destroys the process group, and the main re-raises the original error.
During training. Reporter errors on the main process are deferred and shared with other ranks on log intervals, after eval, and when leaving the loop. See Halt in the training guide.