experiment¶
Entry points for Hydra examples: the generic run_experiment lifecycle plus the training and inference tasks. For the guide, see Running an experiment.
mlkit.util.experiment
¶
InferRun
dataclass
¶
Bases: RunContext
Inference state with trained weights loaded.
model_bundle["model"] is on accelerator.device in eval mode. train_cfg
is the config saved with the checkpoint, or None for checkpoints without one.
The datasets are None without get_dataset. outputs holds what
infer returned.
Source code in src/mlkit/util/experiment/infer.py
24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | |
InferSpec
dataclass
¶
Load trained weights, then run inference hooks.
Weights come from cfg.infer.checkpoint_dir when set, otherwise from the
latest checkpoint under the training save directory for run_suffix. Use the
same run_suffix and training overrides as the training run. get_model
builds the model as for training; its "train_hook_handlers" are disabled.
The global main logs a warning for every model-side config value that differs
from the config saved with the checkpoint.
Hook timing (relative to accelerator.end_training()):
infer: every rank, while the process group is alive (sharded inference,
collectives). Its return value is stored in run.outputs.
on_finish: global main only, inside :meth:finish after end_training
(sampling, plots; single-process work).
Both hooks run under accelerator.autocast(), so infer.mixed_precision
applies to them.
Source code in src/mlkit/util/experiment/infer.py
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 | |
RunContext
dataclass
¶
Process state that :func:run_experiment hands to a :class:Task.
logger and reporter are None off the global main process.
Source code in src/mlkit/util/experiment/runner.py
30 31 32 33 34 35 36 37 38 39 40 | |
Task
¶
Bases: ABC
One job that :func:run_experiment runs inside a single process group.
cfg[section] supplies seed and mixed_precision.
Hook timing (relative to accelerator.end_training()):
run: every rank, while the process group is alive.
finish: global main only, after end_training, while the reporter
is still open.
Source code in src/mlkit/util/experiment/runner.py
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | |
run_name(cfg)
abstractmethod
¶
Return the tracked run name. Called on the global main only.
Source code in src/mlkit/util/experiment/runner.py
56 57 58 | |
resume_run_id(cfg, backend)
¶
Return the tracked run to continue, or None for a new run.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cfg
|
DictConfig
|
experiment config. |
required |
backend
|
str
|
reporter that is about to be opened. |
required |
Source code in src/mlkit/util/experiment/runner.py
60 61 62 63 64 65 66 67 68 | |
run(ctx)
abstractmethod
¶
Do the work on every rank and return the state handed to :meth:finish.
Source code in src/mlkit/util/experiment/runner.py
70 71 72 | |
finish(run)
¶
Do single-process work on the global main after the process group is gone.
Source code in src/mlkit/util/experiment/runner.py
74 75 76 | |
ConfigSnapshotCallback
¶
Bases: BaseCallback
Write the resolved experiment config into each checkpoint before it is published.
The config is read at save time, so rewrites made before training (LR scheduler length) are included.
Source code in src/mlkit/util/experiment/train.py
47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | |
TrainRun
dataclass
¶
Bases: RunContext
Training state built before pipeline.train().
accelerator is pipeline.accelerator: the instance that called prepare
on the model, optimizer, dataloader, and scheduler (use it for save_state,
unwrap_model, clip_grad_norm_, and training collectives). run_experiment
owns a separate session Accelerator for process-group lifecycle only.
Teardown hooks receive the same instance after training; pipeline reflects
post-train state.
Source code in src/mlkit/util/experiment/train.py
77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 | |
TrainSpec
dataclass
¶
Train one model with the Accelerate pipeline.
get_model may return a bare model or a bundle dict with "model" plus
optional "callbacks" (passed to the pipeline constructor, so they see
POST_LOAD) and "train_hook_handlers". run_suffix is appended to
both the run name and the checkpoint directory. Every published checkpoint
also holds the resolved config as :data:CONFIG_SNAPSHOT_NAME.
Hook timing (relative to accelerator.end_training()):
after_train: every rank, after pipeline.train() while the process
group is still alive (optional second training, shared reporter).
on_finish: global main only, inside :meth:finish after end_training.
Source code in src/mlkit/util/experiment/train.py
102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 | |
run_experiment(cfg, task)
¶
Run task inside one Accelerate process group with logging and tracking.
The global main opens the log file and the reporter first; a failure there stops
every rank before task.run. A failure inside task.run destroys the group
without a barrier and skips task.finish. Otherwise every rank waits, the group
is destroyed, a deferred rank-local failure is raised, and task.finish runs on
the global main. A failure to destroy the group is logged, not raised.
Returns:
| Type | Description |
|---|---|
R
|
The state returned by |
Source code in src/mlkit/util/experiment/runner.py
112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 | |
checkpoint_root(cfg, suffix='')
¶
Return the directory holding the checkpoint_* folders of the training run.
Returns:
| Type | Description |
|---|---|
str
|
|
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cfg
|
DictConfig
|
experiment config; |
required |
suffix
|
str
|
the training task's |
''
|
Source code in src/mlkit/util/experiment/train.py
28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | |