[BUG] Autotuner latency metric keeps the slowest config, and FLOPS does not match metrics.json #8766

Describe the bug

Two documented autotuning metrics do not rank the way deepspeed/autotuning/README.md describes. This is on current master (991ccf0).

1. "metric": "latency" keeps the largest latency, so ds_config_optimal.json is the slowest trial. These selectors all use >:

• Autotuner.get_best_space_record
• Autotuner.get_best_space_records (this is what write_optimal_config reads)
• ResourceManager.parse_results
• BaseTuner.tune

Throughput and FLOPS are larger-is-better. Latency is the training step time in milliseconds, so the smaller number is the better config. ModelBasedTuner.find_estimated_top_configs already reverses the sort for latency. The selectors that publish the winner do not.

2. The engine writes the FLOPS number as FLOPS_per_gpu in metrics.json (DeepSpeedEngine._autotuning_exit). The README tells users to set "metric": "FLOPS", and AUTOTUNING_METRIC_FLOPS is "flops". The readers do results[metric], so either string raises KeyError. On the fast path that read sits in run_tuning_micro_batch_sizes, outside BaseTuner.tune's try, and the autotune run aborts. Inside the tuner the same KeyError is caught and logged with logger.info, below the default log level, so the tuner stops on whatever config it already had.

To Reproduce

No GPU run is required. On master, record two finished experiments and ask for the best one:

• latency 30 and latency 8. get_best_space_record returns 30.
• a metrics.json that contains FLOPS_per_gpu and no flops key. parse_results("FLOPS") and parse_results("flops") raise KeyError.

The same > comparison is what deepspeed --autotuning tune uses when the config section is "autotuning": {"enabled": true, "metric": "latency"}.

Expected behavior

"metric": "latency" keeps the lowest latency. "FLOPS" and "flops" both read FLOPS_per_gpu. "throughput" still keeps the largest number.

ds_report output

Not a machine or install failure. The selectors above are on master and do not depend on ds_report.

Screenshots

N/A

System info (please complete the following information):

• OS: source check against deepspeedai/DeepSpeed master 991ccf0
• GPU count and types: none used
• Interconnects (if applicable): N/A
• Python version: the new tests are CPU unit tests
• Any other relevant info about your setup: N/A

Launcher context

deepspeed --autotuning tune (or run). Fast mode is enough to hit both bugs. The latency bug is in the code that writes ds_config_optimal.json. The FLOPS bug is the first time that metric is read back from metrics.json.

Docker context

N/A

Additional context

Proposed fix, in the autotuning package only:

• metric_is_better(metric, candidate, incumbent) in deepspeed/autotuning/utils.py. Latency replaces the incumbent only when it is smaller. Every other metric replaces it only when it is larger. None never wins.
• metric_value(results, metric) so "flops" and "FLOPS" resolve to FLOPS_per_gpu. If the file already has the exact key the user named, that value is used.
• Use those helpers in the four selectors above, in the fast-path metric reads, and in ModelBasedTuner.update.
• The cross-stage check that decides whether to keep searching a ZeRO stage treats a previous metric of 0 as "no previous stage", not as a measured latency of zero. A real latency is not compared against that sentinel.

Throughput ranking is unchanged. This does not change the model-based scheduling bugs, the stale metric_file in the micro-batch follow-up loop, or the swallowed exception in BaseTuner.tune.

The fix is in the pull request linked from this issue. It adds CPU tests in tests/unit/autotuning/test_autotuning.py and does not need a GPU.