intelligence.zenith_tune.evaluators.base

Base class for tuning evaluators.

This module defines the TuningEvaluator abstract base class that is used to evaluate objective values from execution results (logs, outputs, metrics, etc.).

Direction Objects

class Direction(Enum)

Optimization direction for the objective value.

TuningEvaluator Objects

class TuningEvaluator(ABC)

Abstract base class for tuning evaluators.

A tuning evaluator is responsible for evaluating execution results and extracting the objective value to be optimized. This allows the tuning system to work with any framework without requiring users to write custom objective functions.

Each evaluator specifies its optimization direction via the direction property. For example, a LossEvaluator would minimize, while a ThroughputEvaluator would maximize.

The metadata dictionary passed to evaluate contains:

• trial_id (int): The trial number.
• command (str): The command that was executed.
• parameters (dict): The hyperparameters used in this trial.
• duration (float): The wall-clock execution time in seconds.

Example:

class MyEvaluator(TuningEvaluator): direction = Direction.MAXIMIZE

def evaluate(self, stdout: str, metadata: dict[str, Any]) -> float: match = re.search(r"accuracy: (\d+.\d+)", stdout) if match: return float(match.group(1)) raise ValueError("Accuracy not found in output")

direction

Default: minimize

evaluate

@abstractmethod
def evaluate(stdout: str, metadata: dict[str, Any]) -> float

Evaluate the objective value from execution results.

Arguments:

• stdout - The stdout output from the command execution.
• metadata - Trial metadata containing trial_id, command, parameters, and duration.

Returns:

The extracted objective value as a float.

Raises:

• ValueError - If the objective value cannot be extracted.