intelligence.zenith_tune.evaluators.registry

Registry for tuning evaluators.

EvaluatorRegistry Objects

class EvaluatorRegistry()

Registry for tuning evaluator classes.

Example:

@EvaluatorRegistry.register("duration") class DurationEvaluator(TuningEvaluator): ...

evaluator = EvaluatorRegistry.get("duration") evaluator = EvaluatorRegistry.get("regex", pattern=r"loss: (\d+.\d+)")

register

@staticmethod
def register(name: str)

Class decorator to register an evaluator.

Arguments:

• name - Unique name for the evaluator.

Returns:

Decorator that registers the class and returns it unchanged.

get

@staticmethod
def get(name: str, **kwargs) -> TuningEvaluator | None

Get an evaluator instance by name.

Arguments:

• name - Name of the evaluator to retrieve.
• **kwargs - Additional keyword arguments passed to the evaluator constructor.

Returns:

Evaluator instance, or None if not found.

resolve

@staticmethod
def resolve(evaluator: TuningEvaluator | str, **kwargs) -> TuningEvaluator

Resolve an evaluator from an instance or name.

Arguments:

• evaluator - A TuningEvaluator instance (returned as-is) or a string name (looked up in registry).
• **kwargs - Additional keyword arguments passed to the evaluator constructor (only used when evaluator is a string).

Returns:

Resolved evaluator instance.

Raises:

• ValueError - If the evaluator name is not registered.
• TypeError - If evaluator is not a TuningEvaluator or str.

list

@staticmethod
def list() -> list[str]

List all registered evaluator names.

Returns:

List of registered evaluator names.