LEIP Evaluate¶
- leip.evaluate(model: Model, options: EvaluateOptions) EvaluateResults¶
Performs inference of test data on a model and collects accuracy information.
- Parameters
model (Model) – Model object.
options (EvaluateOptions) – Options that configure the inference evaluation.
- Returns
Inference times and scoring.
- Return type
- class leip.EvaluateOptions(*, test_path: Path, batch_size: int = 1, warmups: int = 10, test_size: Optional[int] = None, seed: int = 0, nth_callback: Optional[Callable[[int, str, int, List[int], str, dict], None]] = None, end_callback: Optional[Callable[[dict], None]] = None)¶
Options for model inference evaluation.
- Parameters
test_path (Path) –
batch_size (int) –
warmups (int) –
test_size (Optional[int]) –
seed (int) –
nth_callback (Optional[Callable[[int, str, int, List[int], str, dict], None]]) –
end_callback (Optional[Callable[[dict], None]]) –
- Return type
None
- batch_size: int¶
The number of test images to load into one batch for inference.
- end_callback: Optional[Callable[[dict], None]]¶
A callback that gets called at the end of the evaluation.
- nth_callback: Optional[Callable[[int, str, int, List[int], str, dict], None]]¶
A callback that gets called during each iteration of the evaluation.
- seed: int¶
The seed with which to shuffle the dataset.
- test_path: pathlib.Path¶
The path to a text file containing a list of test examples and their output classification.
- test_size: Optional[int]¶
Size of data subset to evaluate on.
- warmups: int¶
Number of warmup runs to get the model up to speed and cached in memory.
- class leip.EvaluateResults(*, inference: EvaluateInferenceResults, scoring: EvaluateScoringResults)¶
Results of an inference evaluation.
- Parameters
inference (EvaluateInferenceResults) –
scoring (EvaluateScoringResults) –
- Return type
None
- inference: leip.core.operations.evaluate.results.EvaluateInferenceResults¶
Inference evaluation times.
- scoring: leip.core.operations.evaluate.results.EvaluateScoringResults¶
Inference evaluation scoring.
- class leip.EvaluateInferenceResults(*, total_time: float, inference_time: float)¶
Inference evaluation times.
- Parameters
total_time (float) –
inference_time (float) –
- Return type
None
- inference_time: float¶
Net inference time in seconds.
- total_time: float¶
Total inference time in seconds. This includes pre-padding, pre-processing, post-padding, and post-processing.
- class leip.EvaluateScoringResults(*, items: int)¶
Inference evaluation scoring.
- Parameters
items (int) –
- Return type
None
- items: int¶
Number of evaluated items.