LEIP Evaluate

leip.evaluate(model: Model, options: EvaluateOptions) EvaluateResults

Performs inference of test data on a model and collects accuracy information.

Parameters
  • model (Model) – Model object.

  • options (EvaluateOptions) – Options that configure the inference evaluation.

Returns

Inference times and scoring.

Return type

EvaluateResults

class leip.EvaluateOptions(*, test_path: Path, batch_size: int = 1, warmups: int = 10, test_size: Optional[int] = None, seed: int = 0, nth_callback: Optional[Callable[[int, str, int, List[int], str, dict], None]] = None, end_callback: Optional[Callable[[dict], None]] = None)

Options for model inference evaluation.

Parameters
  • test_path (Path) –

  • batch_size (int) –

  • warmups (int) –

  • test_size (Optional[int]) –

  • seed (int) –

  • nth_callback (Optional[Callable[[int, str, int, List[int], str, dict], None]]) –

  • end_callback (Optional[Callable[[dict], None]]) –

Return type

None

batch_size: int

The number of test images to load into one batch for inference.

end_callback: Optional[Callable[[dict], None]]

A callback that gets called at the end of the evaluation.

nth_callback: Optional[Callable[[int, str, int, List[int], str, dict], None]]

A callback that gets called during each iteration of the evaluation.

seed: int

The seed with which to shuffle the dataset.

test_path: pathlib.Path

The path to a text file containing a list of test examples and their output classification.

test_size: Optional[int]

Size of data subset to evaluate on.

warmups: int

Number of warmup runs to get the model up to speed and cached in memory.

class leip.EvaluateResults(*, inference: EvaluateInferenceResults, scoring: EvaluateScoringResults)

Results of an inference evaluation.

Parameters
Return type

None

inference: leip.core.operations.evaluate.results.EvaluateInferenceResults

Inference evaluation times.

scoring: leip.core.operations.evaluate.results.EvaluateScoringResults

Inference evaluation scoring.

class leip.EvaluateInferenceResults(*, total_time: float, inference_time: float)

Inference evaluation times.

Parameters
  • total_time (float) –

  • inference_time (float) –

Return type

None

inference_time: float

Net inference time in seconds.

total_time: float

Total inference time in seconds. This includes pre-padding, pre-processing, post-padding, and post-processing.

class leip.EvaluateScoringResults(*, items: int)

Inference evaluation scoring.

Parameters

items (int) –

Return type

None

items: int

Number of evaluated items.

class leip.EvaluateTopNScoringResults(*, items: int, top1: float, top5: float)

Top1 and Top5 inference evaluation scoring.

Parameters
  • items (int) –

  • top1 (float) –

  • top5 (float) –

Return type

None

top1: float

Score of the top 1 class.

top5: float

Score of the top 5 classes.

class leip.EvaluateMapScoringResults(*, items: int, mAP: dict)

mAP inference evaluation scoring.

Parameters
  • items (int) –

  • mAP (dict) –

Return type

None

mAP: dict

mAP scores.