What one prediction costs, with the settings that produced it.
Carries the compute units and precision because a latency without them says nothing: the same package is three times slower with the accelerator withheld, and float32 forfeits the accelerator entirely.
Examples
>>> import shutil, statistics, tempfile
>>> import lucid, lucid.nn as nn, lucid.coreml as cml
>>> model = nn.Sequential(nn.Conv2d(3, 16, 3, padding=1), nn.ReLU()).eval()
>>> x, room = lucid.randn(1, 3, 32, 32), tempfile.mkdtemp()
>>> package = cml.export(model, x, f"{room}/m.mlpackage",
... precision=cml.Precision.FLOAT16,
... compute_units=cml.ComputeUnits.CPU_AND_NE)
>>> package.benchmark(x) # the first call after an export
Latency(median=...ms, best=...ms, n=30, CPU_AND_NE, FLOAT16)
>>> settled = statistics.median( # and again, once it has settled
... package.benchmark(x).median_ms for _ in range(3)
... )
>>> settled > 0.0
True
>>> package.close()
>>> shutil.rmtree(room)
A package is slower on its first measured runs than it will be after
a few — Core ML is still warming its own caches — so a single reading
taken right after an export is not the number to publish. Measure a
few times and take the median of those.