How a weight is stored, as distinct from how the body computes.
INT8 stores each weight as an integer code plus one scale per
output channel, and Core ML dequantizes it on the way into the
operation that uses it. The arithmetic still runs at the body's
precision — this is a storage decision, not a different network — so
the package halves in size against float16 and the Neural Engine
moves half as much memory to do the same work.
The cost is real and one-directional: eight bits per weight cannot
represent what sixteen did, so a quantized export is further from the
eager model than a float16 one. verify will say by how much.
Examples
>>> import shutil, tempfile
>>> import lucid, lucid.nn as nn, lucid.coreml as cml
>>> model = nn.Sequential(
... nn.Conv2d(3, 64, 3, padding=1), nn.ReLU(), nn.Conv2d(64, 64, 3, padding=1)
... ).eval()
>>> x, room = lucid.randn(1, 3, 16, 16), tempfile.mkdtemp()
>>> with cml.export(model, x, f"{room}/int8.mlpackage",
... weights=cml.WeightPrecision.INT8) as package:
... print(package.verify(model, x, relative=True) < 1e-2) # by how much
True
>>> shutil.rmtree(room)