class

IterableDataset

IterableDataset()

source

Base class for iterable-style (stream) datasets.

Subclasses must implement __iter__, yielding samples one at a time. Unlike Dataset, iterable datasets do not support random access via integer indices and therefore cannot be used with samplers; lucid.utils.data.DataLoader consumes them sequentially.

Notes

Use IterableDataset when:

The data source is a stream (network socket, generator, log tail).
The dataset size is unknown a priori.
Random access is prohibitively expensive.

Otherwise prefer Dataset — it composes with shuffling and distributed sampling, which is impossible for iterable streams.

Examples

>>> class CountUp(IterableDataset):
...     def __init__(self, n): self.n = n
...     def __iter__(self):
...         for i in range(self.n):
...             yield i

Methods (2)

dunder

iter

→Iterator

__iter__()

source

Yield samples one at a time.

Returns

Iterator

Iterator producing individual samples. The iterator should terminate (raise StopIteration) when the stream is exhausted; infinite streams are permitted but require the consumer to externally bound iteration.

dunder

add

→IterableDataset

__add__(other: IterableDataset)

source

Concatenation via + — not supported for iterable datasets.

Use ChainDataset to chain iterable datasets end-to-end.

Raises

NotImplementedError

Always; iterable datasets cannot be concatenated through +.