Video

class byotrack.video.video.VideoTransformConfig(aggregate: bool = False, normalize: bool = False, selected_channel: int | None = None, q_min: float = 0.0, q_max: float = 1.0, smooth_clip: float = 0.0, compute_stats_on: int = 50)

Bases: object

Configuration for video transformations.

aggregate

Aggregate channels

Type:: bool

normalize

Scale and Normalize the video in [0, 1]

Type:: bool

selected_channel

Channel to use for aggregation If None, channel average is done. If any, it performs channel selection

Type:: int | None

q_min

Minimum quantile to use when scaling the video

Type:: float

q_max

Maximum quantile to use when scaling the video

Type:: float

smooth_clip

Smoothness of the clipping process (log clipping) See ScaleAndNormalize: it logs clip the highest values on q_max. If 0.0, hard clipping is done.

Type:: float

compute_stats_on

Number of frames to use to compute the quantiles.

Type:: int

class byotrack.video.video.Video(data_source: str | PathLike | VideoReader | ndarray, **kwargs: Any)

Bases: Sequence[ndarray]

Video: Iterable, indexable and sliceable sequence of frames wrapping a VideoReader.

It wraps VideoReader in order to add video preprocessing (Channel Aggregation, Normalization, Projection, …) and to add useful pythonic protocols (Sliceable, Indexable, Iterable).

Frames are 2D or 3D with a channel axis. It behaves similarly as a 5D/4D numpy array of shape (T[, D], H, W, C).

Example

import byotrack

# Read a video (Usually 2D RGB)
video = byotrack.Video(video_path)

# Normalize the video
video = video.normalize()

# Iterate through the video
for frame in video:
    pass

# Temporal slicing
sliced = video[10:50:3]  # Take one frame every three from frame 10 to frame 50.

# Spatial slicing
sliced = video[:, 100:200, 150:250]  # All frames on the roi (100:200 x 150:250)

ndim

Either 4 (2D) or 5 (3D). (T, H, W, C) in 2D or (T, D, H, W, C) in 3D.

Type:: int

shape

Shape of the video (T, [D, ]H, W, C).

Type:: tuple[int, …]

dtype

Data type of the video.

Type:: np.dtype

reader

Underlying video reader

Type:: byotrack.video.VideoReader

add_preprocessor(preprocessor: VideoPreprocessor) → Video

Add a preprocessor to the video.

Added preprocessors are applied sequentially (you may check the order in _preprocessors).

Note: This may change the shape or dtype of the Video.

Parameters:: preprocessor (byotrack.video.VideoPreprocessor) – The preprocessor to add. Will be initialized with this video.
Returns:: self
Return type:: byotrack.Video

normalize(q_min: float = 0.0, q_max: float = 1.0, smooth_clip: float = 0, compute_stats_on: int = 50) → Video

Normalize each channel of the video into [0, 1].

Copy the video and adds the IntensityNormalizer preprocessor with the given arguments.

Parameters:

q_min (float) – Quantile of the minimum value to consider. Default: 0.0 (min value)
q_max (float) – Quantile of the maximum value to consider. Default: 1.0 (max value)
smooth_clip (float) – Smoothness of the clipping process (a) If 0, values are clipped on the quantiles Else, values above the maximum quantile are log clipped: I = 1 + a log((I - 1)/a + 1) for I > 1, with a the smooth_clip factor Typical values are between 0 and 1. Default: 0 (hard clipping)
compute_stats_on (int) – Max number of frames to compute stats on. It prevents heavy computations that may occur on large videos. Default: 50

Returns:

the normalized video

Return type:

byotrack.Video

set_transform(transform_config: VideoTransformConfig) → None: Deprecated. Will be removed in a future version.

__getitem__(index: int) → ndarray

__getitem__(slice_: slice) → Video

__getitem__(slices: tuple[slice | int | EllipsisType, ...] | EllipsisType) → Video

Indexing and slicing operations.

When indexed, it returns the ith frame in the slice. When sliced, it duplicates the video (wrapper) with the right slicing.

Parameters:: key (int | slice | EllipsisType | tuple[slice | int | EllipsisType, ...]) – index or slice of the video
Returns:: Frame at index or a shallow copy of the video with the right slicing
Return type:: np.ndarray | Video

byotrack.video.video.expand_ellipsis(slices: tuple[int | slice | EllipsisType, ...], ndim: int) → tuple[int | slice, ...]: Expand ellipsis in slices.

byotrack.video.video.compose_slice(slice_1: slice, slice_2: slice, length: int) → slice: Compose two slices in the given order.

Video Reader

byotrack.video.reader.slice_length(slice_: slice, shape: int) → int: Compute the number of element in a slice.

class byotrack.video.reader.MetaVideoReader(cls_name: str, bases: tuple, attributes: dict)

Bases: type

MetaClass for Video Readers.

Each VideoReader has to define a list of supported extensions. The last constructed VideoReader to claim an extension will be used to open the video. If no one has claimed an extension the default OpenCVVideoReader is used.

class byotrack.video.reader.VideoReader(path: str | os.PathLike, **kwargs: Any)

Bases: object

Unified video reader api.

Close to OpenCV API but few key differences:

There is always a frame loaded
Frame ids goes from 0 to length - 1
Frames are loaded in RGB.
It support any number of channels and 2D/3D
Read method is very different:
- It retrieves the current frame then grabs the next (The other way around in opencv)
- It returns therefore a ndarray and a bool rather than a bool and a ndarray
- The boolean returned indicated if we can continue to read and not if the read operation has failed
Easy to check main attributes like:
- frame_id
- length
- channels
- shape
- dtype
- fps if known (-1 otherwise)

supported_extensions

Static attribute used by the open factory to automatically select the right VideoReader subclass for a given file extension.

Type:: tuple[str, …]

path

Path of the current video.

Type:: pathlib.Path

released

True when release has been called (file closed, memory freed).

Type:: bool

fps

Frame rate in frames per second. -1 if unknown.

Type:: int

shape

Spatial dimensions of each frame ([D, ]H, W).

Type:: tuple[int, …]

channels

Number of channels per frame.

Type:: int

length

Total number of frames.

Type:: int

dtype

Data type of the frames.

Type:: np.dtype

frame_id

Index of the current (buffered) frame.

Type:: int

release() → None: Close the file and free memory.

grab() → bool

Grab the next frame.

Can be faster than self.seek(self.frame_id + 1)

Returns:: True if able to grab next frame
Return type:: bool

retrieve() → ndarray

Retrieve the current frame.

Returns:

The current frame: Shape: ([D, ]H, W, C)

Return type:

np.ndarray

read() → tuple[ndarray, bool]

Consume a frame. Is equivalent to retrieve + grab.

As in this implementation there is always a current frame. It reverses OpenCV implementation It first retrieves then grab next frame

Returns:: The current frame - Shape: ([D, ]H, W, C) bool: Whether there is a next frame to read
Return type:: np.ndarray

seek(frame_id: int) → None

Seek to frame_id and update the current frame.

Valid frame ids are in [0, length[.

Parameters:: frame_id (int) – Target frame index.
Raises:: EOFError – If frame_id is not in [0, length[.

tell() → int

Returns self.frame_id.

Returns:: current frame_id
Return type:: int

static open(path: str | os.PathLike, **kwargs: Any) → VideoReader

Open a video file, choosing the right reader from the file extension.

Directories are opened with MultiFrameReader. For files, the extension is looked up in the extension_to_reader registry; if no match is found, OpenCVVideoReader is used.

Parameters:

path (str | os.PathLike) – File or directory to open.
**kwargs – Additional kwargs forwarded to the chosen VideoReader constructor.

Returns:

An open reader positioned at frame 0.

Return type:

VideoReader

class byotrack.video.reader.OpenCVVideoReader(path: str | os.PathLike, **kwargs: Any)

Bases: VideoReader

Wrapper around opencv VideoCapture.

Default VideoReader when opening a file.

It only supports 2D images (grayscale or RGB).

video

VideoCapture from opencv

Type:: cv2.VideoCapture

release() → None: Close the file and free memory.

grab() → bool

Grab the next frame.

Can be faster than self.seek(self.frame_id + 1)

Returns:: True if able to grab next frame
Return type:: bool

retrieve() → ndarray

Retrieve the current frame.

Returns:

The current frame: Shape: ([D, ]H, W, C)

Return type:

np.ndarray

seek(frame_id: int) → None

Seek to frame_id and update the current frame.

Valid frame ids are in [0, length[.

Parameters:: frame_id (int) – Target frame index.
Raises:: EOFError – If frame_id is not in [0, length[.

class byotrack.video.reader.ArrayVideoReader(path: str | os.PathLike, video: np.ndarray, **kwargs: Any)

Bases: VideoReader

Fake VideoReader that directly reads in the given an array-like video.

It’s main goal is to provide a conversion from array-like videos (np.ndarray, zarr, …).

The array-like video is expected to be of shape: T, [D, ]H, W, C.

Note: path is ignored, and the data should be provided via the video argument.

video

Array-like video. Shape: (T, [D, ]H, W, C)

Type:: np.ndarray

release() → None: Close the file and free memory.

grab() → bool

Grab the next frame.

Can be faster than self.seek(self.frame_id + 1)

Returns:: True if able to grab next frame
Return type:: bool

seek(frame_id: int) → None

Seek to frame_id and update the current frame.

Valid frame ids are in [0, length[.

Parameters:: frame_id (int) – Target frame index.
Raises:: EOFError – If frame_id is not in [0, length[.

retrieve() → ndarray

Retrieve the current frame.

Returns:

The current frame: Shape: ([D, ]H, W, C)

Return type:

np.ndarray

class byotrack.video.reader.PILVideoReader(path: str | os.PathLike, **kwargs: Any)

Bases: VideoReader

PIL-based video reader for 2D multi-frame TIFF files not supported by OpenCV.

It only supports 2D videos. Prefer TiffVideoReader for broader TIFF support (3D, any channels).

See VideoReader for inherited attributes.

video

The open PIL image (animated).

Type:: PIL.Image.Image

release() → None: Close the file and free memory.

grab() → bool

Grab the next frame.

Can be faster than self.seek(self.frame_id + 1)

Returns:: True if able to grab next frame
Return type:: bool

retrieve() → ndarray

Retrieve the current frame.

Returns:

The current frame: Shape: ([D, ]H, W, C)

Return type:

np.ndarray

seek(frame_id: int) → None

Seek to frame_id and update the current frame.

Valid frame ids are in [0, length[.

Parameters:: frame_id (int) – Target frame index.
Raises:: EOFError – If frame_id is not in [0, length[.

class byotrack.video.reader.TiffVideoReader(path: str | os.PathLike, level=0, axes: str | None = None, ax_slice: dict[str, slice] | None = None, **kwargs: Any)

Bases: VideoReader

Tiff video reader with tifffile. Handle 2D and 3D videos with any channels.

Axes are inferred from the tifffile metadata and convert into (T, [D, ]H, W, C) (<=> T[Z]YXC). We may not support all formats, or your specific metadata can be wrong/missing. In this case, you can also provide the expected axes of the tifffile using an ordered string.

For example: “TYX” for 2D videos without channel, “TCZYX” for 3D videos with channels (ordered by time, channel then stack), “ZTYX” for 3D videos without channels (ordered by stack then time).

Note

With tifffile syntax, we use X for width, Y for height, Z for depth and C/S for channels (C and S are not supported together) and T for time. Any other letter (I, O, Q, …) is first interpret as T if it is missing, then Z if it is missing, and finally it will yield an error.

It also supports to read the tiff at a specific resolution level

See VideoReader for inherited attributes.

out_axes

Axes order of the outputs

Type:: str

level

Resolution level (if any) Default: 0 (finest level)

Type:: int

in_axes

Parsed in axes

Type:: dict[str, int]

ax_slice

Optional slices to use on axes.

Type:: dict[str, slice]

release() → None: Close the file and free memory.

grab() → bool

Grab the next frame.

Can be faster than self.seek(self.frame_id + 1)

Returns:: True if able to grab next frame
Return type:: bool

seek(frame_id: int) → None

Seek to frame_id and update the current frame.

Valid frame ids are in [0, length[.

Parameters:: frame_id (int) – Target frame index.
Raises:: EOFError – If frame_id is not in [0, length[.

retrieve() → ndarray

Retrieve the current frame.

Returns:

The current frame: Shape: ([D, ]H, W, C)

Return type:

np.ndarray

class byotrack.video.reader.FrameTiffLoader(level=0, axes: str | None = None, ax_slice: dict[str, slice] | None = None)

Bases: object

Load a single frame stored in a TiffFile with tifffile.

It handle 2D and 3D videos with any channels. Axes are inferred from the tifffile metadata and convert into ([D, ]H, W, C) (<=> [Z]YXC).

We may not support all formats, or your specific metadata can be wrong/missing. In this case, you can also provide the expected axes of the tifffile using an ordered string.

For example: “YX” for 2D videos without channel, “CZYX” for 3D videos with channels (ordered by channel then stack).

Note

With tifffile syntax, we use X for width, Y for height, Z for depth and C/S for channels (C and S are not supported together). Any other letter (T, I, O, Q, …) is either interpreted as Z if it is missing, or it will yield an error.

It also supports to read the tiff at a specific resolution level.

out_axes

Axes order of the outputs.

Type:: str

level

Resolution level to read. Default: 0 (finest level).

Type:: int

axes

Override for the axes found in the TIFF metadata. Default: None (infer from metadata).

Type:: str | None

ax_slice

Per-axis slices applied when reading each frame. Default: {} (no slicing).

Type:: dict[str, slice]

byotrack.video.reader.pil_loader(path: str | os.PathLike) → np.ndarray

Load a 2D image with PIL.

Parameters:

path (str | os.PathLike) – Path to the image file.

Returns:

The loaded frame.: Shape: (H, W, C).

Return type:

np.ndarray

Bases: VideoReader

Read video from a list of files inside a folder.

By default, it will find the alphanumerically sorted list of paths that shares the most common extension in the folder. The extension may be provided by the user.

You can provide your own list of paths (absolute paths). The folder path is then ignored.

Finally, you may also provide your own loading function to load each frame as a numpy array.

See VideoReader for inherited attributes.

paths

Sorted list of Paths to each frame of the video.

Type:: list[pathlib.Path]

frame_loader

Loads frame from their associated files.

Type:: Callable[[str | os.PathLike], np.ndarray]

grab() → bool

Grab the next frame.

Can be faster than self.seek(self.frame_id + 1)

Returns:: True if able to grab next frame
Return type:: bool

seek(frame_id: int) → None

Seek to frame_id and update the current frame.

Valid frame ids are in [0, length[.

Parameters:: frame_id (int) – Target frame index.
Raises:: EOFError – If frame_id is not in [0, length[.

retrieve() → ndarray

Retrieve the current frame.

Returns:

The current frame: Shape: ([D, ]H, W, C)

Return type:

np.ndarray

Video Preprocessors

class byotrack.video.preprocessor.preprocessor.VideoPreprocessor

Bases: ABC

Video preprocessor base class.

A preprocessor can both change the intensity (normalization, denoising, frame filtering, …) as well as the shape (Z-projection, channel aggregation, slicing, …) of each video frame.

It first needs to be initialized on the given video (where it can read the video if needed). Then, it is applied online, i.e., for each frame of the video.

shape

Output shape of each frame ([D, ]H, W, C).

Type:: tuple[int, …]

dtype

Output dtype of each frame. Usually independent of the input video.

Type:: np.dtype

initialize(video: Sequence[np.ndarray] | np.ndarray) → None

Initialize the preprocessor for the given video.

The default implementation preserve the video shape and dtype. This should be overwritten by VideoProcessor implementations.

Parameters:: video (Sequence[np.ndarray] | np.ndarray) – The video to preprocess. Sequence of T frames (array). Each array is expected to have a shape ([D, ]H, W, C).

abstractmethod preprocess_frame(frame: ndarray, frame_id=0) → ndarray

Preprocess the given frame.

Parameters:

frame (np.ndarray) – Frame to be preprocessed. Shape: ([D, ]H, W, C)
frame_id (int) – Optional index of the frame in the video. Default to 0.

Returns:

Preprocessed frame.: Shape ([D’, ]H’, W’, C’)

Return type:

np.ndarray

preprocess_video(video: Sequence[np.ndarray] | np.ndarray) → np.ndarray

Preprocess the given video directly.

It will re-initialize the preprocessor at each call.

Warning: Consider using the online version, which is integrated into the Video class.: This will requires much more memory than its online counterpart.

Parameters:: video (Sequence[np.ndarray] | np.ndarray) – The video to preprocess. Sequence of T frames (array). Each array is expected to have a shape ([D, ]H, W, C).
Returns:: The preprocessed video loaded as a np.ndarray.
Return type:: np.ndarray

class byotrack.video.preprocessor.normalizer.IntensityNormalizer(q_min: float, q_max: float, smooth_clip: float = 0, compute_stats_on: int = 50)

Bases: VideoPreprocessor

Normalize each channel intensity into [0, 1].

mini and maxi values are computed using quantile of the video to improve stability. The quantiles are computed using only the first compute_stats_on frames.

Frame shape is preserved, but dtype is changed to float32.

Note: A smooth_clip can be performed by log clipping values above maxi up until the log max.

q_min

Quantile of the minimum value to consider

Type:: float

q_max

Quantile of the maximum value to consider

Type:: float

mini

Minimum value kept (one for each channel) Shape: (C, ), dtype: float32

Type:: np.ndarray

maxi

Maximum value kept (one for each channel) Shape: (C, ), dtype: float32

Type:: np.ndarray

smooth_clip

Smoothness of the clipping process (a) If 0, values are clipped on mini/maxi Else, values above maxi are log clipped: I = 1 + a log((I - 1)/a + 1) for I > 1, with a the smooth_clip factor Typical values are between 0 and 1. Default: 0 (hard clipping)

Type:: float

max

True maximum values (one for each channel) when using smooth clipping Shape: (C, ), dtype: float32

Type:: np.ndarray

compute_stats_on

Max number of frames to compute stats on. It prevents heavy computations that may occur on large videos. Default: 50

Type:: int

initialize(video: Sequence[np.ndarray] | np.ndarray) → None

Initialize the preprocessor for the given video.

It computes mini and maxi values based on the first frames of the video.

Parameters:: video (Sequence[np.ndarray] | np.ndarray) – The video to preprocess. Sequence of T frames (array). Each array is expected to have a shape ([D, ]H, W, C).

preprocess_frame(frame: ndarray, frame_id=0) → ndarray

Preprocess the given frame.

Parameters:

frame (np.ndarray) – Frame to be preprocessed. Shape: ([D, ]H, W, C)
frame_id (int) – Optional index of the frame in the video. Default to 0.

Returns:

Preprocessed frame.: Shape ([D’, ]H’, W’, C’)

Return type:

np.ndarray

class byotrack.video.preprocessor.channel_projection.ChannelProjection(method: Literal['mean', 'min', 'max', 'select'] = 'mean', selected: int = 0)

Bases: VideoPreprocessor

Projection of the video channel.

Allows to reduce multi-channel videos into single channel videos.

method

Projection method. “mean”, “min” and “max” aggregate the channels with the appropriate function. “select” simply selects one specific channel. Default: “mean”.

Type:: Literal[“mean”, “min”, “max”, “select”]

selected

Selected channel if method is “select”. Default: 0.

Type:: int

initialize(video: Sequence[np.ndarray] | np.ndarray) → None

Initialize the preprocessor for the given video.

This will reduce the channel in the shape attribute.

Parameters:: video (Sequence[np.ndarray] | np.ndarray) – The video to preprocess. Sequence of T frames (array). Each array is expected to have a shape ([D, ]H, W, C).

preprocess_frame(frame: ndarray, frame_id=0) → ndarray

Preprocess the given frame.

Parameters:

frame (np.ndarray) – Frame to be preprocessed. Shape: ([D, ]H, W, C)
frame_id (int) – Optional index of the frame in the video. Default to 0.

Returns:

Preprocessed frame.: Shape ([D’, ]H’, W’, C’)

Return type:

np.ndarray

preprocess_video(video: Sequence[np.ndarray] | np.ndarray) → np.ndarray

Preprocess the given video directly.

It will re-initialize the preprocessor at each call.

Warning: Consider using the online version, which is integrated into the Video class.: This will requires much more memory than its online counterpart.

Parameters:: video (Sequence[np.ndarray] | np.ndarray) – The video to preprocess. Sequence of T frames (array). Each array is expected to have a shape ([D, ]H, W, C).
Returns:: The preprocessed video loaded as a np.ndarray.
Return type:: np.ndarray

class byotrack.video.preprocessor.spatial_projection.SpatialProjection(axis: str | int = 'Z', method: Literal['mean', 'min', 'max', 'select'] = 'max', selected: int = 0)

Bases: VideoPreprocessor

Spatial projection of the video along an axis.

It allows to project a 3D video onto a 2D one.

axes_to_int

Mapping from axis name string ("Z", "Y", "X", "D", "H", "W") to integer axis index (0, 1, 2).

Type:: dict[str, int]

axis

Axis on which to project. Order is (Z, Y, X) ~ (D, H, W) ~ (0, 1, 2). Default: 0.

Type:: int

method

Projection method. "mean", "min" and "max" aggregate over the axis with the appropriate function. "select" selects one slice of the volume. Default: "max".

Type:: Literal[“mean”, “min”, “max”, “select”]

selected

Slice index used when method is "select". Default: 0.

Type:: int

initialize(video: Sequence[np.ndarray] | np.ndarray) → None

Initialize the preprocessor for the given video.

This will set the shape attribute correctly, or raise if not 3D.

Parameters:: video (Sequence[np.ndarray] | np.ndarray) – The video to preprocess. Sequence of T frames (array). Each array is expected to have a shape ([D, ]H, W, C).

preprocess_frame(frame: ndarray, frame_id=0) → ndarray

Preprocess the given frame.

Parameters:

frame (np.ndarray) – Frame to be preprocessed. Shape: ([D, ]H, W, C)
frame_id (int) – Optional index of the frame in the video. Default to 0.

Returns:

Preprocessed frame.: Shape ([D’, ]H’, W’, C’)

Return type:

np.ndarray

preprocess_video(video: Sequence[np.ndarray] | np.ndarray) → np.ndarray

Preprocess the given video directly.

It will re-initialize the preprocessor at each call.

Warning: Consider using the online version, which is integrated into the Video class.: This will requires much more memory than its online counterpart.

Parameters:: video (Sequence[np.ndarray] | np.ndarray) – The video to preprocess. Sequence of T frames (array). Each array is expected to have a shape ([D, ]H, W, C).
Returns:: The preprocessed video loaded as a np.ndarray.
Return type:: np.ndarray

class byotrack.video.preprocessor.slicer.FrameSlicer(slices: tuple[slice, ...])

Bases: VideoPreprocessor

Slice the frame with given slices.

slices

slices to apply.

Type:: tuple[slice, …]

initialize(video: Sequence[np.ndarray] | np.ndarray) → None

Initialize the preprocessor for the given video.

This will update the shape attribute to reflect the output shape after slicing.

Parameters:: video (Sequence[np.ndarray] | np.ndarray) – The video to preprocess. Sequence of T frames (array). Each array is expected to have a shape ([D, ]H, W, C).

preprocess_frame(frame: ndarray, frame_id=0) → ndarray

Preprocess the given frame.

Parameters:

frame (np.ndarray) – Frame to be preprocessed. Shape: ([D, ]H, W, C)
frame_id (int) – Optional index of the frame in the video. Default to 0.

Returns:

Preprocessed frame.: Shape ([D’, ]H’, W’, C’)

Return type:

np.ndarray

preprocess_video(video: Sequence[np.ndarray] | np.ndarray) → np.ndarray

Preprocess the given video directly.

It will re-initialize the preprocessor at each call.

Warning: Consider using the online version, which is integrated into the Video class.: This will requires much more memory than its online counterpart.

Parameters:: video (Sequence[np.ndarray] | np.ndarray) – The video to preprocess. Sequence of T frames (array). Each array is expected to have a shape ([D, ]H, W, C).
Returns:: The preprocessed video loaded as a np.ndarray.
Return type:: np.ndarray