Optical Flow

class byotrack.api.optical_flow.optical_flow.OpticalFlow(downscale: float | ndarray = 2.0, blur: None | float | ndarray = None)

Bases: ABC

Optical flow algorithm (Abstract class, see implementations)

It implements or wraps an existing optical flow method.

In ByoTrack, frames are multidimensional arrays of shape ([D, ]H, W, C). Two frames are given as input to the algorithm. It outputs an optical flow map which is a multidimensional floating array (dim, [D, ]H, W) where the first dimension is the flow (displacement) for each spatial dimension.

For instance, with two dimensional frames (H, W, C), the computed flow map has a shape (2, H, W), where the first (resp. second) element of the first dimension is the displacement along height (resp. width). The coordinate system of ByoTrack is using the pixel index (i, j) and not the pixel position (x, y) (which is often use in OpenCV). Note that in 3D, the pixel order is (k, i, j) where k stand for depth.

Additionally, the wrapper allows to downscale (with a blurring) the image to speed up computations.

Usage:

video: Video
optflow: OpticalFlow
detector: BatchDetector

# Preprocess the frames (reference and moving)
ref = optflow.preprocess(video[0])
mov = optflow.preprocess(video[1])

# Compute the flow map from ref to mov
flow_map = optflow.compute(ref, mov)

# flow_map[:, i, j] gives the displacement (di, dj) of the pixel (i, j) of the reference image (scaled)

# You can warp mov into ref with warp
mov_warp = optflow.warp(flow_map, video[1])

# Or use the flow to move points, for instance positions of detected particles
detections = detector.detect(video[0][None])[0]  # Detect with a BatchDetector on frame 0
futur_positions = optflow.transform(detections.position.numpy())  # Expected positions on frame 1

downscale

Downscale factor. Can be provided for each spatial axis. Default: 2.0

Type:: Union[float, np.ndarray]

blur

Std of the Gaussian blur applied before downscaling the images Can be provided for each spatial axis. If not provided, a default one is choosen given the downscale.

Type:: Union[None, float, np.ndarray]

preprocess(frame: ndarray) → ndarray

Downscale a frame before running the optical flow algorithm

It blurs then downscales the frame. It also converts to float32.

Parameters:

frame (np.ndarray) – The frame to pre-preprocess It does not normalize the frame. Integer values are simply converted to floating ones. Shape: ([D, ]H, W, C)

Returns:

The blurred and downscaled frame (converted to float32): Shape: ([D’, ]H’, W’, C), dtype: float32

Return type:

np.ndarray

abstract compute(reference: ndarray, moving: ndarray) → ndarray

Compute the optical flow map from reference to the moving frame

It computes the displacement of each pixel from the reference frame to be in the moving one.

Parameters:

reference (np.ndarray) – Reference frame Shape: ([D’, ]H’, W’, C), dtype: float
moving (np.ndarray) – Moving frame Shape: ([D’, ]H’, W’, C), dtype: float

Returns:

Optical flow map from reference to moving: The flow field is stored in pixel coordinates ([dk’, ]di’, dj’) (!= xyz) Shape: (dim, [D’, ]H’, W’), dtype: float32

Return type:

np.ndarray

flow_at(flow_map: ndarray, points: ndarray) → ndarray

Extract the flow/displacement at the given location

The flow_map is expected to be downscaled, but the input and outputs points are not.

Parameters:

flow_map (np.ndarray) – Optical flow map. Displacement ([dk’, ]di’, dj’) for each pixel ([k’, ]i’, j’) in the downscaled coordinates. Shape: (dim, [D’, ]H’, W’), dtype: float32
points (np.ndarray) – Points ([k, ]i ,j) in the original (non-downscale) coordinates Shape: (N, dim), dtype: float

Returns:

The displacements ([dk, ], di, dj) in the original coordinates (non-downscale): at the given points. Shape: (N, dim), dtype: float

Return type:

np.ndarray

transform(flow_map: ndarray, points: ndarray) → ndarray

Apply the flow to the given points

The flow_map is expected to be downscaled, but the input and outputs points are not.

Parameters:

flow_map (np.ndarray) – Optical flow map. Displacement ([dk’, ]di’, dj’) for each pixel ([k’, ]i’, j’) in the downscaled coordinates. Shape: (dim, [D’, ]H’, W’), dtype: float32
points (np.ndarray) – Points ([k, ]i ,j) in the original (non-downscale) coordinates Shape: (N, dim), dtype: float

Returns:

New positions of the poitns ([k, ]i, j) in the original (non-downscale) coordinates: once the flow is applied. Shape: (N, dim), dtype: float

Return type:

np.ndarray

warp(flow_map: ndarray, moving: ndarray) → ndarray

Warps the moving image onto the reference using the flow map

It warps the non-preprocessed moving image (in the original non-downscaled coordinates).

Note

We only implemented backward warping (It only allows to warp the moving image onto the reference one) Forward warping could be implemented in a future version if needed.

Parameters:

flow_map (np.ndarray) – Optical flow map. Displacement ([dk’, ]di’, dj’) for each pixel ([k’, ]i’, j’) in the downscaled coordinates. Shape: (dim, [D’, ]H’, W’), dtype: float32
moving (np.ndarray) – The moving image to warp (non-preprocessed) Shape: ([D, ]H, W, C), dtype: float

Returns:

Warped image: Shape: ([D, ]H, W, C), dtype: float

Return type:

np.ndarray

class byotrack.api.optical_flow.optical_flow.DummyOpticalFlow(downscale: float | ndarray = 2.0, blur: None | float | ndarray = None)

Bases: OpticalFlow

Dummy optical flow which predict no displacement for each pixel

compute(reference: ndarray, moving: ndarray) → ndarray

Compute the optical flow map from reference to the moving frame

It computes the displacement of each pixel from the reference frame to be in the moving one.

Parameters:

reference (np.ndarray) – Reference frame Shape: ([D’, ]H’, W’, C), dtype: float
moving (np.ndarray) – Moving frame Shape: ([D’, ]H’, W’, C), dtype: float

Returns:

Optical flow map from reference to moving: The flow field is stored in pixel coordinates ([dk’, ]di’, dj’) (!= xyz) Shape: (dim, [D’, ]H’, W’), dtype: float32

Return type:

np.ndarray