Optical Flow

class byotrack.api.optical_flow.optical_flow.OpticalFlow(downscale: float | ndarray = 2.0, blur: None | float | ndarray = None)

Bases: ABC

Optical flow algorithm (Abstract class, see implementations)

It implements or wraps an existing method. In ByoTrack, frames are multidimensional floating arrays, they are the inputs of the optical flow algorithm. It outputs an optical flow map which is a multidimensional floating array where the first dimension is the flow for each spatial dimension.

For instance, with two dimensional frames (H, W, C), the computed flow map has a shape (2, H, W), where the first (resp. second) element of the first dimension is the displacement along height (resp. width). The coordinate system of ByoTrack is using the pixel index (i, j) and not the pixel position (x, y) (which is often use in OpenCv)

Additionally, the wrapper allows to downscale (with a blurring) the image to speed up computations.

Usage:

video: Video
optflow: OpticalFlow

# Preprocess the frames (reference and moving)
ref = optflow.preprocess(video[0])
mov = optflow.preprocess(video[1])

# Compute the flow map from ref to mov
flow_map = optflow.compute(ref, mov)

# flow_map[:, i, j] gives the displacement (di, dj) of the pixel (i, j) of the reference image

# You can warp mov into ref with warp
mov_warp = optflow.warp(flow_map, video[1])

# Or use the flow to move points

downscale

Downscale factor. Can be provided for each spatial axis. Default: 2.0

Type:: Union[float, np.ndarray]

blur

Std of the Gaussian blur applied before downscaling the images Can be provided for each spatial axis. If not provided, a default one is choosen given the downscale.

Type:: Union[None, float, np.ndarray]

preprocess(frame: ndarray) → ndarray

Downscale a frame before running the optical flow algorithm

It blurs then downscales the frame.

Parameters:

frame (np.ndarray) – The frame to pre-preprocess Shape: (H, W, C), dtype: float64

Returns:

The blurred and downscaled frame: Shape: (H’, W’, C), dtype: float32

Return type:

np.ndarray

abstract compute(reference: ndarray, moving: ndarray) → ndarray

Compute the optical flow map from reference to the moving frame

It computes the displacement of each pixel from the reference frame to be in the moving one.

Parameters:

reference (np.ndarray) – Reference frame Shape: (H’, W’, C), dtype: float
moving (np.ndarray) – Moving frame Shape: (H’, W’, C), dtype: float

Returns:

Optical flow map from reference to moving: The pixel coordinates are stored as (i, j) (and not (x, y)), so is their displacement. Shape: (2, H’, W’), dtype: float32

Return type:

np.ndarray

flow_at(flow_map: ndarray, points: ndarray) → ndarray

Extract the flow/displacement at the given location

The output displacement and the given location are expected to be at the frame scale.

Parameters:

flow_map (np.ndarray) – Optical flow map (Displacement in i, j for each pixel) Shape: (2, H’, W’), dtype: float32
points (np.ndarray) – Points where to extract the displacement (i, j) (Not downscaled) Shape: (N, 2), dtype: float

Returns:

The displacement at points (i, j) (Not downscaled): Shape: (N, 2), dtype: float

Return type:

np.ndarray

transform(flow_map: ndarray, points: ndarray) → ndarray

Apply the flow to the given points

The points are expected to be at the frame scale (not downscaled)

Parameters:

flow_map (np.ndarray) – Optical flow map (Displacement in i, j for each pixel) Shape: (2, H’, W’), dtype: float32
points (np.ndarray) – Points to transform (i, j) (Not downscaled) Shape: (N, 2), dtype: float

Returns:

New positions of the points after the flow: Shape: (N, 2), dtype: float

Return type:

np.ndarray

warp(flow_map: ndarray, moving: ndarray) → ndarray

Warps the moving image onto the reference using the flow map

Note

We only implemented backward warping (It only allows to warp the moving image onto the reference one) Forward warping could be implemented in a future version if needed.

Parameters:

flow_map (np.ndarray) – Optical flow map. Displacement (di, dj) for each pixel (i, j) of the reference frame. Shape: (2, H, W), dtype: float32
moving (np.ndarray) – The moving image to warp. Shape: (H, W, C]), dtype: float

Returns:

Warped image: Shape: (H, W, C), dtype: float

Return type:

np.ndarray

class byotrack.api.optical_flow.optical_flow.DummyOpticalFlow(downscale: float | ndarray = 2.0, blur: None | float | ndarray = None)

Bases: OpticalFlow

Dummy optical flow which predict no displacement for each pixel

compute(reference: ndarray, moving: ndarray) → ndarray

Compute the optical flow map from reference to the moving frame

It computes the displacement of each pixel from the reference frame to be in the moving one.

Parameters:

reference (np.ndarray) – Reference frame Shape: (H’, W’, C), dtype: float
moving (np.ndarray) – Moving frame Shape: (H’, W’, C), dtype: float

Returns:

Optical flow map from reference to moving: The pixel coordinates are stored as (i, j) (and not (x, y)), so is their displacement. Shape: (2, H’, W’), dtype: float32

Return type:

np.ndarray