propagation

class byotrack.implementation.refiner.propagation.DirectedPropagate(*args, **kwargs)

Bases: Protocol

Protocol for directed propagate method

Methods that implement this protocol are expected to propagate the tracks matrix in the given direction

byotrack.implementation.refiner.propagation.forward_backward_propagation(tracks_matrix: Tensor, video: Sequence[ndarray] | ndarray, method: str | DirectedPropagate, **kwargs) → Tensor

Fill all NaN values in the tracks matrix by merging a forward and backward propagation

Esimate missing positions for any defined track and frame. First it computes forward estimations, then backward ones. If estimations overlap (for instance with missing positions inside a track), it will merge smoothly the estimations.

We provide two implementations for directed propagation: Constant where we propagate the last known position of the track TPS where we use ThinPlateSpline using the active tracks to estimate the motion of other tracks

Parameters:

tracks_matrix (torch.Tensor) – Tracks data in a single tensor (See Track.tensorize) Shape: (T, N, D), dtype: float32
video (Sequence[np.ndarray] | np.ndarray) – Video on which the tracks are based. (video[i] should match with the points from tracks_matrix[i])
method (str | DirectedPropagate) – Method to use for propagation. Either “constant”, “tps” or “flow” (see constant_directed_propagate, tps_directed_propagate or optical_flow_directed_propagate) or a user defined Callable following the DirectedPropagate protocol. The method will be called with the tracks_matrix, video, a boolean direction and additional kwargs.
**kwargs (Any) – Additional arguments given to the method

Returns:

Extrapolated point for each track and time: Shape: (T, N, D), dtype: float32

Return type:

torch.Tensor

byotrack.implementation.refiner.propagation.merge(tracks_matrix: Tensor, forward_positions: Tensor, backward_positions: Tensor) → Tensor

Merge forward and backward propagation into a single propagation matrix

If both propagation defines a position estimation, a weighted average is performed.

Parameters:

tracks_matrix (torch.Tensor) – Original tracks data (Track.tensorize) Shape: (T, N, D), dtype: float32
forward_positions (torch.Tensor) – Forward estimation of positions Shape: (T, N, D), dtype: float32
backward_positions (torch.Tensor) – backward estimation of positions Shape: (T, N, D), dtype: float32

Returns:

Merged propagation matrix: Shape: (T, N, D), dtype: float32

Return type:

torch.tensor

byotrack.implementation.refiner.propagation.constant_directed_propagate(tracks_matrix: Tensor, video: Sequence[ndarray] | ndarray = (), forward=True, **kwargs) → Tensor

Propagate tracks matrix with the last known position in a single direction

Parameters:

tracks_matrix (torch.Tensor) – Tracks data in a single tensor (See Tracks.tensorize) Shape: (T, N, D), dtype: float32
video (Sequence[np.ndarray] | np.ndarray) – Video (Unused, added for api purposes) Default: () (As unused a default value is provided)
forward (bool) – Forward or backward propagation Default: True (Forward)

Returns:

Estimation of tracks point in a single direction: Shape: (T, N, D), dtype: float32

Return type:

torch.Tensor

byotrack.implementation.refiner.propagation.tps_directed_propagate(tracks_matrix: Tensor, video: Sequence[ndarray] | ndarray = (), forward=True, alpha=10.0, track_mask: Tensor | None = None, **kwargs) → Tensor

Propagate tracks matrix using Thin Plate Spline (TPS) algorithm in a single direction

Note

We use torch-tps which is very fast but not very accurate. For instance, shuflling the tracks may yield different results (<1px deviation). We have not noticed any real impact on EMC2 stitching performances. Alpha > 5.0 is advised (reduces the numerical errors + outliers resilience).

Parameters:

tracks_matrix (torch.Tensor) – Tracks data in a single tensor (See Tracks.tensorize) Shape: (T, N, D), dtype: float32
video (Sequence[np.ndarray] | np.ndarray) – Video (Unused, added for api purposes) Default: () (As unused a default value is provided)
forward (bool) – Forward or backward propagation Default: True (Forward)
alpha (float) – Thin Plate Spline regularization (See torch_tps librarie). We advise to use alpha > 5.0. It improves outliers resiliences and helps reducing numerical errors. Default: 10.0
track_mask (Optional[torch.Tensor]) – Filter out some tracks to build control points Allow to drop uncertain tracks or to simulate propagation with less particles Shape: (N, ), dtype: bool

Returns:

Estimation of tracks point in a single direction: Shape: (T, N, D), dtype: float32

Return type:

torch.Tensor

byotrack.implementation.refiner.propagation.optical_flow_directed_propagate(tracks_matrix: ~torch.Tensor, video: ~typing.Sequence[~numpy.ndarray] | ~numpy.ndarray, forward=True, optflow: ~byotrack.api.optical_flow.optical_flow.OpticalFlow = <byotrack.api.optical_flow.optical_flow.DummyOpticalFlow object>, **kwargs)

Propagate tracks matrix in single direction using optical flow

Parameters:

tracks_matrix (torch.Tensor) – Tracks data in a single tensor (See Tracks.tensorize) Shape: (T, N, D), dtype: float32
video (Sequence[np.ndarray] | np.ndarray) – Video on which the tracks are based. (video[i] should match with the points from tracks_matrix[i]).
forward (bool) – Forward or backward propagation Default: True (Forward)
optflow (byotrack.OpticalFlow) – Optical flow to use Default: DummyOpticalFlow (predict no displacement <=> constant propagate)

Returns:

Estimation of tracks point in a single direction: Shape: (T, N, D), dtype: float32

Return type:

torch.Tensor