Linkers
Review of all the linkers implemented in ByoTrack. For more details, have a look at the documentation or implementation of each linker.
3D visualization of tracks is not mature in ByoTrack and we advise that you save the relabeled segmentation mask to a 3D tiff and visualize them with Fiji/Icy. ___________________________________________________
NearestNeighborLinker (Frame by frame linker using euclidean distance association)
KalmanLinker (Frame by frame linker that models motion with Kalman filters and use maximum likelihood association
KOFTLinker (Frame by frame linker that models motion using Optical Flow enhanced Kalman filters and maximum likelihood association)
IcyEMHTLinker (Wrapper around Icy EMHT algorithm that uses Kalman filters and multiple hypothesis association)
TrackMateLinker (Wrapper around u-track/TrackMate from Fiji. It uses Kalman filters and euclidean distance based association)
[1]:
import matplotlib as mpl
import matplotlib.pyplot as plt
import torch
import byotrack
import byotrack.example_data
import byotrack.visualize
Load a video
[2]:
# Open an example video
video = byotrack.example_data.hydra_neurons()[130:] # Let's start at frame 130 where the animal is contracting
# Or provide a path to one of your video
# video = byotrack.Video("path/to/video.ext")
# Or load manually a video as a numpy array
# video = np.random.randn(50, 500, 500, 3) # T, H, W, C
[3]:
TEST = True # Set to False to analyze a whole video
if TEST:
video = video[:50] # Temporal slicing to analyze only the first 50 frames
[4]:
# A transform can be added to normalize and aggregate channels
transform_config = byotrack.VideoTransformConfig(
aggregate=True, normalize=True, q_min=0.01, q_max=0.999, smooth_clip=1.0
)
video.set_transform(transform_config)
# Show the min max value used to clip and normalize
print(video._normalizer.mini, video._normalizer.maxi)
[0.] [248.]
[5]:
# Display the first frame
frame = video[0]
if video.ndim == 5: # (T, D, H, W, C) (3D video)
frame = frame[frame.shape[0] // 2] # Show the frame in the middle of the stack
plt.figure(figsize=(24, 16), dpi=100)
plt.imshow(frame)
plt.show()
[6]:
# Display the video with opencv
# Use w/x to move forward in time (or space to run/pause the video)
# Use b/n to move inside the stack (For 3D videos)
# Use v to switch on/off the display of the video
byotrack.visualize.InteractiveVisualizer(video).run()
Detections
The linker links detections through time. We use the WaveletDetector from byotrack as an example to produce the detections.
[7]:
# Create the detector object with its hyper parameters
from byotrack.implementation.detector.wavelet import WaveletDetector
detector = WaveletDetector(scale=1, k=3.0, min_area=5.0, batch_size=20, device=torch.device("cpu"))
[8]:
# Run the detection process on the current video
detections_sequence = detector.run(video)
[9]:
# Display the detections with opencv
# Use w/x to move forward in time (or space to run/pause the video)
# Use b/n to move inside the stack (For 3D videos)
# Use v to switch on/off the display of the video
# Use d to switch detection display mode (None, mask, segmentation)
byotrack.visualize.InteractiveVisualizer(video, detections_sequence).run()
NearestNeighborLinker
[10]:
from byotrack.implementation.linker.frame_by_frame.nearest_neighbor import (
AssociationMethod, # noqa: F401
NearestNeighborLinker,
NearestNeighborParameters,
)
[11]:
# See documentation about the Linker
NearestNeighborLinker?
Init signature:
NearestNeighborLinker(
specs: 'NearestNeighborParameters',
optflow: 'byotrack.OpticalFlow | None' = None,
features_extractor: 'byotrack.FeaturesExtractor | None' = None,
*,
save_all=False,
) -> 'None'
Docstring:
Frame by frame linker by associating with the closest detections.
Motion is not modeled, but if an optical flow method is provided, it
will be used to compensate motion online. Matching is done from a simple Euclidean
distance. This can be easily changed by inheriting this class and overriding the `cost` method.
See `FrameByFrameLinker` for the other attributes.
Attributes:
specs (NearestNeighborParameters): Parameters specifications of the algorithm.
See `NearestNeighborParameters`.
active_positions (torch.Tensor): The positions of actives tracks, if undetected it is estimated by
optical flow propagation, or simply propagated if no optical flow is given.
Shape: (N, D), dtype: float32
File: ~/workspace/pasteur/byotrack/src/byotrack/implementation/linker/frame_by_frame/nearest_neighbor.py
Type: ABCMeta
Subclasses:
[12]:
# See documentation about the Linker parameters
NearestNeighborParameters?
Init signature:
NearestNeighborParameters(
association_threshold: 'float' = 5.0,
*,
n_valid=3,
n_gap=3,
association_method: 'str | AssociationMethod' = <AssociationMethod.OPT_SMOOTH: 'opt_smooth'>,
anisotropy: 'tuple[float, float, float]' = (1.0, 1.0, 1.0),
ema=0.0,
fill_gap=False,
split_factor: 'float' = 0.0,
merge_factor: 'float' = 0.0,
)
Docstring:
Parameters of NearestNeighborLinker.
Note:
The merging and splitting features is still experimental.
Attributes:
association_threshold (float): This is the main hyperparameter, it defines the threshold on the distance used
not to link tracks with detections. It prevents to link with false positive detections.
Default: 5 pixels
n_valid (int): Number associated detections required to validate the track after its creation.
Default: 3
n_gap (int): Number of consecutive frames without association before the track termination.
Default: 3
association_method (AssociationMethod): The frame-by-frame association to use. See `AssociationMethod`.
It can be provided as a string. (Choice: GREEDY, [SPARSE_]OPT_HARD, [SPARSE_]OPT_SMOOTH)
Default: OPT_SMOOTH
anisotropy (tuple[float, float, float]): Anisotropy of images (Ratio of the pixel sizes
for each axis, depth first). This will be used to scale distances.
Default: (1., 1., 1.)
ema (float): Optional exponential moving average to reduce detection noise. Detection positions are smoothed
using this EMA. Should be smaller than 1. It use: x_{t+1} = ema x_{t} + (1 - ema) det(t)
As motion is not modeled, EMA may introduce lag that will hinder tracking. It is more effective with
optical flow to compensate motions, in this case, a typical value is 0.5, to average the previous position
with the current measured one. For more advanced modelisation, see `KalmanLinker`.
Default: 0.0 (No EMA)
fill_gap (bool): Fill the gap of missed detections using a forward optical flow
propagation (Only when optical flow is provided). We advise to rather use a
ForwardBackward interpolation using the same optical flow: it will produce
smoother interpolations.
Default: False
split_factor (float): Allow splitting of tracks, using a second association step.
The association threshold in this case is `split_factor * association_threshold`.
Default: 0.0 (No splits)
merge_factor (float): Allow merging of tracks, using a second association step.
The association threshold in this case is `merge_factor * association_threshold`.
Default: 0.0 (No merges)
File: ~/workspace/pasteur/byotrack/src/byotrack/implementation/linker/frame_by_frame/nearest_neighbor.py
Type: type
Subclasses:
[13]:
# Create the linker
# We set only the main parameters.
# You can look at the documentation to see the other parameters and more complete descriptions.
specs = NearestNeighborParameters(
association_threshold=10.0, # Most important parameter: don't link if the euclidean distance is larger than 10 pixels
n_valid=3, # Validate a track after three consecutive detections
n_gap=3, # At most 3 consecutive missed detections
association_method="opt_smooth", # See AssociationMethod?, you can use greedy which is faster but usually less performant
)
linker = NearestNeighborLinker(specs)
[14]:
# Run the linker
tracks = linker.run(video, detections_sequence)
[15]:
# Display the tracks with opencv
# Use w/x to move forward in time (or space to run/pause the video)
# Use b/n to move inside the stack (For 3D videos)
# Use v (resp. t) to switch on/off the display of video (resp. tracks)
# Use d to switch detection display mode (None, mask, segmentation)
byotrack.visualize.InteractiveVisualizer(video, detections_sequence, tracks).run()
[16]:
# Project tracks onto a single image and color by time (Only works with 2D videos)
# Create a list of colors for each time frame
# From cyan (start of the video) to red (end of the video)
hsv = mpl.colormaps["hsv"]
colors = [
tuple(int(c * 255) for c in hsv(0.5 + 0.5 * k / (len(detections_sequence) - 1))[:3])
for k in range(len(detections_sequence))
]
visu = byotrack.visualize.temporal_projection(
byotrack.Track.tensorize(tracks), colors=colors, background=None, color_by_time=True
)
plt.figure(figsize=(24, 16))
plt.imshow(visu)
plt.show()
KalmanLinker
[17]:
from byotrack.implementation.linker.frame_by_frame.kalman_linker import (
Cost, # noqa: F401
KalmanLinker,
KalmanLinkerParameters,
)
[18]:
# See documentation about the Linker
KalmanLinker?
Init signature:
KalmanLinker(
specs: 'KalmanLinkerParameters',
optflow: 'byotrack.OpticalFlow | None' = None,
features_extractor: 'byotrack.FeaturesExtractor | None' = None,
*,
save_all=False,
) -> 'None'
Docstring:
Frame by frame linker using Kalman filters.
Motion is modeled with a Kalman filter of a specified order (See `torch_kf.ckf`)
Matching is done to optimize the given cost. If optical flow is provided, it is used
online to warp the predicted state positions of the kalman filter. This will work, but it
is sub-optimal: consider using `KOFTLinker` that exploits in a finer way optical flow
inside Kalman filters.
This is an implementation of Simple Kalman Tracking (SKT) from KOFT [9].
Note:
This implementation requires torch-kf. (pip install torch-kf)
See `FrameByFrameLinker` for the other attributes.
Attributes:
specs (KalmanLinkerParameters): Parameters specifications of the algorithm.
See `KalmanLinkerParameters`.
kalman_filter (torch_kf.KalmanFilter): The Kalman filter.
active_states (torch_kf.GaussianState): The Kalman filter estimation for each track.
Shape: mean=(N, D * (order + 1), 1), covariance=(N, D * (order + 1), dim * (order + 1))
dtype: float
projections (torch_kf.GaussianState): The Kalman filter projection for each track.
Shape: mean=(N, D, 1), covariance=(N, D, D), precision=(N, D, D)
dtype: float
process_noises (torch.Tensor): The Kalman filter process noise for each track.
Only used when online_process_std > 0.0. It allows to compute an adaptative process_std
and therefore gating for each track.
Shape: (N, D, 1), dtype: float
all_states (list[torch_kf.GaussianState]): The Kalman filter estimation for each track at each seen
frame. States are only registered when save_all=True or if you build tracks from RTS smoothing.
Shape: mean=(N, D * (order + 1), 1), covariance=(N, D * (order + 1), dim * (order + 1))
dtype: float
File: ~/workspace/pasteur/byotrack/src/byotrack/implementation/linker/frame_by_frame/kalman_linker.py
Type: ABCMeta
Subclasses:
[19]:
# See documentation about the Linker parameters
KalmanLinkerParameters?
Init signature:
KalmanLinkerParameters(
association_threshold: 'float' = 5.0,
*,
detection_std: 'float | torch.Tensor' = 3.0,
process_std: 'float | torch.Tensor' = 1.5,
kalman_order: 'int' = 1,
n_valid=3,
n_gap=3,
association_method: 'str | AssociationMethod' = <AssociationMethod.OPT_SMOOTH: 'opt_smooth'>,
anisotropy: 'tuple[float, float, float]' = (1.0, 1.0, 1.0),
cost: 'str | Cost' = <Cost.EUCLIDEAN: 'euclidean'>,
track_building: 'str | TrackBuilding' = <TrackBuilding.FILTERED: 'filtered'>,
split_factor: 'float' = 0.0,
merge_factor: 'float' = 0.0,
online_process_std: 'float' = 0.0,
initial_std_factor: 'float' = 10.0,
)
Docstring:
Parameters of KalmanLinker.
Note:
The merging and splitting features is still experimental.
Attributes:
association_threshold (float): This is the main hyperparameter, it defines the threshold on the distance used
not to link tracks with detections. It prevents to link with false positive detections.
Default: 5 pixels
detection_std (float | torch.Tensor): Expected measurement noise on the detection process.
The detection process is modeled with a Gaussian noise with this given std. (You can provide a different
noise for each dimension). See `torch_kf.ckf.constant_kalman_filter`.
Default: 3.0 pixels
process_std (float | torch.Tensor): Expected process noise. See `torch_kf.ckf.constant_kalman_filter`, the
process is modeled as constant order-th derivative motion. This quantify how much the supposedly "constant"
order-th derivative can change between two consecutive frames. A common rule of thumb is to use
3 * process_std ~= max_t(| dx^(order)(t+1) - dx^(order)(t)|). It can be provided for each dimension).
Default: 1.5 pixels
kalman_order (int): Order of the Kalman filter to use.
0 for brownian motions, 1 for directed brownian motion, 2 for accelerated brownian motions, etc...
Default: 1
n_valid (int): Number associated detections required to validate the track after its creation.
Default: 3
n_gap (int): Number of consecutive frames without association before the track termination.
Default: 3
association_method (AssociationMethod): The frame-by-frame association to use. See `AssociationMethod`.
It can be provided as a string. (Choice: GREEDY, [SPARSE_]OPT_HARD, [SPARSE_]OPT_SMOOTH)
Default: OPT_SMOOTH
anisotropy (tuple[float, float, float]): Anisotropy of images (Ratio of the pixel sizes
for each axis, depth first). This will be used to scale distances. It will only impact
EUCLIDEAN[_SQ] costs. For probabilistic cost, anisotropy should be already integrated
in the stds of the kalman filter (providing one std for each dimension).
Default: (1., 1., 1.)
cost_method (CostMethod): The cost method to use. It can be provided as a string.
See `CostMethod`. It also indicates what is the correct unit of `association_threshold`.
Default: EUCLIDEAN
track_building (TrackBuilding): Tells the linker how to build the final tracks.
Either from detections, or from filtered/smoothed positions computed by the
Kalman filter. See `TrackBuilding`. It can be provided as a string.
Default: FILTERED
split_factor (float): Allow splitting of tracks, using a second association step.
The association threshold in this case is `split_factor * association_threshold`.
Default: 0.0 (No splits)
merge_factor (float): Allow merging of tracks, using a second association step.
The association threshold in this case is `merge_factor * association_threshold`.
Default: 0.0 (No merges)
online_process_std (float): Recomputes the process std online following "A. Genovesio, et al, 2004, October.
Adaptive gating in Gaussian Bayesian multi-target tracking. ICIP'04. (Vol. 1, pp. 147-150). IEEE."
Each track has its own process std depending on the errors made in the past. It automatically adjusts to
process errors, allowing to increase the validation gate. Should be used in conjunction with MAHALANOBIS
or LIKELIHOOD `cost_method`. As this may be detrimental, it is disabled by default.
Default: 0.0 (Process_std is constant)
initial_std_factor (float): The uncertainties on initial velocities/accelerations are set
to initial_std_factor * process_std. See `KalmanLinker.build_initial_covariance`.
Having a small factor will prevent handling correctly starting tracks with large initial velocity
on their first frames. But large values will lead to large uncertainty on the first prediction, making
it hard to associate to a detection with MAHALANOBIS or LIKELIHOOD methods.
Typical values lies between 3.0 to 10.0.
Default: 10.0
File: ~/workspace/pasteur/byotrack/src/byotrack/implementation/linker/frame_by_frame/kalman_linker.py
Type: type
Subclasses:
[20]:
# Create the linker
# We set only the main parameters.
# You can look at the documentation to see the other parameters and more complete descriptions.
specs = KalmanLinkerParameters(
association_threshold=3e-4, # Most important parameter: don't link if the association likelihood is smaller than 3e-4.
detection_std=1.0, # Detections precision in pixels (Usually ~ size of spots / 3)
process_std=2.0, # Kalman filter predictions are precise up to 2.0 pixels (Usually ~ unexpected displacement / 3)
kalman_order=1, # Order of the kalman filter (0: Brownian, 1: Directed, 2: Accelerated, ...)
cost="likelihood", # See Cost? to see which other cost are available, by default it uses Euclidean distance (And association threshold should be express in pixels)
)
linker = KalmanLinker(specs)
[21]:
# Run the linker
tracks = linker.run(video, detections_sequence)
[22]:
# Display the tracks with opencv
# Use w/x to move forward in time (or space to run/pause the video)
# Use b/n to move inside the stack (For 3D videos)
# Use v (resp. t) to switch on/off the display of video (resp. tracks)
# Use d to switch detection display mode (None, mask, segmentation)
byotrack.visualize.InteractiveVisualizer(video, detections_sequence, tracks).run()
[23]:
# Project tracks onto a single image and color by time (Only works with 2D videos)
# Create a list of colors for each time frame
# From cyan (start of the video) to red (end of the video)
hsv = mpl.colormaps["hsv"]
colors = [
tuple(int(c * 255) for c in hsv(0.5 + 0.5 * k / (len(detections_sequence) - 1))[:3])
for k in range(len(detections_sequence))
]
visu = byotrack.visualize.temporal_projection(
byotrack.Track.tensorize(tracks), colors=colors, background=None, color_by_time=True
)
plt.figure(figsize=(24, 16))
plt.imshow(visu)
plt.show()
KOFTLinker
[24]:
from byotrack.implementation.linker.frame_by_frame.koft import KOFTLinker, KOFTLinkerParameters
[25]:
# See documentation about the Linker
KOFTLinker?
Init signature:
KOFTLinker(
specs: 'KOFTLinkerParameters',
optflow: 'byotrack.OpticalFlow | None' = None,
features_extractor: 'byotrack.FeaturesExtractor | None' = None,
*,
save_all=False,
) -> 'None'
Docstring:
Kalman and Optical Flow Tracking [9].
Motion is modeled with a Kalman filter of a specified order >= 1 (See `torch_kf.ckf`)
Positions are measured through the detection process. A second update step is performed
to measure the velocity of all tracks using optical flow.
Matching is done to optimize the given cost.
Note:
This implementation requires torch-kf. (pip install torch-kf)
See `KalmanLinker` for the other attributes.
Attributes:
specs (KOFTLinkerParameters): Parameters specifications of the algorithm.
See `KOFTLinkerParameters`.
last_detections (byotrack.Detections): The last detections used in update.
Optionally used to extract flows at the detection positions and not the track state.
Required for `motion_model`
File: ~/workspace/pasteur/byotrack/src/byotrack/implementation/linker/frame_by_frame/koft.py
Type: ABCMeta
Subclasses:
[26]:
# See documentation about the Linker parameters
KOFTLinkerParameters?
Init signature:
KOFTLinkerParameters(
association_threshold: 'float',
*,
detection_std: 'float | torch.Tensor' = 3.0,
flow_std: 'float | torch.Tensor' = 1.0,
process_std: 'float | torch.Tensor' = 1.5,
kalman_order: 'int' = 1,
n_valid=3,
n_gap=3,
association_method: 'str | AssociationMethod' = <AssociationMethod.OPT_SMOOTH: 'opt_smooth'>,
anisotropy: 'tuple[float, float, float]' = (1.0, 1.0, 1.0),
cost: 'str | Cost' = <Cost.EUCLIDEAN: 'euclidean'>,
track_building: 'str | TrackBuilding' = <TrackBuilding.FILTERED: 'filtered'>,
split_factor: 'float' = 0.0,
merge_factor: 'float' = 0.0,
extract_flows_on_detections=False,
always_measure_velocity=True,
online_process_std=0.0,
initial_std_factor=10.0,
)
Docstring:
Parameters of KOFTLinker.
Note:
The merging and splitting features is still experimental.
Attributes:
association_threshold (float): This is the main hyperparameter, it defines the threshold on the distance used
not to link tracks with detections. It prevents to link with false positive detections.
detection_std (float | torch.Tensor): Expected measurement noise on the detection process.
The detection process is modeled with a Gaussian noise with this given std. (You can provide a different
noise for each dimension). See `torch_kf.ckf.constant_kalman_filter`.
Default: 3.0 pixels
flow_std (float | torch.Tensor): Expected measurement noise on the optical flow process.
The optical flow process is modeled with a Gaussian noise with this given std. (You can provide a different
noise for each dimension).
Default: 1.0 pixels
process_std (float | torch.Tensor): Expected process noise. See `torch_kf.ckf.constant_kalman_filter`, the
process is modeled as constant order-th derivative motion. This quantify how much the supposedly "constant"
order-th derivative can change between two consecutive frames. A common rule of thumb is to use
3 * process_std ~= max_t(| dx^(order)(t+1) - dx^(order)(t)|). It can be provided for each dimension).
Default: 1.5 pixels
kalman_order (int): Order of the Kalman filter to use. 0 is for brownian motion (it predicts a 0 velocity)
1 for directed brownian motion, 2 for accelerated brownian motions, etc...
Default: 1
n_valid (int): Number associated detections required to validate the track after its creation.
Default: 3
n_gap (int): Number of consecutive frames without association before the track termination.
Default: 3
association_method (AssociationMethod): The frame-by-frame association to use. See `AssociationMethod`.
It can be provided as a string. (Choice: GREEDY, [SPARSE_]OPT_HARD, [SPARSE_]OPT_SMOOTH)
Default: OPT_SMOOTH
anisotropy (tuple[float, float, float]): Anisotropy of images (Ratio of the pixel sizes
for each axis, depth first). This will be used to scale distances. It will only impact
EUCLIDEAN[_SQ] costs. For probabilistic cost, anisotropy should be already integrated
in the stds of the kalman filter (providing one std for each dimension).
Default: (1., 1., 1.)
cost_method (CostMethod): The cost method to use. It can be provided as a string.
See `CostMethod`. It also indicates what is the correct unit of `association_threshold`.
Default: EUCLIDEAN
track_building (TrackBuilding): Tells the linker how to build the final tracks.
Either from detections, or from filtered/smoothed positions computed by the
Kalman filter. See `TrackBuilding`. It can be provided as a string.
Default: FILTERED
split_factor (float): Allow splitting of tracks, using a second association step.
The association threshold in this case is `split_factor * association_threshold`.
Default: 0.0 (No splits)
merge_factor (float): Allow merging of tracks, using a second association step.
The association threshold in this case is `merge_factor * association_threshold`.
Default: 0.0 (No merges)
extract_flows_on_detections (bool): If True it extracts the optical flow at the detection location if possible.
Otherwise it extract the flow from the current estimate of the track position.
Default: False
always_measure_velocity (bool): Update velocity for all tracks even non-linked ones.
If set to False, it implements KOFT-- from the paper. This is sub-optimal, you should keep it True.
Default: True
online_process_std (float): Recomputes the process std online following "A. Genovesio, et al, 2004, October.
Adaptive gating in Gaussian Bayesian multi-target tracking. ICIP'04. (Vol. 1, pp. 147-150). IEEE."
Each track has its own process std depending on the errors made in the past. It automatically adjusts to
process errors, allowing to increase the validation gate. Should be used in conjunction with MAHALANOBIS
or LIKELIHOOD `cost_method`. As this may be detrimental, it is disabled by default.
Default: 0.0 (Process_std is constant)
initial_std_factor (float): The uncertainties on initial velocities/accelerations are set
to initial_std_factor * process_std. See `KalmanLinker.build_initial_covariance`.
Having a small factor will prevent handling correctly starting tracks with large initial velocity
on their first frames.
Typical values lies between 3.0 to 100.0.
Default: 10.0
File: ~/workspace/pasteur/byotrack/src/byotrack/implementation/linker/frame_by_frame/koft.py
Type: type
Subclasses:
[27]:
# Koft requires optical flow (NOTE: that optical flow can also be efficiently be used with the two previous linker)
# You could use any optical flow algorithm, but ByoTrack already supports OpenCV and Skimage implementations.
# Let's use Farneback from OpenCV (no extra dependencies)
import cv2
from byotrack.implementation.optical_flow.opencv import OpenCVOpticalFlow
optflow = OpenCVOpticalFlow(cv2.FarnebackOpticalFlow_create(winSize=20), downscale=4)
[28]:
# Before linking, let's check visually that the optical flow algorithm works (Only works with 2D videos)
# We sample a grid of points that are moved by the flow computed.
# The computed flows are good if the points roughly follows the video motion
# Use w/x to move forward in time (or space to run/pause the video)
# Use g to reset the grid of points
byotrack.visualize.InteractiveFlowVisualizer(video, optflow).run()
[29]:
# Create the linker
# We set only the main parameters.
# You can look at the documentation to see the other parameters and more complete descriptions.
specs = KOFTLinkerParameters(
association_threshold=1e-3, # Most important parameter: don't link if the association likelihood is smaller than 1e-3.
detection_std=1.0, # Detections precision in pixels (Usually ~ size of spots / 3)
process_std=2.0, # Kalman filter predictions precision in pixels (Usually ~ unexpected displacement / 3)
flow_std=1.0, # Optical flow precision (Usually ~ max flow errors / 3)
kalman_order=1, # Order of the kalman filter (1: Directed, 2: Accelerated, ...)
n_gap=5, # Allow to link after 5 consecutive missed detections
cost="likelihood", # See Cost? to see which other cost are available, by default it uses Euclidean distance (And association threshold should be express in pixels)
)
linker = KOFTLinker(specs, optflow)
[30]:
# Run the linker
tracks = linker.run(video, detections_sequence)
[31]:
# Display the tracks with opencv
# Use w/x to move forward in time (or space to run/pause the video)
# Use b/n to move inside the stack (For 3D videos)
# Use v (resp. t) to switch on/off the display of video (resp. tracks)
# Use d to switch detection display mode (None, mask, segmentation)
byotrack.visualize.InteractiveVisualizer(video, detections_sequence, tracks).run()
[32]:
# Project tracks onto a single image and color by time (Only works with 2D videos)
# Create a list of colors for each time frame
# From cyan (start of the video) to red (end of the video)
hsv = mpl.colormaps["hsv"]
colors = [
tuple(int(c * 255) for c in hsv(0.5 + 0.5 * k / (len(detections_sequence) - 1))[:3])
for k in range(len(detections_sequence))
]
visu = byotrack.visualize.temporal_projection(
byotrack.Track.tensorize(tracks), colors=colors, background=None, color_by_time=True
)
plt.figure(figsize=(24, 16))
plt.imshow(visu)
plt.show()
EMHT (Icy)
Icy software must be installed
[33]:
from byotrack.implementation.linker.icy_emht import EMHTParameters, IcyEMHTLinker, Motion
[34]:
# See documentation about the Linker
IcyEMHTLinker?
Init signature:
IcyEMHTLinker(
icy_path: 'str | os.PathLike | None' = None,
full_specs: 'EMHTParameters | None' = None,
timeout: 'float | None' = None,
) -> 'None'
Docstring:
Run EMHT [4] from Icy [1].
It is a wrapper around Icy's tracking code. It supports 2D and 3D data.
About EMHT:
It is a probabilistic tracking that uses statistical motion model on particles. It uses multiple
kalman filters for each particle allowing a particle to have several mode of motions. It also keeps
multiple hypothesis of tracking at each frame so that a final detections linking decision is made
after seeing several frames in the past and future of these detections.
Here we rely on the handmade protocol "emht_protocol" that expects detections as a valid rois file
for icy and some hyperparameters.
The workflow is:
1. Detections to rois in Icy format
2. Run the Icy protocol
3. [In ICY] Read rois, estimate or load emht parameters, run emht, export tracks to xml
4. Read Icy tracks and return
Note:
This implementation requires Icy to be installed (https://icy.bioimageanalysis.org/download/)
You should also install the Spot Tracking Blocks plugin
(https://icy.bioimageanalysis.org/tutorial/how-to-install-an-icy-plugin/)
Attributes:
runner (byotrack.icy.IcyRunner): Icy runner
motion (Motion): Prior on the underlying motion model (Brownian vs Directed vs Both)
Given to the Icy block that estimates EMHT parameters. (See `full_specs` and `EMHTParameters` to have
a finegrained control over the algorithm parameters)
Default: Motion.BROWNIAN
full_specs (EMHTParameters | None): Full specification of the algorithm. If not provided,
we use the estimation of EMHT parameters provided by Icy (with `motion` the only parameter to set).
Init docstring:
Constructor.
Args:
icy_path (str | os.PathLike): Path to the icy jar (Icy is called with java -jar <icy_jar>)
If not given, icy is searched in the PATH
timeout (float | None): Optional timeout in seconds as EMHT may enter an infinite loop.
File: ~/workspace/pasteur/byotrack/src/byotrack/implementation/linker/icy_emht/icy_emht.py
Type: ABCMeta
Subclasses:
[35]:
# See documentation about the Linker parameters
EMHTParameters?
Init signature:
EMHTParameters(
detections_fpr: 'float' = 0.1,
detections_fnr: 'float' = 0.1,
expected_track_length: 'int' = -1,
expected_initial_particles: 'int' = -1,
expected_new_particles: 'int' = 10,
existence_prob: 'float' = 0.5,
termination_prob: 'float' = 0.0001,
motion: 'Motion' = <Motion.BROWNIAN: (0, 0)>,
use_most_likely_model: 'bool' = True,
update_motion_1: 'bool' = False,
update_motion_2: 'bool' = False,
xy_std_1: 'float' = 3.0,
z_std_1: 'float' = 3.0,
xy_std_2: 'float' = 3.0,
z_std_2: 'float' = 3.0,
inertia: 'float' = 0.8,
gate_factor: 'float' = 4.0,
tree_depth: 'int' = 4,
) -> None
Docstring:
Parameters of EMHT algorithm [4].
Attributes:
detections_fpr (float): Estimation of the false positive rate of the detection process
Default: 0.1
detections_fnr (float): Estimation of the false negative rate of the detection process
Default: 0.1
expected_track_length (int): Expected length of tracks. If negative, it defaults to the sequence size
Default: -1
expected_initial_particles (int): Estimation of the number of particles in the first frames. If negative, it
defaults to the average number of detections by frame in the sequence
Default: -1
expected_new_particles (int): Estimation of the number of new particle by frame
Default: 0
existence_prob (float): Minimum probability to confirm a track existence
Default: 0.5
termination_prob (float): Minimum probability before terminating a track
Default: 1e-4
motion (Motion): Motion of the particles (Brownian vs Directed vs Multi). If MULTI, motion 1 is
brownian and motion 2 is directed.
Default: Motion.BROWNIAN
use_most_likely_model (bool): Use the most likely model to predict rather than a weighted predictions
Default: True
update_motion_{1, 2} (bool): Use an adaptative process covariance, depending on previous prediction errors.
xy_std_{1,2} (float): Used to define the covariance (Q) of the process. (Looking at Icy code,
it is not clear what they truly are, depending on the motion the covariance Q is set a bit weirdly)
Default: 3.0
z_std_{1,2} (float): Same as `xy_std` but for the 3d axes (if any)
Default: 3.0
inertia (float): Probability to not switch of motion model (For MULTI motion)
Default: 0.8
gate_factor (float): Max mahalanobis distance for potential association
Default: 4.0
tree_depth (int): Number of frames to consider in the search tree
Default: 4
File: ~/workspace/pasteur/byotrack/src/byotrack/implementation/linker/icy_emht/icy_emht.py
Type: type
Subclasses:
[36]:
# Create the linker object with icy path
# This Linker requires to install Icy software first
icy_path = "path/to/icy/icy.jar"
motion = Motion.MULTI # Can also be DIRECTED or MULTI (both)
if True: # Set full specs with EMHTParameters
# You can choose to set manually the parameters. See EMHTParameters?
# the more important ones are:
# - gate_factor: How greedy the linking is. (Default to 4.0) more or less equivalent to the association_threshold
# of KalmanLinker with a Mahalanobis Cost.
# - motion: Motion model to consider: Can be BROWNIAN, DIRECTED or MULTI. (Default is BROWNIAN)
# Brownian <=> kalman_order = 0, Directed <=> kalman_order = 1 (MULTI uses both)
# - tree_depth: MHT tree depth. Higher values are usually more performant, but much more expensive
# If the tracking is too slow or too ram intensive, you may reduce this value. (Default 4)
parameters = EMHTParameters(gate_factor=4.0, motion=motion, tree_depth=2)
linker = IcyEMHTLinker(icy_path, parameters)
else: # Do not provide specs, parameters will be estimated by Icy (We do not advise this solution)
linker = IcyEMHTLinker(icy_path)
linker.motion = motion # Set motion afterwards if no parameters are provided
[37]:
# Run the linker
tracks = linker.run(video, detections_sequence)
Calling Icy with: java -jar icy.jar -hl -x plugins.adufour.protocols.Protocols protocol="/home/rreme/workspace/pasteur/byotrack/src/byotrack/implementation/linker/icy_emht/emht_protocol_with_full_specs.xml" rois="/home/rreme/workspace/pasteur/byotrack/docs/source/run_examples/_tmp_rois.xml" parameters="/home/rreme/workspace/pasteur/byotrack/docs/source/run_examples/_tmp_parameters.xml" tracks="/home/rreme/workspace/pasteur/byotrack/docs/source/run_examples/_tmp_tracks.xml"
[DEBUG] 2026-03-16 15:42:47 - Initializing...
/snap/core20/current/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /lib/x86_64-linux-gnu/libproxy.so.1)
Failed to load module: /home/rreme/snap/code/common/.cache/gio-modules/libgiolibproxy.so
JarResources.loadJar(/home/rreme/workspace/pasteur/icy-app-v3/plugins/nchenouard/particletracking/sequenceGenerator/._tracking-benchmark-generator-2.0.0.jar) error:
java.util.zip.ZipException: zip END header not found
[INFO] 2026-03-16 15:42:48 - Java(TM) SE Runtime Environment 21.0.1+12-LTS-29 (64 bit)
[INFO] 2026-03-16 15:42:48 - Running on Linux 6.8.0-101-generic (amd64)
[INFO] 2026-03-16 15:42:48 - System total memory: 32.6 GB
[INFO] 2026-03-16 15:42:48 - System available memory: 4.8 GB
[INFO] 2026-03-16 15:42:48 - Max Java memory: 8.2 GB
[INFO] 2026-03-16 15:42:48 - Headless mode.
[INFO] 2026-03-16 15:42:48 - Icy v3.0.0a started.
[DEBUG] 2026-03-16 15:42:49 - Magic name is Spot Tracking
[DEBUG] 2026-03-16 15:42:49 - Magic icon is /plugins/nchenouard/particletracking/simplifiedMHT/detectionIcon.png
[DEBUG] 2026-03-16 15:42:49 - Magic name is EzPlug tutorial
[DEBUG] 2026-03-16 15:42:49 - Magic icon is /plugins/adufour/ezplug/ezplug.png
Loading workflow...
Error(s) while loading protocol:
Variable 'useLPSolver' not found in block 'Spot tracking do tracking'. It may have been removed or renamed.
--
Converted from 31443 ROI(s)
Non binaryaction at frame 3
Non binaryaction at frame 9
Non binaryaction at frame 46
Track extraction at frame 49
[INFO] 2026-03-16 15:43:15 - Exiting...
[INFO] 2026-03-16 15:43:15 - Done.
[38]:
# Display the tracks with opencv
# Use w/x to move forward in time (or space to run/pause the video)
# Use b/n to move inside the stack (For 3D videos)
# Use v (resp. t) to switch on/off the display of video (resp. tracks)
# Use d to switch detection display mode (None, mask, segmentation)
byotrack.visualize.InteractiveVisualizer(video, detections_sequence, tracks).run()
[39]:
# Project tracks onto a single image and color by time (Only works with 2D videos)
# Create a list of colors for each time frame
# From cyan (start of the video) to red (end of the video)
hsv = mpl.colormaps["hsv"]
colors = [
tuple(int(c * 255) for c in hsv(0.5 + 0.5 * k / (len(detections_sequence) - 1))[:3])
for k in range(len(detections_sequence))
]
visu = byotrack.visualize.temporal_projection(
byotrack.Track.tensorize(tracks), colors=colors, background=None, color_by_time=True
)
plt.figure(figsize=(24, 16))
plt.imshow(visu)
plt.show()
TrackMate (Fiji)
ImageJ/Fiji software must be installed
[40]:
from byotrack.implementation.linker.trackmate import TrackMateLinker, TrackMateParameters
[41]:
# See documentation about the Linker
TrackMateLinker?
Init signature:
TrackMateLinker(
fiji_path: 'str | os.PathLike',
specs: 'TrackMateParameters',
) -> 'None'
Docstring:
Run TrackMate [7, 8] from Fiji [6].
Wrapper around TrackMate, using Fiji headless call. Supports 2D and 3D frames.
About TrackMate:
It is a global distance minimization tracking. It supports multiple algorithms. We have wrapped the more advanced
ones (SparseLapTracker and AdvancedKalmanTracker that both follows [7]). It solves frame-to-frame GDM between
detections. And then solve a GDM between tracklets to correct them (stitch, merge, split).
The AdvancedKalmanTracker additionally uses kalman filters to estimate velocities of particles.
Here we rely on the handmade ImageJ script "_trackmate.py" that expects detections as instance segmentation
and the hyperparameters of the linking process.
We do not support track splitting and merging yet, neither the use of feature-based cost.
The workflow is:
1. Save detections to segmentation format
2. Run the Fiji script
3. [In Fiji] Read segmentation, load parameters, extract detections, run linking [5], export tracks to xml
4. Read Fiji tracks and return
Note:
This implementation requires Fiji to be installed (https://imagej.net/downloads)
Note:
In case of missed detections, positions are filled with nan. To fill nan with true values, use an Interpolator
Attributes:
runner (byotrack.fiji.FijiRunner): Fiji runner
specs (TrackmateParameters): Parameters specifications of the algorithm
Init docstring:
Constructor.
Args:
fiji_path (str | os.PathLike): Path to the fiji executable
The executable can be found inside the installation folder of Fiji.
Linux: Fiji.app/ImageJ-<os>
Windows: Fiji.app/ImageJ-<os>.exe
MacOs: Fiji.app/Contents/MacOs/ImageJ-<os>
File: ~/workspace/pasteur/byotrack/src/byotrack/implementation/linker/trackmate/trackmate.py
Type: ABCMeta
Subclasses:
[42]:
# See documentation about the Linker parameters
TrackMateParameters?
Init signature:
TrackMateParameters(
allow_gap_closing: 'bool' = True,
allow_track_splitting: 'bool' = False,
allow_track_merging: 'bool' = False,
alternative_linking_cost_factor: 'float' = 1.05,
blocking_value: 'float' = inf,
cutoff_percentile: 'float' = 0.9,
max_frame_gap: 'int' = 2,
linking_max_distance: 'float' = 15.0,
gap_closing_max_distance: 'float' = 15.0,
merging_max_distance: 'float' = 15.0,
splitting_max_distance: 'float' = 15.0,
kalman_search_radius: 'float | None' = None,
) -> None
Docstring:
Parameters of TrackMate [7, 8].
We do *not* support Features penalties yet. Moreover track splitting and merging is currently *not* supported in
ByoTrack.
To use kalman filtering in addition to sparse lap tracking, set the `kalman_search_radius` value. By default,
no kalman filtering is used.
The main parameters to set are:
* `linking_max_distance`: The distance threshold for frame-to-frame linking.
* `max_frame_gap` and `gap_closing_max_distance`: The temporal and spatial distances for track stitching.
* `kalman_search_radius`: The distance threshold for kalman linking (Switch to AdvancedKalmanTracker).
Attributes:
allow_gap_closing (bool): Use a second lap to solve tracklet stitching (Closing temporal gap
between previously established tracks)
Default: True
allow_track_splitting (bool): NOT SUPPORTED (Useful for cell tracking)
Default: False
allow_track_merging (bool): NOT SUPPORTED (To allow split and remerge and handle split detections)
Default: False
alternative_linking_cost_factor (float): Alternative linking cost
Default: 1.05
blocking_value (float): Cost for mon-physical, forbidden links.
Default: inf
cutoff_percentile (float): Cutoff percentile
Default: 0.9
max_frame_gap (int): The max difference in frames between two spots to allow for linking.
For instance a value of 2 means that the tracker will be able to make a link between a spot in frame t
and a successor spots in frame t+2, effectively bridging over one missed detection in one frame.
Default: 2 (1 missed detections)
linking_max_distance (float): The max distance between two consecutive spots, in physical units, allowed
for creating links. If using kalman filters: this is the initial search radius, in physical units,
specifying how far two spots can be apart when initiating new tracks. (See `kalman_search_radius`)
Default: 15.0
gap_closing_max_distance (float): Gap-closing max spatial distance. The max distance between two spots,
in physical units, allowed for creating links over missing detections.
Default: 15.0
merging_max_distance (float): Unused. Track merging max spatial distance.
Default: 15.0
splitting_max_distance (float): Unused. Track splitting max spatial distance.
Default: 15.0
kalman_search_radius (float | None): Set this parameter to use the AdvancedKalmanTracker. The max search
radius specifying how far from a predicted position the tracker should look for candidate spots.
Default: None (Without kalman filters)
File: ~/workspace/pasteur/byotrack/src/byotrack/implementation/linker/trackmate/trackmate.py
Type: type
Subclasses:
[43]:
# Create the linker object with fiji path
# This Linker requires to install Fiji software first
# We set only the main parameters.
# You can look at the documentation to see the other parameters and more complete descriptions.
fiji_path = "path/to/Fiji.app/ImageJ-os"
specs = TrackMateParameters(
linking_max_distance=10.0, # Max linking euclidean distance (pixels) between consecutive spots
max_frame_gap=4, # Max diff in frames to allow gap closing. Here it allows 3 consecutives missed detections
gap_closing_max_distance=15.0, # Max gap closing euclidean distance (pixels).
kalman_search_radius=10.0, # When set, it enables Kalman filters, and replace the linking_max_distance (except for the first two spots association)
)
linker = TrackMateLinker(fiji_path, specs)
[44]:
# Run the linker
tracks = linker.run(video, detections_sequence)
Calling Fiji with: ./ImageJ-linux64 --ij2 --headless --console --run "/home/rreme/workspace/pasteur/byotrack/src/byotrack/implementation/linker/trackmate/_trackmate.py" "detections='/home/rreme/workspace/pasteur/byotrack/docs/source/run_examples/_tmp_detections.tif',parameters='/home/rreme/workspace/pasteur/byotrack/docs/source/run_examples/_tmp_parameters.json',tracks='/home/rreme/workspace/pasteur/byotrack/docs/source/run_examples/_tmp_tracks.xml'"
OpenJDK 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
[WARNING] 2 exceptions occurred during plugin discovery.
Hello from ImageJ/Fiji
('Loading detections from', u'/home/rreme/workspace/pasteur/byotrack/docs/source/run_examples/_tmp_detections.tif')
Settings:
{u'MAX_FRAME_GAP': 4, u'ALTERNATIVE_LINKING_COST_FACTOR': 1.05, u'KALMAN_SEARCH_RADIUS': 10.0, u'LINKING_FEATURE_PENALTIES': {}, u'LINKING_MAX_DISTANCE': 10.0, u'GAP_CLOSING_MAX_DISTANCE': 15.0, u'MERGING_FEATURE_PENALTIES': {}, u'SPLITTING_MAX_DISTANCE': 15.0, u'BLOCKING_VALUE': inf, u'ALLOW_GAP_CLOSING': True, u'ALLOW_TRACK_SPLITTING': False, u'ALLOW_TRACK_MERGING': False, u'MERGING_MAX_DISTANCE': 15.0, u'SPLITTING_FEATURE_PENALTIES': {}, u'CUTOFF_PERCENTILE': 0.9, u'GAP_CLOSING_FEATURE_PENALTIES': {}}
Starting detection process using 16 threads.
Detection processes 16 frames simultaneously and allocates 1 thread per frame.
Detection...
Found 31443 spots.
Starting initial filtering process.
Computing spot features over 16 frames simultaneously and allocating 1 thread per frame.
Calculating 31443 spots features...
Computation done in 4 ms.
Starting spot filtering process.
Starting tracking process.
Creating the segment linking cost matrix...
Creating the main cost matrix...
Completing the cost matrix...
Solving the cost matrix...
Creating links...
Computing edge features:
Computation done in 0 ms.
Computing track features:
Computation done in 6 ms.
Starting track filtering process.
[45]:
# Display the tracks with opencv
# Use w/x to move forward in time (or space to run/pause the video)
# Use b/n to move inside the stack (For 3D videos)
# Use v (resp. t) to switch on/off the display of video (resp. tracks)
# Use d to switch detection display mode (None, mask, segmentation)
byotrack.visualize.InteractiveVisualizer(video, detections_sequence, tracks).run()
[46]:
# Project tracks onto a single image and color by time (Only works with 2D videos)
# Create a list of colors for each time frame
# From cyan (start of the video) to red (end of the video)
hsv = mpl.colormaps["hsv"]
colors = [
tuple(int(c * 255) for c in hsv(0.5 + 0.5 * k / (len(detections_sequence) - 1))[:3])
for k in range(len(detections_sequence))
]
visu = byotrack.visualize.temporal_projection(
byotrack.Track.tensorize(tracks), colors=colors, background=None, color_by_time=True
)
plt.figure(figsize=(24, 16))
plt.imshow(visu)
plt.show()
