TrackOnSTra Linker

Online implementation of TrackAstra.

byotrack.implementation.linker.frame_by_frame.trackonstra.build_cost_dict(nodes: list[dict[str, Any]], weights: Iterable[tuple[tuple[int, int], float]]) → dict[tuple[int, int, int], dict[int, float]]

Build the cost dictionary from Trackastra data format.

It converts the feasible tracking graph predicted by trackastra into a mapping from edge to cost. Where an edge is a link between two detections (node). It converts the probability weight into a cost = -log weight.

Parameters:

nodes (list[dict[str, Any]]) – Nodes of the graph (detections), with their features Format: { “id”: int, “coords”: (float, float), “time”: int, “label”: int}
weights (Iterable[tuple[tuple[int, int], float]]) – Considered edges, with their weight Format: [((node_id, node_id_2), weight), …]

Returns:

Cost dictionary of the feasible tracking graph. For every (frame1, node1, frame2) there is a dictionary with key node2 and the value the cost of the edge.

Return type:

dict[tuple[int, int, int], dict[int, float]]

class byotrack.implementation.linker.frame_by_frame.trackonstra.TrackastraFlex(transformer: TrackingTransformer, train_args: dict[str, Any], delta_t: int = 4, intra_weight: float = 0, device=None)

Bases: Trackastra

Trackastra with ability to modify delta_t.

predict_with_gap(imgs: ndarray, masks: ndarray, edge_threshold: float = 0.05, n_workers: int = 0) → dict: Same function as the original _predict but it calls predict_windows with the model delta_t.

class byotrack.implementation.linker.frame_by_frame.trackonstra.TrackOnStraParameters(*, association_threshold: float = 0.05, positional_cutoff: float = 256.0, n_valid=2, n_gap=3, association_method: str | AssociationMethod = AssociationMethod.SPARSE_OPT_SMOOTH, anisotropy: tuple[float, float, float] = (1.0, 1.0, 1.0), split_factor: float = 0.0, merge_factor: float = 0.0)

Bases: FrameByFrameLinkerParameters

Parameters of TrackOnStraLinker.

Note

Most parameters can be estimated automatically from the detections using estimate.

association_threshold

Minimum probability to consider a link. We advise to keep the default value. Default: 0.05 (default value from Trackstra)

Type:: float

positional_cutoff

Remove links based on an Euclidean thresholding. We use the default value provided by Trackastra. Tuning it may improve performance. Default: 256.0

Type:: float

n_valid

Number of detections required to validate the track after its creation. If a track is missed during its first n_valid frames, it is dropped. This provides robustness to false positive detections. With no false positives, it can be set to 1 (a detection always belongs to a track). Highers values allow to remove non time-consistent false positives, but may prune real tracks that have been miss-detected. Default: 2

Type:: int

n_gap

Number of consecutive frames without any association (miss-detected) before the track termination. This provides robustness to false negative detections. Without any false negatives, it can be set to 0. Higher values allow to support larger gaps in the track, but may lead to wrong assignments. Default: 3

Type:: int

association_method

The frame-by-frame association to use. See AssociationMethod. It can be provided as a string. (Choice: GREEDY, OPT_HARD, OPT_SMOOTH, SPARSE_OPT_HARD, SPARSE_OPT_SMOOTH) Default: SPARSE_OPT_SMOOTH

Type:: AssociationMethod

anisotropy

Anisotropy of images (Ratio of the pixel sizes for each axis, depth first). This will be used to scale distances. Default: (1., 1., 1.)

Type:: tuple[float, float, float]

split_factor

Allow splitting of tracks, using a second association step. The association threshold in this case is split_factor * association_threshold. Default: 0.0 (No splits)

Type:: float

merge_factor

Allow merging of tracks, using a second association step. The association threshold in this case is merge_factor * association_threshold. Default: 0.0 (No merges)

Type:: float

check() → None: Check the specification for invalid values.

estimate(detections_sequence) → TrackOnStraParameters

Estimate parameters from the given detections.

Estimation is triggered by providing negative dummy values for positive parameters. The dummy values are then replaced by their estimate.

Estimators:

positional_cutoff: max(3 * statistics.average_radius, statistics.average_min_dist)
anisotropy: Computed from statistics.anisotropy.
split_factor: 1.0 if the number of detection increase by more than 30% over the full sequence.
merge_factor: 1.0 if the number of detection decrease by more than 30% over the full sequence.

Parameters:: detections_sequence (Sequence[byotrack.Detections]) – Detections for the current sequence.
Returns:: self with updated parameters.
Return type:: TrackOnStraParameters

class byotrack.implementation.linker.frame_by_frame.trackonstra.TrackOnStraLinker(specs: TrackOnStraParameters, model: TrackastraFlex | None = None, optflow: byotrack.OpticalFlow | None = None, features_extractor: byotrack.FeaturesExtractor | None = None)

Bases: FrameByFrameLinker

Online TrackAstra.

It uses a trained TrackAstra model to predict linking costs. But it replaces the graph optimization from TrackAstra by our online FrameByFrame linking. This allows a simple support for false negative detections that TrackAstra do not support by itself.

Warning: This implementation is not yet Online. Indeed the linker has to be setup with: the full video and detections_sequence before being usable.

Note

This implementation requires trackastra. (pip install trackastra) trackastra is only available for python >= 3.10

See FrameByFrameLinker for the other attributes.

specs

Parameters specifications of the algorithm. See TrackOnStraParameters.

Type:: TrackOnStraParameters

model

Model of Trackastra used to compute the association costs.

Type:: TrackastraFlex

cost_dict

Cost dictionary of the feasible tracking graph.

Type:: dict[tuple[int, int, int], dict[int, float]

setup(video: Sequence[np.ndarray] | np.ndarray, detections_sequence: Sequence[byotrack.Detections]) → None

Offline setup of the linker by computing all the linking costs with Trackastra.

Linking costs are stored into cost_dict.

This function needs to be called on each video for the linking to be able to run.

Parameters:

video (Sequence[np.ndarray] | np.ndarray) – Sequence of T frames (array). Each frame is expected to have a shape ([D, ]H, W, C)
detections_sequence (Sequence[byotrack.Detections]) – Detections for each frame Detections is expected for each frame of the video, in the same order. (Note that for a given frame, the Detections can be empty)

motion_model() → None

Optional modelisation of motion for tracks.

It can be used to update some internal state of the tracker after the optical flow computation and before the distance computation.

post_association(frame: np.ndarray | None, detections: byotrack.Detections, active_mask: torch.Tensor) → None

Update the internal state of the tracker after update_active_tracks.

It should update any internal model/data. It is also responsible to register the position of each active track in all_positions for the current time frame.

Parameters:

frame (np.ndarray | None) – The optional current frame of the video Shape: (H, W, C), dtype: float
detections (byotrack.Detections) – Detections for the given frame
active_mask (torch.Tensor) – Boolean tensor indicating True for still active tracks Shape: (N_tracks), dtype: bool

cost(frame: np.ndarray | None, detections: byotrack.Detections) → tuple[torch.Tensor, float]

Compute the association cost between active tracks and detections.

It also returns the threshold to use (Depending on the dist you use, association_threshold could be related to a more meaning full quantity than the cost itself). For instance, when using a squared Euclidean distance, the association threshold could be express as the distance in pixel, and this function could square it. For likelihood association, you could provide the association threshold as a probability and use -log(threshold) as the true threshold. (See KalmanLinker and NearestNeighborLinker)

Parameters:

frame (np.ndarray | None) – The current optional frame of the video Shape: (H, W, C), dtype: float
detections (byotrack.Detections) – Detections for the given frame

Returns:

The cost matrix between active tracks and detections: Shape: (n_tracks, n_dets), dtype: float
float: The association threshold to use.: It can be different than self.association_threshold depending on the dist build here

Return type:

torch.Tensor