MixSORT: Adaptive Hybrid Real-time Tracking System

1. Tracking-by-Detection Overview

Currently, MOT largely follows a detection-based procedure that takes detection results as input to update tracks. In each frame, when the detector outputs bounding boxes and confidence scores, the tracker performs data association between consecutive frames, followed by state estimation and track management. The advantage of this approach lies in its modularity. It offers benefits in terms of reproducibility and scalability because it enables the independent improvement of individual modules such as detection, association, state estimation, and track management [10,11].

The characteristics of the detection stage, such as the rates of false positives and false negatives, and the distribution of low-confidence detections, are reflected in the association design. Specifically, these become core factors that define the cost function design, gating intensity, and track management scheduling strategies. Detectors have evolved into methods such as anchor-based Faster R-CNN [20], single-stage YOLO detectors [21], and transformer-based DETR and Deformable-DETR [26,27]. Post-processing techniques like non-maximum suppression (NMS), soft-NMS, and Optimal Transport Assignment (OTA), as well as Distance Intersection over Union (DioU) and Complete Intersection over Union (CioU) family loss functions and feature parameter design, significantly impact the final precision and recall. Particularly, in adverse conditions such as fog, night, or backlighting, low-confidence detections increase, leading to a rapid expansion of the association search space and a higher risk of mismatches. Therefore, techniques such as dehazing, image restoration, domain adaptation, and test time adaptation (TTA) are used concurrently. These pre-processing and adaptation methods enhance the possibility of recovering true positives in low-confidence regions and provide a foundation for subsequent score-aware cascade association to function effectively [13].

2. Association and Online Trackers

The success of online MOT systems is contingent upon the design of data association strategies. Given real-time constraints, most methodologies perform global one-to-one assignments by integrating IoU based proximity via Kalman filters with detection confidence derived from appearance descriptors for target identity. SORT [10] achieved simple and fast matching through a Hungarian algorithm assignment method based on the minimum cost of IoU predictions via Kalman filters; however, it reported limitations, namely an increase in identity switches during close proximity interactions or long-term occlusions. As a complementary approach, DeepSORT [11] enhanced identity stability by incorporating an appearance descriptor and adjusting matching priorities to improve the accuracy of re-identification for target identities. Subsequent research has proceeded in a direction that compromises and expands upon the advantages of these two methods. BoT-SORT [12] restored the reliability of IoU costs by alleviating inter-frame geometric variations from moving cameras through global motion compensation (GMC). ByteTrack [13] structurally absorbed true positives in low-confidence regions while suppressing mis-associations using a two-stage score-aware strategy of high-confidence first matching followed by low-confidence matching. StrongSORT [14] reinforced the temporal consistency of appearance costs via a robust re-identification backbone, gallery quality gating, and budget management. OC-SORT [15] compensated for the jitter and non-linear motion vulnerabilities of pure IoU-based estimation by utilizing observation-centric updates, as well as velocity and direction information. These trackers commonly reduce error propagation and computational load by pre-masking impossible combinations through gating, and maintain a balance of accuracy, robustness, and cost in online environments by combining motion, appearance, and detection score information in a stepwise or weighted manner.

In scenes with small bounding boxes at long range, the IoU between consecutive frames tends toward zero, and many online trackers adopt nearest-neighbor association based on the Euclidean distance between bounding-box centers as a geometric fallback [28]. This center-distance heuristic relies exclusively on two-dimensional coordinates and has several limitations. A one-pixel shift in the image plane can correspond to a large change in physical space, so camera motion, perspective distortion, and localization noise strongly affect the measured distance. In crowded scenes where many objects move along similar trajectories, multiple targets produce similar center distances and identity cues disappear. Appearance information accumulated along each track over time also remains unused at the matching stage.

The distance correlation (dCor) based association cost used in this study addresses these weaknesses of geometry-only matching. This method compares the high-dimensional embedding distribution of a candidate detection through pairwise distance statistics and measures nonlinear dependence that remains stable under monotone rescaling [23]. Re-identification embeddings capture clothing color, silhouette, and local texture, and dCor uses these cues to preserve reliable identities for small and distant objects. In such regimes, geometric proximity causes center-distance matching to merge neighboring trajectories, whereas dCor-based appearance association provides additional discriminative signals. This analysis motivates a design that combines IoU-based geometric information with dCor-based appearance cost, and Section IV describes the resulting structure and algorithms in detail.

3. Real-Time Optimization and Appearance Representation

The tracking quality of a real-time system is largely determined by the track lifecycle policy and inference light-weighting. Following track initialization, subsequent stages are typically classified into a ‘tentative’ state (where the matching status is not yet confirmed), a ‘confirmed’ state (due to a certain period of consistent matching), and a ‘missed’ state (where the target identity has not been found for a certain duration). Hyperparameters such as appearance distance thresholds, along with confidence thresholds, as well as measurements, such as IoU determine the balance of tracking quality due to trade-offs concerning identification stability, processing latency, and the rates of false positives and false negatives for the target identity.

Re-identification typically involves a procedure of comparing distances using L2-normalized embeddings derived from CNNs or Vision Transformers (ViTs). In the training phase, multi-loss configurations that combine triplet loss, contrastive loss, and softmax classification loss, along with hard sample mining, label refinement, and domain adaptation, are known to contribute to generalization performance.

Furthermore, during the inference phase, system optimizations are jointly applied to reduce computational load. These optimizations include extracting embeddings primarily for candidate pairs that pass through a gate, prototype aggregation based on recent or representative values, preventing memory inflow for costs below a certain threshold, and employing techniques such as batching, catching, mixed precision, quantization, and kernel fusion. The combination of this score-aware cascade and gating-based masking presents a practical solution that simultaneously pursues the robust identification characteristic of the DeepSORT family, and the low-confidence recall advantages of the ByteTrack family.

1. State Estimation

We continuously estimate the object state by applying a Kalman filter model based on the latest position and size information of the tracked bounding boxes [29,30]. While aspect ratio or area could be used as state variables, as in previous studies [10,11], this approach has limitations in directly predicting and estimating the fine-grained changes of the actual bounding box coordinates, even if advantageous for estimating macroscopic changes such as the shape or size of the bounding box. Therefore, in this study, we represent the motion state using a linear constant velocity model, utilizing the bounding box's center coordinates (𝑥, 𝑦), width 𝑤, and height ℎ. The state vector is as follows.

The initial state is set by the first observation, with velocity components initialized to zero. The state error covariance, P and the process and observation noises, Q and R are constructed as diagonal components using axis-specific standard deviation coefficients proportional to the size of the bounding box and are recalculated every frame based on the current width and height based on constant velocity assumption. When an observation is received, the standard Kalman filter update procedure is followed; if no observation is available, only the prediction step is performed. For numerical stability, a minimum lower bound is applied to the width and height. If a bounding box is expected to collapse in the next frame, the corresponding velocity components are clipped to zero. During initialization, the variances of the size and velocity axes are set relatively large to prevent oversharing (overconfidence), and whenever an observation is received, the observation noise covariance is recalculated based on that observation's width and height.

2. Appearance Descriptor

Appearance feature extraction follows the deep appearance embedding procedure proposed in DeepSORT, where a CNN is pre-trained offline using the large-scale dataset MARS (approximately 1.1 million images, 11,261 identities) for pedestrian re-identification [31]. The network follows a Wide Residual Network [32] backbone consisting of two initial convolutional layers and six residual blocks, yielding a 128-dimensional feature embedding in the upper layers.

Batch normalization and L2 normalization are sequentially applied to the resulting embeddings to project them to unit norm, thereby securing appearance representations that are less sensitive to scale variations [33]. In the online stage, detection boxes are normalized to a fixed resolution and then processed through the same preprocessing steps for inference. This process generates a normalized 128-dimensional appearance feature vector every frame.

3. Distance Correlation

In this study, we adapt a distance correlation as an appearance similarity metric. Distance correlation, R(X,Y) is a probabilistic measure representing either linear or a non-linear association between two vectors, which can be calculated as (2). 𝑅²(𝑋,𝑌) becomes zero only when the two random vectors are independent, and its value ranges from [0, 1], representing the strength of dependency. The definition of distance correlation is as follows.

Here, 𝜐²(𝑋, 𝑌) is the distance covariance, which is defined as follows using the joint characteristic function, 𝑓_𝑋𝑌(𝑡, 𝑠) and the marginal characteristic functions 𝑓_𝑋(𝑡) and 𝑓_𝑌(𝑠).

In the special case of a bivariate normal distribution, the relationship with the traditional correlation coefficient 𝜌 is such that 𝑅 = 1 when |𝜌| = 1, and the following relationship is satisfied.

Therefore, distance correlation can be considered a generalized concept of the Pearson correlation coefficient, capturing not only linear relationships but also nonlinear dependencies [23].

The actual calculation uses the sample distance correlation of the double-centered Euclidean distance matrices. We compute the pairwise Euclidean distance matrices for the set of detection embeddings and the set of track gallery embeddings, obtain the sample distance covariance and sample distance variance through double-centering (removing row and column means), and then acquire the value 𝑅∈[0,1] via the normalization in (2). The sample distance correlation operates stably even with high-dimensional embeddings, and the measured latency can be sufficiently suppressed by combining efficient computation techniques such as mini-batch calculation, partial sampling, cumulative updating, and approximation methods [25].

In this study, we define the appearance-based cost as 1-R, utilizing the characteristic that dependency is stronger when R is larger. We also implement an adaptive association strategy that actively selects and combines appearance-based metrics and IoU-based metrics depending on the situation of the tracklet.

4. Gating Parameters

To pre-emptively eliminate candidates that are physically unlikely to be valid between the object's predicted state and the newly received measurement, we use scale-adaptive gating based on the elliptical Euclidean principle. Gating is the procedure of retaining only those candidates for computation whose distance measurement between the track and the detection candidate pair is below a certain threshold. An elliptical gate is used because circular or fixed-scale Euclidean gates can cause proportional distortion when object size changes or aspect ratios differ. Specifically, an elliptical gate is positioned relative to each track's predicted box center (𝑥_𝑝, 𝑦_𝑝) , with its minor and major axes proportional to the predicted width (𝑤), and height (ℎ). The normalized squared distance to the detection center (𝑥_𝑑, 𝑦_𝑑) is defined as follows:

Here, 𝛼 and 𝛽 are scale adjustment coefficient for width and height, respectively where 𝛼, 𝛽 > 0 . In Eq (5), a detection is maintained as an association candidate only if 𝑑² ≤ 𝑑_th. This implements anisotropic (elliptical) gating that is stricter for smaller targets and more lenient for larger targets, even with the same pixel error. The size of the ellipse is updated every frame with the predicted 𝑤 and ℎ to immediately reflect size changes due to perspective changes, zoom, and acceleration. In implementation, track-detection pairs where 𝑑² > 𝑑_th are masked in the cost matrix using a large constant value, where 𝑑_th is adjusted by the validation procedure. For those entries which pass this gate, the appearance costs are computed and then passed to the Hungarian optimization algorithm. This scale-adaptive elliptical gating effectively reduces false associations and lowers the noise in the association cost matrix in scenes where objects of different sizes and shapes coexist or where object size changes significantly due to perspective effects, thereby reducing the computational burden of subsequent optimization and appearance cost calculations. While 𝛼 and 𝛽 basically use the same value, they can be set differently for each axis depending on scene characteristics, such as vertical and horizontal uncertainty deviations.

5. Track Handling

To manage track confidence, MixSORT uses three finite states: tentative, confirmed, and deleted. When a frame is input to the system, a Kalman filter prediction is first performed for all active tracks, and a global matching process performs one-to-one assignment with the detection results. Unmatched detections are immediately initialized as tentative tracks based on the observation. At this point, the mean of the initial state is set to the first observation, and the velocity components are set to zero. Each track is managed with two counters: hits, which represents the number of successful matches so far, and time_since_update, which indicates the number of frames since the last successful association. Tentative tracks must pass a short confirmation period; if they meet min_hits (e.g., three consecutive successful matches), they are promoted to the confirmed state. If a tentative track fails to match even once during the initial confirmation period, it is immediately deleted.

For successfully matched tracks, the Kalman filter is updated, the number of hits is incremented, and time_since_update is reset to 0. The latest appearance embedding is added to the track gallery. Conversely, tracks that fail to match are only predicted without being updated, and time_since_update is incremented by 1. Regardless of the track's state, if time_since_update exceeds a predefined maximum allowed un-updated frames, max_age, the track is considered to have left the scene and is terminated. New tracks are not regenerated from detections that have already been associated in the same frame, and track identifiers are not reused throughout the sequence. This simple state machine and counter-based policy are advantageous for their easy implementation in real-world online environments and clear tuning points.

6. Assignment and Matching

Data association is separated into high-confidence (high) and low-confidence (low) based on detection confidence, and matching is performed sequentially through a two-stage global optimization. Both stages construct a cost matrix and find the optimal assignment that minimizes the total cost using the Hungarian algorithm [34].

In the first stage, the cost is defined based on the IoU between the predicted tracks and only the high-confidence detections.

Additionally, global matching is performed only for pairs that pass the IoU threshold mask.

Matched pairs are immediately confirmed with a state update, and the IoU-unmatched tracks, detections, and low-confidence detections are carried over to the second stage. This process aligns with a score-aware strategy that conservatively absorbs high-confidence detections first and then handles remaining and low-confidence detections in a subsequent stage [13].

The second stage handles the remaining unmatched tracks and detections from the first stage, along with low-confidence detections. This stage focuses on improving the precision of appearance information. First, elliptical Euclidean gating is applied to pre-emptively exclude spatially inappropriate combinations. For candidates that pass the gate, an appearance-based matching is performed using the distance correlation similarity between the track gallery embeddings and the detection embeddings as the cost. The distance correlation R between the track embedding set 𝐹_𝑖 and the detection embedding 𝑓_𝑗 is as follows.

Here, the appearance-based cost is defined as 1 - 𝑅.

Additionally, by setting a quality threshold, a cost matrix is constructed using only the pairs that satisfy it, and global optimization is performed using the Hungarian algorithm.

By actively combining detection confidence, IoU, and appearance information and separating the process into two stages, this configuration allows MixSORT to minimize identity switches and secure re-detection recovery even in crowded scenes.

7. Architecture

At its core, MixSORT employs a score-aware, two-stage matching process for every frame. When a new frame is ingested, the active tracks from the previous frame are predicted by a Kalman filter and propagated to the next state. The detection results are branched into high-confidence and low-confidence groups according to their confidence scores. Fig. 2 illustrates the entire pipeline.

Fig. 2.

Overview of the proposed MixSORT pipeline. A confidence-aware two-stage association first matches high-confidence detections using IoU and then performs appearance matching with scale-adaptive gating, deep appearance descriptors, and distance correlation while managing tracks through tentative, confirmed, and deleted states.

Data association is divided into two sequential stages in MixSORT. Stage 1 performs IoU-based matching between high-confidence detections and predicted tracks to minimize initial computational latency. Stage 2 performs precise matching for the remaining tracks and detections, including low-confidence detections, by first using spatial gating to eliminate implausible candidates, and then applying distance correlation-based appearance costs. This hierarchical structure ensures both real-time processing and ID consistency in scenes with crowding, occlusion, and viewpoint changes, by quickly assigning high-confidence items and precisely correcting the remaining items with appearance information. Track management uses tentative, confirmed, and deleted states, and the latest embedding is added to the gallery upon update to increase the reliability of subsequent matching.

1. Dataset

The experiments in this study were conducted using the MOT17 dataset, a standard benchmark in the field of multi-object tracking. MOT17 consists of 14 video sequences filmed in various urban environments such as intersections, roads, and commercial districts, and it includes both static and moving camera scenarios. The dataset contains a wide range of challenges that make tracking difficult in real-world situations, such as changes in lighting and background, partial and full occlusion, differences in crowd density, and distant pedestrians. Each frame provides official annotations with bounding boxes and consistent object IDs, allowing for a fair performance comparison between algorithms [22,35].

Instead of directly using the official test split for final submission, a fixed validation sequence was separated and utilized from within the training split to stably conduct iterative ablation and hyperparameter search. Specifically, MOT17-02, MOT17-04, MOT17-05, MOT17-09, MOT17-10, and MOT17-13 were used as training sequences, while MOT17-11 was reserved as a separate validation sequence. This configuration allows for a consistent comparison of various settings under the same validation conditions without external server submission, and it naturally extends to the procedure of submitting the same model to the official test split if needed.

All experiments were conducted on the single pedestrian class, and the original resolution, frame rate, frame order, and encoding of each sequence were preserved. Only the official annotations provided by MOT17 were used for training and validation. An internal benchmark procedure was followed, where parameters were estimated using the training sequences, and then performance was compared and a model was selected using the separated MOT17-11 sequence. Unless otherwise noted, the results presented in the main text were derived using MOT17-11 as the validation set.

2. Evaluation Metrics

To objectively compare and analyze the performance of the MOT system, this study utilized comprehensive evaluation metrics based on the CLEAR MOT framework [36]. The common evaluation metrics adopted were MOTA, MOTP, IDF1, IDSW, FP, FN, Frag, MT, and ML. These metrics correspond to the framework, allowing for the quantitative evaluation of the design goals of the MixSORT methodology: improving accuracy, stabilizing IDs, suppressing unnecessary detections and missed detections, and reducing track fragmentation. In the main text, we describe the core metrics among these—MOTA, MOTP, IDF1, IDSW, MT, and ML—while FP, FN, and Frag are also presented in the table to provide a complementary understanding of the system's error scale and track fragmentation patterns.

1. Selection of Object Detectors Based on Performance Comparison

The final performance of multi-object tracking is directly dependent on the quality of the object detector, which is the initial input stage. In detection-based tracking, recall is crucial for creating new tracks and maintaining existing ones, while precision is a key indicator for suppressing false associations. In this study, to ensure system robustness, we conducted a comparison and analysis of DPM, Faster R-CNN (FRCNN), and YOLOv8 to select the final detector. A total of 9,436 GT boxes were used in the experiment, and data consistency was verified as each detector satisfied TP+FN=9,436.

The quantitative results are summarized in Table 1. DPM performed the worst with 4,659 TPs and a recall of 0.4937. This means a very high FN of 4,777, likely causing track interruptions and frequent ID re-initializations during actual system application. FRCNN most effectively suppressed FPs with a precision of 0.9696, reducing the risk of incorrect associations during the linking stage. YOLOv8 had slightly higher FPs than FRCNN, making it less favorable in terms of precision, but its recall of 0.6141 resulted in the fewest missed objects among the three models. Consequently, YOLOv8 achieved an F1-score of 0.7398, showing a slight edge in the overall metric. In summary, we confirmed that YOLOv8 and FRCNN have contrasting characteristics: YOLOv8 focuses on recall (not missing detections), and FRCNN focuses on precision (not making false detections).

Table 1.

Comparison of object detector performance on MOT17-11.

The same trend is observed in Fig. 3. YOLOv8 captures more pedestrians in long-distance scenarios and partial occlusion situations, reducing omissions, while FRCNN uses a conservative suppression strategy to aggressively remove low-confidence candidates, thus lowering FPs, but some TPs are missed. DPM frequently experiences object omissions in adverse conditions such as dense crowds, reflections, and motion blur.

Fig. 3.

Comparison of pedestrian detection results of GT, DPM, FRCNN, and YOLOv8 in the same frame.

Since this study aims to ensure track continuity (minimizing interruptions and re-initializations), YOLOv8, which reduces omissions with high recall, is the most suitable. YOLOv8 directly contributes to trajectory maintenance stability and showed superior performance in F1-score, confirming its advantage in overall balance.

2. Tracking Performance Comparison

In this study, we performed a comparative analysis of real-time tracking algorithms, including SORT [8], DeepSORT [9], and the proposed MixSORT, using standard evaluation metrics on the MOT-11 validation sequence. According to Table 2, the MOTA, which represents overall accuracy, is identical for DeepSORT and MixSORT at 60.1 each, which is 1.8 percentage points higher than SORT's 58.3. MixSORT had the highest MOTP (position precision metric) at 83.7, suggesting relatively superior bounding box location approximation in the matched segments. In terms of ID consistency, MixSORT showed the highest IDF1 at 59.2 and the lowest IDSW at 32, stably maintaining the ID of the same object. Long-term tracking metrics also favor MixSORT, with MT ratio of 30.7% and ML ratio of 44.0%, demonstrating excellent track maintenance and a low tendency for track loss. Regarding error composition, FN recorded the minimum value at 3506, but FP 224 and Frag 112 showed relatively high values. These results are interpreted as aggressive track activation and re-activation strategies for reducing omissions but partially accompanying increased over-detection and fragmentation. Nevertheless, the simultaneous improvement in IDF1, reduction in IDSW, and enhancement of MOTP and MT while maintaining the same MOTA level clearly demonstrate that the proposed association method and track management work towards prioritizing and strengthening ID stability and position precision.

Table 2.

Comparison of online trackers on the MOT17-11 validation sequence (↑: higher is better, ↓: lower is better).

Fig. 4 presents an enlarged view of a scene in which a target reappears after occlusion, enabling a detailed comparison of the results from each tracker. The figure visualizes the tracking outputs, where different box colors denote distinct tracking IDs, and red boxes indicate locations of ID switches. Unlike SORT and DeepSORT, which assigned new IDs to the same individual upon reappearance, the proposed MixSORT successfully maintained and restored the original ID. This observation demonstrates MixSORT’s ability to ensure trajectory continuity across occlusion and reappearance, based on a two-stage association procedure, further supporting the previously reported improvements in IDF1 and reductions in IDSW.

Fig. 4.

ID switches in an occlusion–reappearance case. (a) SORT. (b) DeepSORT. (c) MixSORT. The proposed MixSORT preserves the identity of the same pedestrian, reducing ID switches that occur in SORT and DeepSORT.

On the other hand, the introduction of a distance-correlation based appearance association increases computational cost; when the number of candidate pairs is k, distance correlation requires pairwise computation of L2 distance at first, which results in O(k²) proportionally to the number of candidate pairs, potentially causing latency in applications with stringent real-time constraints. This study primarily focused on performance comparison; however, future work will explore strategies to alleviate such challenges -such as candidate reduction, approximate distance correlation computation, and lightweight batch processing- as well as post-processing techniques aimed at balancing different metrics, including conservative reactivation, fragmented track merging, and selective handling of low-confidence detections. Furthermore, since the present evaluation was conducted on a single validation sequence, it is necessary to extend the experiments to additional sequences encompassing diverse scene compositions, crowd densities, and occlusion patterns to assess the generalizability of the proposed approach.