If you’ve ever watched a self-driving car demo and thought, “Cool… but what happens when a pedestrian walks behind a truck for a few seconds?”, you’re thinking about one of the hardest problems in autonomy: multiple object tracking (MOT) through occlusion.
A new Nature Communications paper proposes a surprisingly elegant upgrade: put an “embeddable Ising machine” inside a vehicle-mountable tracking system to solve the gnarly matching problem that happens every video frame. In other words, the system doesn’t just detect objects; it keeps their identities consistent even when they cross, overlap, or disappear behind another object for multiple frames.
The core problem: “Which detection belongs to which track?”
Modern MOT often uses a “tracking-by-detection” pipeline. The camera (or sensor) detects objects in each frame, then the tracker decides which new detections match previously tracked objects (“tracks”).
When nothing overlaps, matching is pretty straightforward. But during occlusion (two objects cross, one hides behind the other), the system can easily swap identities or drop the hidden object entirely. That’s not just an academic issue—identity swaps can ripple into prediction, planning, and safety decisions.
Why today’s common solver hits a wall
Many widely used trackers assume one-to-one matching between tracks and detections (one track gets one detection). That makes the math a linear assignment problem, solvable efficiently with the Hungarian algorithm.
However, occlusion often looks more like many-to-one (multiple tracks plausibly map to one visible detection because one object is hidden). The moment you allow that flexibility, the assignment becomes a harder class of problem—NP-hard combinatorial optimization.
The big idea: Flexible assignment solved by an embeddable Ising machine
The authors introduce a flexible assignment function that can label a track as:
- matched
- unmatched
- potentially-match (the key new state that means “this object is probably still there, just hidden”)
How the system decides “potentially-match”
Instead of solving matching once, the system solves it twice per video frame using a quadratic optimization formulation (QUBO):
- First run: a “strict” version that heavily enforces one-to-one matching (large penalty weight λ).
- Second run: a “looser” version that tolerates many-to-one matches (small λ).
Then it compares the two results:
- If a track is “unmatched” in the strict solution,
- but it does match something in the loose solution,
- the system flags it as potentially-match, which effectively marks an occlusion and even localizes where it happened.
This is the clever part: the tracker learns when it should keep believing in an object even without a clean detection.
What is an “Ising machine” here, really?
An Ising machine is a special-purpose optimizer designed to find low-energy states of an Ising model (equivalent to QUBO), which is powerful because many hard combinatorial problems can be mapped into that format.
In this work, the optimizer is quantum-inspired, based on simulated bifurcation (SB)—not a cryogenic quantum computer. SB simulates the dynamics of coupled nonlinear oscillators, where each oscillator’s “branch” corresponds to a binary choice.
That matters because it’s embeddable: it can run on vehicle-friendly hardware like FPGAs.
The hardware win: real-time speed on vehicle-mountable boards
The prototype uses two vehicle-mountable SoC-FPGA boards: one for object detection and one for the Ising-machine assignment.
Key numbers from the paper:
- The embedded Ising machine supports 512 spins, enabling up to 22 tracks × 22 detections = 484 variables.
- It solves the QUBO twice per frame in a deterministic 568 µs total.
- System-wide throughput exceeds 20 FPS (example: ~23 FPS on a benchmark sequence).
- The Ising machine operating power is reported as 3.4 W.
So the “hard math” becomes a small slice of the frame budget.
Does it actually track better?
The paper evaluates tracking using HOTA (Higher Order Tracking Accuracy) and reports that improvements are driven mainly by AssA (association accuracy)—exactly what you’d expect when the matching logic gets smarter.
They also created custom benchmark sequences designed to stress long and complex occlusions, including multi-object crossings.
Why this is cool (even if you’re not a math person)
Because it’s a rare combo of:
- a real autonomy pain point (occlusion identity loss),
- a genuinely different computation approach (NP-hard optimization per frame),
- and a plausible embedded path (FPGAs + quantum-inspired solver).
It’s not “quantum hype.” It’s specialized optimization hardware showing up as a practical tool inside the autonomy stack.
When will people actually see this?
Short-term, you’ll most likely see this approach first in:
- advanced driver assistance systems (ADAS) stacks where identity stability matters,
- robotics in cluttered environments (warehouses, delivery bots),
- and research prototypes that need robust tracking without sending compute to the cloud.
Longer-term, if embeddable Ising accelerators scale (more variables, tighter integration onto one board), this kind of NP-hard “decision helper” could expand into other vehicle tasks like scheduling and path planning.
Check out the cool NewsWade YouTube video about this article!
Article derived from:Tatsumura, K., Hamakawa, Y., Yamasaki, M., Oya, K., & Fujimoto, H. (2025). Enhancing vehicle-mountable multiple object tracking systems with embeddable Ising machines. Nature Communications. https://doi.org/10.1038/s41467-025-67282-7
