From-scratch C++ correlation-filter tracker for object detection, tracking and redetection (without OpenCV) for Raspberry Pi 5 targeting 100+ FPS - looking for advice from people who've pushed similar systems further.

Hey everyone,

I'm currently building a **real-time object tracker from scratch in C++17** for the **Raspberry Pi 5**, with the goal of achieving **100+ FPS** on CPU-only hardware. The project is based on **correlation filters** and **FFT-based signal processing**, with **no machine learning, no neural networks, and no OpenCV** in the core tracking pipeline.

The motivation is simple: a straightforward OpenCV-based implementation on the Pi only gets me around **15 FPS**, which seems far below what this class of algorithms should be capable of. From both the literature and projects I've come across, the gap appears to be in implementation and system overhead rather than the underlying tracking method itself.

# Current approach

Right now, my plan is to build the pipeline around:

* A custom image loading and preprocessing path to avoid unnecessary OpenCV decode/resize overhead.
* **FFT-based correlation in the frequency domain** for fast target localization.
* Adaptive online filter updates so the tracker learns appearance changes over time.
* **PSR (Peak-to-Sidelobe Ratio)** based confidence estimation for occlusion and tracking failure detection.
* A modular architecture that can later be extended with features like scale estimation and automatic re-acquisition.

The area I'm currently spending the most time researching is the FFT layer. I'm trying to determine whether the best approach on the Pi 5 is:

* a hand-written radix-2 FFT,
* aggressive **NEON/SIMD** optimization,
* or using an existing library such as **FFTW** or **kissFFT**.

# Other approaches I've been studying

To better understand the design space, I've also been looking into modern transformer-based visual trackers. They jointly process information from a target template and a search region, making them much more semantically aware and capable of handling challenging scenarios such as partial occlusions or target disappearance with automatic re-acquisition. The downside is that they are significantly heavier computationally and can be difficult to deploy efficiently on constrained edge hardware.

On the more classical side, I'm currently reading about **Discriminative Scale Space Tracking (DSST)**. One of the main limitations of basic correlation-filter trackers is that they often assume the target size remains constant. DSST addresses this by learning a separate correlation filter across multiple image scales, allowing the tracker to estimate changes in object size efficiently while still maintaining real-time performance. It seems like an elegant way to improve robustness without giving up the speed advantages that make correlation filters attractive in the first place.

Exploring these different approaches has been interesting because they represent very different trade-offs: transformer-based methods emphasize robustness and semantic understanding, while correlation-filter methods prioritize simplicity, efficiency, and extremely high throughput.

# Looking for advice from people who've built similar systems

If you've worked on **correlation-filter trackers**, **embedded computer vision**, **real-time image processing**, **high-performance C++**, **drone tracking**, or **ARM optimization**, I'd really appreciate your perspective.

Some questions I'm hoping to get insight on:

* Where did the biggest performance bottleneck actually end up being? The FFT itself, memory layout, cache locality, camera capture, frame copies, synchronization, or something else entirely?
* On Raspberry Pi 5 specifically, is hand-vectorizing FFTs and pointwise complex operations with **NEON** worth the effort, or do mature FFT libraries generally outperform custom implementations?
* If you've implemented trackers such as **MOSSE**, **ASEF**, **UMACE**, **DSST**, or related adaptive correlation-filter methods, what optimizations made the biggest practical