Designing a plugin "collector" interface for a telemetry module — sanity check on C-ABI vs C++, self-describing schemas, and not over-engineering
I'm designing the interface for a telemetry/observability module in a C++ machine-vision inspection platform, and I'd love a sanity check from anyone who's built plugin/collector systems. I've made a bunch of decisions and mostly want to hear *where I'm wrong* or *what I'm not considering*.
**Context.** The platform is plugin-based — a host app dynamically loads modules and device adapters as DLLs/.so. The telemetry module collects HW/OS metrics and correlates them with the app's own activity (which stage is running, how many run concurrently, etc.) into a forensic log, under a hard constraint: **\~zero impact on the hot inspection path**. I want collection to be extensible — add a new metric source *without touching the core interface*.
**What I have so far — a "collector" abstraction with two tiers:**
1. **Internal collectors (in-process, C++ abstract interface).** These subscribe to the host's typed event bus. The subscribe API is a compile-time template keyed on the C++ event type, so these *can't* sit behind a C ABI — they're a C++ `ICollector` compiled into the module.
2. **External collectors (pure C-ABI plugin DLLs).** These only poll OS/HW/kernel (no host types). They implement a small C ABI (`init/finalize/open/close/describe/collect`), are `dlopen`'d at runtime, and run on a dedicated thread.
Boundary rule: *needs the event bus / a host type → internal C++; OS/HW polling only → external C-ABI.*
**Specific decisions I'm unsure about:**
* **Pull-based C-ABI:** host calls `collect(handle, row_sink_cb, userdata)` on a dedicated thread; the collector pushes result rows into the sink callback (scalar = 1 row, top-N = N rows).
* **Self-describing schema:** `describe(handle)` returns a JSON string declaring each field's `{key, type, unit}` \+ gating metadata (`minLevel`, `osMask`). The host's writer/viewer need no code changes for a new metric.
* **"Day-1" safety contracts** baked into the header: (1) every C function wraps its body in try/catch so no C++ exception crosses the `extern "C"` boundary; (2) the `describe()` string is owned by the collector, valid only until the next `describe`/`close`; (3) `close()` must quiesce the collector's threads before returning.
**Questions I'd love opinions on:**
1. **Two parallel interfaces (C++ internal + C-ABI external) — smell or justified?** Maintaining two shapes of the same idea bugs me, but forcing everything through a C ABI loses typed event subscription. Reasonable seam, or is there a cleaner approach (thin C++ wrapper over one C ABI, message-passing instead of typed events, …)?
2. **Self-describing JSON from** `describe()` **vs a versioned C struct.** JSON is flexible and avoids ABI breaks when fields change, but it's stringly-typed. How do you evolve a plugin schema/ABI safely — version constants, capability negotiation at init, a real IDL (flatbuffers/protobuf) at the boundary?
3. **C-ABI safety idioms.** Are try/catch-everywhere + string-ownership + teardown-quiesce the standard playbook for C++ behind a C ABI, or am I missing cleaner patterns (error codes + out-params, never returning owned pointers)? Is documenting these in header comments enough, or should they be enforced structurally?
4. **Pull vs push, mixed cadence.** Some collectors are cheap/fast (NVML poll), others slow/blocking (kernel queues). Everything is "host pulls on one thread" now. Would you isolate blocking collectors (thread per impact class? async with a deadline?), or is one slow-probe thread + an "impact" self-declaration in `describe()` enough?
5. **Versioning a plugin C-ABI for real.** What's worked for forward/backward compat as the ABI grows — a version int from `init`, additive-only rules, a vtable-of-function-pointers struct instead of named exports?
6. **Avoiding over-engineering.** I have exactly *one* real external collector today (kernel nonpaged pool) and a couple internal ones; the team is wary of speculative plugin machinery.
Post #25437
9