feat(observability): supervisor OTLP telemetry relay by rhuss · Pull Request #3196 · NVIDIA/OpenShell

rhuss · 2026-09-05T04:26:57Z

Summary

Add an OTLP relay to the sandbox supervisor that accepts agent-emitted traces on 127.0.0.1:4318, enriches spans with sandbox resource attributes, and forwards them to the gateway over the existing session protocol
Enable zero-config observability for any OTel-instrumented agent inside network-isolated sandboxes without requiring egress policy exceptions
Validate via spike (feat(observability): agent-level trace collection via supervisor OTLP relay #2641) with zero measurable latency impact up to 10,000 spans/s on the shared session channel

Related Issue

Closes #2641

Changes

Proto (`proto/openshell.proto`)

TelemetryData message with sandbox_id, trace_data (serialized OTLP), and ocsf_events
capabilities field on SupervisorHello and SessionAccepted for feature negotiation
Telemetry variant on SupervisorMessage

Supervisor OTLP Module (`crates/openshell-supervisor-network/src/otlp/`)

receiver.rs: hyper-based HTTP server accepting POST /v1/traces (protobuf + JSON), TCP_NODELAY, dual spawn variants for netns vs direct bind
enrichment.rs: Injects sandbox resource attributes (openshell.sandbox.id, .workspace_id, .policy, .user, .image, .driver) and openshell.telemetry.source: "agent" routing marker
buffer.rs: Bounded channel (4096 slots) with drop-newest semantics, atomic drop counter and queue depth gauge
mod.rs: TelemetryRelay orchestrator, RelayHandle for lifecycle, RateLimitedOcsfSink for OCSF events, forwarder with non-blocking try_send

Supervisor Process Integration

child_env.rs: OTEL env var injection helper
process.rs / ssh.rs: Set OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL for agent processes
supervisor_session.rs: Advertise telemetry_relay capability, gate forwarding on gateway confirmation, log capability negotiation outcome

Sandbox Lifecycle (`crates/openshell-sandbox/src/lib.rs`)

Relay startup with driver-appropriate binding (netns for Docker/Podman, direct for K8s/VM)
Graceful shutdown: drain relay before network teardown so short-lived agents don't lose final spans

Gateway (`crates/openshell-server/`)

telemetry_relay.rs: Dedicated TelemetryRelayExporter with separate gRPC client to OTLP collector (preserves supervisor-enriched resource attributes), info-level logging on connect
supervisor_session.rs: Capability negotiation with logging (confirm telemetry_relay only when OTLP is configured, log why when not confirmed)

Configuration

deploy/docker/gateway.toml: Commented-out [openshell.gateway.otlp] section as setup reference

Supporting

openshell-otel/propagation.rs: HeaderMapInjector and inject_traceparent_if_missing() for W3C context propagation
openshell-ocsf: OcsfRelaySink trait and OcsfRelayLayer (implemented but not wired into subscriber, follow-up)
architecture/sandbox.md: Telemetry relay section with data flow diagram

Testing

cargo check passes for all relay crates
17 unit tests covering buffer, enrichment, rate limiter, and forwarder
Local Docker smoke test: traces flow from sandbox agent curl through supervisor relay to Jaeger via gateway
End-to-end validated on SAW cluster (demo.redhat.com OCP 4.22.8) with traces flowing to MLflow + Jaeger
Performance: zero latency impact at sustained 10k spans/s (spike report)

Checklist

Tests pass
Documentation updated (architecture/sandbox.md)
Changes are scoped to the issue
No secrets or credentials committed

copy-pr-bot · 2026-09-05T04:27:00Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Add an OTLP relay to the sandbox supervisor that accepts agent-emitted traces on 127.0.0.1:4318, enriches spans with sandbox resource attributes, and forwards them to the gateway over the existing session protocol. The relay enables zero-config observability for OTel-instrumented agents inside network-isolated sandboxes. Agents export to the standard OTLP endpoint; the supervisor handles enrichment, buffering, and forwarding without requiring egress policy exceptions. Key components: - OTLP HTTP receiver (protobuf + JSON) with per-driver binding (netns for Docker/Podman, direct for K8s/VM) - Span enrichment with sandbox identity (sandbox_id, workspace_id, policy, user, image, driver) and telemetry source marker - Bounded buffer (4096 slots) with non-blocking try_send forwarding - Gateway-side TelemetryRelayExporter with dedicated gRPC client that preserves supervisor-enriched resource attributes - Capability negotiation: supervisor advertises telemetry_relay, gateway confirms only when OTLP export is configured - Graceful shutdown ordering: relay drains before network teardown Proto: TelemetryData message, capabilities on SupervisorHello and SessionAccepted. Closes: NVIDIA#2641 Signed-off-by: Roland Huß <rhuss@redhat.com>

Fix Critical and Important findings from multi-agent code review: - Fix shutdown deadlock: drop telemetry sender before awaiting forwarder to allow the mpsc channel to close - Limit OTLP request body to 4 MiB (http_body_util::Limited), return 413 Payload Too Large when exceeded - Replace expect() with ConnectError on malformed OTLP endpoint URI to prevent gateway panic on startup - Add 10s timeout on telemetry export gRPC calls to prevent unbounded task accumulation during collector brownout - Validate OCSF events as JSON before logging and emit as structured field to prevent log injection via embedded newlines - Cap concurrent OTLP receiver connections at 64 via semaphore - Strip existing trusted keys before enrichment to prevent agent-supplied attribute spoofing - Fix rate limiter TOCTOU race with fetch_update CAS loop - Log session drop counter during shutdown for observability - Log on supervisor session try_send failure instead of silent discard Signed-off-by: Roland Huß <rhuss@redhat.com>

Clarify in architecture/sandbox.md that the OTLP receiver only binds port 4318 when the relay is active (gateway has OTLP configured and confirms the telemetry_relay capability). Document topology constraints: all current topologies keep the process supervisor co-located with the agent, so 127.0.0.1 is correct. Note that future topologies moving the process supervisor out of the workload pod would need a different address. Remove the netns_fd gate on OTEL env var injection in process.rs since all topologies keep the supervisor co-located and the env vars are harmless when the relay is not running (OTel SDKs handle unreachable endpoints gracefully). Signed-off-by: Roland Huß <rhuss@redhat.com>

The cherry-pick conflict resolution for podman container.rs used --theirs which in a cherry-pick context takes the spike's version, not main's. This accidentally included unrelated spike changes (TLS secret prefix removals, SPIFFE mount changes). Revert to upstream/main's version since our PR has no podman changes. Also revert mise.lock changes that were auto-generated when the pre-commit hook installed tools. Signed-off-by: Roland Huß <rhuss@redhat.com>

Replace hardcoded "127.0.0.1:4318" with constants in sandbox_env.rs (OTLP_RECEIVER_ADDR and OTLP_RECEIVER_ENDPOINT) so the bind address and agent env var are defined in one place. Future topologies that move the process supervisor out of the agent's network namespace can update these constants or derive the address from the topology. Signed-off-by: Roland Huß <rhuss@redhat.com>

…, and forwarder Add 13 new tests bringing the relay test suite from 4 to 17 tests: Buffer (4 new): - depth_tracks_send_and_recv: queue depth accuracy across operations - drop_count_increments_on_each_overflow: counter accuracy under sustained overflow - metrics_shared_across_clones: shared metrics between sender clones - recv_returns_none_when_all_senders_dropped: channel close behavior Enrichment (4 new): - enrichment_strips_agent_supplied_trusted_keys: dedup prevents spoofing - enrichment_preserves_non_trusted_agent_attributes: custom attrs kept - enrichment_handles_json_content_type: JSON input, protobuf output - enrichment_rejects_invalid_protobuf: error handling for bad input Rate limiter (3 new): - rate_limiter_acquires_initial_tokens: exact token count - rate_limiter_drops_when_exhausted: OcsfRelaySink drop counting - rate_limiter_refills_after_time: time-based token refill Forwarder (2 new): - forwarder_constructs_telemetry_data_messages: correct message construction - forwarder_increments_session_drop_counter: session channel backpressure Signed-off-by: Roland Huß <rhuss@redhat.com>

The OTLP relay was silently disabled when the gateway config lacked an [openshell.gateway.otlp] section, with no log output at any decision point. This made it impossible to diagnose why telemetry data was not flowing from supervisor to gateway. Add info-level logging when the relay exporter connects and when capabilities are confirmed, and debug-level logging for the normal inactive paths. Add a commented-out OTLP section to the Docker gateway config as a setup reference. Fix pre-existing rustfmt and clippy issues in OTLP relay code. Signed-off-by: Roland Huß <rhuss@redhat.com>

rhuss · 2026-09-05T13:43:28Z

@krishicks @drew this is a PR for introducing an OTEL relay to the supervisor, forwarding OTLP traces to a collector via the gateway. This is the hardened OTEL-relay spike that I did for creating this report in #2641 (comment)

It would be great if you could put your review agent army on this PR (I did some self-review, but we all know the more reviews the merrier :)

See also https://github.com/rhuss/OpenShell/blob/48e9ece979889aa49b1df146c3d0d08e7118496a/architecture/sandbox.md#telemetry-relay for a recap of the overall architecture.

Rename the proto message and all related types to clarify that this carries OpenTelemetry signal data destined for an OTLP collector, not product analytics telemetry. Add a `oneof signal` wrapper around `trace_data` so future metrics and logs signals can be added without breaking the wire format. Renames: - Proto: TelemetryData -> OtelExportData, oneof payload field telemetry -> otel_export - Capability: "telemetry_relay" -> "otel_export" - Gateway: TelemetryRelayExporter -> OtelRelayExporter, telemetry_relay.rs -> otel_relay.rs - Supervisor: TelemetryRelay -> OtelRelay - All related variables, functions, and log messages Signed-off-by: Roland Huß <rhuss@redhat.com>

2000krysztof · 2026-09-09T16:54:05Z

+        // OTEL relay: bind OTLP receiver for all Linux topologies.
+        // All current topologies keep the process supervisor co-located with
+        // the agent, so 127.0.0.1 is reachable from agent processes. Future
+        // topologies that move the supervisor out of the workload pod would
+        // need to derive the bind address from the topology (e.g., pod IP
+        // via downward API) and update OTEL_EXPORTER_OTLP_ENDPOINT to match.
+        let otel_rx = {
+            #[cfg(target_os = "linux")]
+            {
+                let otlp_addr = openshell_core::sandbox_env::OTLP_RECEIVER_ADDR;
+
+                let (otel_session_tx, otel_session_rx) =
+                    tokio::sync::mpsc::channel::<openshell_core::proto::SupervisorMessage>(64);
+
+                let relay_config = openshell_supervisor_network::otlp::RelayConfig::default();
+                let metadata = openshell_supervisor_network::otlp::SandboxMetadata {
+                    sandbox_id: sandbox_id.clone().unwrap_or_default(),
+                    workspace_id: workspace_rx.borrow().clone(),
+                    policy: sandbox_name_for_agg.clone().unwrap_or_default(),
+                    user: resolved_process_identity
+                        .uid()
+                        .map_or_else(String::new, |uid| uid.to_string()),
+                    image: std::env::var("OPENSHELL_CONTAINER_IMAGE").unwrap_or_default(),
+                    driver: std::env::var(openshell_core::sandbox_env::SUPERVISOR_TOPOLOGY)
+                        .unwrap_or_else(|_| "container".to_string()),
+                };
+                let relay = openshell_supervisor_network::otlp::OtelRelay::new(
+                    relay_config,
+                    metadata,
+                    otel_session_tx,
+                );
+
+                let bind_addr: std::net::SocketAddr = otlp_addr.parse().unwrap();
+
+                if let Some(ns) = netns.as_ref() {
+                    match ns.bind_tcp_in_netns(otlp_addr).await {
+                        Ok(listener) => {
+                            let handle = relay.start_with_listener(listener);
+                            tracing::info!(bind = %bind_addr, "OTEL relay started (netns)");
+                            otel_relay_handle = Some(handle);
+                        }
+                        Err(e) => {
+                            tracing::warn!(error = %e, "OTEL relay failed to bind in netns; continuing without relay");
+                        }
+                    }
+                } else {
+                    match relay.start(bind_addr).await {
+                        Ok(handle) => {
+                            tracing::info!(bind = %bind_addr, "OTEL relay started");
+                            otel_relay_handle = Some(handle);
+                        }
+                        Err(e) => {
+                            tracing::warn!(error = %e, "OTEL relay failed to start; continuing without relay");
+                        }
+                    }
+                }
+                Some(otel_session_rx)


The OTLP listener appears to bind before the gateway confirms the otel_export capability, and it remains active when the gateway declines it. This reserves the standard 127.0.0.1:4318 port even when OTLP export is not configured. The receiver can also return 200 OK for telemetry that will not be forwarded because the session does not drain otel_rx unless the capability is confirmed. Could listener startup be gated on the negotiated capability, or could the listener be shut down when the capability is declined?

Confirmed. The bind in crates/openshell-sandbox/src/lib.rs:962-1011 is unconditional and happens before run_process opens the supervisor session, while the drain arm in crates/openshell-supervisor-process/src/supervisor_session.rs:461 is gated on otel_confirmed. So a declined capability leaves 4318 reserved and the receiver answering 200 for spans that then die on try_send into the 64-slot channel. architecture/sandbox.md already describes bind-on-confirm as the intended behavior, so this is the code catching up with the docs.

Planned route: bind only on confirmation, no open-then-close. The relay moves into the supervisor session task (the otlp module relocates to openshell-supervisor-process, which is its only consumer). The session binds the listener on the first SessionAccepted that confirms otel_export, keeps it across reconnects, and drains the telemetry buffer directly into the outbound stream. That removes the forwarder task and the intermediate otel_session_tx channel, so the pipeline becomes receiver -> buffer -> session -> gateway with one bounded buffer instead of two, and the silent drop point you found goes away with it.

Two things worth calling out. The agent starts before the handshake (run.rs:247 vs :374), so spans exported in that window get a refused connection instead of a 200; OTel exporters retry with backoff, so in practice they land once the port is up. And the existing handle.shutdown().await in lib.rs:1211 runs after run.rs:490 has already aborted the session, so it drains into a channel nobody reads. With the session owning the relay, run.rs asks it to stop the receiver and flush the buffer after the entrypoint exits and before report_main_process_exit, with a bounded wait, so final spans actually reach the gateway.

Applied in 0890eee. The receiver is now bound by the session on the first confirming SessionAccepted (RelayLifecycle::ensure_started in crates/openshell-supervisor-process/src/otlp/mod.rs), the forwarder task and intermediate channel are gone, and run_process flushes buffered telemetry before reporting the exit.

2000krysztof · 2026-09-09T17:02:50Z

+                            tokio::spawn(async move {
+                                let svc = service_fn(move |req| {
+                                    let buf_tx = buf_tx.clone();
+                                    let metadata = metadata.clone();
+                                    async move {
+                                        handle_request(req, &buf_tx, &metadata, enrichment_enabled).await
+                                    }
+                                });


I think an accepted keep-alive connection can block shutdown. Each detached serve_connection task retains a TelemetrySender, but receiver_handle tracks only the accept loop. RelayHandle::shutdown() drops only its own sender and then waits for the forwarder. The forwarder cannot observe channel closure until every connection task exits, so this wait is unbounded. Should connection tasks receive shutdown cancellation or be tracked and explicitly drained or aborted?

OpenShell/crates/openshell-supervisor-network/src/otlp/mod.rs

Line 133 in f8f7efc

pub struct RelayHandle {

P.S sorry for the two reviews I selected the wrong lines earlier.

Confirmed. Each connection task in receiver.rs:85-100 holds a cloned TelemetrySender, receiver_handle only covers the accept loop, and spawn_forwarder exits only when the last sender drops. With HTTP/1.1 keep-alive and no idle timeout on http1::Builder, RelayHandle::shutdown() can wait forever, and it sits on the sandbox teardown path.

Planned fix: connections are tracked in a JoinSet and watched by hyper_util::server::graceful::GracefulShutdown, so shutdown disables keep-alive on every open connection (idle ones close immediately), waits a bounded 2s for in-flight requests, and aborts stragglers. http1::Builder also gets a header read timeout. The final drain no longer waits for senders to drop at all; it uses the buffer's non-blocking drain(), so a straggler cannot stall it. Regression test: keep-alive connection held open, shutdown must complete within the deadline.

No need to apologise for the two reviews, both were worth filing.

Applied in 0890eee. Connections are tracked in a JoinSet under GracefulShutdown with a header read timeout; ReceiverHandle::shutdown is bounded at 2s plus abort. Regression test: shutdown_completes_with_idle_keepalive_connection in otlp/receiver.rs.

Resolve Cargo.lock conflict by regenerating the lockfile against the merged manifests. The branch-side lock still carried the removed openshell-router crate and a divergent dependency graph. Signed-off-by: Roland Huß <rhuss@redhat.com>

The supervisor bound the OTLP receiver on 127.0.0.1:4318 before the session handshake, regardless of whether the gateway confirmed the otel_export capability. When the capability was declined the port stayed reserved and the receiver answered 200 OK for spans that were then dropped on the way to a session that never drained them. Shutdown could also hang: each HTTP connection task held a buffer sender and the forwarder only exited once the last sender dropped, so a single keep-alive connection blocked sandbox teardown. Move the relay into openshell-supervisor-process and make the supervisor session own it. The session binds the receiver on the first SessionAccepted that confirms otel_export, keeps it across reconnects, and drains the buffer straight into the session stream. This removes the forwarder task and the intermediate session channel, leaving one bounded buffer between receiver and gateway. The receiver now tracks connections in a JoinSet under hyper-util's GracefulShutdown with a header read timeout, so shutdown disables keep-alive, waits a bounded 2s for in-flight requests, and aborts stragglers. run_process asks the session to stop the receiver and flush buffered telemetry after the entrypoint exits and before the exit is reported, bounded at 3s, so final spans still reach the gateway. The previous lib.rs drain ran after the session task had been aborted and flushed into a channel nobody read. Also drop the never-read RelayConfig.enabled field, factor the netns bind into bind_tcp_in_netns_fd, and update the architecture doc. Signed-off-by: Roland Huß <rhuss@redhat.com>

The network-sidecar path in openshell-sandbox spawns its own supervisor session and only compiles on Linux, so the SessionHandle return type change was missed by the macOS check. Also drop a redundant path qualification flagged by the Linux build. Signed-off-by: Roland Huß <rhuss@redhat.com>

rhuss · 2026-09-10T09:43:15Z

@2000krysztof thanks for the review, that was very helpful! I've did some reordering and it should be ensured now that the port is not opened if otel is not enabled.

2000krysztof

All my concerns were addressed LGMT, nice work @rhuss

rhuss requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners September 5, 2026 04:26

rhuss marked this pull request as draft September 5, 2026 05:44

rhuss added 4 commits September 5, 2026 08:50

rhuss force-pushed the feat/supervisor-otlp-relay branch from 2038d0d to e6cada9 Compare September 5, 2026 06:51

rhuss added 3 commits September 5, 2026 08:55

rhuss marked this pull request as ready for review September 5, 2026 13:38

rhuss mentioned this pull request Sep 5, 2026

feat(observability): OpenTelemetry span emission from the sandbox supervisor #2508

Open

2 tasks

rhuss added 2 commits September 5, 2026 17:15

Merge branch 'main' into feat/supervisor-otlp-relay

f8f7efc

2000krysztof suggested changes Sep 9, 2026

View reviewed changes

rhuss added 3 commits September 10, 2026 10:19

2000krysztof approved these changes Sep 10, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat(observability): supervisor OTLP telemetry relay - #3196

feat(observability): supervisor OTLP telemetry relay#3196
rhuss wants to merge 12 commits into
NVIDIA:mainfrom
rhuss:feat/supervisor-otlp-relay

rhuss commented Sep 5, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Sep 5, 2026

Uh oh!

rhuss commented Sep 5, 2026 •

edited

Loading

Uh oh!

2000krysztof Sep 9, 2026

Uh oh!

rhuss Sep 10, 2026

Uh oh!

rhuss Sep 10, 2026

Uh oh!

2000krysztof Sep 9, 2026

Uh oh!

rhuss Sep 10, 2026

Uh oh!

rhuss Sep 10, 2026

Uh oh!

rhuss commented Sep 10, 2026

Uh oh!

2000krysztof left a comment •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

rhuss commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Proto (proto/openshell.proto)

Supervisor OTLP Module (crates/openshell-supervisor-network/src/otlp/)

Supervisor Process Integration

Sandbox Lifecycle (crates/openshell-sandbox/src/lib.rs)

Gateway (crates/openshell-server/)

Configuration

Supporting

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Sep 5, 2026

Uh oh!

rhuss commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

2000krysztof Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

rhuss Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

rhuss Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

2000krysztof Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

rhuss Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

rhuss Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

rhuss commented Sep 10, 2026

Uh oh!

2000krysztof left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rhuss commented Sep 5, 2026 •

edited

Loading

Proto (`proto/openshell.proto`)

Supervisor OTLP Module (`crates/openshell-supervisor-network/src/otlp/`)

Sandbox Lifecycle (`crates/openshell-sandbox/src/lib.rs`)

Gateway (`crates/openshell-server/`)

rhuss commented Sep 5, 2026 •

edited

Loading

2000krysztof left a comment •

edited

Loading