OpenChoreo uses OpenTelemetry to persist Kubernetes events and traces

OpenChoreo, a CNCF Sandbox project, has integrated the OpenTelemetry Collector into its Internal Developer Platform to solve common observability challenges. Released in late October 2026, this update enables platform teams to capture distributed traces and persist Kubernetes events that would otherwise vanish from the API server.

What happened

Internal Developer Platforms aim to provide a streamlined path from code to production, but observability often remains a fragmented burden. Developers face a proliferation of tools, complex wiring requirements, and disjointed dashboards. OpenChoreo addresses this by embedding observability directly into its architecture, using the OpenTelemetry Collector as the core engine for two critical features: distributed tracing and persistent Kubernetes events.

The platform is structured into distinct planes running on Kubernetes. A control plane manages state, while data planes handle workloads. An optional workflow plane manages builds and tests, and a dedicated observability plane collects telemetry data. This separation allows the observability layer to operate independently, collecting logs, metrics, traces, and events without interfering with application performance. The system uses a hexagonal architecture, exposing standard APIs for ingestion and adapter interfaces for storage backends, allowing teams to swap providers like OpenSearch or AWS X-Ray via Helm charts rather than code changes.

How it works

For distributed tracing, OpenChoreo leverages the standard OpenTelemetry SDKs. Applications export spans over OTLP to a local Collector in each data plane. The pipeline first receives these spans, then enriches them using the k8sattributes processor. This step matches spans to their originating pods and attaches Kubernetes metadata, mapping technical details to OpenChoreo concepts like project and environment. A tail_sampling processor then evaluates entire traces before deciding which to keep, ensuring context-aware sampling decisions. Finally, enriched traces are exported to a central collector or directly to cloud backends.

Kubernetes events present a different challenge because they are short-lived and not tied to specific pods in the same way traces are. The API server typically retains them for only an hour. OpenChoreo deploys a purpose-built Collector distribution to solve this. A single-replica collector uses the k8s_events receiver to watch cluster-wide events. A custom processor, k8seventenrich, looks up object labels and annotations from in-memory informer caches, adding them as resource attributes without hitting the API server repeatedly. These enriched events are then batched and sent to durable log backends, making them available for post-incident analysis long after the default retention period expires.

Key details

• OpenChoreo is a modular, open-source Internal Developer Platform currently in the CNCF Sandbox.
• The observability plane uses a hexagonal architecture with adapter APIs for swapping storage backends.
• Distributed tracing uses tail_sampling to make retention decisions based on full trace context.
• A custom k8seventenrich processor adds Kubernetes labels and owner references to events using in-memory caches.
• Supported tracing backends include OpenSearch, OpenObserve, AWS X-Ray, Azure Application Insights, and Google Cloud Trace.
• The events collector distribution supports exporters for OpenSearch, OTLP, AWS CloudWatch Logs, and debug outputs.

Why it matters

For platform engineers, the primary benefit is the reduction of operational friction. By standardizing on OpenTelemetry, OpenChoreo avoids vendor lock-in and reduces the need for custom instrumentation code. The ability to swap backends via configuration means teams can adapt to changing cost or compliance requirements without refactoring the platform. Furthermore, the integration of AI agents via MCP servers allows automated debugging tools to access the same rich trace data that human developers see, potentially accelerating incident resolution.

The persistence of Kubernetes events is particularly valuable for diagnosing infrastructure-level issues. Problems like resource limit throttling or scheduling failures often leave no trace in application logs. By capturing and enriching these events, platform teams gain visibility into the lifecycle of deployments and jobs. This transforms ephemeral cluster noise into a reliable audit trail, helping teams understand why a deployment failed or why a service scaled unexpectedly during peak traffic.

What you can do

• Deploy the GCP microservices demo on OpenChoreo to test distributed tracing across multiple services.
• Install the observability-events-otel-collector module using the provided Helm chart to start capturing persistent events.
• Configure the tail_sampling processor to retain only traces with errors or high latency, optimizing storage costs.
• Use the k8sattributes processor to map raw Kubernetes metadata to your internal project and environment labels.
• Explore the OpenChoreo portal’s Traces and Events tabs to correlate application performance with cluster-level activities.
• Review the custom k8seventenrich processor code if you need to extend event enrichment logic for your specific cluster objects.