6 min read
Let customers choose the observability backend
Make export part of the product boundary
If a customer has to change application code to send telemetry somewhere else, the backend has become part of the product. I would rather define telemetry export as a supported capability: the application produces signals through a stable, vendor-neutral contract, and the customer chooses where those signals go.
That boundary matters for self-hosted software in particular. Customers may run it in their own data centre, in a cloud account they control, or on a platform that the vendor does not operate. They may also have requirements about where telemetry is stored or which team can access it. The OpenTelemetry post on OTel-native product design makes the same case for products that need to send logs, traces, and metrics to a customer’s observability stack.
My preference is OpenTelemetry instrumentation in the application and OTLP as the export contract. The application should describe what happened in terms of the product and its operations, not encode assumptions about a particular dashboard, query language, or vendor’s ingestion API. OpenTelemetry’s OTLP exporter documentation for .NET describes OTLP exporters as designed for the OpenTelemetry data model, which is a useful basis for keeping the application-facing boundary portable.
Portable does not mean responsibility-free. Once I say a destination is supported, I own the compatibility boundary: the signals, attributes, configuration, and failure behaviour customers can reasonably expect. The customer gets choice; the product team takes on the work of making that choice usable.
Keep destination details out of application code
The application should know how to produce telemetry and how to send it to a configured endpoint. It should not contain a branch for each customer’s backend. Put endpoint, credentials, and transport settings in deployment configuration or in a Collector that the customer operates. Keep credentials out of source control, and make the configuration surface explicit enough to document and validate.
For example, an application can be configured to send OTLP over HTTP to an endpoint supplied at deployment time:
OTEL_EXPORTER_OTLP_ENDPOINT="https://collector.example.invalid"
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
The hostname here is illustrative, not a real service. The important design choice is that the application’s endpoint can change without rebuilding it. For a product that supports a customer-managed Collector, the application can send to that Collector, and the customer can configure its downstream destination there. The OpenTelemetry Collector configuration guide describes pipelines built from receivers, processors, and exporters; that separation gives customers a place to route and process telemetry without adding backend-specific branches to product code.
I would keep the product’s own configuration contract smaller than the full range of options a Collector can expose. A product might document the endpoint, protocol, authentication mechanism, and whether the customer can choose a Collector. It should not pretend that every Collector extension or backend-specific feature is part of the product’s support promise. A clear, limited contract is easier to test than an unbounded promise of compatibility.
There is a trade-off here. Sending directly from the application may be simpler to deploy, but puts more destination and credential configuration close to the process. Sending through a customer-managed Collector gives the customer a control point for routing and processing, but introduces another component to operate. I would support the deployment model that fits the product’s trust and operational boundaries, and state plainly which one is covered. I would not silently require a vendor-managed service just because it is the easiest path for the product team.
Portability is not identical behaviour
An OTLP endpoint can accept telemetry without making the downstream experience equivalent. A backend may interpret, store, index, or expose signals differently. The customer can still encounter differences in dashboards, alerting, query capabilities, retention, or cost. Those are consequences of the destination, not problems a wire format can erase.
The contract therefore needs to say more than “we support OpenTelemetry.” It should describe the signals the product emits, the meaning of important attributes, and which fields the product considers stable. OpenTelemetry’s resource specification describes a Resource as the observed entity associated with telemetry. Consistent resource identity gives a customer a useful way to recognize which service or deployment produced a signal, regardless of the backend receiving it.
I would also be conservative about changing names or meanings once customers build queries and alerts against them. The OpenTelemetry versioning and stability guidance explains that semantic conventions form a contract between instrumentation and analysis tools. That is the right mental model for product telemetry too: a field that looks internal to the application may become an input to customer-owned monitoring.
This is where vendor neutrality can be misunderstood. It does not require the product team to guarantee that every backend renders the same view, nor that every vendor-specific feature has an equivalent elsewhere. It requires the application’s core instrumentation and export path not to depend on one backend. I would document the limits: portable signal delivery is supported; backend-specific queries, dashboards, and alert rules belong to the destination or to separately maintained integrations.
Test a destination change, not just an endpoint
A test that proves telemetry reaches one endpoint is a start, not a portability test. The useful question is what happens when the customer changes the destination while leaving the application unchanged. I would exercise that change in the product’s integration tests or release checks, using an OTLP-compatible test receiver or Collector and verifying the signals the product promises to emit.
The checks should cover the application-facing contract: configuration is read from the documented source, the intended signals arrive, resource identity and important attributes are present, and sensitive values are not accidentally included. I would also test the operational edges that matter to the product: a destination that is unavailable, invalid configuration, and a credentials change. The expected outcome should be explicit for each case, such as a clear startup error or a bounded retry policy, rather than a silent loss that customers discover only after an incident.
A destination-change check can be simple in principle:
- Run the same application build against a test Collector configured for one destination.
- Change only the destination configuration and run it again.
- Compare the received signal names, resource attributes, and product-specific attributes.
- Confirm that the application code and its instrumentation configuration did not need a backend-specific change.
That test does not prove compatibility with every backend. It proves something narrower and more valuable: the product’s own export contract survives a change in destination. I would add backend-specific integrations only when customers need them, and keep their support boundaries separate from the portable path.
Treating telemetry export as a product capability is not a promise that all observability stacks are interchangeable. It is a decision to keep the product’s instrumentation stable, put destination choice at the deployment boundary, and test that boundary when it changes. The cost is owning compatibility and explaining the limits. I would pay that cost rather than make a customer’s observability backend a permanent product fork.
Drafted with AI assistance, reviewed and edited by Ali.