OpenTelemetry in .NET

OpenTelemetry (OTel) is the closest thing our industry has to a shared language for observability: traces, metrics and logs, generated in your app, exported in one standard shape (OTLP) and correlated so you can answer the only question production ever asks. What is happening, and why, right now?

Getting the wiring in place takes an afternoon, and the official docs will walk you through it. What they do not tell you is which decisions are expensive to reverse and which order to make them in.

I am setting up observability for a new project as I write this. I have also spent a lot of hours grepping log files trying to reconstruct an incident after the fact, which is the state most projects are in when someone finally asks for telemetry. The order below is how I avoid ending up back there.

Why OpenTelemetry

Debugging a distributed system without telemetry is archaeology. You sift through fragments and hope a story falls out.

One user action can hop through APIs, queues, databases and third parties. Traces give you the story, metrics give you system health, logs carry the details. .NET is a good host for this because Activity, Meter and ILogger are already in the runtime. You are not adopting a foreign model, you are exporting what your app already emits, which is why instrumenting an existing .NET service is mostly configuration rather than a rewrite.

Two arguments carry the decision for me.

You instrument once and export anywhere. Backends change, and I have been through three in the last few years. Instrumentation tied to a vendor SDK turns every one of those moves into a pass through application code.

Correlation is the multiplier. When log records carry TraceId and SpanId, one error line becomes a jump to the exact trace and its dependency timings. That is the difference between "I think the database was slow" and "this call took 4.2 seconds, here is the query." The rest of OTel is useful. This is the part that changes how an incident goes.

The wiring (ASP.NET Core)

Traces, metrics and logs, all three exported over OTLP. This is the shape I start a service with.

Program.cs

using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

var serviceName = builder.Environment.ApplicationName;
var serviceVersion = typeof(Program).Assembly.GetName().Version?.ToString() ?? "unknown";

// Configure the resource ONCE, on the builder. It then applies to traces,
// metrics and logs alike. Calling SetResourceBuilder on an individual
// provider overrides this and is the usual way people end up with two
// resources that disagree.
builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(serviceName: serviceName, serviceVersion: serviceVersion)
        .AddAttributes(new[]
        {
            new KeyValuePair<string, object>("deployment.environment.name", builder.Environment.EnvironmentName),
        }))
    .WithTracing(tracing => tracing
        // For custom spans later (ActivitySource):
        .AddSource(serviceName)
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        // Add DB instrumentation depending on your stack (SqlClient, EF Core, etc.)
        .AddOtlpExporter()
    )
    .WithMetrics(metrics => metrics
        // For custom metrics later (Meter):
        .AddMeter(serviceName)
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()
        .AddOtlpExporter()
    );

// Logs go through ILogger and share the resource configured above.
builder.Logging.AddOpenTelemetry(logging =>
{
    logging.IncludeFormattedMessage = true;
    logging.IncludeScopes = true;
    logging.ParseStateValues = true;
    logging.AddOtlpExporter();
});

var app = builder.Build();
app.MapGet("/ping", () => "pong");
app.Run();

One thing worth being deliberate about: configure the resource in exactly one place. ConfigureResource on the builder covers every signal. SetResourceBuilder on an individual provider silently replaces it, so if you use both you end up shipping traces and metrics tagged differently from your logs and wondering why the backend will not correlate them.

Configure the exporter from the environment, not from code

Nothing in that sample names an endpoint, and that is deliberate. The SDK reads its exporter configuration from environment variables, so one image runs in every environment and pointing a service at a different collector during an incident is a restart rather than a release.

export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4317"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
export OTEL_EXPORTER_OTLP_HEADERS="api-key=REDACTED"
export OTEL_RESOURCE_ATTRIBUTES="service.namespace=mycompany,deployment.environment.name=prod"

OTEL_EXPORTER_OTLP_HEADERS usually carries an API key. That makes it a secret, so it belongs wherever the rest of your secrets live and not in a compose file.

What I set up first, and in what order

The wiring above is the easy part. What follows are the decisions that are cheap to make now and expensive to unwind in six months, roughly in the order I make them.

1. Put a Collector in front of the backend

Applications do not talk to the observability backend. They talk to an OpenTelemetry Collector, and the Collector talks to the backend. It listens for OTLP on 4317 (gRPC) and 4318 (HTTP).

That one extra hop is what makes dropping a noisy attribute, redacting a field, adding a second destination or switching vendor a config change on one component instead of a release across every service. It also means one component holds the backend credentials and one component sends data out of the network, which is a far shorter conversation with a security team than "every service egresses to a SaaS endpoint."

2. Agree the telemetry contract before the second service

This one bit me. Naming has to be settled before more than one team is emitting:

• service.name, plus service.namespace once you have more than a handful of services
• deployment.environment.name
• which attributes are required, which are forbidden because they are personal data, and which are forbidden because they will eat your cardinality budget

The reason this goes early is that renaming does not backfill. Rename an attribute in month six and every dashboard, alert and saved query splits into a before and an after, while the data you already paid to store keeps the old name forever. Start from the semantic conventions rather than inventing a vocabulary. They are boring, and boring is the point: anyone who has used OTel elsewhere already knows your schema.

3. Correlate logs and traces on day one

Inside a request you get this for free. Where you lose it is at every asynchronous boundary: a message on a queue, a background job, a retry out of an outbox. Trace context does not cross those hops unless you carry it across yourself.

Do that while the system is small. Every uninstrumented hop becomes a blind spot in exactly the place the hard bugs live, and retrofitting propagation later means touching every producer and every consumer in one go.

4. Instrument less than you want to

Inbound HTTP, outbound HTTP, runtime metrics and a small number of business counters will answer most of the questions you actually ask at 3am. Add more when an incident makes you wish you had it. You cannot usefully guess in advance which signal you will want, and everything you add unused is a bill.

The cost is not only storage. Attributes are the real trap: put a user id or a request id on a metric and you have created one time series per user, which is how a metrics backend gets taken down from the inside. High cardinality belongs on spans, never on metrics.

5. Decide the sampling story before you need it

• Dev: AlwaysOn. See everything while you are still learning the system.
• Prod: head sampling to bring volume down to something you can afford.
• Later: tail sampling in the Collector, keeping errors and slow traces and discarding the successful health checks.

Pick the direction early even if you start at 100 percent, because sampling changes what the data means. Once traces are sampled you can no longer count them, so "how often does this happen" has to come from metrics, which are not sampled. Alert on metrics, explain with traces. Alerts built on trace counts have to be rebuilt the day the bill forces sampling on you.

6. Treat telemetry as production data

Traces and logs carry request paths, query strings, user identifiers and whatever ended up in an exception message. That makes your observability backend a partial copy of production data, usually with wider access and a different retention policy than the database it came from.

In regulated work this is a design constraint, not a hardening pass at the end. Before the first span leaves the network I want three answers: what personal data can appear in an attribute, who can read the backend, and how long it is kept. Redaction goes in the Collector, so the rule lives in one place instead of being reimplemented slightly differently in every service. It is also the honest answer to a question that always arrives later than it should: when someone exercises their right to erasure, telemetry is one more place their data lives.

What I skip

Custom spans on everything, a metrics taxonomy designed in a meeting, and dashboards built before the first incident. The first incident tells you what the dashboard should have been.

Resources

The docs I actually send people to:

• Getting Started (.NET) for the wiring
• SDK Environment Variables for the full exporter configuration list
• Semantic Conventions before you name anything
• Collector Configuration for the processors that do your redaction and filtering
• Sampling for head versus tail
• Security Best Practices for locking the pipeline down
• Observability with OTel for how this maps onto the .NET primitives