OpenTelemetry and hosted services

Question

I have a .NET 6 web application with HostedServices doing background processing. When the HostedService starts Activity just after the application starts, this Activity is not exported by the OpenTelemetry exporters like a console or a Jeager exporter.

It can be reproduced using this app and test:

using System.Diagnostics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddSingleton<Func<Task>>(() => Task.CompletedTask);
builder.Services.AddHostedService<TestBackgroundService>();
builder.Services.AddOpenTelemetryTracing(b =>
{
    b
        .AddSource(ServiceName)
        .SetResourceBuilder(ResourceBuilder
            .CreateDefault()
            .AddService(serviceName: ServiceName))
        .AddInMemoryExporter(ExportedActivities);
});
var app = builder.Build();
app.Run();

public partial class Program
{
    public static readonly ActivitySource ActivitySource = new("OpenTelemetryTestApp");
    public static readonly List<Activity> ExportedActivities = new();
}

public class TestBackgroundService : BackgroundService
{
    private readonly Func<Task> _notify;

    public TestBackgroundService(Func<Task> notify)
    {
        _notify = notify;
    }
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using var activity = Program.ActivitySource.StartActivity("some trace");
        await _notify();
    }
}

using System.Threading.Channels;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.Extensions.Hosting;

namespace OpenTelemetryBackgroundProcessesTests;

public class Tests : IClassFixture<TestApplicationFactory>
{
    private readonly TestApplicationFactory _app;

    public Tests(TestApplicationFactory app)
    {
        _app = app;
        // Start app
        _app.CreateClient();
    }
    
    [Fact]
    public async Task ActivityShouldBeExportedFromHostedService()
    {
        await _app.NotifyChannel.Reader.ReadAsync(new CancellationTokenSource(TimeSpan.FromSeconds(10)).Token);
        Assert.NotEmpty(Program.ExportedActivities);
    }
}

public class TestApplicationFactory : WebApplicationFactory<Program>
{
    public readonly Channel<object> NotifyChannel = Channel.CreateUnbounded<object>();
    
    protected override IHost CreateHost(IHostBuilder builder)
    {
        builder.ConfigureServices(services =>
        {
            services.Replace(ServiceDescriptor.Singleton<Func<Task>>(async () =>
            {
                await NotifyChannel.Writer.WriteAsync("notify");
            }));
        });
        return base.CreateHost(builder);
    }
}

I tested it with the latest stable (1.3.2) and prerelease (1.4.0-rc4) OpenTelemetry libraries.

Answer 1

The problem is with the timing when the OpenTelemetry SDK registers the ActivityListener that does the export. It does it in the constructor of TracerProviderSdk. When using OpenTelemetry extensions for ASP.NET Core setup, TracerProviderSdk is registered as a singleton and will be instantiated when something asks for its instance for the first time. OpenTelemetry registers its own HostedService, which is there to ensure that all instrumentations, exporters, etc., are created and started. .NET instantiates HostedServices in the order they were registered, and it does it before the web application starts. So the described problem happens only with the HostedServices registered before the OpenTelemetry one.

Solutions I found out:

1. Register OpenTelemetry before any HostedServices. It might be hard to ensure in bigger applications.

builder.Services.AddOpenTelemetryTracing(b =>
{
    b
        .AddSource(ServiceName)
        .SetResourceBuilder(ResourceBuilder
            .CreateDefault()
            .AddService(serviceName: ServiceName))
        .AddInMemoryExporter(ExportedActivities);
});
builder.Services.AddHostedService<TestBackgroundService>();

2. Instantiate TraceProvider manually and then register it to a DI container to make the container manage its lifetime. This solution was recommended to me on the CNCF slack.

var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddSource(ServiceName)
    .SetResourceBuilder(ResourceBuilder
        .CreateDefault()
        .AddService(serviceName: ServiceName))
    .AddInMemoryExporter(ExportedActivities)
    .Build();
builder.Services.AddSingleton(tracerProvider);

3. Ensure that the TraceProvider is instantiated before the app runs.

var app = builder.Build();
app.Services.GetRequiredService<TracerProvider>();
app.Run();