Building an AI Gateway in .NET

.NET applications usually begin their AI integration in a sensible way but that soon changes once the application has ten or twenty AI-backed features. One feature retries on a timeout while another fails immediately. One records token usage and another records only elapsed time. Some prompts include a version number. Others are hard coded inside handlers. Model names leak into application code. Streaming is handled differently in each endpoint. Nobody can answer which feature caused last month's increase in AI spend. At that point, the problem isn't calling a model. The problem is making every call behave like part of the same system.

An AI gateway gives a .NET application one controlled path to its model providers. It can apply consistent execution policies, route requests to suitable models, record telemetry and capture usage without forcing every feature to rebuild the same plumbing. The idea is already established at infrastructure level. Azure API Management describes its AI gateway capabilities as a way to secure, scale, monitor and govern models, agents and tools. Inside a .NET application, the same pressure appears at a smaller boundary. The application needs somewhere to own the behaviour shared by every AI request.

That doesn't mean every application needs a separate gateway service. For many systems, an in-process abstraction around IChatClient is enough. The important decision is the boundary, not whether it runs in another container.

Where the duplication starts

Imagine an application with three vertical slices:

SummariseSubmission
ClassifyIncomingEmail
ExtractClaimDetails

The first implementation might be perfectly reasonable:

public sealed class SummariseSubmissionHandler(
    IChatClient chatClient)
{
    public async Task<string> HandleAsync(
        SummariseSubmission command,
        CancellationToken cancellationToken)
    {
        var messages = new[]
        {
            new ChatMessage(
                ChatRole.System,
                "Summarise insurance submissions clearly and accurately."),
            new ChatMessage(
                ChatRole.User,
                command.SubmissionText)
        };

        var response = await chatClient.GetResponseAsync(
            messages,
            cancellationToken: cancellationToken);

        return response.Text;
    }
}

There is nothing obviously wrong with this code. The handler asks for a chat client and uses it.

The trouble starts when the next handler needs slightly different behaviour. Perhaps classification should use a cheaper model. Extraction needs a longer timeout. Summarisation needs token usage recorded against a customer. A compliance requirement then says prompts and responses must be traceable by feature and version.

The application begins to grow small, slightly different copies of the same execution logic.

That is the point where a gateway earns its place.

What an AI gateway actually does

The gateway sits between application features and model clients.

Each feature still decides what its trying to achieve. It builds the prompt, supplies any structured output schema and validates the result against its own rules. The gateway owns behaviour that should remain consistent across features. That includes model resolution, timeout handling, retry policy, provider access, telemetry, usage capture and operational limits.

If the gateway starts deciding how a claim should be summarised, what fields belong in a submission or whether an underwriting answer is acceptable, it has crossed the boundary. Those decisions belong with the feature that understands them.

A useful rule is:

The feature owns intent. The gateway owns execution.

Start with a request contract

A gateway API should carry enough information to apply policy without becoming a bag of arbitrary settings. The feature shouldn't usually send a provider name or deployment ID. Those are operational details. It should describe the capability it needs.

public enum AiCapability
{
    FastText,
    StructuredExtraction,
    ComplexReasoning,
    Vision
}

public sealed record AiExecutionContext(
    string Feature,
    string Operation,
    string PromptVersion,
    string? TenantId = null,
    string? UserId = null,
    string? CorrelationId = null);

public sealed record AiRequest(
    AiCapability Capability,
    IReadOnlyList<ChatMessage> Messages,
    AiExecutionContext Context,
    ChatOptions? Options = null);

The capability gives the router something stable to work with. A feature can ask for structured extraction without knowing whether that maps to an Azure OpenAI deployment, another provider or a model that has changed since the feature was written. The execution context provides dimensions that matter outside the model call. They can be attached to traces, usage records and logs. A prompt version is especially useful because behaviour changes are often caused by prompt changes rather than code deployments.

The gateway contract can remain small:

public interface IAiGateway
{
    Task<ChatResponse> GetResponseAsync(
        AiRequest request,
        CancellationToken stopToken = default);

    IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        AiRequest request,
        CancellationToken stopToken = default);
}

Returning the Microsoft.Extensions.AI response types preserves useful metadata and avoids inventing an incomplete response abstraction.

Use IChatClient as the provider boundary

Microsoft.Extensions.AI gives .NET applications a common abstraction for chat-based AI services. IChatClient supports complete and streaming responses and can be composed through a builder pipeline. That makes it a good provider boundary underneath the gateway. The gateway adds application level policy. IChatClient removes direct coupling to a particular provider SDK. Those are related responsibilities, but they operate at different levels.

A simple resolver can map capabilities to named clients:

public interface IChatClientResolver
{
    IChatClient Resolve(AiCapability capability);
}

public sealed class ChatClientResolver(
    IReadOnlyDictionary<AiCapability, IChatClient> clients)
    : IChatClientResolver
{
    public IChatClient Resolve(AiCapability capability)
    {
        if (clients.TryGetValue(capability, out var client))
        {
            return client;
        }

        throw new InvalidOperationException(
            $"No chat client is configured for capability '{capability}'.");
    }
}

For a real application, the dictionary would normally be assembled in the composition root. The important part is that feature code doesn't know which deployment sits behind ComplexReasoning.

A first gateway implementation

The gateway can now coordinate resolution, telemetry and usage capture.

using System.Diagnostics;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Logging;

public sealed class AiGateway(
    IChatClientResolver clientResolver,
    IAiUsageRecorder usageRecorder,
    ILogger<AiGateway> logger)
    : IAiGateway
{
    private static readonly ActivitySource ActivitySource =
        new("DotNetDigest.AiGateway");

    public async Task<ChatResponse> GetResponseAsync(
        AiRequest request,
        CancellationToken stopToken = default)
    {
        ArgumentNullException.ThrowIfNull(request);

        using var activity = StartActivity(request);
        var startedAt = Stopwatch.GetTimestamp();

        try
        {
            var client = clientResolver.Resolve(request.Capability);

            var response = await client.GetResponseAsync(
                request.Messages,
                request.Options,
                cancellationToken);

            var elapsed = Stopwatch.GetElapsedTime(startedAt);

            await usageRecorder.RecordAsync(
                AiUsageRecord.From(request, response, elapsed),
                cancellationToken);

            activity?.SetStatus(ActivityStatusCode.Ok);

            return response;
        }
        catch (OperationCanceledException)
            when (stopToken.IsCancellationRequested)
        {
            activity?.SetStatus(
                ActivityStatusCode.Error,
                "The caller cancelled the AI request.");

            throw;
        }
        catch (Exception exception)
        {
            activity?.SetStatus(
                ActivityStatusCode.Error,
                exception.Message);

            logger.LogError(
                exception,
                "AI execution failed for {Feature}.{Operation} using {Capability}.",
                request.Context.Feature,
                request.Context.Operation,
                request.Capability);

            throw;
        }
    }

    public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        AiRequest request,
        [System.Runtime.CompilerServices.EnumeratorCancellation]
        CancellationToken stopToken = default)
    {
        ArgumentNullException.ThrowIfNull(request);

        using var activity = StartActivity(request);
        var client = clientResolver.Resolve(request.Capability);

        await foreach (var update in client.GetStreamingResponseAsync(
                           request.Messages,
                           request.Options,
                           stopToken))
        {
            yield return update;
        }

        activity?.SetStatus(ActivityStatusCode.Ok);
    }

    private static Activity? StartActivity(AiRequest request)
    {
        var activity = ActivitySource.StartActivity(
            "ai.execute",
            ActivityKind.Client);

        activity?.SetTag("ai.feature", request.Context.Feature);
        activity?.SetTag("ai.operation", request.Context.Operation);
        activity?.SetTag("ai.prompt.version", request.Context.PromptVersion);
        activity?.SetTag("ai.capability", request.Capability.ToString());
        activity?.SetTag("enduser.id", request.Context.UserId);
        activity?.SetTag("tenant.id", request.Context.TenantId);

        return activity;
    }
}

This is deliberately modest. It creates one trace around execution, resolves the right client and records usage once a response is available. The gateway doesn't build prompts. It doesn't inspect claim data. It doesn't decide whether a model answer can update business state.

Those decisions remain in the feature.

Keep prompt construction close to the feature

Prompt centralisation can sound attractive because it removes strings from handlers. It can also create a distant prompt catalogue that is hard to change safely because the prompt and the feature no longer evolve together.

A feature specific prompt builder keeps the relationship visible:

public interface ISubmissionSummaryPrompt
{
    AiRequest Create(
        Submission submission,
        string correlationId);
}

public sealed class SubmissionSummaryPrompt
    : ISubmissionSummaryPrompt
{
    private const string Version = "3";

    public AiRequest Create(
        Submission submission,
        string correlationId)
    {
        var messages = new[]
        {
            new ChatMessage(
                ChatRole.System,
                """
                Produce a concise submission summary for an underwriter.
                Use only the supplied material.
                State clearly when a material fact is missing.
                """),
            new ChatMessage(
                ChatRole.User,
                submission.ExtractedText)
        };

        return new AiRequest(
            Capability: AiCapability.FastText,
            Messages: messages,
            Context: new AiExecutionContext(
                Feature: "Submissions",
                Operation: "Summarise",
                PromptVersion: Version,
                TenantId: submission.TenantId,
                CorrelationId: correlationId));
    }
}

The handler becomes straightforward:

public sealed class SummariseSubmissionHandler(
    IAiGateway aiGateway,
    ISubmissionSummaryPrompt prompt)
{
    public async Task<string> HandleAsync(
        Submission submission,
        string correlationId,
        CancellationToken stopToken)
    {
        var request = prompt.Create(submission, correlationId);

        var response = await aiGateway.GetResponseAsync(
            request,
            stopToken);

        return response.Text;
    }
}

This arrangement gives the gateway the metadata it needs without asking it to understand the submission domain. Prompt versions can still be stored externally if the team needs runtime rollout or experimentation. The same ownership rule applies. A feature should choose the prompt identity and understand the output contract. The gateway can retrieve the requested version or attach its resolved version to telemetry.

Model routing should follow capability

Hard coding model names in features makes model changes expensive. It also encourages developers to choose deployments based on guesswork. A capability router can make the operational decision centrally:

public sealed record AiModelRoute(
    string ClientName,
    TimeSpan Timeout,
    int MaximumAttempts);

public interface IAiRouteProvider
{
    AiModelRoute GetRoute(AiCapability capability);
}

public sealed class ConfigurationAiRouteProvider(
    IOptions<AiRoutingOptions> options)
    : IAiRouteProvider
{
    public AiModelRoute GetRoute(AiCapability capability)
    {
        if (options.Value.Routes.TryGetValue(
                capability,
                out var route))
        {
            return route;
        }

        throw new InvalidOperationException(
            $"No AI route is configured for '{capability}'.");
    }
}

Configuration might look like this:

{
  "AiRouting": {
    "Routes": {
      "FastText": {
        "ClientName": "fast",
        "TimeoutSeconds": 20,
        "MaximumAttempts": 2
      },
      "StructuredExtraction": {
        "ClientName": "structured",
        "TimeoutSeconds": 45,
        "MaximumAttempts": 2
      },
      "ComplexReasoning": {
        "ClientName": "reasoning",
        "TimeoutSeconds": 90,
        "MaximumAttempts": 1
      }
    }
  }
}

This gives operations teams room to move a capability to another deployment without editing every slice. It also makes cost control less intrusive. A fast classification feature can remain mapped to a cheaper model while a small number of operations use a more capable one. Routing should still be observable. Every trace and usage record should capture the resolved client and model, not only the requested capability. Otherwise, a configuration change can alter quality or cost without leaving enough evidence to explain it.

Be careful with retries

Model providers are remote dependencies, so transient failures are inevitable. That doesn't mean every failure should be retried. A timeout before any response arrives may be safe to retry. A streaming response that fails halfway through is different because the caller may already have received content. Tool calling flows can be more dangerous again because a previous attempt may have performed a side effect. The retry decision needs to consider the operation, not only the HTTP status code.

For simple, non-streaming generation, a small retry policy can be reasonable. Microsoft.Extensions.Resilience and Microsoft.Extensions.Http.Resilience provide modern .NET resilience pipelines built on Polly, including retry, timeout and circuit-breaker strategies.

The gateway can apply an execution timeout around a request:

public sealed class ResilientAiGateway(
    IAiGateway innerGateway,
    ResiliencePipelineProvider<string> pipelines)
    : IAiGateway
{
    public Task<ChatResponse> GetResponseAsync(
        AiRequest request,
        CancellationToken stopToken = default)
    {
        var pipeline = pipelines.GetPipeline(
            request.Capability.ToString());

        return pipeline.ExecuteAsync(
            async token => await innerGateway.GetResponseAsync(
                request,
                token),
            stopToken).AsTask();
    }

    public IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        AiRequest request,
        CancellationToken stopToken = default)
    {
        return innerGateway.GetStreamingResponseAsync(
            request,
            stopToken);
    }
}

The streaming path is left alone here on purpose. Retrying a stream invisibly can duplicate output. A production design needs an explicit restart behaviour that the caller understands. The same caution applies to tools. If an AI operation can call a tool that changes data, idempotency belongs at the tool boundary. A gateway retry should never be expected to make a non-idempotent operation safe.

Put telemetry at the boundary

AI features are difficult to support when every trace ends at IChatClient.GetResponseAsync. The gateway knows which feature made the request. It knows the capability, prompt version and resolved model. It can also capture latency and usage. That makes it the natural place to create a span around model execution. Microsoft.Extensions.AI includes an OpenTelemetry chat client implementation that follows the OpenTelemetry semantic conventions for generative AI systems. That instrumentation can sit underneath the application gateway, while the gateway adds domain-relevant tags such as feature and operation.

A useful trace should let an engineer answer which feature made the request, which prompt version was used and which deployment handled it. It should also show how long the call took, how many tokens were consumed and whether the caller cancelled it or the provider failed. The feature should record what happened after the model returned. Operational telemetry and product-quality evaluation meet at that point.

Treat prompt and response logging as sensitive

It is tempting to log full prompts because they make debugging easy. They can also contain personal data, commercial information, secrets, retrieved documents and internal instructions. The Microsoft.Extensions.AI logging implementation warns that trace-level logging can include chat messages and options, which may hold sensitive application data. Trace logging should not be switched on casually in production. A safer default is to record metadata without recording content:

public sealed record AiUsageRecord(
    string Feature,
    string Operation,
    string PromptVersion,
    string Capability,
    string? ModelId,
    long? InputTokens,
    long? OutputTokens,
    TimeSpan Duration,
    DateTimeOffset OccurredAt);

When full content is required for a controlled evaluation environment, make that an explicit data path with retention rules and access controls. Don't let verbose logging quietly become the organisation's prompt archive.

Cost tracking needs business dimensions

Provider dashboards can show total token consumption. They rarely know why the application spent it.

The gateway can attach usage to dimensions the business understands:

Feature: Submissions
Operation: ExtractRiskDetails
Tenant: 42
PromptVersion: 7
Capability: StructuredExtraction
Model: resolved-model-name

This makes cost questions answerable. A team can see whether summarisation, extraction or an agent workflow caused an increase. It can compare prompt versions. It can identify one tenant generating unusually large contexts. It can decide whether a capability belongs on a cheaper model. Avoid calculating an authoritative financial value inside the request path unless pricing data is reliable and versioned. Token prices can change and some providers use more complex billing rules. Recording raw usage and the resolved model gives a later billing process enough information to calculate cost consistently.

Caching is more awkward than it looks

AI response caching sounds like an obvious gateway responsibility. Sometimes it is. A deterministic classification over immutable input can be a good candidate. A response based on changing retrieval data may become stale quickly. A user-specific answer can leak data if the cache key doesn't include the right identity and authorisation context. A safe cache key needs more than the prompt text. It should account for the feature, operation, prompt version, model route, normalised input, security scope and output schema. Even that doesn't make every request cacheable. Let the feature declare whether an operation is eligible, then let the gateway implement the caching mechanism. That keeps the business decision close to the feature and the storage plumbing inside the gateway.

public sealed record AiCachePolicy(
    bool Enabled,
    TimeSpan? TimeToLive = null);

public sealed record AiRequest(
    AiCapability Capability,
    IReadOnlyList<ChatMessage> Messages,
    AiExecutionContext Context,
    ChatOptions? Options = null,
    AiCachePolicy? Cache = null);

The default should be no cache. Reuse should be deliberate.

Rate limits belong at more than one level

Provider quotas protect the provider account. They don't necessarily protect one feature from consuming the application's entire allowance. A gateway can apply limits by capability, tenant or operation. A background document pipeline may be allowed more concurrency than an interactive endpoint, while a single tenant may need a daily usage ceiling. For larger estates, Azure API Management can enforce policies outside the application and provide a shared gateway across several systems. The in-process gateway still has value because it understands feature names, prompt versions and application-level outcomes.

The two layers can work together:

The local gateway owns application context and feature policy. The infrastructure gateway owns organisation wide access, quotas, backend pools and external governance. Building both layers on day one would be excessive for many teams. The diagram describes an evolution path, not a minimum architecture.

Structured output still needs feature validation

A gateway can request structured output, but it cannot decide whether the output makes sense for a particular domain.

Suppose a model produces this result:

public sealed record ClaimExtraction(
    string ClaimReference,
    decimal EstimatedLoss,
    DateOnly LossDate);

JSON deserialisation can prove the response has the right shape. It cannot prove the loss date is allowed, that the claim reference exists or that the current user can access it.

The feature should validate the response before it affects state:

public sealed class ExtractClaimDetailsHandler(
    IAiGateway aiGateway,
    IClaimExtractionPrompt prompt,
    IClaimRepository claims,
    IValidator<ClaimExtraction> validator)
{
    public async Task<ClaimExtraction> HandleAsync(
        ExtractClaimDetails command,
        CancellationToken stopToken)
    {
        var request = prompt.Create(command);

        var response = await aiGateway.GetResponseAsync(
            request,
            stopToken);

        var extraction = response.Text.Deserialize<ClaimExtraction>()
            ?? throw new InvalidOperationException(
                "The model returned an empty extraction.");

        await validator.ValidateAndThrowAsync(
            extraction,
            cancellationToken);

        var claimExists = await claims.ExistsAsync(
            extraction.ClaimReference,
            stopToken);

        if (!claimExists)
        {
            throw new InvalidOperationException(
                "The extracted claim reference does not exist.");
        }

        return extraction;
    }
}

An LLM response remains untrusted input, even when the provider guarantees valid JSON.

Don't turn the gateway into an agent framework

Once a gateway exists, it can become the place where every AI-related concern is deposited. Tool orchestration appears. Retrieval follows. Then memory, workflow state and approval logic arrive. Before long, every feature goes through a central service that understands the entire system. That creates the same coupling the gateway was supposed to reduce. A gateway should stay focused on model execution and shared policy. An agent runtime can call the gateway. A retrieval feature can call the gateway. A Durable Functions orchestration can call the gateway. None of those workflows needs to live inside it. Think of the gateway as the controlled road to model providers. It doesn't need to become the destination.

When an in-process gateway is enough

A separate service adds deployment, networking, authentication and operational overhead. Keep the gateway inside the application when the model access belongs to one system, the team owns the features together and there is no requirement to share policy across many applications. An in-process gateway still provides a clean boundary. It can be extracted later if the organisation genuinely needs a shared platform. A remote gateway becomes more attractive when many applications use the same provider estate, central governance is required or model credentials must be isolated from application teams. It can also help when a platform team owns routing and quotas for the wider organisation. The transition should be driven by ownership and scale rather than the idea that gateways must be network services.

Registering the gateway

A composition root can register named clients and the gateway without leaking provider SDKs into feature projects.

The exact provider registration will vary, but the overall shape can remain stable:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing.AddSource("DotNetDigest.AiGateway");
    });

builder.Services.AddSingleton<IChatClientResolver>(services =>
{
    var clients = new Dictionary<AiCapability, IChatClient>
    {
        [AiCapability.FastText] =
            services.GetRequiredKeyedService<IChatClient>("fast"),

        [AiCapability.StructuredExtraction] =
            services.GetRequiredKeyedService<IChatClient>("structured"),

        [AiCapability.ComplexReasoning] =
            services.GetRequiredKeyedService<IChatClient>("reasoning")
    };

    return new ChatClientResolver(clients);
});

builder.Services.AddScoped<IAiUsageRecorder, AiUsageRecorder>();
builder.Services.AddScoped<IAiGateway, AiGateway>();

builder.Services.AddScoped<
    ISubmissionSummaryPrompt,
    SubmissionSummaryPrompt>();

builder.Services.AddScoped<SummariseSubmissionHandler>();

Keyed services are a useful fit when several implementations of the same abstraction are registered. Feature code still asks for IAiGateway; only the composition root needs to understand the concrete client arrangement. For smaller applications, a single IChatClient can sit behind the resolver. The abstraction doesn't require multiple models. It simply leaves room for them.

A practical rollout path

The worst time to build a platform is before the application has shown what it needs. Start by introducing a gateway around the calls you already have. Capture feature, operation, prompt version, resolved model, latency and usage. Keep prompt construction and validation in their current slices. Once the traffic is visible, model routing can follow. You may discover that only two capabilities are needed. You may also find that the expensive model is being used for work a smaller model handles just as well. Add resilience after classifying failure modes. Don't apply one retry policy to streaming, tool execution and ordinary generation. Introduce caching only for operations with a clear freshness and security model. Move the gateway out of process only when organisational ownership or shared governance makes the extra boundary worthwhile.

That sequence lets the architecture grow from evidence.

What the gateway should leave alone

A healthy gateway has a narrow set of reasons to change. It changes when provider integration changes. It changes when routing, resilience, telemetry or usage policy changes. It shouldn't change because the claims team added a validation rule. It shouldn't change because the submission prompt gained another instruction. It shouldn't know how an underwriting decision is made. That separation is what keeps it useful.

Calling an AI model from .NET is now straightforward. Keeping twenty AI-backed features consistent is where the engineering begins. An AI gateway gives the application a place to own that consistency. It can hide provider details, route capabilities, apply measured resilience and produce telemetry that operations teams can use. The boundary works when it remains disciplined. Features describe the work, build the prompt and validate the answer. The gateway executes the request under a common set of operational rules.

For a small application, that can be one in-process class around IChatClient. For a larger system, it may grow into a shared platform backed by Azure API Management or another infrastructure gateway. Either way, the value comes from giving every AI call the same controlled path before duplicated plumbing becomes part of the architecture.

• Microsoft.Extensions.AI libraries

• Use the IChatClient interface

• AI gateway capabilities in Azure API Management

• Build resilient HTTP apps with .NET

• .NET observability with OpenTelemetry