For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Protect provider credentials with virtual keys, then add optional token budgets and usage tracking.

Keep provider API keys on the gateway and give each client a revocable virtual key. Add token budgets and usage tracking as needed.

About

Clients use virtual keys to authenticate to agentgateway. The gateway replaces each virtual key with the provider key before forwarding the request. Clients never receive the provider key, and the provider never receives the virtual key.

Each client has its own key, so you can revoke one client’s access without rotating the provider key or reconfiguring other clients. Provider-key rotation happens on the gateway host.

How virtual keys work

    sequenceDiagram
    participant C as Client<br/>(holds a virtual key)
    participant AGW as Agentgateway<br/>(holds the provider key)
    participant P as LLM provider
    C->>AGW: Request with the virtual key
    AGW->>AGW: Validate the key and check model access
    AGW->>AGW: Replace the virtual key with the provider key
    AGW->>P: Request with the provider key
    P-->>AGW: Response
    AGW-->>C: Response
  

Security considerations

The gateway host holds every provider key. Protect it as you would any other secret store.

• Restrict access to the files. Run agentgateway as a dedicated user, and make the key file and the configuration file readable only by that user.
• Do not store raw virtual keys. Store a keyHash for each virtual key instead of the raw key, so that a copy of the configuration file does not contain usable keys.
• Encrypt client traffic. Virtual keys are bearer credentials. Serve clients over TLS, or keep the gateway on a private network.
• Protect the UI and admin API. Anyone who can reach them can create and read virtual keys. For more information, see Secure the UI.
• Keep agentgateway up to date with security patches.

Before you begin

You also need an OpenAI API key. The examples read it from OPENAI_API_KEY. For file-based credentials and rotation without a restart, see Provider API keys.

Set up virtual keys

Turn on API key authentication, and create two virtual keys, one for Alice and one for Bob. You can do this interactively in the built-in UI, or declaratively in your config file. The UI is convenient for exploring and one-off changes. The config file is the source of truth for GitOps workflows.

Step 1: Create the virtual keys

cat <<'EOF' > config.yaml
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
llm:
  policies:
    apiKey:
      mode: strict
      keys:
      - key: vk-alice-4f9c2a
        metadata:
          user: alice
      - key: vk-bob-8d1e7b
        metadata:
          user: bob
  models:
  - name: "*"
    provider: openAI
    params:
      apiKey: "$OPENAI_API_KEY"
EOF

agentgateway -f config.yaml

Step 2: Test the virtual keys

You can test the keys in the UI playground or with curl. Both work the same way, whether you created the keys in the UI or in the config file.

Manage client access

Control access for individual clients without reconfiguring the others. You can also rotate the provider key without changing any client.

Restrict model access

The * model serves every model the provider offers. To restrict access for all clients, replace it with named model entries.

For model patterns and budget interactions, see Limit model access per key.

Revoke a virtual key

Revoke one client’s access without restarting agentgateway or changing other keys.

Rotate a provider key

Replace the provider credential on the gateway host. Clients keep their virtual keys.

1. Follow Rotate a provider key to replace the credential and revoke the old provider key.
2. Send a request with an existing virtual key. Verify that it still succeeds.

Configure token budgets

Token budgets are optional. Continue here to limit token usage across clients, then give each key its own budget. Keep your existing keys, provider credentials, and model restrictions when adding these policies.

How rate limiting works

Agentgateway checks token-based rate limits in two phases:

At request time:

• When tokenize: true is not set or is set to false on the AI backend, the number of tokens that are used for the request cannot be calculated. Because of this, the request is always allowed, unless the rate limit is set to 0 tokens. The LLM typically returns the number of tokens that were used for the request when sending the response. Agentgateway verifies the number of tokens that were used in the request and the response to determine whether the rate limit was reached. By default, tokenize is set to false.
• When tokenize: true is set, agentgateway estimates the number of tokens at request time. Because of that, the request is only allowed if the estimated number of tokens does not exceed the set rate limit.

At response time:

When the LLM returns a response, it typically provides the number of tokens that were used during the request and response. Agentgateway uses these numbers to determine if the rate limit was reached.

Note that this determination happens after the response is returned. Even, if the number of tokens that are used in the response exceeds the number of allowed tokens, the response is still returned to the user. Only subsequent requests are rate limited. If tokenize: true is set on the AI backend and tokens were estimated during the request, agentgateway verifies the actual number of tokens that were used for the request when the LLM returns its response. In the case the initial estimation was off, agentgateway adjusts the number of used tokens to count these against the set rate limit.

Step 1: Add a token budget

llm:
  policies:
    localRateLimit:
    - maxTokens: 10
      tokensPerFill: 1
      fillInterval: 60s
      type: tokens

Step 2: Verify rate limits

1. Start agentgateway with the updated configuration.

   agentgateway -f config.yaml

2. Send a prompt to the LLM. At the time the prompt is sent, the number of tokens required for the completion is unknown. Make sure to include a virtual key in the authorization header. Because tokenize: true is not set on the model, the prompt count is not estimated. As a result, the prompt is allowed.

   Note

   The LLM typically returns the number of tokens required for completion in its response. Agentgateway uses this number and counts it against the rate limit.

   curl http://localhost:4000/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -H 'Authorization: Bearer vk-alice-4f9c2a' \
     -d '{
       "model": "gpt-3.5-turbo",
       "messages": [
         {
           "role": "user",
           "content": "Tell me a short story"
         }
       ]
     }'

   Example output:

   {
     "choices": [
       {
         "message": {
           "content": "Once upon a time, in a small village nestled between towering mountains...",
           "role": "assistant"
         },
         "finish_reason": "stop"
       }
     ],
     "usage": {
       "prompt_tokens": 12,
       "completion_tokens": 248,
       "total_tokens": 260
     }
   }

3. Repeat the same request. This time, the request is rate limited because the tokens used in the first request exceeded the budget.

   curl http://localhost:4000/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -H 'Authorization: Bearer vk-alice-4f9c2a' \
     -d '{
       "model": "gpt-3.5-turbo",
       "messages": [
         {
           "role": "user",
           "content": "Tell me a short story"
         }
       ]
     }'

   Example output:

   rate limit exceeded

Step 3: Enable request-time token estimation

By default, agentgateway does not estimate token counts at request time. To reject requests before they reach the LLM, set tokenize: true on your model.

For more information about rate limiting configuration options, see Rate limits.

Then, verify the change.

1. Send a request. Use Bob’s virtual key because Alice’s virtual key already reached the rate limit in the previous step. This time, the request is rate limited because the estimated tokens exceed the budget.

   curl http://localhost:4000/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -H 'Authorization: Bearer vk-bob-8d1e7b' \
     -d '{
       "model": "gpt-3.5-turbo",
       "messages": [
         {
           "role": "user",
           "content": "Tell me a short story"
         }
       ]
     }'

   Example output:

   rate limit exceeded

Step 4: Give each key its own budget

So far, all of the virtual keys share one token budget, so the busiest key exhausts the budget for everyone else. To give each key its own budget, add a key field to the rate limit rule. The key field takes a CEL expression, and each distinct value that the expression returns gets its own token bucket with the limits of that rule. Use apiKey.user to read the user metadata from the authenticated key.

Then, verify that each key has its own budget.

1. Send a prompt with Alice’s virtual key. The request is allowed, because the keyed rule starts Alice with a full bucket.

   curl http://localhost:4000/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -H 'Authorization: Bearer vk-alice-4f9c2a' \
     -d '{
       "model": "gpt-3.5-turbo",
       "messages": [
         {
           "role": "user",
           "content": "Tell me a short story"
         }
       ]
     }'

   Example output:

   {
     "choices": [
       {
         "message": {
           "content": "Once upon a time, in a small village nestled between towering mountains...",
           "role": "assistant"
         },
         "finish_reason": "stop"
       }
     ],
     "usage": {
       "prompt_tokens": 12,
       "completion_tokens": 248,
       "total_tokens": 260
     }
   }

2. Repeat the same request to exhaust Alice’s budget. The tokens that the first response used exceed the 10-token bucket, so this request is rate limited.

   curl http://localhost:4000/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -H 'Authorization: Bearer vk-alice-4f9c2a' \
     -d '{
       "model": "gpt-3.5-turbo",
       "messages": [
         {
           "role": "user",
           "content": "Tell me a short story"
         }
       ]
     }'

   Example output:

   rate limit exceeded

3. Send the same request with Bob’s virtual key. Unlike in Step 3, where Alice’s usage rate limited Bob, the request succeeds, because Bob’s key now has its own token bucket that Alice’s requests do not draw from.

   curl http://localhost:4000/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -H 'Authorization: Bearer vk-bob-8d1e7b' \
     -d '{
       "model": "gpt-3.5-turbo",
       "messages": [
         {
           "role": "user",
           "content": "Tell me a short story"
         }
       ]
     }'

   Example output:

   {
     "choices": [
       {
         "message": {
           "content": "Once upon a time, in a small village nestled between towering mountains...",
           "role": "assistant"
         },
         "finish_reason": "stop"
       }
     ],
     "usage": {
       "prompt_tokens": 12,
       "completion_tokens": 248,
       "total_tokens": 260
     }
   }

Review the following behavior before you rely on a keyed budget.

• Keys that cannot be evaluated: Requests whose key is empty, or whose expression cannot be evaluated, such as an unauthenticated request when the key reads apiKey.user, all share one bucket. An empty key does not exempt a request from the limit.
• How many buckets are kept, and where they live: Each rule keeps up to 65,536 buckets and drops the least used ones, which for that key is the same as never having been seen. Buckets are held in memory by a single agentgateway instance, so each instance enforces the limit separately. For a virtual key whose bucket was dropped, the next request starts from a full bucket. For a budget that is shared across instances, use remote rate limits.

For more information, including how to key a limit on a JWT claim or on the requested model, see Per-key limits.

Monitor per-key spending

Usage tracking is optional and does not require a token budget.

Agentgateway exposes token usage as Prometheus metrics on its stats endpoint, which listens on port 15020 by default (not the 15000 admin port). The agentgateway_gen_ai_client_token_usage metric is a histogram that records the tokens used per request.

By default, this metric is broken down by dimensions such as the model (gen_ai_request_model) and token type (gen_ai_token_type), but not by key. To attribute usage to each virtual key, add a label such as user_id that reads the user metadata from the authenticated key, then query Prometheus.

Add a per-key metric label

You can add a per-key metric label such as to track metrics by user ID. Note that the user_id label is high cardinality: every unique value creates a new metric series, which increases Prometheus memory and storage. This is acceptable for tens or hundreds of keys, but avoid attaching unbounded identifiers at large scale. Prefer lower-cardinality dimensions like tier or team when possible.

1. Update your configuration to add a user_id metric label. The add field maps a label name to a CEL expression that is evaluated per request. Use apiKey.user to read the user metadata from the authenticated key. Add this block to config.yaml without replacing the existing authentication or budget policies.

   config:
     metrics:
       fields:
         add:
           user_id: apiKey.user

   Setting	Description
   config.metrics.fields.add	A map of metric label names to CEL expressions. Each expression is evaluated per request and its result is attached as a Prometheus label on the metrics.
   user_id: apiKey.user	Adds a user_id label whose value is the user metadata from the authenticated API key. If the expression cannot be evaluated (for example, on an unauthenticated request), the label value is unknown.

2. Restart agentgateway with the updated configuration, then send successful requests with each key so the metrics have per-key data to report. Because the earlier rate-limit budget is small, raise maxTokens or wait for the bucket to refill so the requests are not rejected.

3. Verify that the user_id label is attached to the token usage metric. Read the metrics endpoint and filter for the token usage sum.

   curl -s http://localhost:15020/metrics | grep gen_ai_client_token_usage_sum

   Each series now carries a user_id label that matches the user metadata of the key that made the request. For example, after sending requests with Alice’s and Bob’s keys:

   agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="input",gen_ai_request_model="gpt-3.5-turbo",...,user_id="alice"} 21.0
   agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="output",gen_ai_request_model="gpt-3.5-turbo",...,user_id="alice"} 14.0
   agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="input",gen_ai_request_model="gpt-3.5-turbo",...,user_id="bob"} 9.0
   agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="output",gen_ai_request_model="gpt-3.5-turbo",...,user_id="bob"} 9.0

Query with Prometheus

The raw curl output in the previous step is a quick sanity check, but it returns only Prometheus exposition text. To run aggregations such as totals over time or estimated cost, use PromQL. PromQL runs inside a Prometheus server that scrapes the agentgateway metrics endpoint. You cannot send PromQL to the /metrics endpoint directly, so in standalone mode you run your own Prometheus.

1. Create a Prometheus scrape configuration that targets the agentgateway stats endpoint.

   cat <<'EOF' > prometheus.yml
   global:
     scrape_interval: 5s
   
   scrape_configs:
   - job_name: agentgateway
     static_configs:
     - targets: ["host.docker.internal:15020"]
   EOF

   Note

   Use host.docker.internal:15020 when you run Prometheus in Docker, as in the next step. If you run the Prometheus binary directly on your machine, use localhost:15020 instead.

2. Start Prometheus with this configuration.

   docker run --rm -p 9090:9090 \
     -v "$(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml" \
     prom/prometheus

3. Verify that Prometheus is scraping agentgateway. Open http://localhost:9090/targets and confirm that the agentgateway target is UP. You might need to restart the container or refresh the page.

   

   

4. Query token usage per key. The following query returns the total tokens consumed by each virtual key. The token usage metric carries a separate series per token type, so match both input and output in a single selector with =~ and sum them.

   QueryPrometheus UIcurl

   sum by (user_id) (agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type=~"input|output"})

   Open the Prometheus expression browser and run a PromQL query

   

   

   curl -s http://localhost:9090/api/v1/query \
     --data-urlencode 'query=sum by (user_id) (agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type=~"input|output"})' \
     | jq .

   Example output:

   {
     "status": "success",
     "data": {
       "resultType": "vector",
       "result": [
         {
           "metric": {
             "user_id": "bob"
           },
           "value": [
             1783540709.896,
             "253"
           ]
         },
         {
           "metric": {
             "user_id": "alice"
           },
           "value": [
             1783540709.896,
             "262"
           ]
         }
       ]
     }
   }

5. Estimate costs by multiplying token counts by your provider’s pricing. Input and output tokens are usually priced differently, so aggregate each type separately, then add the two results. Each sum by (user_id) collapses the gen_ai_token_type label, so the + matches on user_id.

   Note

   Queries that use rate() or increase() over a range such as [24h] need that much scrape history to return meaningful values. While testing locally, query the raw _sum counters or use a shorter range such as [5m].

   This query estimates cost for OpenAI GPT-3.5 by applying your own pricing to token counts. To have agentgateway compute the realized USD cost per request from a model cost catalog, see Model costs.

   QueryPrometheus UIcurl

   # Estimated cost per key (assuming $0.50 per 1M input tokens, $1.50 per 1M output tokens)
   sum by (user_id) (agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="input"} / 1000000 * 0.50)
   +
   sum by (user_id) (agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="output"} / 1000000 * 1.50)

   Open the Prometheus expression browser and run a PromQL query

   

   

   curl -s http://localhost:9090/api/v1/query \
     --data-urlencode 'query=(sum by (user_id) (agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="input"} / 1000000 * 0.50))+(sum by (user_id) (agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="output"} / 1000000 * 1.50))' \
     | jq .

   Example output:

   {
     "status": "success",
     "data": {
       "resultType": "vector",
       "result": [
         {
           "metric": {
             "user_id": "bob"
           },
           "value": [
             1783541276.199,
             "0.0003675"
           ]
         },
         {
           "metric": {
             "user_id": "alice"
           },
           "value": [
             1783541276.199,
             "0.000381"
           ]
         }
       ]
     }
   }