Configuration

Two things configure omnigate:

• One JSON file, read by the control plane. It holds gateway keys, upstreams, routes, the media section and retention.
• Environment variables for each process: addresses, tokens, file paths, timeouts, and every provider secret.

Gateway keys can also be created at run time through the admin API or the dashboard. Those live in a separate keys file, not in the config file; see Admin.

Provider secrets are never in the JSON file. Each upstream names an environment variable (api_key_env) and the process that calls that provider reads it: text upstream keys are read by the data plane, media upstream keys and bucket credentials by the control plane.

Contents:

• The config file
  • Gateway keys
  • Text upstreams
  • Text routes
  • Response cache
  • Retention
  • Media
• Environment variables
• What the data plane receives

The config file

The control plane loads the file named by OMNIGATE_CONFIG at startup and checks its modification time every 2 seconds. When the file changes it is loaded again; no restart is needed. The data plane picks the change up on its next poll (OMNIGATE_POLL_MS, 2 seconds by default).

Behaviour on a bad file:

• At startup, a file that is missing, is not valid JSON, lacks a required field or has an invalid value stops the control plane from starting.
• On reload, the control plane logs config reload failed, keeping previous: ... and keeps serving the previous config. It does not try that version of the file again; fix the file and save it.

Templates: config/omnigate.dev.json (mock providers, used by the quickstart) and config/omnigate.example.json (real providers; replace every <...> placeholder).

Top-level shape:

{
  "keys": [],
  "upstreams": [],
  "routes": [],
  "media": {},
  "retention": {},
  "credits": {},
  "payments": {}
}

Every top-level section is optional and defaults to empty.

Gateway keys

keys[] are the credentials applications present to the data plane. Keys written here are "file keys": they are read-only for the admin API, and are listed before the managed keys created through it (Admin).

max_concurrent is new with the close-out commit (the one after 55a294d). Until then every key had the data plane's OMNIGATE_MAX_CONCURRENT_PER_KEY (see Environment variables) and nothing else; a key without the field still does.

A key is any string; it is only ever compared by its SHA-256. The LiteLLM importer generates keys as og- followed by random characters, which is a reasonable convention:

python -c "import secrets; print('og-' + secrets.token_urlsafe(32))"

To get the value for key_sha256:

python -c "import hashlib,sys; print(hashlib.sha256(sys.argv[1].encode()).hexdigest())" 'og-your-key-here'

The hash must be lowercase hex, which is what the command prints.

Example:

{
  "name": "my-app",
  "key_sha256": "<sha256 hex of the gateway key you hand to the app>",
  "budget_usd": 25.0,
  "budget_period": "monthly",
  "rpm": 600,
  "max_concurrent": 64,
  "max_open_jobs": 20,
  "max_jobs_per_day": 2000,
  "models": ["claude", "gpt", "gemini", "image"],
  "webhook_secret_env": "MY_APP_WEBHOOK_SECRET"
}

Text upstreams

upstreams[] are the LLM providers the data plane forwards to.

| attestation | object | absent | The provider proves what hardware and configuration it runs on, and its targets are only used while that is verified: {"provider": "near", "accepted_tcb": [...], "gpu": "if_present", "max_age_s": 900, "measurements": {...}}. Every field, what is verified and what is not: Attested upstreams. A setting that is not understood is rejected when the file is loaded. |

How the key is sent: openai upstreams get Authorization: Bearer <key>; anthropic upstreams get x-api-key: <key> and an anthropic-version header (the client's, or 2023-06-01 if the client sent none). If the named variable is not set in the data plane's environment, an empty key is sent and the provider will refuse the request.

kind is checked when the file is loaded: a value other than openai or anthropic is refused with a message naming the upstream (at startup the control plane does not start; on a reload the previous config stays in force). Media providers do not go here; they go under media.upstreams.

Text routes

routes[] map the model name a client sends to one or more targets.

Each target:

A config that sets only the input and output prices bills every input token at the input price, cached or not. How the four token counts are told apart is in Operations.

Whole-number settings are read strictly. rpm, timeout_ms, cache_ttl_s and weight go into 32-bit fields in the data plane, where one value that does not fit makes the whole snapshot unreadable. Each is therefore a whole number within the range in its table, and so are the whole-number settings of the media section below. A fraction (600.0), a string, a negative number or a value above the range is rejected when the file is loaded, with a message that names the setting; until commit d79406b a fraction was cut to a whole number without a word. A price above 1,000,000,000, or a budget_usd above 1,000,000,000,000, is rejected too.

A usage event for a route or target the config does not have (or no longer has) is recorded at $0, counted in omnigate_ledger_unpriced_events_total and logged once per route and target.

Strategies:

• failover: config order. The first target takes all the traffic; the others are used when it fails.
• round_robin: the starting target rotates with every request; the rest follow in ring order as failover.
• weighted: the starting target is drawn at random in proportion to each target's weight; the rest follow in config order as failover. At least one target must have a weight above 0.

Whatever the strategy, targets whose upstream is cooling down after repeated failures are moved to the end of the order. Rotation counters and cooldowns live in memory in each data-plane process.

A route may mix openai and anthropic upstreams. For /v1/chat/completions and /v1/messages the gateway translates when the target speaks the other protocol; for the other endpoints a target of the other protocol is skipped (protocol_mismatch), and if that leaves no target at all the request is refused with 400 endpoint_not_supported.

Example with two providers behind one name:

{
  "model": "claude",
  "strategy": "weighted",
  "targets": [
    { "upstream": "anthropic", "model": "claude-sonnet-5-5", "weight": 3, "price_in_per_mtok": 2.0, "price_cached_in_per_mtok": 0.1, "price_out_per_mtok": 10.0 },
    { "upstream": "openai", "model": "<openai model id>", "weight": 1, "price_in_per_mtok": 0.0, "price_out_per_mtok": 0.0 }
  ]
}

The prices above are the ones in config/omnigate.example.json; check them against your provider's price list before relying on the spend figures.

Response cache

A text route with cache_ttl_s above 0 lets each data plane remember successful answers and reuse them:

{ "model": "faq", "cache_ttl_s": 600, "targets": [ { "upstream": "openai", "model": "<openai model id>" } ] }

• What is cached. Non-streaming answers with a 2xx status on the metered endpoints (/v1/chat/completions, /v1/messages, /v1/completions, /v1/embeddings, /v1/responses). Requests with "stream": true are never looked up or stored.
• What counts as the same request. The same gateway key, the same route name, the same endpoint path and the same JSON body. Key order inside objects and whitespace do not matter; array order and every value do. The match is exact: there is no semantic matching.
• Never across keys. The gateway key is part of the identity, so one key is never served an answer another key paid for.
• Checks still apply. A hit is looked up after authentication, the budget, the key's concurrency and memory limits, the rate limit and the model list, and counts against the rate limit like any request.
• Some requests are never cached. The cache key is the request in canonical form, which takes parsing it into a tree. A body with more than 1,024 members in one object, or one whose tree would pass its memory cap or the tree budget (see OMNIGATE_PARSE_MEMORY_MB), is served normally but neither looked up nor stored, and its response carries no x-omnigate-cache header.
• Billing. A hit is recorded in the ledger as a request with no tokens and no cost, marked as served from cache.
• Bounds. The cache lives in the memory of each data-plane process: OMNIGATE_CACHE_MAX_MB (64 by default; 0 turns caching off whatever the routes say). The least recently used answers are evicted first, an answer larger than a quarter of that budget is not kept, and a restart starts empty. Several data planes each have their own cache.
• There is no per-request switch to force a refresh. Removing cache_ttl_s from the route stops lookups as soon as the data planes have polled the new config. A key with privacy mode's no_store or redact, and a request that sends x-omnigate-privacy: no-store (or redact, or on), is never answered from the cache and never stored in it.

Responses carry x-omnigate-cache: hit or miss; see API. This is separate from the media job cache (media.routes[].cache_ttl_s, below), which stores output files in the control plane and is shared across keys.

Retention

"retention": { "jobs_days": 30, "ledger_ids_days": 3, "ledger_days": 90, "delete_bucket_objects": true, "private_outputs_hours": 24 }

No job, file or ledger segment is deleted unless you set these, with two exceptions that have a default. One only concerns jobs created under privacy mode's no_store, whose outputs are deleted after private_outputs_hours (a deployment that does not use privacy mode has no such job). The other is ledger_ids_days: with no retention section at all, duplicate-detection ids older than 3 days are dropped from memory and their per-day files (ledger.ids.YYYY-MM-DD) are deleted. That touches no total and no file of events; what it costs, and when to raise it, is described in Operations. A negative number, or a value that is not a number, for one of the three _days fields is rejected when the file is loaded. Details and caveats are in Operations; read the one about ledger_days and backups before you set it.

Media

The media section configures async jobs. It never leaves the control plane (apart from the route names, which data planes list in GET /v1/models).

media.upstreams[]

An unknown kind, or a key variable that is not set, does not stop the file from loading. A job that reaches such a target records the attempt as unsupported_kind:<kind> or missing_api_key_env:<VARIABLE> and moves on to the next target.

media.routes[]

Each target has upstream, model (the provider's model id), an optional price object, on a sume upstream a required router ("image" or "video": which of Sume's two generate endpoints the model is on; refused on any other kind), on an elevenlabs upstream an optional api ("speech", "sound", "music", "image" or "video": which ElevenLabs product the model belongs to; speech when it is left out; refused on any other kind), and, for per-second prices, three optional settings:

min_seconds above max_seconds, default_seconds above max_seconds and a default_seconds of 0 are rejected when the file is loaded.

Per-second prices and providers that take no duration. Only the video kinds (replicate, fal, xai_video, sume) and an elevenlabs target whose api is sound, music or video are sent a duration from the request. An upstream of kind openai_images, gemini or gemini_interactions, and an elevenlabs target for speech or images, never is, so there a per_second_usd price has nothing from the request to multiply: the job is billed default_seconds (raised to min_seconds), whatever input.duration says, and max_seconds refuses nothing. A per-second price on such a target without default_seconds is rejected when the file is loaded (since the close-out commit; before, such a job was refused or billed nothing per second depending on the request). Set default_seconds, or price the target with per_job_usd or per_output_usd.

The three only matter on a target whose per_second_usd is above 0. They change what is billed and what is accepted, not what is sent to the provider, and a job is checked against every per-second target of its route when it is created, not only the one that ends up delivering.

All four price fields default to 0 and are added together. A price that is negative, or not a number, is rejected when the file is loaded. A target with no price at all records its jobs at $0, which is counted in omnigate_ledger_unpriced_events_total and logged once per target.

Seconds and characters are counted from what the adapter sends, not from one field the client chose to fill in, so a request cannot lower its own price by moving its text or its duration into another field:

For the adapters that send no duration, the seconds of a per_second_usd price come from the target's default_seconds and min_seconds alone. Each of the four parts is at least zero by itself. A job whose input.duration or input.seconds is negative or not a duration, or whose input.n, input.num_outputs or input.num_images is not a whole number of 0 or more, is refused with 400 invalid_request when it is created (API).

Cost is the configured price of the target that delivered, times the route's price multipliers. A failed or canceled job costs nothing in the ledger, and a cache hit costs nothing.

Price variants

Some providers charge a rate that depends on the request: a video model priced per second by resolution, more with audio than without. A target's price may carry variants for that:

"price": {
  "per_job_usd": 0.0,
  "variants": [
    { "when": { "resolution": "720p", "generate_audio": false }, "per_second_usd": 0.08 },
    { "when": { "resolution": "720p", "generate_audio": true },  "per_second_usd": 0.10 },
    { "when": { "resolution": "720p", "generate_audio": null },  "per_second_usd": 0.10 },
    { "when": { "resolution": "1080p" }, "per_second_usd": 0.12 }
  ]
}

(The figures are an illustration of the shape, not anyone's price list: take yours from your provider.)

• A variant applies when every field in its when equals the value of that top-level input field. Values are JSON scalars compared as JSON: strings, numbers (24 equals 24.0), true, false, and null, which stands for a field the request does not send. true does not equal "true".
• The first variant that applies is used. Its rates (per_job_usd, per_output_usd, per_second_usd, per_input_char_usd) replace the rates of the same name in the price around it; the others stay.
• A request to which no variant applies is refused with 400 invalid_request when the job is created, naming the priced combinations. This is checked against every target of the route that has variants, because the job may end up on any of them. A price with variants therefore never records a job at a rate nobody set. To price "everything else", end the list with { "when": {}, ... }.
• If the config changes while a job waits and its target no longer has a variant for it, that target is skipped (unpriced_variant) before anything is sent to its provider. A job already running there is charged the dearest of the target's variants.
• A per-second rate in a variant follows the same rules as per_second_usd outside one: the job needs a duration (from the request or default_seconds), and on a kind that is never sent a duration default_seconds is required.
• At most 500 variants per price, at most 8 fields in a when. A variant without when, without a rate, with a rate name that does not exist, or with the same when as an earlier one is rejected when the file is loaded.
• Variants and the route's price_multipliers can be combined: the multipliers apply to the price the variant gave.

Variants are covered by the control plane's unit tests (control/test/sume_test.exs) and the end-to-end suite against mock providers. They are not specific to one kind of upstream.

Price multipliers

price_multipliers maps an input field to the factor each of its values puts on the whole price of the job:

"price_multipliers": {
  "quality": { "low": 1, "medium": 4, "high": 16, "default": 16 },
  "size": { "1024x1024": 1, "1024x1536": 1.5, "1536x1024": 1.5, "default": 1.5 }
}

• Each factor is a number from 0 to 1,000,000; the factors of several fields are multiplied together.
• A job that sends a listed field must use one of the listed values; anything else is refused with 400 invalid_request, naming the allowed values. Numbers and booleans are matched by their text (4 matches "4").
• "default" is the factor for a job that does not send the field. Without a "default", such a job is charged a factor of 1.
• If the config changes while a job runs and the job's value is no longer listed, the job is charged the highest factor of that field.
• An entry that is not an object of value to factor is rejected when the file is loaded.

Use it together with allowed_input, so that an option that changes what the provider charges cannot be sent without the price following it.

media.storage

Local disk (the default):

S3-compatible bucket:

Objects are stored under <prefix><job id>/<name>. The README records that bucket storage was checked against Cloudflare R2 only; other S3-compatible services, Amazon S3 included, have only met the mock.

media.webhooks

The secret used for a job is the first one that exists: the key's own (webhook_secret / webhook_secret_env), then media.webhooks.secret / secret_env, then the OMNIGATE_WEBHOOK_SECRET environment variable. A job that asks for a webhook when none of these is set is refused (webhook_not_configured).

Credits

credits sets what prepaid credit burns and earns. All of it is optional, and with nothing set a prepaid key behaves as it always did. The rule itself is in Prepaid credit.

A field credits does not know, or a value outside these ranges, is rejected when the file is loaded.

Payments

payments lets the holder of a prepaid key buy credit through a payment provider. Absent (the default) there are no payment endpoints at all. Every field, and what the gateway does and does not do with money, is in Payments.

"payments": { "provider": "near_intents", "recipient": "your-account.near" }

Environment variables

Both processes

Data plane

The spool and the config cache are relative to the data plane's working directory unless given as absolute paths.

The three limits OMNIGATE_MAX_CONCURRENT_PER_KEY, OMNIGATE_REQUEST_MEMORY_MB and OMNIGATE_PARSE_MEMORY_MB are checked before a request body is read. The two memory settings are ceilings on what the data plane admits, not on what the process uses in total: add the response cache (OMNIGATE_CACHE_MAX_MB) and a few tens of megabytes of its own. The binary's defaults (1024 + 1024 + 64) are meant for a host without a memory limit. compose.yaml limits the data-plane container to 1 GiB and therefore sets the three itself, to 256 + 256 + 64, from .env (below); see Deployment.

omnigate-dp healthcheck is the binary's one subcommand: it asks the address in OMNIGATE_LISTEN for /healthz and exits 0 if that answers 200 within three seconds, 1 otherwise. It reads no other variable, needs no token and prints nothing; the container image's HEALTHCHECK runs it. Any other argument stops the data plane with exit status 2.

Numeric variables are read strictly. A value must be a whole number written in digits only, within the variable's range (the millisecond and count settings go up to at least two billion; no sign, no _, no unit, no decimal point). Anything else stops the data plane at startup with a message that names the variable, as does a log level or format that is not one of the values above, or an OMNIGATE_LISTEN without a valid port. A numeric variable that is set but empty counts as unset. A typo never quietly becomes a default.

Control plane

Relative defaults are resolved against the control plane's working directory. When it is started from control/ as in the quickstart, the config default is the repository's config/omnigate.dev.json and state goes to control/data/.

The control plane checks these before it opens its port, and logs the reason and stops if one fails: OMNIGATE_INTERNAL_TOKEN is set, the numeric variables (OMNIGATE_CONTROL_PORT, OMNIGATE_AUDIT_MAX_MB) and the bind address can be read, a config path is known (see OMNIGATE_CONFIG), and an admin token that was set is at least 16 characters long. A token, a config path or a numeric variable that is set but empty counts as unset.

Not configurable through the environment: when the ledger seals a segment (at 64 MiB, or after the UTC day turns) and how often it writes a snapshot (every 50,000 events, and every five minutes while there is something new).

The container images set some of these differently; see Deployment.

Compose only

Read by Compose from .env for compose.yaml itself: the proxy's environment, the two memory limits, and the data plane's three memory budgets, which compose.yaml passes to the data-plane container.

With these defaults the three add up to 576 MiB of the container's 1,024. If you raise one, raise OMNIGATE_DATAPLANE_MEMORY by at least as much.

Which file a variable goes in (Compose)

compose.yaml gives each container its own environment file, so that the process that faces the internet holds as little as possible:

Each has an .example file next to it. compose.yaml itself sets OMNIGATE_TRUST_FORWARDED=1 for the control plane. The steps are in Deployment.

What the data plane receives

The data plane never reads the config file. It fetches a snapshot from the control plane (GET /internal/config) that contains:

• for each key that is not disabled, from the config file or created through the admin API: its SHA-256, name, rpm, models, and whether it is currently blocked by its budget (a disabled key is left out, so the data plane treats it as unknown), and its max_concurrent (0 when the key has none of its own);
• for each of those keys, the parts of its privacy mode that a data plane applies (no_store, the classes to redact, unlink, attested), or null; never the encrypt_to recipient, which only the control plane uses;
• the text upstreams as configured (the api_key_env name, never a secret; privacy_store_flag when set), without their attestation setting, which only the control plane uses;
• for each text route: model, strategy, cache_ttl_s, and each target's upstream, model and weight; for a target on an attested upstream, also the verdict (attestation: verified, max_age_s, tcb, since);
• the names of the media routes, which the data plane only uses to list them in GET /v1/models.

Budgets, budget periods, prices, raw keys, webhook secrets and the rest of the media section stay in the control plane.

While the control plane's ledger is starting or restarting, the endpoint answers 503 not_ready with retry-after: 1 instead of a snapshot: the snapshot says which keys are over budget, and that cannot be known without the ledger. A data plane keeps the config it has and asks again.

This page on GitHub