Skip to main content

Temporal Proxy

View Markdown

The Temporal Proxy is a gRPC proxy that sits between your Temporal SDK Clients, Workers, and the Temporal Web UI on one side and one or more upstream Temporal Services on the other. It handles Namespace translation, TLS termination, and optional payload encryption so your applications can target a single local endpoint while the proxy routes each request to the right upstream, whether that is a local development Service, a self-hosted Service, or Temporal Cloud.

Why use it​

Without the proxy, connection details leak into your application code. Every Worker and Client has to know the upstream's host, TLS material, credentials, and the exact Namespace name the upstream expects. That couples your code to an environment: moving between a local Service, a self-hosted deployment, and Temporal Cloud becomes a code change.

The proxy owns that concern instead. Workers talk plaintext to a single local endpoint using a short Namespace name, and the proxy adds TLS, credentials, and Namespace translation on the way out. Point a Worker at a different Namespace and it reaches a different upstream, with no change to the Worker.

How it works​

The proxy is built from a gateway and one proxy per upstream, connected by unix sockets:

  • The gateway is the single inbound endpoint that every Worker, SDK Client, and the Web UI connects to.
  • Each upstream has its own proxy that handles communication with that destination.

For each request, the gateway:

  1. refuses the call if its gRPC service is not one the configuration allows, before any upstream work. By default that allows WorkflowService and OperatorService and nothing else.
  2. peeks the target Namespace without parsing the payload; it is codec-transparent and relays raw frames in both directions.
  3. picks an upstream: the first matching routing rule, otherwise the system upstream for Namespace-less calls, otherwise the default.
  4. hands the request to that upstream's proxy over a unix socket.

The per-upstream proxy then rewrites the local Namespace to the name the upstream expects, attaches that upstream's TLS and credentials, forwards to the Temporal Service, and translates the Namespace back on responses. When payload encryption is enabled, it also seals codec-capable Payloads on the way out and opens them on the way back. New Payloads sent through the proxy reach the upstream as ciphertext. See Encrypt payloads for the scope and limits of this fail-closed behavior.

Terms​

TermMeaning
gatewayThe single inbound gRPC endpoint that every SDK Client, Worker, and the Web UI connects to. It routes each request to an upstream by Namespace and request metadata, and never parses payloads.
upstreamA configured destination the proxy forwards to: a Temporal Service (local dev, self-hosted, or Temporal Cloud), or another Temporal Proxy.
system upstreamThe upstream that handles Namespace-less requests, such as the SDK's GetSystemInfo call on connect.
extension serverA gRPC service you run that the proxy calls out to for a capability it has no built-in backend for: wrapping data encryption keys, or deciding whether an inbound call may proceed.
Temporal ServiceA Temporal frontend the proxy connects to.

Prerequisites​

  • One or more upstream Temporal Services to route to, such as a local development Service, a self-hosted Service, or Temporal Cloud.
  • The hostPort address for each upstream.
  • Any credentials the upstreams require, such as a Temporal Cloud API key or mTLS certificates.
  • Go installed, if you build the proxy from source. The container image and Helm chart do not require a local Go toolchain.

Install the proxy​

Install the proxy binary with Go:

go install github.com/temporalio/temporal-proxy/cmd/proxy@latest

Pin an explicit version instead of @latest, using a tag from the releases page:

go install github.com/temporalio/temporal-proxy/cmd/proxy@vX.Y.Z

Pull the container image:

docker pull temporalio/temporal-proxy:latest

Install with Helm from the Temporal Helm repo, optionally pinning a chart version with --version:

helm install temporal-proxy temporal-proxy \
--repo https://go.temporal.io/helm-charts

Each chart release deploys a proxy version by default. Override it with --set image.tag=vX.Y.Z. Supply the proxy configuration under the config key of a Helm values file, as described in Deploy to Kubernetes.

Run the proxy with a configuration file passed through the -c (or --config) flag:

proxy serve -c config.yaml

--config also reads the PROXY_CONFIG environment variable, which is how the Helm chart points the proxy at its mounted configuration. See Observability for the log level flag and the metrics settings.

Observability​

The proxy serves Prometheus metrics at /metrics on :9090 by default and logs JSON to stderr. The gateway also serves the standard gRPC health service, with an entry for every allowed service, which is what a Client health check or a liveness probe reads. See Report health for what it reports.

proxy serve takes two flags, each with an environment variable equivalent:

FlagEnvironment variableDefaultSets
--config, -cPROXY_CONFIGnonePath to the configuration file. Required.
--levelLOG_LEVELinfoLog level: debug, info, warn, error.

Everything else about metrics comes from the metrics block in the configuration file:

metrics:
hostPort: :9090 # Address serving /metrics.
namespace: tmprl_proxy # Prometheus namespace prefixed onto every metric.
labels:
namespace: false # Report the Temporal Namespace on metrics that can name one.
fixed: # Constant labels stamped on every series.
region: us-east-1
metadata: # Inbound headers reported as labels, written <header>:<name>.
- x-tenant:tenant
Version note

metrics.hostPort and metrics.namespace replace the --metrics-addr and --metrics-namespace flags, removed in v0.6.0 along with their METRICS_ADDR and METRICS_NAMESPACE environment variables. The defaults are the same, so a deployment that never set them needs no change.

  • labels.namespace decides whether the metrics that can name a Temporal Namespace report it. It is false by default. The value comes from the request, so every Namespace a Client names adds another series. The label is declared either way and only its value is dropped, so a query that already ignores it reads the same before and after you turn enable it. The name reported is the local, pre-translation one, the same key you write under encryption.overrides.
  • labels.fixed maps a label name to a constant value stamped on every series the proxy publishes, such as the region or cluster a deployment runs in.
  • labels.metadata reports an inbound request header as a label on the request-scoped series, written <header>:<name>. Values are truncated at 256 bytes.
caution

Do not list a header that carries a credential, such as an API key or bearer token, under labels.metadata. The /metrics endpoint doesn't require authentication, so anyone who can reach the port can read the header's value.

Also avoid headers whose values the caller can set freely. Each distinct value creates a new copy of every request-scoped series, so an unbounded header causes unbounded metric cardinality.

Metric names are <namespace>_<subsystem>_<name>, so with the default namespace the routing counter is tmprl_proxy_router_decisions_total. There are three subsystems:

SubsystemMetricLabelsReports
serverrequests_totalmethod, codeRPCs served, by gRPC status code
serverrequest_duration_secondsmethodEnd-to-end time serving an RPC
routerdecisions_totalupstream, outcomeRouting decisions, by chosen upstream
routerforwarding_errors_totalupstream, reasonForwarding failures the router originated
encryptionvault_ops_totaloperation, result, namespaceEnvelope operations, sealing and opening payloads
encryptionvault_ops_duration_secondsoperation, namespaceTime per envelope operation, end to end
encryptiondek_ops_totaloperation, resultOutcome of the AES-256-GCM step alone
encryptiondek_ops_duration_secondsoperationTime in the AES-256-GCM step alone
encryptionkek_ops_totalprovider, operation, resultDEK wrap and unwrap calls to your KMS
encryptionkek_ops_duration_secondsprovider, operationTime spent wrapping and unwrapping DEKs
encryptiondek_rotations_totalreasonDEK rotations, by why the DEK was replaced
encryptiondek_cache_hits_totalnoneReads served from the decrypted-DEK cache
encryptiondek_cache_misses_totalnoneReads that required a KMS unwrap
encryptiondek_cache_sizenoneCurrent entries in the decrypted-DEK cache
Version note

v0.6.0 renamed the three *_duration_secs histograms to *_duration_seconds to follow the Prometheus convention. Update any dashboard or alert that names the old spelling.

The namespace label reports a value only when metrics.labels.namespace is on. It is declared either way, so the series exist regardless.

The encryption metrics only move when payload encryption is configured. They are layered, so pick the one that matches the question you are asking:

  • vault_ops_* is the whole envelope operation end to end, including any KEK call and cache lookup, and is the pair to alert on.
  • dek_ops_* is the symmetric AES-256-GCM step by itself, with the KEK work excluded. Its result is that step's own outcome, so a payload that encrypts cleanly and then fails to wrap its DEK counts as a success here and an error under kek_ops_total, which keeps the blame with the KMS.
  • kek_ops_* is the calls to your KMS. Watch kek_ops_total{result="error"}, since a failure to wrap or unwrap a DEK fails the request that needed it.

dek_rotations_total splits by reason: initial for a Namespace's first DEK, scheduled for the renewBefore pre-rotation, and on_demand for a DEK replaced at request time because no fresh one was ready. A rising on_demand rate means rotation is falling behind, so raise renewBefore. Compare the cache counters against cacheSize to see whether the cache is absorbing read traffic.