Skip to content

Latest commit

 

History

History
115 lines (82 loc) · 8.22 KB

README.md

File metadata and controls

115 lines (82 loc) · 8.22 KB
Raw

Label Filter Gateway (lfgw)

lfgw is a simple reverse proxy aimed at PromQL / MetricsQL metrics filtering based on OIDC roles.

It relies on VictoriaMetrics/metricsql for label filters manipulation in metric expressions (according to an ACL) before a request is proxied to Prometheus/VictoriaMetrics by httputil. The process is described in more details here.

Target setup: grafana (with OIDC integration) <-> lfgw <-> Prometheus/VictoriaMetrics.

Key features

  • ACL-based request rewrites with implicit deny;
  • a user can have multiple roles;
  • support for autoconfiguration in environments, where OIDC-role names match names of namespaces ("assumed roles" mode; thanks to @aberestyak for the idea);
  • automatic expression optimizations for non-full access requests;
  • support for different headers with access tokens (Authorization, X-Forwarded-Access-Token, X-Auth-Request-Access-Token), which can be useful for tools like oauth2-proxy;
  • requests to both /api/* and /federate endpoints are protected (=rewritten);
  • requests to sensitive endpoints are blocked by default;
  • compatible with both PromQL and MetricsQL.

Similar projects

Similar projects are described here.

Docker images

Docker images are published on ghcr.io/weisdd/lfgw.

Configuration

Example of keycloak + grafana + lfgw setup is described here.

Requirements for jwt-tokens

  • OIDC-roles must be present in roles claim;
  • Client ID specified via OIDC_CLIENT_ID must be present in aud claim (more details in environment variables section), otherwise token verification will fail.

Environment variables

Basic settings

Variable Default Value Description
UPSTREAM_URL Prometheus URL, e.g. http://prometheus.localhost.
OIDC_REALM_URL OIDC Realm URL, e.g. https://keycloak.localhost/auth/realms/monitoring
OIDC_CLIENT_ID OIDC Client ID (1*)
ACL_PATH ./acl.yaml Path to a file with ACL definitions (OIDC role to namespace bindings). Skipped if ACL_PATH is empty (might be useful when autoconfiguration is enabled through ASSUMED_ROLES=true).
ASSUMED_ROLES false In environments, where OIDC-role names match names of namespaces, ACLs can be constructed on the fly (e.g. ["role1", "role2"] will give access to metrics from namespaces role1 and role2). The roles specified in acl.yaml are still considered and get merged with assumed roles. Role names may contain regular expressions, including the admin definition .*.

(1*): since it's grafana who obtains jwt-tokens in the first place, the specified client id must also be present in the forwarded token (the aud claim).

Additional settings

Variable Default Value Description
ENABLE_DEDUPLICATION true Whether to enable deduplication, which leaves some of the requests unmodified if they match the target policy. Examples can be found in the "acl.yaml syntax" section.
OPTIMIZE_EXPRESSIONS true Whether to automatically optimize expressions for non-full access requests. More details
SAFE_MODE true Whether to block requests to sensitive endpoints like /api/v1/admin/tsdb, /api/v1/insert.
SET_PROXY_HEADERS false Whether to set proxy headers (X-Forwarded-For, X-Forwarded-Proto, X-Forwarded-Host).
SET_GOMAXPROCS true Automatically set GOMAXPROCS to match Linux container CPU quota.
DEBUG false Whether to print out debug log messages.
LOG_FORMAT pretty Log format (pretty, json)
LOG_NO_COLOR false Whether to disable colors for pretty format
LOG_REQUESTS false Whether to log HTTP requests
PORT 8080 Port the web server will listen on.
READ_TIMEOUT 10s ReadTimeout covers the time from when the connection is accepted to when the request body is fully read (if you do read the body, otherwise to the end of the headers). More details
WRITE_TIMEOUT 10s WriteTimeout normally covers the time from the end of the request header read to the end of the response write (a.k.a. the lifetime of the ServeHTTP). More details
GRACEFUL_SHUTDOWN_TIMEOUT 20s Maximum amount of time to wait for all connections to be closed. More details

ACL syntax

The file with ACL definitions (./acl.yaml by default) has a simple structure:

role: namespace, namespace2

For example:

team0: .*                # all metrics
team1: min.*, .*, stolon # all metrics, it's the same as .*
team2: minio             # only those with namespace="minio"
team3: min.*             # only those matching namespace=~"min.*"
team4: minio, stolon     # only those matching namespace=~"minio|stolon"
team5: min.*, stolon     # only those matching namespace=~"min.*|stolon"

To summarize, here are the key principles used for rewriting requests:

  • .* - all requests are simply forwarded to an upstream;
  • minio - all label filters with the namespace label are removed, then namespace="minio" is added;
  • min.* - positive regex-match label filters (namespace=~"X") are removed, then namespace=~"mi.*" is added;
  • minio, stolon - positive regex-match label filters (namespace=~"X") are removed, then namespace=~"minio|stolon" is added;
  • min.*, stolon - positive regex-match label filters (namespace=~"X") are removed, then namespace=~"min.*|stolon" is added.

When deduplication is enabled, these queries will stay unmodified:

  • min.*, stolon, query: request_duration{namespace="minio"} - a non-regexp label filter that matches policy;
  • min.*, stolon, query: request_duration{namespace=~"minio"} - a "fake" regexp (no special symbols) label filter that matches policy;
  • min.*, stolon, query: request_duration{namespace=~"min.*"} - a label filter is a subfilter of the policy.

Note: Regex matches are fully anchored. A match of env=~"foo" is treated as env=~"^foo$" (Source). Please, be careful, they are not expected to be used in ACLs.

Note: a user is free to have multiple roles matching the contents of acl.yaml. Basically, there are 3 cases:

  • one role => a prepopulated LF is returned;
  • multiple roles, one of which gives full access => a prepopulated LF, corresponding to the full access role, is returned;
  • multiple "limited" roles => definitions of all those roles are merged together, and then lfgw generates a new LF. The process is the same as if this meta-definition was loaded through acl.yaml.

Licensing

lfgw code is licensed under MIT, though its dependencies might have other licenses. Please, inspect the modules listed in go.mod if needed.