Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add session storage clustering support #3555

Merged
merged 14 commits into from
Nov 22, 2024
+1,697 −366
Merged

Add session storage clustering support #3555

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
acfd175
replaced dual session DB impl to go-cache impl
woutslakhorst Nov 11, 2024
877531b
added memcached impl
woutslakhorst Nov 11, 2024
39aa9aa
link memcached
woutslakhorst Nov 11, 2024
9272fa0
add e2e test with redis sentinel clustered session memory
woutslakhorst Nov 11, 2024
93aa329
added e2e-tests
woutslakhorst Nov 11, 2024
57d1f62
reuse oauth2.js
woutslakhorst Nov 12, 2024
e9f3eaa
remove unused files
woutslakhorst Nov 12, 2024
5a3013a
reduce added lines by removing comments from redis conf files
woutslakhorst Nov 18, 2024
ae2e0c6
add documentation on clustering
woutslakhorst Nov 18, 2024
1126341
some godoc
woutslakhorst Nov 18, 2024
bb6abcd
deduplicate configs
woutslakhorst Nov 19, 2024
2aa24c0
deduplicate configs
woutslakhorst Nov 19, 2024
5e7844d
add session DB to health check
woutslakhorst Nov 19, 2024
0d38af5
added comment on default memcached behaviour
woutslakhorst Nov 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,7 @@ The following options can be configured on the server:
pki.maxupdatefailhours 4 Maximum number of hours that a denylist update can fail
pki.softfail true Do not reject certificates if their revocation status cannot be established when softfail is true
**Storage**
storage.session.memcached.address [] List of Memcached server addresses. These can be a simple 'host:port' or a Memcached connection URL with scheme, auth and other options.
reinkrul marked this conversation as resolved.
Show resolved Hide resolved
storage.session.redis.address Redis session database server address. This can be a simple 'host:port' or a Redis connection URL with scheme, auth and other options. If not set it, defaults to an in-memory database.
storage.session.redis.database Redis session database name, which is used as prefix every key. Can be used to have multiple instances use the same Redis instance.
storage.session.redis.password Redis session database password. If set, it overrides the username in the connection URL.
Expand Down
1 change: 1 addition & 0 deletions docs/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ Nuts documentation
pages/deployment/configuration.rst
pages/deployment/migration.rst
pages/deployment/recommended-deployment.rst
pages/deployment/clustering.rst
pages/deployment/certificates.rst
pages/deployment/docker.rst
pages/deployment/storage.rst
Expand Down
16 changes: 16 additions & 0 deletions docs/pages/deployment/clustering.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
.. _clustering:

Clustering
##########

With the introduction of a SQL database and separate session storage, clustering with HA is now possible.
woutslakhorst marked this conversation as resolved.
Show resolved Hide resolved
Clustering is currently limited to nodes that have the ``did:nuts`` method disabled.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe state what NOT to do (use default on-disk/in-memory stores for private keys and stuff)?

To enable clustering, you must support the following:

- A clustered SQL database (SQLite is not supported)
- A clustered session storage (Redis sentinel is recommended)
- A clustered private key storage (Hashicorp Vault or Azure Keyvault)
- Read only mounts for configuration, policy, discovery and JSON-LD context files.

It's recommended to use a level 4 load balancer to distribute the load across the nodes.
Each node should have a reverse proxy for TLS termination.
1 change: 1 addition & 0 deletions docs/pages/deployment/server_options.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
pki.maxupdatefailhours 4 Maximum number of hours that a denylist update can fail
pki.softfail true Do not reject certificates if their revocation status cannot be established when softfail is true
**Storage**
storage.session.memcached.address [] List of Memcached server addresses. These can be a simple 'host:port' or a Memcached connection URL with scheme, auth and other options.
storage.session.redis.address Redis session database server address. This can be a simple 'host:port' or a Redis connection URL with scheme, auth and other options. If not set it, defaults to an in-memory database.
storage.session.redis.database Redis session database name, which is used as prefix every key. Can be used to have multiple instances use the same Redis instance.
storage.session.redis.password Redis session database password. If set, it overrides the username in the connection URL.
Expand Down
62 changes: 62 additions & 0 deletions docs/pages/deployment/storage.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,68 @@ Refer to the documentation of the driver for the database you are using for the
Usage of SQLite is not recommended for production environments.
Connections to a SQLite DB are restricted to 1, which will lead to severe performance reduction.

Session storage
***************

Session storage is used for storing access tokens, nonces and other volatile data.
Data is stored in-memory only. There are 3 supported session storage types:
woutslakhorst marked this conversation as resolved.
Show resolved Hide resolved

- local
- memcached
- redis (standalone, cluster, sentinel)
woutslakhorst marked this conversation as resolved.
Show resolved Hide resolved

Local
=====

This is the default and will store data in-memory. Any restart will wipe all data.
Data is also not shared if you run multiple nodes.

Memcached
=========

Memcached can be enabled with the following config:
woutslakhorst marked this conversation as resolved.
Show resolved Hide resolved

.. code-block:: yaml

storage.session.memcached.address:
- localhost:11211

You can add multiple memcached servers to the list.
memcached is not capable of clustering. Each piece of data is stored on a single instance.
If you want true HA, you'll need to use Redis.

Redis
=====

Redis is the only option if you want to run multiple nodes and the cache as HA.
Redis can be configured in standalone or sentinel mode.
Standalone:

.. code-block:: yaml

storage:
session:
redis:
address: localhost:6379
username: user
password: pass
db: 0

Sentinel:

.. code-block:: yaml

storage:
session:
redis:
sentinel:
master: mymaster
nodes:
- localhost:26379
- localhost:26380
- localhost:26381


Private Keys
************

Expand Down
60 changes: 60 additions & 0 deletions e2e-tests/clustering/memcached/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
services:
memcached:
image: memcached
command:
- --conn-limit=1024
- --memory-limit=64
- --threads=4
nodeA-backend:
image: "${IMAGE_NODE_A:-nutsfoundation/nuts-node:master}"
ports:
- "18081:8081"
environment:
NUTS_CONFIGFILE: /opt/nuts/nuts.yaml
volumes:
- "./node-A/nuts.yaml:/opt/nuts/nuts.yaml:ro"
- "../../tls-certs/nodeA-backend-certificate.pem:/opt/nuts/certificate-and-key.pem:ro"
- "../../tls-certs/truststore.pem:/opt/nuts/truststore.pem:ro"
# did:web resolver uses the OS CA bundle, but e2e tests use a self-signed CA which can be found in truststore.pem
# So we need to mount that file to the OS CA bundle location, otherwise did:web resolving will fail due to untrusted certs.
- "../../tls-certs/truststore.pem:/etc/ssl/certs/Nuts_RootCA.pem:ro"
- "./node-A/presentationexchangemapping.json:/opt/nuts/policies/presentationexchangemapping.json:ro"
- "./shared/discovery:/nuts/discovery:ro"
healthcheck:
interval: 1s # Make test run quicker by checking health status more often
nodeA:
image: nginx:1.25.1
ports:
- "10443:443"
volumes:
- "./node-A/nginx.conf:/etc/nginx/nginx.conf:ro"
- "../../tls-certs/nodeA-certificate.pem:/etc/nginx/ssl/server.pem:ro"
- "../../tls-certs/nodeA-certificate.pem:/etc/nginx/ssl/key.pem:ro"
- "../../tls-certs/truststore.pem:/etc/nginx/ssl/truststore.pem:ro"
- "../../scripts/oauth2.js:/etc/nginx/oauth2.js:ro"
nodeB-backend:
image: "${IMAGE_NODE_B:-nutsfoundation/nuts-node:master}"
ports:
- "28081:8081"
environment:
NUTS_CONFIGFILE: /opt/nuts/nuts.yaml
volumes:
- "./node-B/nuts.yaml:/opt/nuts/nuts.yaml:ro"
- "../../tls-certs/nodeB-certificate.pem:/opt/nuts/certificate-and-key.pem:ro"
- "../../tls-certs/truststore.pem:/opt/nuts/truststore.pem:ro"
- "../../tls-certs/truststore.pem:/etc/ssl/certs/truststore.pem:ro"
# did:web resolver uses the OS CA bundle, but e2e tests use a self-signed CA which can be found in truststore.pem
# So we need to mount that file to the OS CA bundle location, otherwise did:web resolving will fail due to untrusted certs.
- "../../tls-certs/truststore.pem:/etc/ssl/certs/Nuts_RootCA.pem:ro"
- "./shared/discovery:/nuts/discovery:ro"
healthcheck:
interval: 1s # Make test run quicker by checking health status more often
nodeB:
image: nginx:1.25.1
ports:
- "20443:443"
volumes:
- "../../shared_config/nodeB-http-nginx.conf:/etc/nginx/conf.d/nuts-http.conf:ro"
- "../../tls-certs/nodeB-certificate.pem:/etc/nginx/ssl/server.pem:ro"
- "../../tls-certs/nodeB-certificate.pem:/etc/nginx/ssl/key.pem:ro"
- "../../tls-certs/truststore.pem:/etc/nginx/ssl/truststore.pem:ro"
71 changes: 71 additions & 0 deletions e2e-tests/clustering/memcached/node-A/nginx.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
load_module /usr/lib/nginx/modules/ngx_http_js_module.so;
woutslakhorst marked this conversation as resolved.
Show resolved Hide resolved

user nginx;
worker_processes 1;

error_log /var/log/nginx/error.log debug;
pid /var/run/nginx.pid;

events {
worker_connections 1024;
}

http {
js_import oauth2.js;
include /etc/nginx/mime.types;
default_type application/octet-stream;

log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

access_log /var/log/nginx/access.log main;

keepalive_timeout 65;

include /etc/nginx/conf.d/*.conf;

upstream nodeA-internal {
server nodeA-backend:8081;
}
upstream nodeA-external {
server nodeA-backend:8080;
}

server {
server_name nodeA;
listen 443 ssl;
http2 on;
ssl_certificate /etc/nginx/ssl/server.pem;
ssl_certificate_key /etc/nginx/ssl/key.pem;
ssl_client_certificate /etc/nginx/ssl/truststore.pem;
ssl_verify_client optional;
ssl_verify_depth 1;
ssl_protocols TLSv1.3;

location / {
proxy_set_header X-Ssl-Client-Cert $ssl_client_escaped_cert;
proxy_pass http://nodeA-external;
}

# check access via token introspection as described by https://www.nginx.com/blog/validating-oauth-2-0-access-tokens-nginx/
location /resource {
js_content oauth2.introspectAccessToken;
}

# Location in javascript subrequest.
# this is needed to set headers and method
location /_oauth2_send_request {
internal;
proxy_method POST;
proxy_set_header Content-Type "application/x-www-form-urlencoded";
proxy_pass http://nodeA-internal/internal/auth/v2/accesstoken/introspect;
}
location /_dpop_send_request {
internal;
proxy_method POST;
proxy_set_header Content-Type "application/json";
proxy_pass http://nodeA-internal/internal/auth/v2/dpop/validate;
}
}
}
101 changes: 101 additions & 0 deletions e2e-tests/clustering/memcached/node-A/presentationexchangemapping.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
{
"test": {
"organization": {
"format": {
"ldp_vp": {
"proof_type": [
"JsonWebSignature2020"
]
},
"ldp_vc": {
"proof_type": [
"JsonWebSignature2020"
]
}
},
"id": "pd_any_care_organization_with_employee",
"name": "Care organization with employee",
"purpose": "Finding a care organization with logged in user for authorizing access to medical metadata",
"input_descriptors": [
{
"id": "id_nuts_care_organization_cred",
"constraints": {
"fields": [
{
"path": [
"$.type"
],
"filter": {
"type": "string",
"const": "NutsOrganizationCredential"
}
},
{
"path": [
"$.credentialSubject.organization.name"
],
"filter": {
"type": "string"
}
},
{
"path": [
"$.credentialSubject.organization.city"
],
"filter": {
"type": "string"
}
}
]
}
},
{
"id": "id_employee_credential_cred",
"constraints": {
"fields": [
{
"path": [
"$.type"
],
"filter": {
"type": "string",
"const": "EmployeeCredential"
}
},
{
"id": "employee_identifier",
"path": [
"$.credentialSubject.identifier",
"$.credentialSubject[0].identifier"
],
"filter": {
"type": "string"
}
},
{
"id": "employee_name",
"path": [
"$.credentialSubject.name",
"$.credentialSubject[0].name"
],
"filter": {
"type": "string"
}
},
{
"id": "employee_role",
"path": [
"$.credentialSubject.roleName",
"$.credentialSubject[0].roleName"
],
"filter": {
"type": "string"
}
}
]
}
}
]
}
}
}
Loading
Loading