Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

override configuration keys with environment variables #663

Merged
merged 3 commits into from
Sep 27, 2024
Merged

override configuration keys with environment variables #663

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
795bdc7
override configuration keys with environment variables
arner Aug 31, 2024
63f2c1f
add documentation about environment variables
arner Sep 19, 2024
c241fc8
fix parsing of pkcs11 options
arner Sep 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 36 additions & 22 deletions docs/core-fabric.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ The following example provides descriptions for the various keys required for a
---
# ------------------- Logging section ---------------------------
logging:
# format is same as fabric [<logger>[,<logger>...]=]<level>[:[<logger>[,<logger>...]=]<level>...]
# available loggers: TBD
spec: debug
# format is same as fabric [<logger>[,<logger>...]=]<level>[:[<logger>[,<logger>...]=]<level>...]
format: '%{color}%{time:15:04:05.000} [%{module}] %{shortfunc} %{level:.4s}%{color:reset} %{message}'
spec: debug

# ------------------- FSC Node Configuration -------------------------
fsc:
Expand Down Expand Up @@ -397,6 +397,22 @@ fabric:
addresses:
```

## Overriding configuration keys

Any value that is not a (grand-)child of a list can be overridden with an environment variable that is all uppercase, prefixed with `CORE_`,
and traversing the path in the yaml with underscores. This means that a key like fsc.endpoint.resolvers[0].name cannot be changed via environment variables. Examples:

```sh
CORE_LOGGING_LEVEL=debug
CORE_FSC_P2P_LISTENADDRESS=/ip4/0.0.0.0/tcp/9001
CORE_FSC_IDENTITY_KEY_FILE=/my/private.key
CORE_FSC_KVS_PERSISTENCE_OPTS_DATASOURCE=/mydb.sqlite
CORE_FSC_TRACING_OPTL_ADDRESS=jaeger.example.com:4318
CORE_FABRIC_MYNETWORK_KEEPALIVE_TIMEOUT=120s
```

And so on.

## HSM Support

In order to use a hardware HSM for x.509 identities, you have to build the application with
Expand All @@ -416,7 +432,7 @@ reasons to choose sql may include:
The driver has been tested with the following sql drivers:

- SQLite: (pure go): modernc.org/sqlite
- Postgres (pure Go): github.com/lib/pq
- Postgres (pure Go): github.com/jackc/pgx/v5/stdlib

In theory you can use any [sql driver](https://github.com/golang/go/wiki/SQLDrivers) if you import it in your application.
To try a new sql driver, add a test here: `token/services/db/driver/sql/sql_test.go`.
Expand All @@ -433,15 +449,7 @@ persistence:
dataSource: /some/path/fsc.sqlite
```

Make sure that in your main.go, you `import _ "modernc.org/sqlite"`. This uses the following settings:

```sql
PRAGMA journal_mode = WAL;
PRAGMA busy_timeout = 5000;
PRAGMA synchronous = NORMAL;
PRAGMA cache_size = 1000000000;
PRAGMA temp_store = memory;
```
We use one connection for writes, and an unlimited number for concurrent read connections
(see the excellent https://kerkour.com/sqlite-for-servers for more information).

Expand All @@ -452,23 +460,32 @@ persistence:
type: sql
opts:
driver: sqlite
dataSource: file:/some/path/fsc.sqlite?_pragma=journal_mode(WAL)&_pragma=busy_timeout(1000)
dataSource: file:/some/path/fsc.sqlite&_txlock=immediate
tablePrefix: fsc # optional
skipCreateTable: true # tells FSC _not_ to create a table when starting up (because it already exists).
skipPragmas: true # if this is false, the pragmas we set in the datasource will be overridden with the defaults.
maxOpenConns: 50 # optional: max open read connections to the database. Defaults to unlimited.
```

Set any [pragmas](https://www.sqlite.org/pragma.html) as per the example above. Make sure that journal mode is always WAL.
By default we set the following pragmas (unless you do `skipPragmas: true`. Make sure you always have `_pragma=journal_mode(WAL`):

```sql
PRAGMA journal_mode = WAL;
PRAGMA busy_timeout = 5000;
PRAGMA synchronous = NORMAL;
PRAGMA cache_size = 1000000000;
PRAGMA temp_store = memory;
```

### Config example for postgres

`import _ "github.com/lib/pq"` in main.go. The same configuration flags as above apply,
but for Postgres we always use one connection pool for reads and writes, and the sqlite pragmas don't apply.
The same configuration flags as above apply, but for Postgres we always use one connection pool for reads and writes,
and the sqlite pragmas don't apply.

> [!WARNING]
> The 'dataSource' field is sensitive because it contains a password. Instead of in this file, set it in an
> `FSC_DB_DATASOURCE` environment variable.
> [!WARNING]
> The 'dataSource' field is sensitive because it contains a password. Instead of in this file, set it in the
> `CORE_FSC_KVS_PERSISTENCE_OPTS_DATASOURCE` and `CORE_FABRIC_MYNETWORK_VAULT_PERSISTENCE_OPTS_DATASOURCE` environment
> variables (where mynetwork is the name of your network in core.yaml).

```yaml
persistence:
Expand All @@ -477,6 +494,3 @@ persistence:
driver: postgres
dataSource: host=localhost port=5432 user=postgres password=example dbname=tokendb sslmode=disable
```

See [pq docs](https://pkg.go.dev/github.com/lib/pq#hdr-Connection_String_Parameters) for more information about the
postgres connection string.
50 changes: 25 additions & 25 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ require (
github.com/IBM/idemix/bccsp/types v0.0.0-20240816143710-3dce4618d760
github.com/IBM/mathlib v0.0.3-0.20231011094432-44ee0eb539da
github.com/dgraph-io/badger/v3 v3.2103.5
github.com/docker/docker v27.1.2+incompatible
github.com/docker/docker v27.2.0+incompatible
github.com/docker/go-connections v0.5.0
github.com/fsouza/go-dockerclient v1.12.0
github.com/gin-contrib/cors v1.7.2
Expand Down Expand Up @@ -47,17 +47,17 @@ require (
github.com/pkg/errors v0.9.1
github.com/prometheus/client_golang v1.20.0
github.com/spf13/cobra v1.8.1
github.com/spf13/viper v1.12.0
github.com/spf13/viper v1.19.0
github.com/stretchr/testify v1.9.0
github.com/sykesm/zap-logfmt v0.0.4
github.com/tedsuo/ifrit v0.0.0-20230516164442-7862c310ad26
github.com/test-go/testify v1.1.4
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.53.0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.53.0
go.opentelemetry.io/otel v1.28.0
go.opentelemetry.io/otel v1.29.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.28.0
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.28.0
go.opentelemetry.io/otel/trace v1.28.0
go.opentelemetry.io/otel/trace v1.29.0
go.uber.org/atomic v1.11.0
go.uber.org/dig v1.18.0
go.uber.org/zap v1.27.0
Expand All @@ -72,11 +72,10 @@ require (

require (
github.com/alecthomas/kingpin/v2 v2.4.0 // indirect
github.com/libp2p/go-libp2p-routing-helpers v0.7.2 // indirect
github.com/pelletier/go-toml v1.9.5 // indirect
github.com/libp2p/go-libp2p-routing-helpers v0.7.4 // indirect
github.com/pion/datachannel v1.5.8 // indirect
github.com/pion/dtls/v2 v2.2.12 // indirect
github.com/pion/ice/v2 v2.3.32 // indirect
github.com/pion/ice/v2 v2.3.34 // indirect
github.com/pion/interceptor v0.1.29 // indirect
github.com/pion/logging v0.2.2 // indirect
github.com/pion/mdns v0.0.12 // indirect
Expand All @@ -87,10 +86,12 @@ require (
github.com/pion/sdp/v3 v3.0.9 // indirect
github.com/pion/srtp/v2 v2.0.20 // indirect
github.com/pion/stun v0.6.1 // indirect
github.com/pion/transport/v2 v2.2.9 // indirect
github.com/pion/transport/v2 v2.2.10 // indirect
github.com/pion/turn/v2 v2.1.6 // indirect
github.com/pion/webrtc/v3 v3.2.50 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/pion/webrtc/v3 v3.3.0 // indirect
github.com/sagikazarmark/locafero v0.6.0 // indirect
github.com/sagikazarmark/slog-shim v0.1.0 // indirect
github.com/sourcegraph/conc v0.3.0 // indirect
github.com/wlynxg/anet v0.0.3 // indirect
github.com/xhit/go-str2duration/v2 v2.1.0 // indirect
)
Expand Down Expand Up @@ -143,7 +144,7 @@ require (
github.com/flynn/noise v1.1.0 // indirect
github.com/francoispqt/gojay v1.2.13 // indirect
github.com/fsnotify/fsnotify v1.7.0 // indirect
github.com/gabriel-vasile/mimetype v1.4.3 // indirect
github.com/gabriel-vasile/mimetype v1.4.4 // indirect
github.com/ghodss/yaml v1.0.0 // indirect
github.com/gin-contrib/sse v0.1.0 // indirect
github.com/go-logfmt/logfmt v0.5.1 // indirect
Expand Down Expand Up @@ -173,22 +174,22 @@ require (
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
github.com/googleapis/gax-go/v2 v2.12.5 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.22.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/golang-lru v0.5.4 // indirect
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/huin/goupnp v1.3.0 // indirect
github.com/hyperledger/fabric-amcl v0.0.0-20230602173724-9e02669dceb2 // indirect
github.com/hyperledger/fabric-config v0.1.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/ipfs/boxo v0.10.0 // indirect
github.com/ipfs/boxo v0.21.0 // indirect
github.com/ipfs/go-cid v0.4.1 // indirect
github.com/ipfs/go-datastore v0.6.0 // indirect
github.com/ipfs/go-log v1.0.5 // indirect
github.com/ipfs/go-log/v2 v2.5.1 // indirect
github.com/ipld/go-ipld-prime v0.20.0 // indirect
github.com/ipld/go-ipld-prime v0.21.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
github.com/jackc/puddle/v2 v2.2.1 // indirect
Expand Down Expand Up @@ -250,7 +251,7 @@ require (
github.com/opencontainers/runtime-spec v1.2.0 // indirect
github.com/opentracing/opentracing-go v1.2.0 // indirect
github.com/pbnjay/memory v0.0.0-20210728143218-7b4eea64cf58 // indirect
github.com/pelletier/go-toml/v2 v2.2.2 // indirect
github.com/pelletier/go-toml/v2 v2.2.3 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/polydawn/refmt v0.89.0 // indirect
github.com/prometheus/client_model v0.6.1
Expand All @@ -265,7 +266,7 @@ require (
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/spaolacci/murmur3 v1.1.0 // indirect
github.com/spf13/afero v1.11.0 // indirect
github.com/spf13/cast v1.6.0 // indirect
github.com/spf13/cast v1.7.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
Expand All @@ -281,27 +282,26 @@ require (
go.etcd.io/etcd/raft/v3 v3.5.1 // indirect
go.etcd.io/etcd/server/v3 v3.5.1 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.28.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.21.0 // indirect
go.opentelemetry.io/otel/metric v1.28.0 // indirect
go.opentelemetry.io/otel/sdk v1.28.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.29.0
go.opentelemetry.io/otel/metric v1.29.0 // indirect
go.opentelemetry.io/otel/sdk v1.29.0
go.opentelemetry.io/proto/otlp v1.3.1 // indirect
go.uber.org/fx v1.22.1 // indirect
go.uber.org/mock v0.4.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/arch v0.8.0 // indirect
golang.org/x/crypto v0.26.0 // indirect
golang.org/x/mod v0.20.0 // indirect
golang.org/x/oauth2 v0.21.0 // indirect
golang.org/x/oauth2 v0.22.0 // indirect
golang.org/x/sys v0.24.0 // indirect
golang.org/x/text v0.17.0 // indirect
golang.org/x/time v0.5.0 // indirect
golang.org/x/tools v0.24.0 // indirect
gonum.org/v1/gonum v0.13.0 // indirect
gonum.org/v1/gonum v0.15.0 // indirect
google.golang.org/api v0.186.0 // indirect
google.golang.org/genproto v0.0.0-20240624140628-dc46fd24d27d // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240701130421-f6361c86f094 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240701130421-f6361c86f094 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240822170219-fc7c04adadcd // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240822170219-fc7c04adadcd // indirect
gopkg.in/alecthomas/kingpin.v2 v2.2.6 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
Expand Down
Loading