Discovery: SQLite-based server implementation #2589

reinkrul · 2023-11-06T16:00:40Z

Base maintainer implementation, not including:

HTTP API
Checking revocations
Registration of the module
Diagnostics

Requires

SQL: introduce SQLite storage #2595 (SQLite support)
PEX: Support matching JWT VCs #2598 (PEX JWT support)
PEX: Validate presentation submission #2566 (PEX validation)

woutslakhorst

This PR is about the usecase maintainer impl but introduces SQL on the fly. The migration towards sql storage is a big one and needs a proper setup. We risk in introducing it with add-ons/patches and forget to properly design it.

Issues that come to mind:

put the migrations in sub dirs per supported DB
are the migration prefixes with 1, 2, 3 going to break when using 10 ("10"<"2_" stringwise)
where are we going to put the models for the tabels, all here or in their own packages?
migration file naming
connectionstring documentation
which DBs to support?
additional config for those DBs and can that be added to later on?

SQL introduction should be done in its own PR

storage/config.go

reinkrul · 2023-11-09T12:10:09Z

Extracted SQL functionality into a separate PR: #2595

reinkrul · 2023-11-20T06:06:30Z

Refactored with server and client having 1 store.

README.rst

discoveryservice/api/v1/api.go

discoveryservice/config.go

vcr/pe/schema/v2/schema.go

discoveryservice/module.go

discoveryservice/store.go

reinkrul · 2023-11-20T16:08:48Z

note to self: check that presentation.ID is scoped to the signer DID

reinkrul · 2023-11-20T16:09:09Z

First feedback processed

gerardsn

A PR like this takes me a full day to review to cover about 80% of the content. Adding more comment along the way or summarizing behavior of longer functions/methods (also private ones) would help me a lot.

discovery/test/invalid_definition/README.md

storage/sql_migrations/2_discoveryservice.up.sql

discovery/definition.go

discovery/interface.go

discovery/module.go

vcr/pe/presentation_definition.go

storage/sql_migrations/2_discoveryservice.up.sql

discovery/store.go

discovery/config.go

discovery/module.go

vcr/pe/presentation_definition.go

vcr/pe/presentation_submission.go

vcr/pe/schema/v2/schema.go

gerardsn

comments are suggestions. Requesting changes because retracted_jti is not a DID (see previous comments)

discovery/module.go

gerardsn · 2023-11-30T11:25:55Z

discovery/store.go

+	if err := s.prune(); err != nil {
+		return err
+	}


it depends how care orgs will be allowed to access discovery services. As long as care orgs can't use use-cases that aren't configured in a service definition I suppose there is no issue.

rationale: If a node only acts as client (applies to most) and all service definitions are gone after a restart (config error), the db will only contain outdated info. The client will also no longer publish its clients to the discovery services. This could result in weird situations where orgs can find orgs managed by other vendors but not their own, etc..

additional tests additional test PR feedback PR feedback test fix remove failing test (#2618) added e2e test for OpenID4VP s2s flow (#2617) PEX: Provide ParseEnvelope to correctly parse PEX VP envelopes (#2620) Bump schneider.vip/problem from 1.8.1 to 1.9.0 (#2622) Bumps [schneider.vip/problem](https://github.com/mschneider82/problem) from 1.8.1 to 1.9.0. - [Commits](mschneider82/problem@v1.8.1...v1.9.0) --- updated-dependencies: - dependency-name: schneider.vip/problem dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump golang.org/x/crypto from 0.15.0 to 0.16.0 (#2628) Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.15.0 to 0.16.0. - [Commits](golang/crypto@v0.15.0...v0.16.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump golang.org/x/time from 0.4.0 to 0.5.0 (#2627) Bumps [golang.org/x/time](https://github.com/golang/time) from 0.4.0 to 0.5.0. - [Commits](golang/time@v0.4.0...v0.5.0) --- updated-dependencies: - dependency-name: golang.org/x/time dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Missing rfc021 e2e tests changes (#2623) Discovery: SQLite-based server implementation (#2589) Bump github.com/nuts-foundation/go-leia/v4 from 4.0.0 to 4.0.1 (#2632) Bumps [github.com/nuts-foundation/go-leia/v4](https://github.com/nuts-foundation/go-leia) from 4.0.0 to 4.0.1. - [Release notes](https://github.com/nuts-foundation/go-leia/releases) - [Commits](nuts-foundation/go-leia@v4.0.0...v4.0.1) --- updated-dependencies: - dependency-name: github.com/nuts-foundation/go-leia/v4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Root URL server config property to replace auth.publicurl (#2633) change idToDID to use did:web in iam API (#2635) Bump github.com/lestrrat-go/jwx/v2 from 2.0.17 to 2.0.18 (#2645) Bumps [github.com/lestrrat-go/jwx/v2](https://github.com/lestrrat-go/jwx) from 2.0.17 to 2.0.18. - [Release notes](https://github.com/lestrrat-go/jwx/releases) - [Changelog](https://github.com/lestrrat-go/jwx/blob/develop/v2/Changes) - [Commits](lestrrat-go/jwx@v2.0.17...v2.0.18) --- updated-dependencies: - dependency-name: github.com/lestrrat-go/jwx/v2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump github.com/nats-io/nats-server/v2 from 2.10.5 to 2.10.6 (#2644) Bumps [github.com/nats-io/nats-server/v2](https://github.com/nats-io/nats-server) from 2.10.5 to 2.10.6. - [Release notes](https://github.com/nats-io/nats-server/releases) - [Changelog](https://github.com/nats-io/nats-server/blob/main/.goreleaser.yml) - [Commits](nats-io/nats-server@v2.10.5...v2.10.6) --- updated-dependencies: - dependency-name: github.com/nats-io/nats-server/v2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump alpine from 3.18.4 to 3.18.5 (#2636) Bumps alpine from 3.18.4 to 3.18.5. --- updated-dependencies: - dependency-name: alpine dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> more NO_CONTENT timeout godoc

reinkrul force-pushed the usecaselist-maintainer branch from c1c4acf to 8108827 Compare November 6, 2023 16:02

reinkrul requested review from woutslakhorst and gerardsn and removed request for woutslakhorst November 6, 2023 16:02

woutslakhorst requested changes Nov 8, 2023

View reviewed changes

storage/config.go Outdated Show resolved Hide resolved

reinkrul force-pushed the usecaselist-maintainer branch from bcae7aa to 64d1cf4 Compare November 12, 2023 13:14

reinkrul requested a review from woutslakhorst November 12, 2023 13:15

reinkrul marked this pull request as draft November 17, 2023 07:02

reinkrul removed request for gerardsn and woutslakhorst November 17, 2023 07:02

reinkrul force-pushed the usecaselist-maintainer branch 3 times, most recently from b84d289 to 5932201 Compare November 19, 2023 08:08

gerardsn and others added 2 commits November 20, 2023 07:05

Discovery: SQL based server implementation

865d95e

credential properties index

9dacfc4

reinkrul force-pushed the usecaselist-maintainer branch from eb54df9 to 9dacfc4 Compare November 20, 2023 06:05

regen

ff9b719

reinkrul requested a review from woutslakhorst November 20, 2023 06:05

reinkrul changed the title ~~Usecase: SQLite-based maintainer implementation~~ Discovery: SQLite-based server implementation Nov 20, 2023

reinkrul requested a review from gerardsn November 20, 2023 06:06

reinkrul marked this pull request as ready for review November 20, 2023 06:06

woutslakhorst requested changes Nov 20, 2023

View reviewed changes

reinkrul added 5 commits November 20, 2023 14:01

sqlite concurrency

984f8aa

SQL schema feedback fixes

fe929e3

feedback

8b7bce7

unusedfunc

6d8a88d

godoc, moved errors to interface.go

55dcede

reinkrul requested a review from woutslakhorst November 20, 2023 16:08

gerardsn requested changes Nov 23, 2023

View reviewed changes

reinkrul added 7 commits November 24, 2023 12:12

PEX: Changed path_nested to be object instead of slice (#2609)

e44fa5f

wip

4d815cb

PR feedback

28f2795

fixed tests

f6f4caa

Storage: do not write sqlite.db to source dir

ffd37cb

added VP verify, more tests, organized tests

b6957d3

feedback

442cab1

woutslakhorst requested changes Nov 27, 2023

View reviewed changes

reinkrul added 7 commits November 29, 2023 06:47

feedback

4eaadb5

config flags

91b0bef

copyriht

e751c88

documentation

96d172d

Merge branch 'master' into usecaselist-maintainer

1cd95f6

tests, todo

3a759fc

feedback

2467cc7

reinkrul requested review from woutslakhorst and gerardsn November 29, 2023 14:12

woutslakhorst approved these changes Nov 29, 2023

View reviewed changes

gerardsn requested changes Nov 30, 2023

View reviewed changes

reinkrul requested a review from gerardsn November 30, 2023 13:38

reinkrul added 2 commits November 30, 2023 14:39

tweak error msg

7c25c86

remove requirement retraction ID is scoped to issuer

0306b39

gerardsn approved these changes Nov 30, 2023

View reviewed changes

reinkrul merged commit d1bc3af into master Nov 30, 2023
9 checks passed

reinkrul deleted the usecaselist-maintainer branch November 30, 2023 14:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Discovery: SQLite-based server implementation #2589

Discovery: SQLite-based server implementation #2589

reinkrul commented Nov 6, 2023 •

edited

Loading

woutslakhorst left a comment

reinkrul commented Nov 9, 2023

reinkrul commented Nov 20, 2023

reinkrul commented Nov 20, 2023

reinkrul commented Nov 20, 2023

gerardsn left a comment

gerardsn left a comment •

edited

Loading

gerardsn Nov 30, 2023

Discovery: SQLite-based server implementation #2589

Discovery: SQLite-based server implementation #2589

Conversation

reinkrul commented Nov 6, 2023 • edited Loading

woutslakhorst left a comment

Choose a reason for hiding this comment

reinkrul commented Nov 9, 2023

reinkrul commented Nov 20, 2023

reinkrul commented Nov 20, 2023

reinkrul commented Nov 20, 2023

gerardsn left a comment

Choose a reason for hiding this comment

gerardsn left a comment • edited Loading

Choose a reason for hiding this comment

gerardsn Nov 30, 2023

Choose a reason for hiding this comment

reinkrul commented Nov 6, 2023 •

edited

Loading

gerardsn left a comment •

edited

Loading