Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Add trace input via JSON #1265

Closed
wants to merge 45 commits into from
Closed

test: Add trace input via JSON #1265

Show file tree
Hide file tree
Changes from 42 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
039fa81
Fix merge
p-offtermatt Apr 25, 2023
bc18a8c
Lint file
p-offtermatt Sep 6, 2023
de89a18
Clean filepath before usage
p-offtermatt Sep 6, 2023
a4347f7
Wait downtime_duration, not single block
p-offtermatt Sep 6, 2023
65fdd40
Revert formatting change
p-offtermatt Sep 6, 2023
21a6f55
Readd comment string
p-offtermatt Sep 6, 2023
b7217c8
Revert formatting change
p-offtermatt Sep 6, 2023
55aa940
Add back comment string
p-offtermatt Sep 6, 2023
ef41ba8
Revert formatting change
p-offtermatt Sep 6, 2023
47a8970
Regenerate test data, better action registry
p-offtermatt Sep 6, 2023
199ca5c
Add tests and handle proposals
p-offtermatt Sep 7, 2023
935e2c9
fix: e2e trace format fails on slashthrottlesteps (#903)
p-offtermatt Apr 28, 2023
9b07d2c
Fix merge
p-offtermatt Sep 7, 2023
b204f92
Fix ibc-go version
p-offtermatt Sep 7, 2023
5b82e96
Fix custom marshal: no pointer receiver
p-offtermatt Sep 7, 2023
c8fd833
Add test for proposal unwrapping and fix bugs in it
p-offtermatt Sep 8, 2023
0114a81
Merge branch 'main' into ph/875-revived-json-trace-format
p-offtermatt Sep 8, 2023
0f6537c
Fix tests
p-offtermatt Sep 8, 2023
91c97aa
Attempt to fix json parsing trouble
p-offtermatt Sep 8, 2023
89182b7
Add todos to make pickup easier
p-offtermatt Sep 8, 2023
4bab3dd
Change registries to switch statements
p-offtermatt Sep 12, 2023
e4ae9d5
test: add rapid pbt for trace parser (#913)
p-offtermatt May 12, 2023
d88deec
Update dependency versions
p-offtermatt Sep 12, 2023
81a504e
Add rapid tests for actions
p-offtermatt Sep 12, 2023
0a165d2
Add comment to action marshalling
p-offtermatt Sep 12, 2023
3a6d9f1
Don't capitalize function args
p-offtermatt Sep 12, 2023
a81f92d
Fix marshal tests
p-offtermatt Sep 13, 2023
d24f732
Merge branch 'main' into ph/875-revived-json-trace-format
p-offtermatt Sep 13, 2023
ddea97c
Regenerate traces
p-offtermatt Sep 13, 2023
4865984
Merge branch 'main' into ph/875-revived-json-trace-format
p-offtermatt Sep 13, 2023
bb24be3
Fix change in denom action
p-offtermatt Sep 13, 2023
7ab4f2c
Add light client attacks to action gens
p-offtermatt Sep 13, 2023
d882ad1
Add light client attacks to switch-case in marshall function
p-offtermatt Sep 13, 2023
4dcc9e0
Start rewriting logic for choosing test cases
p-offtermatt Sep 13, 2023
54c7f61
Finish map for test runner choice
p-offtermatt Sep 15, 2023
fbec9d8
Fix usage infos and negate contains
p-offtermatt Sep 15, 2023
799f866
Fix Makefiles with new test case format
p-offtermatt Sep 15, 2023
efd6d53
Rename steps to be more explicit
p-offtermatt Sep 15, 2023
60d510f
Change steps/test run delimiter
p-offtermatt Sep 15, 2023
c09ddee
Add support for reading traces from files
p-offtermatt Sep 15, 2023
53deeff
Add trace test to automated tests
p-offtermatt Sep 15, 2023
04e319f
Fix typo: frAction
p-offtermatt Sep 15, 2023
b00b22a
Remove mistakenly introduced arg from CometMock
p-offtermatt Sep 15, 2023
e2175c2
Remove trace folder
p-offtermatt Sep 15, 2023
d271175
Update tests/e2e/actions.go
p-offtermatt Sep 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .github/workflows/automated-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,3 +56,17 @@ jobs:
go-version: "1.20"
- name: E2E tests
run: make test-e2e-short-cometmock
Trace-Tests:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a test that reads traces to the CI to check that trace-reading does not break. Could be expanded in the future

runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
lfs: true
- name: Checkout LFS objects
run: git lfs checkout
- name: Setup Go
uses: actions/setup-go@v4
with:
go-version: "1.20"
- name: E2E tests
run: make test-trace
12 changes: 6 additions & 6 deletions .github/workflows/manual-e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
with:
go-version: "1.20" # The Go version to download (if necessary) and use.
- name: E2E happy-path test
run: go run ./tests/e2e/... --tc happy-path
run: go run ./tests/e2e/... --tc happy-path::default
changeover-test:
runs-on: ubuntu-latest
timeout-minutes: 20
Expand All @@ -36,7 +36,7 @@ jobs:
with:
go-version: "1.20" # The Go version to download (if necessary) and use.
- name: E2E changeover test
run: go run ./tests/e2e/... --tc changeover
run: go run ./tests/e2e/... --tc changeover::changeover
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From description

for test-file, specifying the test runner separately is necessary

It seems to me like specifying testrunner is mostly relevant to the --test-file flag. Is it neccessary to also require that callers specify the testrunner when using --tc?

Asking because the cli for e2e tests is becoming quite verbose to run a simple test from the manual (golang defined) test cases

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also we should document the required format for each flag in a readme if that's not already done

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also we should document the required format for each flag in a readme if that's not already done

You can see it by running --help, do you think that is good enough?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From description

for test-file, specifying the test runner separately is necessary

It seems to me like specifying testrunner is mostly relevant to the --test-file flag. Is it neccessary to also require that callers specify the testrunner when using --tc?

Asking because the cli for e2e tests is becoming quite verbose to run a simple test from the manual (golang defined) test cases

Okay, I can see that. I'm between making it optional and removing it for the test cases, and I think for now I will remove it for --tc. Can be added back later if we need it.

democracy-reward-test:
runs-on: ubuntu-latest
timeout-minutes: 20
Expand All @@ -52,7 +52,7 @@ jobs:
with:
go-version: "1.20" # The Go version to download (if necessary) and use.
- name: E2E democracy-reward tests
run: go run ./tests/e2e/... --tc democracy-reward
run: go run ./tests/e2e/... --tc democracy-reward::democracy-reward
democracy-test:
runs-on: ubuntu-latest
timeout-minutes: 20
Expand All @@ -68,7 +68,7 @@ jobs:
with:
go-version: "1.20" # The Go version to download (if necessary) and use.
- name: E2E democracy tests
run: go run ./tests/e2e/... --tc democracy
run: go run ./tests/e2e/... --tc democracy::democracy
slash-throttle-test:
runs-on: ubuntu-latest
timeout-minutes: 20
Expand All @@ -84,7 +84,7 @@ jobs:
with:
go-version: "1.20" # The Go version to download (if necessary) and use.
- name: E2E slash-throttle tests
run: go run ./tests/e2e/... --tc slash-throttle
run: go run ./tests/e2e/... --tc slash-throttle::slash-throttle
multiconsumer-test:
runs-on: ubuntu-latest
timeout-minutes: 40
Expand All @@ -100,4 +100,4 @@ jobs:
with:
go-version: "1.20" # The Go version to download (if necessary) and use.
- name: E2E multi-consumer tests
run: go run ./tests/e2e/... --tc multiconsumer
run: go run ./tests/e2e/... --tc multiconsumer::multiconsumer
12 changes: 6 additions & 6 deletions .github/workflows/nightly-e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ jobs:
with:
go-version: "1.20" # The Go version to download (if necessary) and use.
- name: E2E happy-path test
run: go run ./tests/e2e/... --tc happy-path
run: go run ./tests/e2e/... --tc happy-path::default
changeover-test:
runs-on: ubuntu-latest
timeout-minutes: 20
Expand All @@ -49,7 +49,7 @@ jobs:
with:
go-version: "1.20" # The Go version to download (if necessary) and use.
- name: E2E changeover test
run: go run ./tests/e2e/... --tc changeover
run: go run ./tests/e2e/... --tc changeover::changeover
democracy-reward-test:
runs-on: ubuntu-latest
timeout-minutes: 20
Expand All @@ -65,7 +65,7 @@ jobs:
with:
go-version: "1.20" # The Go version to download (if necessary) and use.
- name: E2E democracy-reward tests
run: go run ./tests/e2e/... --tc democracy-reward
run: go run ./tests/e2e/... --tc democracy-reward::democracy-reward
democracy-test:
runs-on: ubuntu-latest
timeout-minutes: 20
Expand All @@ -81,7 +81,7 @@ jobs:
with:
go-version: "1.20" # The Go version to download (if necessary) and use.
- name: E2E democracy tests
run: go run ./tests/e2e/... --tc democracy
run: go run ./tests/e2e/... --tc democracy::democracy
slash-throttle-test:
runs-on: ubuntu-latest
timeout-minutes: 20
Expand All @@ -97,7 +97,7 @@ jobs:
with:
go-version: "1.20" # The Go version to download (if necessary) and use.
- name: E2E slash-throttle tests
run: go run ./tests/e2e/... --tc slash-throttle
run: go run ./tests/e2e/... --tc slash-throttle::slash-throttle
multiconsumer-test:
runs-on: ubuntu-latest
timeout-minutes: 40
Expand All @@ -113,7 +113,7 @@ jobs:
with:
go-version: "1.20" # The Go version to download (if necessary) and use.
- name: E2E multi-consumer tests
run: go run ./tests/e2e/... --tc multiconsumer
run: go run ./tests/e2e/... --tc multiconsumer::multiconsumer

nightly-test-fail:
needs:
Expand Down
12 changes: 8 additions & 4 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,12 @@ test-diff:

# run only happy path E2E tests
test-e2e-short:
go run ./tests/e2e/... --tc happy-path
go run ./tests/e2e/... --tc happy-path::default

# run only happy path E2E tests with cometmock
# this set of traces does not test equivocation but it does check downtime
test-e2e-short-cometmock:
go run ./tests/e2e/... --tc happy-path-short --use-cometmock --use-gorelayer
go run ./tests/e2e/... --tc happy-path-short::default --use-cometmock --use-gorelayer

# run full E2E tests in sequence (including multiconsumer)
test-e2e-multi-consumer:
Expand All @@ -52,7 +52,7 @@ test-gaia-e2e:

# run only happy path E2E tests using latest tagged gaia
test-gaia-e2e-short:
go run ./tests/e2e/... --tc happy-path --use-gaia
go run ./tests/e2e/... --tc happy-path::default --use-gaia

# run full E2E tests in parallel (including multiconsumer) using latest tagged gaia
test-gaia-e2e-parallel:
Expand All @@ -66,7 +66,7 @@ test-gaia-e2e-tagged:
# run only happy path E2E tests using latest tagged gaia
# usage: GAIA_TAG=v9.0.0 make test-gaia-e2e-short-tagged
test-gaia-e2e-short-tagged:
go run ./tests/e2e/... --tc happy-path --use-gaia --gaia-tag $(GAIA_TAG)
go run ./tests/e2e/... --tc happy-path::default --use-gaia --gaia-tag $(GAIA_TAG)

# run full E2E tests in parallel (including multiconsumer) using specific tagged version of gaia
# usage: GAIA_TAG=v9.0.0 make test-gaia-e2e-parallel-tagged
Expand All @@ -77,6 +77,10 @@ test-gaia-e2e-parallel-tagged:
test-no-cache:
go test ./... -count=1 && go run ./tests/e2e/...

# test reading a trace from a file
test-trace:
go run ./tests/e2e/... --test-file tests/e2e/tracehandler_testdata/happyPath.json::default

###############################################################################
### Linting ###
###############################################################################
Expand Down
6 changes: 3 additions & 3 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ require (
github.com/cosmos/rosetta-sdk-go v0.10.0 // indirect
github.com/creachadair/taskgroup v0.4.2 // indirect
github.com/danieljoos/wincred v1.1.2 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/davecgh/go-spew v1.1.1
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.1.0 // indirect
github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f // indirect
github.com/dgraph-io/badger/v2 v2.2007.4 // indirect
Expand All @@ -88,7 +88,7 @@ require (
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/btree v1.1.2 // indirect
github.com/google/go-cmp v0.5.9 // indirect
github.com/google/go-cmp v0.5.9
github.com/google/orderedcode v0.0.1 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.2.3 // indirect
Expand Down Expand Up @@ -161,7 +161,7 @@ require (
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
nhooyr.io/websocket v1.8.6 // indirect
pgregory.net/rapid v0.5.5 // indirect
pgregory.net/rapid v0.5.7
sigs.k8s.io/yaml v1.3.0 // indirect
)

Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1882,8 +1882,8 @@ honnef.co/go/tools v0.0.1-2020.1.4/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9
honnef.co/go/tools v0.1.3/go.mod h1:NgwopIslSNH47DimFoV78dnkksY2EFtX0ajyb3K/las=
nhooyr.io/websocket v1.8.6 h1:s+C3xAMLwGmlI31Nyn/eAehUlZPwfYZu2JXM621Q5/k=
nhooyr.io/websocket v1.8.6/go.mod h1:B70DZP8IakI65RVQ51MsWP/8jndNma26DVA/nFSCgW0=
pgregory.net/rapid v0.5.5 h1:jkgx1TjbQPD/feRoK+S/mXw9e1uj6WilpHrXJowi6oA=
pgregory.net/rapid v0.5.5/go.mod h1:PY5XlDGj0+V1FCq0o192FdRhpKHGTRIWBgqjDBTrq04=
pgregory.net/rapid v0.5.7 h1:p7/XbOgyFY1I/3Q12UTXfos70VZTcgc3WeoyiEru5cs=
pgregory.net/rapid v0.5.7/go.mod h1:PY5XlDGj0+V1FCq0o192FdRhpKHGTRIWBgqjDBTrq04=
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4=
rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0=
Expand Down
1 change: 1 addition & 0 deletions tests/e2e/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
testdata
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ignore temporary folder that rapid puts test failures in

Loading