Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

state: Concurrent Map Writes #1174

Closed
crypto-services opened this issue Mar 5, 2024 · 8 comments · Fixed by #1176
Closed

state: Concurrent Map Writes #1174

crypto-services opened this issue Mar 5, 2024 · 8 comments · Fixed by #1176

Comments

@crypto-services
Copy link

crypto-services commented Mar 5, 2024 •
edited
Loading

We have noticed a rare race condition that results in either a concurrent write panic or a locked state on the main thread. This is reported as stemming from the following line:

bor/core/state/state_object.go

Line 173 in 241af1f

return s.GetCommittedState(key)

It happens infrequently and results in either a restart or the main thread no longer progressing.

Version: v1.2.3
OS: Ubuntu 22.04
Start of error log:

Mar 05 00:21:29 ovhc-eu-uk-poly-node-01 bor[316583]: fatal error: concurrent map writes
Mar 05 00:21:29 ovhc-eu-uk-poly-node-01 bor[316583]: goroutine 155060 [running]:
Mar 05 00:21:29 ovhc-eu-uk-poly-node-01 bor[316583]: github.com/ethereum/go-ethereum/core/state.(*stateObject).GetCommittedState(0xc054d9a0e0, {0x74, 0x39, 0x82, 0x3b, 0x81, 0xeb, 0x7f, 0xa6, 0x22, ...})
Mar 05 00:21:29 ovhc-eu-uk-poly-node-01 bor[316583]:         /home/runner/work/bor/bor/core/state/state_object.go:237 +0x5f3
Mar 05 00:21:29 ovhc-eu-uk-poly-node-01 bor[316583]: github.com/ethereum/go-ethereum/core/state.(*stateObject).GetState(...)
Mar 05 00:21:29 ovhc-eu-uk-poly-node-01 bor[316583]:         /home/runner/work/bor/bor/core/state/state_object.go:173
Mar 05 00:21:29 ovhc-eu-uk-poly-node-01 bor[316583]: github.com/ethereum/go-ethereum/core/state.(*StateDB).GetState.func1(0x3253382007295565?)
Mar 05 00:21:29 ovhc-eu-uk-poly-node-01 bor[316583]:         /home/runner/work/bor/bor/core/state/statedb.go:667 +0x135
Mar 05 00:21:29 ovhc-eu-uk-poly-node-01 bor[316583]: github.com/ethereum/go-ethereum/core/state.MVRead[...](0xc0b75d7000?, {0x37, 0x95, 0xab, 0xd1, 0x70, 0xcd, 0xb7, 0x16, 0xc7, ...}, ...)
Mar 05 00:21:29 ovhc-eu-uk-poly-node-01 bor[316583]:         /home/runner/work/bor/bor/core/state/statedb.go:292 +0x4ed
Mar 05 00:21:29 ovhc-eu-uk-poly-node-01 bor[316583]: github.com/ethereum/go-ethereum/core/state.(*StateDB).GetState(0xc0b9a39c00?, {0x37, 0x95, 0xab, 0xd1, 0x70, 0xcd, 0xb7, 0x16, 0xc7, ...}, ...)
@ty-sentio-xyz
Copy link

Same issue here. In my case this also just happened around 00:16 UTC.

It seems at the point of panicking, there are 2 goroutines that are related to stateObject:

Mar 05 00:16:23 goroutine 874088378 [running]:
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/state.(*stateObject).GetCommittedState(0xc0fc1517a0, {0xf3, 0xe4, 0x3, 0x3c, 0xb2, 0x7a, 0xc3, 0x0, 0x56, ...})
Mar 05 00:16:23         /home/ubuntu/bor/core/state/state_object.go:183 +0x94
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/state.(*stateObject).GetState(...)
Mar 05 00:16:23         /home/ubuntu/bor/core/state/state_object.go:173
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/state.(*StateDB).GetState.func1(0x7b1030a8380286a4?)
Mar 05 00:16:23         /home/ubuntu/bor/core/state/statedb.go:667 +0x128
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/state.MVRead[...](0xc09b68f600?, {0x37, 0x95, 0xab, 0xd1, 0x70, 0xcd, 0xb7, 0x16, 0xc7, ...}, ...)
Mar 05 00:16:23         /home/ubuntu/bor/core/state/statedb.go:318 +0x373
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/state.(*StateDB).GetState(0xc0fc770b08?, {0x37, 0x95, 0xab, 0xd1, 0x70, 0xcd, 0xb7, 0x16, 0xc7, ...}, ...)
Mar 05 00:16:23         /home/ubuntu/bor/core/state/statedb.go:664 +0x171
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.opSload(0xc0fc75c120?, 0xc79d580ea0, 0xc222dc7c50?)
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/instructions.go:573 +0x14c
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).Run(0xc79d580ea0, 0xc0fc169d40, {0xc26d0ab400, 0x1e4, 0x200}, 0x0, {0x0?, 0x0?})
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/interpreter.go:354 +0xaff
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).PreRun(0x5101000000004600?, 0xb5fd547b79898700?, {0xc26d0ab400?, 0x75acfb4eb9c6d8d4?, 0x4a9495a08db46d70?}, 0x9c?, {0x0?, 0x0?})
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/interpreter.go:192 +0x110
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.(*EVM).DelegateCall(0xc0fc75c120, {0x3aea440, 0xc0fc169c80?}, {0x0, 0x46, 0x0, 0x0, 0x0, 0x0, 0x1, ...}, ...)
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/evm.go:353 +0x5ed
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.opDelegateCall(0xc0fc75c120?, 0xc79d580ea0, 0xc222dc7c38)
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/instructions.go:836 +0x476
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).Run(0xc79d580ea0, 0xc0fc169c80, {0xc0fc83a2a0, 0x1e4, 0x760}, 0x0, {0x0?, 0x0?})
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/interpreter.go:354 +0xaff
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).PreRun(0x16b7cd70d1ab9537?, 0x595b4869bb1c64c7?, {0xc0fc83a2a0?, 0x2db3515aa9a3ae49?, 0x52380c8429d26f1e?}, 0x1e?, {0x0?, 0x0?})
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/interpreter.go:192 +0x110
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.(*EVM).Call(0xc0fc75c120, {0x3aea440, 0xc0fc169b00}, {0x37, 0x95, 0xab, 0xd1, 0x70, 0xcd, 0xb7, ...}, ...)
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/evm.go:245 +0xbc5
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.opCall(0xc0fc75c120?, 0xc79d580ea0, 0xc222dc7c08)
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/instructions.go:763 +0x61a
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).Run(0xc79d580ea0, 0xc0fc169b00, {0xc3570f23a0, 0x84, 0xe0}, 0x0, {0x0?, 0x0?})
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/interpreter.go:354 +0xaff
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).PreRun(0xd9782de9000000?, 0xda6f0082007f0000?, {0xc3570f23a0?, 0x8664da32890cca02?, 0x2b8bfa639a1aebec?}, 0x6d?, {0x0?, 0x0?})
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/interpreter.go:192 +0x110
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.(*EVM).Call(0xc0fc75c120, {0x3aea440, 0xc0fc169a40}, {0x0, 0x0, 0x0, 0xe9, 0x2d, 0x78, 0xd9, ...}, ...)
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/evm.go:245 +0xbc5
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.opCall(0xc0fc75c120?, 0xc79d580ea0, 0xc222dc7bf0)
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/instructions.go:763 +0x61a
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).Run(0xc79d580ea0, 0xc0fc169a40, {0xc65ea9b040, 0xe4, 0xec0}, 0x0, {0x0?, 0x0?})
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/interpreter.go:354 +0xaff
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).PreRun(0x60bbedea3084c97f?, 0x507298d7395bb370?, {0xc65ea9b040?, 0x611bafea918081ae?, 0x89a659234741ee75?}, 0xf3?, {0x0?, 0x0?})
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/interpreter.go:192 +0x110
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.(*EVM).Call(0xc0fc75c120, {0x3aea440, 0xc0fc169980}, {0x7f, 0xc9, 0x84, 0x30, 0xea, 0xed, 0xbb, ...}, ...)
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/evm.go:245 +0xbc5
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.opCall(0xc0fc75c120?, 0xc79d580ea0, 0xc222dc7bd8)
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/instructions.go:763 +0x61a
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).Run(0xc79d580ea0, 0xc0fc169980, {0xc567264000, 0x12c4, 0x12c4}, 0x0, {0x0?, 0x0?})
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/interpreter.go:354 +0xaff
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).PreRun(0x49cdfdb0d437f15f?, 0x8a577ef57c0ca3dc?, {0xc567264000?, 0xb50083736e803cc9?, 0x43d671e9c2cd7e35?}, 0x8d?, {0x0?, 0x0?})
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/interpreter.go:192 +0x110
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/vm.(*EVM).Call(0xc0fc75c120, {0x3ae9aa0, 0xc558bc6990}, {0x5f, 0xf1, 0x37, 0xd4, 0xb0, 0xfd, 0xcd, ...}, ...)
Mar 05 00:16:23         /home/ubuntu/bor/core/vm/evm.go:245 +0xbc5
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core.(*StateTransition).TransitionDb(0xc0fc773650, {0x0, 0x0})
Mar 05 00:16:23         /home/ubuntu/bor/core/state_transition.go:464 +0x92f
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core.ApplyMessageNoFeeBurnOrTip(0xc0fc75c120, {0xc563dcae58, {0x54, 0xaf, 0x39, 0xeb, 0xab, 0x5d, 0x13, 0x70, ...}, ...}, ...)
Mar 05 00:16:23         /home/ubuntu/bor/core/state_transition.go:207 +0x105
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core.(*ExecutionTask).Execute(0xc0fb0a1d40, 0xc1f7d51f40, 0x2)
Mar 05 00:16:23         /home/ubuntu/bor/core/parallel_state_processor.go:119 +0x693
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/blockstm.(*ExecVersionView).Execute(0xc0fc773e68)
Mar 05 00:16:23         /home/ubuntu/bor/core/blockstm/executor.go:42 +0x66
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/blockstm.(*ParallelExecutor).Prepare.func1.1({{0x47, 0x2}, {0x3b17fb8, 0xc0fb0a1d40}, 0xc1f7d51f40, {0x54, 0xaf, 0x39, 0xeb, 0xab, ...}})
Mar 05 00:16:23         /home/ubuntu/bor/core/blockstm/executor.go:343 +0x6e
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/blockstm.(*ParallelExecutor).Prepare.func1(0xd59b33?)
Mar 05 00:16:23         /home/ubuntu/bor/core/blockstm/executor.go:373 +0x17b
Mar 05 00:16:23 created by github.com/ethereum/go-ethereum/core/blockstm.(*ParallelExecutor).Prepare in goroutine 874087574
Mar 05 00:16:23         /home/ubuntu/bor/core/blockstm/executor.go:334 +0x2c5

Mar 05 00:16:23 goroutine 874088379 [runnable]:
Mar 05 00:16:23 github.com/syndtr/goleveldb/leveldb/cache.(*Cache).Get(0xc0017144c0, 0x0?, 0xc472c52070?, 0xc0fbd06578)
Mar 05 00:16:23         /home/ubuntu/.go/github.com/syndtr/[email protected]/leveldb/cache/cache.go:401 +0x2b9
Mar 05 00:16:23 github.com/syndtr/goleveldb/leveldb/cache.(*NamespaceGetter).Get(...)
Mar 05 00:16:23         /home/ubuntu/.go/github.com/syndtr/[email protected]/leveldb/cache/cache.go:58
Mar 05 00:16:23 github.com/syndtr/goleveldb/leveldb/table.(*Reader).readBlockCached(0xc7fff92000, {0x43d73b?, 0xc0fbd06798?}, 0x1, 0x20?)
Mar 05 00:16:23         /home/ubuntu/.go/github.com/syndtr/[email protected]/leveldb/table/reader.go:625 +0xa5
Mar 05 00:16:23 github.com/syndtr/goleveldb/leveldb/table.(*Reader).getIndexBlock(0xc9314f5977?, 0x60?)
Mar 05 00:16:23         /home/ubuntu/.go/github.com/syndtr/[email protected]/leveldb/table/reader.go:712 +0x52
Mar 05 00:16:23 github.com/syndtr/goleveldb/leveldb/table.(*Reader).find(0xc7fff92000, {0xc6f123b3b0, 0x49, 0x49}, 0x1, 0x0, 0x0)
Mar 05 00:16:23         /home/ubuntu/.go/github.com/syndtr/[email protected]/leveldb/table/reader.go:829 +0x114
Mar 05 00:16:23 github.com/syndtr/goleveldb/leveldb/table.(*Reader).Find(...)
Mar 05 00:16:23         /home/ubuntu/.go/github.com/syndtr/[email protected]/leveldb/table/reader.go:922
Mar 05 00:16:23 github.com/syndtr/goleveldb/leveldb.(*tOps).find(0xc0fbd06928?, 0x7f6269?, {0xc6f123b3b0, 0x49, 0x49}, 0xd06035?)
Mar 05 00:16:23         /home/ubuntu/.go/github.com/syndtr/[email protected]/leveldb/table.go:450 +0x106
Mar 05 00:16:23 github.com/syndtr/goleveldb/leveldb.(*version).get.func1(0x3, 0xc6f123b3b0?)
Mar 05 00:16:23         /home/ubuntu/.go/github.com/syndtr/[email protected]/leveldb/version.go:180 +0x209
Mar 05 00:16:23 github.com/syndtr/goleveldb/leveldb.(*version).walkOverlapping(0xc8c01f7500, {0x0, 0x0, 0xc000e8e290?}, {0xc6f123b3b0, 0x49, 0x49}, 0xc0fbd06b88, 0xc0fbd06b58)
Mar 05 00:16:23         /home/ubuntu/.go/github.com/syndtr/[email protected]/leveldb/version.go:128 +0x252
Mar 05 00:16:23 github.com/syndtr/goleveldb/leveldb.(*version).get(0xc8c01f7500, {0x0, 0x0, 0x0}, {0xc6f123b3b0, 0x49, 0x49}, 0x0, 0x0)
Mar 05 00:16:23         /home/ubuntu/.go/github.com/syndtr/[email protected]/leveldb/version.go:164 +0x2f8
Mar 05 00:16:23 github.com/syndtr/goleveldb/leveldb.(*DB).get(0xc000226000, 0x0, {0x0, 0x0, 0x0}, {0xc7206bad20?, 0x41, 0xc0fbd06dd8?}, 0x14c92a1581, 0x0)
Mar 05 00:16:23         /home/ubuntu/.go/github.com/syndtr/[email protected]/leveldb/db.go:785 +0x547
Mar 05 00:16:23 github.com/syndtr/goleveldb/leveldb.(*DB).Get(0xc000226000, {0xc7206bad20, 0x41, 0x60}, 0x7f1fa61a3140?)
Mar 05 00:16:23         /home/ubuntu/.go/github.com/syndtr/[email protected]/leveldb/db.go:851 +0x14b
Mar 05 00:16:23 github.com/ethereum/go-ethereum/ethdb/leveldb.(*Database).Get(0xc7206bad20?, {0xc7206bad20?, 0xc0fbd06ec8?, 0x457b0f?})
Mar 05 00:16:23         /home/ubuntu/bor/ethdb/leveldb/leveldb.go:231 +0x1e
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/rawdb.ReadStorageSnapshot({0x7f1eaf3fc4b0, 0xc0012aa5d0}, {0x66, 0xaa, 0x5f, 0x3b, 0x76, 0x26, 0x82, 0xcc, ...}, ...)
Mar 05 00:16:23         /home/ubuntu/bor/core/rawdb/accessors_snapshot.go:98 +0x1a2
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/state/snapshot.(*diskLayer).Storage(0xc1986f13b0, {0x66, 0xaa, 0x5f, 0x3b, 0x76, 0x26, 0x82, 0xcc, 0x7f, ...}, ...)
Mar 05 00:16:23         /home/ubuntu/bor/core/state/snapshot/disklayer.go:157 +0x211
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/state/snapshot.(*diffLayer).Storage(0xc266d04500, {0x66, 0xaa, 0x5f, 0x3b, 0x76, 0x26, 0x82, 0xcc, 0x7f, ...}, ...)
Mar 05 00:16:23         /home/ubuntu/bor/core/state/snapshot/difflayer.go:401 +0x253
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/state.(*stateObject).GetCommittedState(0xc0fc1517a0, {0xf9, 0x8b, 0xe1, 0x9b, 0xe8, 0xfb, 0xe5, 0x83, 0xff, ...})
Mar 05 00:16:23         /home/ubuntu/bor/core/state/state_object.go:204 +0x21a
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/state.(*stateObject).GetState(...)
Mar 05 00:16:23         /home/ubuntu/bor/core/state/state_object.go:173
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/state.(*StateDB).GetState.func1(0xc201d8aabb1ae6e5?)
Mar 05 00:16:23         /home/ubuntu/bor/core/state/statedb.go:667 +0x128
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/state.MVRead[...](0xc09b68f200?, {0x37, 0x95, 0xab, 0xd1, 0x70, 0xcd, 0xb7, 0x16, 0xc7, ...}, ...)
Mar 05 00:16:23         /home/ubuntu/bor/core/state/statedb.go:292 +0x4bf
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/state.(*StateDB).GetState(0xc34c78da00?, {0x37, 0x95, 0xab, 0xd1, 0x70, 0xcd, 0xb7, 0x16, 0xc7, ...}, ...)
Mar 05 00:16:23         /home/ubuntu/bor/core/state/statedb.go:664 +0x171
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/state.(*StateDB).ApplyMVWriteSet(0xc0fbd07f38?, {0xc0fb566000, 0x32, 0xa39805dfa6f5e906?})
Mar 05 00:16:23         /home/ubuntu/bor/core/state/statedb.go:387 +0x4f0
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core.(*ExecutionTask).Settle(0xc0fb0a18c0)
Mar 05 00:16:23         /home/ubuntu/bor/core/parallel_state_processor.go:181 +0x305
Mar 05 00:16:23 github.com/ethereum/go-ethereum/core/blockstm.(*ParallelExecutor).Prepare.func2()
Mar 05 00:16:23         /home/ubuntu/bor/core/blockstm/executor.go:383 +0x3c
Mar 05 00:16:23 created by github.com/ethereum/go-ethereum/core/blockstm.(*ParallelExecutor).Prepare in goroutine 874087574
Mar 05 00:16:23         /home/ubuntu/bor/core/blockstm/executor.go:381 +0x3c7

@ty-sentio-xyz
Copy link

In fact this may not be as rare as we thought.

After panic, when the node has synced to roughly the same height, it panicked again.

However, the third sync attempt proceeded without issue.

@ericjohncarlson
Copy link

Same here. We had 3 separately nodes all panic at around 00:16 UTC as well. One of those panicked again at the same height, but made it past the second time.

@pratikspatil024
Copy link
Member

Thank you for informing, folks. We are having a look at this and will post a fix soon.

@pratikspatil024 pratikspatil024 linked a pull request Mar 5, 2024 that will close this issue
fix: concurrent map writes #1176
Merged
18 tasks
@stasb
Copy link

stasb commented Mar 5, 2024

Experienced this several times as well. I think it's related to #1006, and the block-stm code. I'm also doing a lot of tracing so not sure if thats related.

@pratikspatil024
Copy link
Member

This fix has been applied in this PR#1176 and is included in the latest release, would recommend upgrading your bor to v1.2.7.
Thank you.

@crypto-services
Copy link
Author

That was amazingly fast. Thanks!!

@gaia
Copy link

gaia commented Mar 6, 2024

Also saw the issue here.

@shorsher shorsher mentioned this issue Mar 8, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: concurrent map writes maticnetwork/bor
6 participants
@gaia @stasb @ericjohncarlson @pratikspatil024 @crypto-services @ty-sentio-xyz