Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Something went really wrong! OT-node shutting down... TypeError: Cannot read properties of undefined (reading 'assertionId') #3394

Open
haroldboom opened this issue Nov 5, 2024 · 2 comments
Open

Something went really wrong! OT-node shutting down... TypeError: Cannot read properties of undefined (reading 'assertionId') #3394

haroldboom opened this issue Nov 5, 2024 · 2 comments

Comments

@haroldboom
Copy link
Contributor

Issue description

Freshy rebuilt node, new paranets and node is in a reboot loop

Expected behavior

Normal startup and run

Actual behavior

Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] TRACE: Storing new session stream for remotePeerId: QmbKh9rxcfEeysLu3DbwDfAVBRNPtjMkk1ngqs2jEPEUqf with operation id: cce218e5-a8a6-4663-a7a5-af591e8e63cc
Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] DEBUG: Receiving message from QmbKh9rxcfEeysLu3DbwDfAVBRNPtjMkk1ngqs2jEPEUqf to QmPs2wjFAtb4uRuPZyKF2dt2gjG7vBdRu1bdbv2zW66iSv: protocol: /get/1.0.0, messageType: PROTOCOL_INIT;
Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] USERLVL: {Operation ID: cce218e5-a8a6-4663-a7a5-af591e8e63cc} [v1_0_0HandleGetInitCommand] (Command ID: ea436511-3be2-4efd-bc6f-d58230093af5) level-change
Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] TRACE: {Operation ID: cce218e5-a8a6-4663-a7a5-af591e8e63cc} [v1_0_0HandleGetInitCommand] (Command ID: ea436511-3be2-4efd-bc6f-d58230093af5) Checking if assertion 0xe388cf288822784ae94b2e5cb481ce6c6862c4d958392c1f9fd0d0921beed320 exists for state 0xe388cf288822784ae94b2e5cb481ce6c6862c4d958392c1f9fd0d0921beed320, on blockchain: base:84532, contract: 0xb8B904c73D2fB4D8c173298A51c27Fab70222c32, and tokenId: 5683591
Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] TRACE: {Operation ID: cce218e5-a8a6-4663-a7a5-af591e8e63cc} [v1_0_0HandleGetInitCommand] (Command ID: ea436511-3be2-4efd-bc6f-d58230093af5) Checking if assertion exists in pending storage at path: /root/ot-node/data/pending_storage_cache/public/base:84532:0xb8b904c73d2fb4d8c173298a51c27fab70222c32:5683591/0xe388cf288822784ae94b2e5cb481ce6c6862c4d958392c1f9fd0d0921beed320
Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] TRACE: {Operation ID: cce218e5-a8a6-4663-a7a5-af591e8e63cc} [v1_0_0HandleGetInitCommand] (Command ID: ea436511-3be2-4efd-bc6f-d58230093af5) Node connected to QmNpDvED1ud6N8HpH2KUG1X6v2Vcp3h4YsFYbki4SWxHTQ, updating sharding table last seen and last dialed.
Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] DEBUG: Sending response from QmPs2wjFAtb4uRuPZyKF2dt2gjG7vBdRu1bdbv2zW66iSv to QmbKh9rxcfEeysLu3DbwDfAVBRNPtjMkk1ngqs2jEPEUqf: protocol: /get/1.0.0, messageType: NACK;
Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] TRACE: Session found remotePeerId: QmbKh9rxcfEeysLu3DbwDfAVBRNPtjMkk1ngqs2jEPEUqf, operation id: cce218e5-a8a6-4663-a7a5-af591e8e63cc
Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] TRACE: Removed session for remotePeerId: QmbKh9rxcfEeysLu3DbwDfAVBRNPtjMkk1ngqs2jEPEUqf, operationId: cce218e5-a8a6-4663-a7a5-af591e8e63cc.
Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] TRACE: Storing new session stream for remotePeerId: QmaG78PiYa97z6ArM7uYZkSXaX1SCGWpwBGPspDq1xgm71 with operation id: e6cfb4e6-2397-456f-8ec0-b05272c43b76
Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] DEBUG: Receiving message from QmaG78PiYa97z6ArM7uYZkSXaX1SCGWpwBGPspDq1xgm71 to QmPs2wjFAtb4uRuPZyKF2dt2gjG7vBdRu1bdbv2zW66iSv: protocol: /store/1.0.0, messageType: PROTOCOL_REQUEST;
Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] DEBUG: {Operation ID: cce218e5-a8a6-4663-a7a5-af591e8e63cc} [v1_0_0HandleGetInitCommand] (Command ID: ea436511-3be2-4efd-bc6f-d58230093af5) Reading operation id: e6cfb4e6-2397-456f-8ec0-b05272c43b76 cached data from file
Nov 05 01:07:48 curated2 node[697867]: Something went really wrong! OT-node shutting down... TypeError: Cannot read properties of undefined (reading 'assertionId')
Nov 05 01:07:48 curated2 node[697867]: at PublishController.v1_0_0HandleRequest (file:///root/ot-node/8.1.1+beta.3/src/controllers/rpc/publish-rpc-controller.js:33:45)
Nov 05 01:07:48 curated2 systemd[1]: otnode.service: Main process exited, code=exited, status=1/FAILURE
Nov 05 01:07:48 curated2 systemd[1]: otnode.service: Failed with result 'exit-code'.
Nov 05 01:07:48 curated2 systemd[1]: otnode.service: Consumed 10.861s CPU time.
Nov 05 01:07:48 curated2 systemd[1]: otnode.service: Scheduled restart job, restart counter is at 6.
Nov 05 01:07:48 curated2 systemd[1]: Stopped OriginTrail V8 Node.
Nov 05 01:07:48 curated2 systemd[1]: otnode.service: Consumed 10.861s CPU time.

Steps to reproduce the problem

  1. Only seems to be on 1 of my nodes, all identical hardware and publishing scripts, paranets etc. I tried fully removing ot-node folder and also the mysql databases and starting over but somehow it keeps crashing with the same error

Specifications

  • Node version: v8.1.1 beta 3
  • Platform: Ubuntu 22
  • Node wallet: 0xaE601Cf4074F9C455c0d2675Fc0622Eb5ee3e25c
  • Node libp2p identity: 347

Error logs

Nov 05 01:05:34 curated2 node[697766]: [2024-11-05 01:05:34] TRACE: {Operation ID: 74543a8d-b2a5-4d1f-899b-a24719f51710} [findNodesCommand] (Command ID: 2989ce18-299b-4ed5-8924-c22b315fa3e5) Node connected to QmbKh9rxcfEeysLu3DbwDfAVBRNPtjMkk1ngqs2jEPEUqf, updating sharding table last seen and last dialed. Nov 05 01:05:35 curated2 node[697766]: [2024-11-05 01:05:35] TRACE: {Operation ID: 74543a8d-b2a5-4d1f-899b-a24719f51710} [findNodesCommand] (Command ID: 2989ce18-299b-4ed5-8924-c22b315fa3e5) Node connected to Qmbe7mDtqgaXVgMjUT6BgeAUQK7ybtFeue7Sw21NYYRVwF, updating sharding table last seen and last dialed. Nov 05 01:05:35 curated2 node[697766]: [2024-11-05 01:05:35] TRACE: {Operation ID: 74543a8d-b2a5-4d1f-899b-a24719f51710} [findNodesCommand] (Command ID: 2989ce18-299b-4ed5-8924-c22b315fa3e5) Node connected to QmVikRKmkqrxEg2q8uXza3uLtRE5WLNEMCfX7WgrCBVAUw, updating sharding table last seen and last dialed. Nov 05 01:05:35 curated2 node[697766]: [2024-11-05 01:05:35] TRACE: {Operation ID: 74543a8d-b2a5-4d1f-899b-a24719f51710} [findNodesCommand] (Command ID: 2989ce18-299b-4ed5-8924-c22b315fa3e5) Node connected to QmemEAoebqoXnv8ZqRjn25WtuBkH45MDU1gFyFARbudNhy, updating sharding table last seen and last dialed. Nov 05 01:05:35 curated2 node[697766]: [2024-11-05 01:05:35] TRACE: {Operation ID: 74543a8d-b2a5-4d1f-899b-a24719f51710} [findNodesCommand] (Command ID: 2989ce18-299b-4ed5-8924-c22b315fa3e5) Node connected to Qmcx6RG1bA4yaZz2tDyDfep3h8xDYDyEgX6dmnNCRkVAnU, updating sharding table last seen and last dialed. Nov 05 01:05:35 curated2 node[697766]: [2024-11-05 01:05:35] TRACE: Storing new session stream for remotePeerId: QmaG78PiYa97z6ArM7uYZkSXaX1SCGWpwBGPspDq1xgm71 with operation id: c2679a89-ff15-4886-ae74-c46ebdd2fd1b Nov 05 01:05:35 curated2 node[697766]: [2024-11-05 01:05:35] DEBUG: Receiving message from QmaG78PiYa97z6ArM7uYZkSXaX1SCGWpwBGPspDq1xgm71 to QmPs2wjFAtb4uRuPZyKF2dt2gjG7vBdRu1bdbv2zW66iSv: protocol: /store/1.0.0, messageType: PROTOCOL_REQUEST; Nov 05 01:05:35 curated2 node[697766]: [2024-11-05 01:05:35] DEBUG: {Operation ID: 74543a8d-b2a5-4d1f-899b-a24719f51710} [findNodesCommand] (Command ID: 2989ce18-299b-4ed5-8924-c22b315fa3e5) Reading operation id: c2679a89-ff15-4886-ae74-c46ebdd2fd1b cached data from file Nov 05 01:05:35 curated2 node[697766]: Something went really wrong! OT-node shutting down... TypeError: Cannot read properties of undefined (reading 'assertionId') Nov 05 01:05:35 curated2 node[697766]: at PublishController.v1_0_0HandleRequest (file:///root/ot-node/8.1.1+beta.3/src/controllers/rpc/publish-rpc-controller.js:33:45) Nov 05 01:05:35 curated2 systemd[1]: otnode.service: Main process exited, code=exited, status=1/FAILURE Nov 05 01:05:35 curated2 systemd[1]: otnode.service: Failed with result 'exit-code'. Nov 05 01:05:35 curated2 systemd[1]: otnode.service: Consumed 10.178s CPU time. Nov 05 01:05:35 curated2 systemd[1]: otnode.service: Scheduled restart job, restart counter is at 4. Nov 05 01:05:35 curated2 systemd[1]: Stopped OriginTrail V8 Node. Nov 05 01:05:35 curated2 systemd[1]: otnode.service: Consumed 10.178s CPU time.

Disclaimer

Please be aware that the issue reported on a public repository allows everyone to see your node logs, node details, and contact details. If you have any sensitive information, feel free to share it by sending an email to [email protected].
@botnumberseven
Copy link

Do all your nodes use the same RPC?

@haroldboom
Copy link
Contributor Author

Do all your nodes use the same RPC?

Split between 2 different RPCs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@botnumberseven @haroldboom