panic: some epoch 5851784562212864 older than epoch to seal 5851806452875264

[hzxa21]

Describe the bug

The panic happens when running a longevity test with commit ee444e0e4c54e4fdb5765d123bbfd5a68c9e3eaa

Buildkite
Grafana metric
Log

Error message/log

2024-01-29 19:18:35.496	
thread 'rw-main' panicked at src/storage/src/hummock/event_handler/uploader.rs:816:13:

2024-01-29 19:18:35.496	
some epoch 5851784562212864 older than epoch to seal 5851806452875264

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response

[hzxa21]

The panic happened on 19:18 but we did saw another log 7mins ago in 19:11 on the same compute node with prev epoch == the epoch in the above log

2024-01-29T19:11:10.832 INFO risingwave_storage::hummock::event_handler::hummock_event_handler: handle clear event. max_committed_epoch: 5851779319136256, max_synced_epoch: 5851779319136256, max_sealed_epoch: 5851781940707328

2024-01-29T19:11:10.905 ERROR risingwave_stream::task::stream_manager: actor exit with error actor_id=20070 error=failed to send message to actor 20065: Barrier(
    Barrier {
        epoch: EpochPair {
            curr: 5851785872736256,
            prev: 5851784562212864,
        },
        mutation: None,
        kind: Barrier,
        tracing_context: TracingContext(
            Context {
                entries: 0,
            },
        ),
        passed_actors: [
            20253,
            20070,
        ],
    },
)

[wenym1]

The panic never happens again. May have been resolved in the bidi-stream refactor. Close the issue for now. May reopen it when we see the panic again.