Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky test t_core_node_down #113

Open
ieQu1 opened this issue Feb 1, 2023 · 1 comment
Open

Flaky test t_core_node_down #113

ieQu1 opened this issue Feb 1, 2023 · 1 comment

Comments

@ieQu1
Copy link
Member

ieQu1 commented Feb 1, 2023 

2023-01-31T13:14:23.336240+00:00 [email protected]
    '$kind': Connected to the core node
    node: '[email protected]'
    seqno: 4
    shard: '$mria_meta_shard'
2023-01-31T13:14:23.338843+00:00 [email protected]
    '$kind': Connected to the core node
    node: '[email protected]'
    seqno: 0
    shard: test_shard
2023-01-31T13:14:23.345780+00:00 [email protected]
    '$kind': Bootstrap of the shard is complete
    checkpoint: 1675170863345
    shard: '$mria_meta_shard'
2023-01-31T13:14:23.347808+00:00 [email protected]
    '$kind': Shard fully up
    node: '[email protected]'
    shard: '$mria_meta_shard'
2023-01-31T13:14:23.349204+00:00 [email protected]
    '$kind': Setting RLOG shard config
    shard: test_shard
    tables: [mria_helper_tab,test_bag,test_tab]
2023-01-31T13:14:23.352690+00:00 [email protected]
    '$kind': Setting RLOG shard config
    shard: test_shard
    tables: [mria_helper_tab,test_bag,test_tab]
2023-01-31T13:14:23.355637+00:00 [email protected]
    '$kind': Setting RLOG shard config
    shard: test_shard
    tables: [mria_helper_tab,test_bag,test_tab]
2023-01-31T13:14:23.357486+00:00 [email protected]
    '$kind': Bootstrap of the shard is complete
    checkpoint: 1675170863356
    shard: test_shard
2023-01-31T13:14:23.359137+00:00 [email protected]
    '$kind': Shard fully up
    node: '[email protected]'
    shard: test_shard
2023-01-31T13:14:24.451728+00:00 [email protected]
    '$kind': Remote RLOG agent died
    reason: shutdown
    repl_state: normal
2023-01-31T13:14:24.453281+00:00 [email protected]
    '$kind': Mria is stopped
=NOTICE REPORT==== 31-Jan-2023::13:14:24.453281 ===
    '$kind': Mria is stopped
2023-01-31T13:14:24.455053+00:00 [email protected]
    '$kind': Connected to the core node
    node: '[email protected]'
    seqno: 0
    shard: '$mria_meta_shard'
=INFO REPORT==== 31-Jan-2023::13:14:24.456627 ===
    application: mria
    exited: stopped
    type: temporary

2023-01-31T13:14:24.458026+00:00 [email protected]
    '$kind': Remote RLOG agent died
    reason: shutdown
    repl_state: normal
2023-01-31T13:14:24.460281+00:00 [email protected]
    '$kind': Mria is stopped
=NOTICE REPORT==== 31-Jan-2023::13:14:24.460281 ===
    '$kind': Mria is stopped
=INFO REPORT==== 31-Jan-2023::13:14:24.462131 ===
    application: mria
    exited: stopped
    type: temporary

2023-01-31T13:14:24.463300+00:00 [email protected]
    '$kind': Remote RLOG agent died
    reason: shutdown
    repl_state: bootstrap
=SUPERVISOR REPORT==== 31-Jan-2023::13:14:24.464206 ===
    supervisor: {local,mria_shards_sup}
    errorContext: child_terminated
    reason: shutdown
    offender: [{pid,<0.230.0>},
               {id,'$mria_meta_shard'},
               {mfargs,
                   {mria_replicant_shard_sup,start_link,['$mria_meta_shard']}},
               {restart_type,permanent},
               {significant,false},
               {shutdown,infinity},
               {child_type,supervisor}]

2023-01-31T13:14:31.318336+00:00 [email protected]
    '$kind': Starting mria
=NOTICE REPORT==== 31-Jan-2023::13:14:31.318336 ===
    '$kind': Starting mria
2023-01-31T13:14:31.319511+00:00 [email protected]
    '$kind': Starting mnesia
=NOTICE REPORT==== 31-Jan-2023::13:14:31.319511 ===
    '$kind': Starting mnesia
2023-01-31T13:14:31.320993+00:00 [email protected]
    '$kind': Creating new mnesia schema
    result: {error,{'[email protected]',{already_exists,'[email protected]'}}}
=NOTICE REPORT==== 31-Jan-2023::13:14:31.320993 ===
    '$kind': Creating new mnesia schema
    result: {error,{'[email protected]',{already_exists,'[email protected]'}}}
2023-01-31T13:14:31.322163+00:00 [email protected]
    '$kind': Starting shards
=NOTICE REPORT==== 31-Jan-2023::13:14:31.322163 ===
    '$kind': Starting shards
2023-01-31T13:14:31.324431+00:00 [email protected]
    '$kind': Setting RLOG shard config
    shard: '$mria_meta_shard'
    tables: [mria_schema]
2023-01-31T13:14:31.325200+00:00 [email protected]
    '$kind': Converging schema
2023-01-31T13:14:31.326006+00:00 [email protected]
    '$kind': Setting RLOG shard config
    shard: test_shard
    tables: [mria_helper_tab,test_bag,test_tab]
2023-01-31T13:14:31.327014+00:00 [email protected]
    '$kind': Setting RLOG shard config
    shard: test_shard
    tables: [mria_helper_tab,test_bag,test_tab]
2023-01-31T13:14:31.327968+00:00 [email protected]
    '$kind': Setting RLOG shard config
    shard: test_shard
    tables: [mria_helper_tab,test_bag,test_tab]
2023-01-31T13:14:31.329811+00:00 [email protected]
    '$kind': Mria is running
=NOTICE REPORT==== 31-Jan-2023::13:14:31.329811 ===
    '$kind': Mria is running
2023-01-31T13:14:31.331054+00:00 [email protected]
    '$kind': Shard fully up
    node: '[email protected]'
    shard: '$mria_meta_shard'
2023-01-31T13:14:31.331230+00:00 [email protected]
    '$kind': Starting mria
=NOTICE REPORT==== 31-Jan-2023::13:14:31.331230 ===
    '$kind': Starting mria
2023-01-31T13:14:31.332632+00:00 [email protected]
    '$kind': Starting mnesia
=NOTICE REPORT==== 31-Jan-2023::13:14:31.332632 ===
    '$kind': Starting mnesia
2023-01-31T13:14:31.334722+00:00 [email protected]
    '$kind': Starting shards
=NOTICE REPORT==== 31-Jan-2023::13:14:31.334722 ===
    '$kind': Starting shards
2023-01-31T13:14:31.335845+00:00 [email protected]
    '$kind': Shard fully up
    node: '[email protected]'
    shard: test_shard
2023-01-31T13:14:31.337794+00:00 [email protected]
    '$kind': Setting RLOG shard config
    shard: '$mria_meta_shard'
    tables: [mria_schema]
2023-01-31T13:14:31.338519+00:00 [email protected]
    '$kind': Converging schema
2023-01-31T13:14:31.339247+00:00 [email protected]
    '$kind': Setting RLOG shard config
    shard: test_shard
    tables: [mria_helper_tab,test_bag,test_tab]
2023-01-31T13:14:31.340108+00:00 [email protected]
    '$kind': Setting RLOG shard config
    shard: test_shard
    tables: [mria_helper_tab,test_bag,test_tab]
2023-01-31T13:14:31.340924+00:00 [email protected]
    '$kind': Setting RLOG shard config
    shard: test_shard
    tables: [mria_helper_tab,test_bag,test_tab]
2023-01-31T13:14:31.342724+00:00 [email protected]
    '$kind': Mria is running
=NOTICE REPORT==== 31-Jan-2023::13:14:31.342724 ===
    '$kind': Mria is running
2023-01-31T13:14:31.343971+00:00 [email protected]
    '$kind': Shard fully up
    node: '[email protected]'
    shard: '$mria_meta_shard'
2023-01-31T13:14:31.344673+00:00 [email protected]
    '$kind': Shard fully up
    node: '[email protected]'
    shard: test_shard
2023-01-31T13:14:31.401021+00:00 [email protected]
    '$kind': Connected to the core node
    node: '[email protected]'
    seqno: 0
    shard: test_shard
2023-01-31T13:14:31.401805+00:00 [email protected]
    '$kind': Connected to the core node
    node: '[email protected]'
    seqno: 1
    shard: '$mria_meta_shard'
2023-01-31T13:14:31.404558+00:00 [email protected]
    '$kind': Bootstrap of the shard is complete
    checkpoint: 1675170871404
    shard: '$mria_meta_shard'
2023-01-31T13:14:31.406252+00:00 [email protected]
    '$kind': Shard fully up
    node: '[email protected]'
    shard: '$mria_meta_shard'
2023-01-31T13:14:31.407648+00:00 [email protected]
    '$kind': Bootstrap of the shard is complete
    checkpoint: 1675170871407
    shard: test_shard
2023-01-31T13:14:31.409006+00:00 [email protected]
    '$kind': Shard fully up
    node: '[email protected]'
    shard: test_shard
2023-01-31T13:14:50.367968+00:00 
Run stage timed out.
Stacktrace: [{mria_SUITE,'-t_core_node_down/1-fun-11-',1,
                         [{file,"/__w/mria/mria/test/mria_SUITE.erl"},
                          {line,664}]},
             {mria_SUITE,t_core_node_down,1,
                         [{file,"/__w/mria/mria/test/mria_SUITE.erl"},
                          {line,635}]}]
Trace dump: "/__w/mria/mria/_build/test/logs/[email protected]_13.13.45/snabbkaffe/1675170890267940814.log"

2023-01-31T13:14:50.371998+00:00 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%                                                                            %%
%% Complete t_core_node_down                                                  %%
%%                                                                            %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@ieQu1
Copy link
Member Author

ieQu1 commented Feb 7, 2023

Root cause:

  1. Replicant starts a RW transaction T on a core node C.
  2. C rejoins the cluster via autoheal. Mnesia is restarted.
  3. Mnesia forgets about transaction T, it never returns.
  4. Transaction on the replicant also never returns.

@ieQu1 ieQu1 closed this as completed in baec6a8 Feb 7, 2023
@ieQu1 ieQu1 reopened this Feb 7, 2023
ieQu1 added a commit to ieQu1/mria that referenced this issue Jun 27, 2023
@ieQu1
fix: Fix flaky t_core_node_down testcase
b268f36 
Fixes: emqx#113
ieQu1 added a commit that referenced this issue Sep 6, 2023
@ieQu1
fix: Fix flaky t_core_node_down testcase
efd70ca 
Fixes: #113
ieQu1 added a commit that referenced this issue Sep 20, 2023
@ieQu1
fix: Fix flaky t_core_node_down testcase
ea086b9 
Fixes: #113
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ieQu1