failure on {ok, not_found} in rafter_consensus_fsm:600

[d0rc]

Here is full failure log and log of peer3 after restart attempt:

95:rafter Aniks$ bin/start-node peer3
Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Eshell V5.10.4  (abort with ^G)
([email protected])1> 16:35:49.642 [info] Application lager started on node '[email protected]'
16:35:49.642 [info] Application rafter started on node '[email protected]'
sys:get_state(peer3_log).
{state,{file_descriptor,prim_file,{#Port<0.3609>,16}},
       1,"./data/rafter_peer3.meta",182,
       {config,stable,
               [{peer1,'[email protected]'},
                {peer2,'[email protected]'},
                {peer3,'[email protected]'}],
               []},
       1,
       {meta,{peer1,'[email protected]'},1},
       {rafter_entry,config,1,1,
                     {config,stable,
                             [{peer1,'[email protected]'},
                              {peer2,'[email protected]'},
                              {peer3,'[email protected]'}],
                             []}},
       1,0}
([email protected])2> rafter:read_op(peer3, {list_keys, sometable}).
{ok,[somekey]}
([email protected])3> rafter:read_op(peer3, {put, sometable, onemorekey, onemoreval}).
{error,ets_read_badarg}
([email protected])4> rafter:read_op(peer3, {list_keys, sometable}).
{ok,[somekey]}
([email protected])5> rafter:read_op(peer3, {list_tables, sometable}).
{error,ets_read_badarg}
([email protected])6> rafter:read_op(peer3, {list_tables}).
{error,ets_read_badarg}
([email protected])7> rafter:read_op(peer3, list_tables).
{ok,[sometable]}
([email protected])8> rafter:read_op(peer3, {put, sometable, onemorekey, onemoreval}).
{error,ets_read_badarg}
([email protected])9>  rafter:op(peer3, {put, sometable, somekey, someval}).
{error,timeout}
([email protected])10>  rafter:op(peer3, {put, sometable, somekey, someval}).
{ok,someval}
([email protected])11>  rafter:op(peer3, {put, sometable, somekey0, someval}).
{error,timeout}
([email protected])12>  rafter:op(peer3, {put, sometable, somekey0, someval}).
{error,election_in_progress}
([email protected])13>  rafter:op(peer3, {put, sometable, somekey0, someval}).
{ok,someval}
([email protected])14>  rafter:op(peer3, {put, sometable, somekey0, someval}).
{ok,someval}
([email protected])15>  rafter:op(peer3, {put, sometable, somekey0, someval2}).
{ok,someval2}
([email protected])16>  rafter:op(peer3, {put, sometable, somekeyx, someval2}).
{ok,someval2}
([email protected])17>  rafter:op(peer3, {put, sometable, somekeyx, 1}).
18:12:28.491 [info] {append_entries,27,{peer1,'[email protected]'},11,25,[{rafter_entry,noop,27,12,noop}],10,1} {state,undefined,27,undefined,10,undefined,#Ref<0.0.6.181226>,{dict,2,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[[{peer1,'[email protected]'}|13]],[[{peer2,'[email protected]'}|13]]}}},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},0,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},[{client_req,<<220,158,4,211,180,244,75,175,180,128,40,185,59,121,191,167>>,{1389172348910056,#Ref<0.0.6.181194>},{<0.38.0>,#Ref<0.0.6.181191>},11,24,undefined}],[],{peer3,'[email protected]'},{config,stable,[{peer1,'[email protected]'},{peer2,'[email protected]'},{peer3,'[email protected]'}],[]},rafter_backend_ets,{state,{peer3,'[email protected]'}}}
{error,timeout}
([email protected])18> 18:12:29.156 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
Magic Number found at 1305
18:12:29.157 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:29.157 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.77.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:29.185 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
Magic Number found at 1382
18:12:29.185 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:29.186 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29248.11> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
Magic Number found at 1459
18:12:29.211 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:29.211 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:29.212 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29250.11> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:29.238 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:29.238 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
Magic Number found at 1536
18:12:29.238 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29252.11> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
Magic Number found at 1613
18:12:29.263 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:29.263 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:29.264 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29254.11> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:29.292 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:29.292 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
Magic Number found at 1690
18:12:29.293 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29256.11> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:29.293 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29256.11> exit with reason reached_max_restart_intensity in context shutdown
18:12:29.293 [error] Supervisor rafter_sup had child peer3_consensus_sup started with rafter_consensus_sup:start_link({peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.75.0> exit with reason shutdown in context child_terminated
18:12:29.319 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
Magic Number found at 1767
18:12:29.319 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:29.319 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29259.11> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:29.340 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
Magic Number found at 1844
18:12:29.341 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:29.341 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29261.11> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:29.370 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
Magic Number found at 1921
18:12:29.370 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:29.371 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29263.11> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:29.393 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
Magic Number found at 1998
18:12:29.393 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:29.393 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29265.11> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:29.421 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
Magic Number found at 2075
18:12:29.421 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:29.421 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29267.11> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:29.445 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
Magic Number found at 2152
18:12:29.445 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:29.446 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29269.11> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:29.446 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29269.11> exit with reason reached_max_restart_intensity in context shutdown
18:12:29.446 [error] Supervisor rafter_sup had child peer3_consensus_sup started with rafter_consensus_sup:start_link({peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29257.11> exit with reason shutdown in context child_terminated
18:12:29.474 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
Magic Number found at 2229
18:12:29.474 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:29.474 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29272.11> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:29.500 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
Magic Number found at 2306
18:12:29.500 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:29.500 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29274.11> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:29.523 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
Magic Number found at 2383
18:12:29.523 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:29.524 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29276.11> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:29.549 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:29.549 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
Magic Number found at 2460
18:12:29.550 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29278.11> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:29.580 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
Magic Number found at 2537
18:12:29.580 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
Magic Number found at 2614
Magic Number found at 2691
Magic Number found at 2768
Magic Number found at 2845
Magic Number found at 2922
Magic Number found at 2999
Magic Number found at 3076
Magic Number found at 3153
Magic Number found at 3230
Magic Number found at 3307
Magic Number found at 3384
Magic Number found at 3461
Magic Number found at 3538
Magic Number found at 3615
Magic Number found at 3692
Magic Number found at 3769
18:12:30.020 [warning] lager_error_logger_h dropped 19 messages in the last second that exceeded the limit of 50 messages/sec
18:12:30.020 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
Magic Number found at 3846
18:12:30.020 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:30.021 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29317.11> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:30.046 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:30.046 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:30.047 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29319.11> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
Magic Number found at 3923
18:12:30.080 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:30.080 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:30.080 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29321.11> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:30.081 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29321.11> exit with reason reached_max_restart_intensity in context shutdown
18:12:30.081 [error] Supervisor rafter_sup had child peer3_consensus_sup started with rafter_consensus_sup:start_link({peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29309.11> exit with reason shutdown in context child_terminated
18:12:30.082 [error] Supervisor rafter_sup had child peer3_consensus_sup started with rafter_consensus_sup:start_link({peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.29309.11> exit with reason reached_max_restart_intensity in context shutdown
18:12:30.082 [info] Application rafter exited with reason: shutdown

([email protected])18>
BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded
       (v)ersion (k)ill (D)b-tables (d)istribution
^C95:rafter Aniks$ bin/start-node peer3
Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Eshell V5.10.4  (abort with ^G)
([email protected])1> 18:12:45.501 [info] Application lager started on node '[email protected]'
18:12:45.501 [info] Application rafter started on node '[email protected]'
Magic Number found at 4000
Magic Number found at 4077
18:12:45.572 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:45.572 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:45.572 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.93.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
Magic Number found at 4154
18:12:45.593 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:45.594 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:45.594 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.95.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:45.633 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:45.633 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:45.634 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.97.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
Magic Number found at 4231
18:12:45.665 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:45.665 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
Magic Number found at 4308
18:12:45.666 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.99.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:45.700 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
Magic Number found at 4385
18:12:45.701 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:45.701 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.101.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
Magic Number found at 4462
18:12:45.738 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:45.738 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:45.738 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.103.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:45.738 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.103.0> exit with reason reached_max_restart_intensity in context shutdown
18:12:45.739 [error] Supervisor rafter_sup had child peer3_consensus_sup started with rafter_consensus_sup:start_link({peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.90.0> exit with reason shutdown in context child_terminated
18:12:45.752 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:45.752 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
Magic Number found at 4539
18:12:45.752 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.106.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:45.786 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:45.786 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
Magic Number found at 4616
18:12:45.786 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.108.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
Magic Number found at 4693
Magic Number found at 4770
Magic Number found at 4847
Magic Number found at 4924
Magic Number found at 5001
Magic Number found at 5078
Magic Number found at 5155
Magic Number found at 5232
18:12:46.016 [warning] lager_error_logger_h dropped 8 messages in the last second that exceeded the limit of 50 messages/sec
18:12:46.016 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
Magic Number found at 5309
18:12:46.016 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:46.016 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.127.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
Magic Number found at 5386
18:12:46.039 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:46.039 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:46.040 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.129.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:46.040 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.129.0> exit with reason reached_max_restart_intensity in context shutdown
18:12:46.040 [error] Supervisor rafter_sup had child peer3_consensus_sup started with rafter_consensus_sup:start_link({peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.117.0> exit with reason shutdown in context child_terminated
18:12:46.074 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
Magic Number found at 5463
18:12:46.074 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:46.074 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.132.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
Magic Number found at 5540
18:12:46.093 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:46.093 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:46.093 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.134.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
Magic Number found at 5617
18:12:46.124 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:46.125 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:46.125 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.136.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:46.143 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
Magic Number found at 5694
18:12:46.143 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:46.143 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.138.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:46.174 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:46.174 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:46.174 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.140.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
Magic Number found at 5771
Magic Number found at 5848
18:12:46.196 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
18:12:46.196 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:46.197 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.142.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
18:12:46.197 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.142.0> exit with reason reached_max_restart_intensity in context shutdown
18:12:46.197 [error] Supervisor rafter_sup had child peer3_consensus_sup started with rafter_consensus_sup:start_link({peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.130.0> exit with reason shutdown in context child_terminated
18:12:46.224 [error] gen_fsm peer3 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600
Magic Number found at 5925
18:12:46.224 [error] CRASH REPORT Process peer3 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in gen_fsm:terminate/7 line 622
18:12:46.225 [error] Supervisor peer3_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer3, {peer3,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.145.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 600 in context child_terminated
Magic Number found at 6002
Magic Number found at 6079
Magic Number found at 6156
Magic Number found at 6233
Magic Number found at 6310
Magic Number found at 6387
Magic Number found at 6464
Magic Number found at 6541
Magic Number found at 6618
Magic Number found at 6695

[andrewjstone]

@d0rc I'm having trouble reproducing this issue. I've killed nodes, given bad commands, restarted all nodes etc... I saw something similar to this with bugs in the past, but not recently. If you have the logs and the nodes are still crashing like this can you stop the nodes, tar up the data directory and email it to me? astone AT basho Dot Com

Additionally, did you have existing data, stop the nodes, pull a new version from github and start again? I'm just trying to nail down potential causes. You can likely clear this up and hopefully not seeing it again by using the latest code, wiping your data (which appears to be test data anyway) and starting again.

Thanks for reporting this. Hopefully I'll get it sorted out.

[ahltorp]

Without knowing about this issue, I ran into this problem. I wrote an expect script to reproduce it.

Basically, I tried to append an entry to the log when the leader had lost contact with its two followers.

Unfortunately, it's not possible to attach files to comments, so I have to paste it inline:

spawn ./bin/start-node peer1
set p1 $spawn_id

spawn ./bin/start-node peer2
set p2 $spawn_id

spawn ./bin/start-node peer3
set p3 $spawn_id

expect -i $p1 "1>"
expect -i $p2 "1>"
expect -i $p3 "1>"

sleep 3

send -i $p1 "Peers = [{peer1, '[email protected]'}, {peer2, '[email protected]'}, {peer3, '[email protected]'}].\n"
expect -i $p1 "2>"
send -i $p1 "rafter:set_config(peer1, Peers).\n"
expect -i $p1 "{ok,{config,stable" {} timeout { exit }
expect -i $p1 "3>"

sleep 2

send -i $p1 "rafter:get_leader(peer1).\n"
expect -i $p1 -re ".\n(.)\r\n.*4>"
set leader $expect_out(1,string)
send_user "\nleader: leader\n"

send -i $p1 "rafter:op(rafter:get_leader(peer1), {new, ourtable}).\n"
expect -i $p1 "5>"

send -i $p1 "rafter:op(rafter:get_leader(peer1), {put, ourtable, foo, 1}).\n"
expect -i $p1 "6>"

sleep 2

if { $leader eq "{peer1,'[email protected]'}" } {
set leader_p $p1
set leader_name "peer1"
} elseif { $leader eq "{peer2,'[email protected]'}" } {
set leader_p $p2
set leader_name "peer2"
} elseif { $leader eq "{peer3,'[email protected]'}" } {
set leader_p $p3
set leader_name "peer3"
}

foreach { id } [list $p1 $p2 $p3] {
if { $leader_p != $id } {
expect -i $id ">"
send -i $id "halt().\n"
}
}

send_user "\nkilled everyone except leader\n"

sleep 3

send -i $leader_p "rafter:op(rafter:get_leader(peer1), {put, ourtable, foo, 2}).\n"
expect -i $leader_p "7>"

sleep 2

send_user "\nall commands executed\n"
send -i $leader_p "halt().\n"
expect -i $leader_p ">"
sleep 1

foreach { name } [list peer1 peer2 peer3] {
if { $leader_name != $name } {
spawn ./bin/start-node $name
set id_$name $spawn_id
}
}

foreach { name } [list peer1 peer2 peer3] {
if { $leader_name != $name } {
expect -i id_$name "1>"
}
}

sleep 3

foreach { name } [list peer1 peer2 peer3] {
if { $leader_name != $name } {
send -i id_$name "{}.\n"
}
}

foreach { name } [list peer1 peer2 peer3] {
if { $leader_name != $name } {
expect -i id_$name "2>"
}
}

spawn ./bin/start-node $leader_name
set $leader $spawn_id
expect -i $leader "1>"

sleep 3

foreach { name } [list peer1 peer2 peer3] {
if { $leader_name != $name } {
send -i id_$name "{}.\n"
}
}

foreach { name } [list peer1 peer2 peer3] {
if { $leader_name != $name } {
expect -i id_$name "3>"
}
}

send -i $leader "{}.\n"
expect -i $leader "2>"

foreach { name } [list peer1 peer2 peer3] {
if { $leader_name != $name } {
send -i id_$name "halt().\n"
}
}

foreach { name } [list peer1 peer2 peer3] {
if { $leader_name != $name } {
expect -i id_$name ">"
}
}

[ln5]

I can confirm this, using Erlang/OTP 17 and rafter 4dbbb75.
I see things like these:

08:44:35.501 [error] gen_fsm peer1 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 599
Magic Number found at 514
08:44:35.502 [error] CRASH REPORT Process peer1 with 0 neighbours exited with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 599 in gen_fsm:terminate/7 line 620
08:44:35.503 [error] Supervisor peer1_sup had child rafter_consensus_fsm started with rafter_consensus_fsm:start_link(peer1, {peer1,'[email protected]'}, {rafter_opts,rafter_backend_ets,"./data"}) at <0.85.0> exit with reason no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 599 in context child_terminated
08:44:35.527 [error] gen_fsm peer1 in state follower terminated with reason: no case clause matching {ok,not_found} in rafter_consensus_fsm:'-commit_entries/2-fun-0-'/5 line 599
Magic Number found at 591

[andrewjstone]

Honestly, at this point, I've pretty much ceased development on rafter. I'm not sure when I'll really have time to dig into this issue. Rafter is definitely not a production ready project. There are a lot of rough edges, and I don't really have a use case at the moment enticing me to work on it more.

Since I've stopped working on rafter I've poured a lot of my energy into Riak Ensemble. It is a production ready consensus protocol that is in use in Riak 2.0 to provide atomic single key operations. While it is different from rafter in that it doesn't provide a globally ordered log, there is no reason a log cannot be built on top of riak ensemble. Additionally, Riak Ensemble provides leader leases allowing 0 round trip reads, and built in integrity trees that protect against some byzantine failure scenarios. It also manages multiple ensemble groups instead of the one managed by rafter. On the downside it requires rewriting active keys on epoch changes and maybe isn't quite as user friendly to get started. The big advantage however, is that it is production ready now and in use in a soon to be released Riak 2.0.