From d5924dea604b33962585b0643b95940d50714d16 Mon Sep 17 00:00:00 2001 From: iosmanthus Date: Tue, 7 Jun 2022 19:16:25 +0800 Subject: [PATCH 1/4] stale read for rawkv with read ts Signed-off-by: iosmanthus --- media/lease.png | Bin 0 -> 40215 bytes media/rawkv-stale-read.svg | 3 ++ media/replica-read.svg | 3 ++ media/txnkv-stale-read.svg | 3 ++ text/96-rawkv-stale-read.md | 85 ++++++++++++++++++++++++++++++++++++ 5 files changed, 94 insertions(+) create mode 100644 media/lease.png create mode 100644 media/rawkv-stale-read.svg create mode 100644 media/replica-read.svg create mode 100644 media/txnkv-stale-read.svg create mode 100644 text/96-rawkv-stale-read.md diff --git a/media/lease.png b/media/lease.png new file mode 100644 index 0000000000000000000000000000000000000000..4b49f4b30c39db9013cf8e2d061635734c0de7ce GIT binary patch literal 40215 zcmeFZcQ}{t-#`9pNh&lDiZsZU85yNg35D!cA+oc#hEkzrRfrO@vp1E<9%b*nM+n)! z$5o&E_2kf!>pY*&$2zaaR}{|gq+p;Rkw`l)UO1~nB5f8V zkv0mGZ^WM*^uN3xzihL*pkYHIQGO%-zafB=k`aGJx_I{VRmYI=4kz^h!`}7j7tZW zp_;B!@ijiVSA1VuS~OZF2L^=j52GyX1}k4HDk?Jgmi)Oo(VFh<>npTx$1{m}^_sU5 zRY^3vcCjQi$oa9#Q&3V0OFat;YWmevbm`J1-|bC}jZYbbVqF#{2L}=uMXf)G7sYF& zI?s(7>K;6HY&apalfN)OMmd^+Zgy_2ssTml=+?IX^uV2o_&3yXw$$p}hero5A z9V_FhHSrz{_y;MOnd=joU2iz7bfo+mC1xuHaCs|hYktvy+*}tqxysIb7nPa4tm~qJ zWcPcV*NaG`I~j2W1#81~5nHxw8LA0$u(ho>N$>3J6br2`DH*iyV$l2%c`3+!X(ZZa zs9ID^Y<{%C^jG(nr%#{KUTSP;SpC!YY;A3gw$JdAsLQ;Wv-8qa;kx@kInCwEHIp5= zTeoa+a(3>_;f(!RBegDWJ6xNwHfBHe_MR270#DJj+8b}e7gpa7hTs>{l`UVl5x=VZ z?{EKiHvV@p{NLRRpK?1yNT+lZLdDYEmhD--p+?w=bxah5nsTzUFZO!#dQ$J&wJSo# zH>V@>ey--li)HW5c+JerjE;_aQs+C*y*Y8aw!KnC`>Qg+VISo`zg2Evu=yfE2N~P^gmljF|3gLTDG>jtmnRH$a40=1@fNFpAX=Q zq|a5FpRV1F|FTEg(aEW$t!+C?j@_8OmL&zLm5C@cA_LNGTDR}o zMM_h#eYBO-9ku_2)hV9JV3B7A8%ZL6uO^@EIQ?FGxFvOHB7YCbc+9wQOE;O2u<+o( zKtNE?$&)9SmzUdGTZLyoZ>QtY&a!$}P|#UaRMgg%yQo$_HkX~9t(K@g(w=Sej7h?x z`zaW9q`gKbMq5;cpTX6PsKd6&33+fuSKG8#itoA72~W zB9qjFi|OIEl9Q9qcu}Hi3k$ohQlwp68Qqv=vAi%9CU{Hf`_luftE&<3-*4Qs3IDjn zmoXEUjG`>7saanrT$iwDKY^>DxYyR!mX=;%W@g6MqBesel5VX_G`xBRD>D_m4r7hn z2M-?P;3#iuYO1I(ZAmdqPfxGP?paw`DMUlm)zy9b_AQ!oPfrgTCOH+=`mZ9Yv9U33 zrkQVhE}uO|3>8(?#`^la)%i9_Nl6Lk-_L7;pFgh+8S8!Ou=IQ6ABs9YF z=gy(Qns?@Dush-QUke!JqW7Age#gJLS-FpyISe-=EG!%q9c_t9 zfro>8`5Aqg3Ufq7B_bLd6*c(Ii>bV(hECei#f4Wl&k>M@!FRCx%YDnj)p-XZCpJqk-*stiZEa=WQ-jr^!T$dB zsYW%piQf}#neX2p5*1w})BMo=w7;(})oFGx-L%PleRVI^gqB8uj`^CAFRoW=O^0O6 z_wF|)Hx~iJC%OvP-?~h`mTJp$f4=5WdYExn`RULXV zNjKkQ`pNp*LgB!z^S%s*1r8%WBCRcrjfVk5eH?VMt&fU{X)c=H)U~n3MqUg^@bmjg zCAD%b1ye5V^h`V3IYIlc!G;KYrxvw8Jxg{+v47p*8JRN$zjw{7McKB38CM zJf;WKsJ%0v>+7%f9+fFy2yH>t@aU7Fli#xEei(ES?9>5trwx|-%EJK79_b}Kml9#+)RanpHrKi7B8>irHuSClDrvol!tr!_vew6yed zuaJGcXkn2k6Y=789(u`&b>Z4l99MXqv0TA<$3Hz@1ERi#ZpXN|{28T|#|1Q!XQkF> zrAp41O_7g}kINO@QBN^=EdHxOO;4zB=lw5NP`zW6*;vkEbzttMd5(sl>MnkLu$`7l zO>+MCZ(Hvzq_lHrwY6N!^AnuMj!{d?g`7x^dA*?#>%kxuR9y)9u!|%L2sh#uJ zunJ4s`uh63pFHs-7V6~B=x2*}(H8bex#tzzjLt2|2McdwX>V`$_rLL8E7M{ow+;8z z-mdL!NXw;uEwX~bN`7YO3i))UK#jT?^$FwJ@Q)vlv79wDjJ&-mmT?xeiD3sB%QvjZ z{Jgx%?Y}2GwAPv#caZr#ybO%z`pt6*A0`l~If3}_F zeME%(o0|!m>9XWK-QCxcL!+tR>wkZWx)*bb zP!O`R`^$=TJ_-%m=Jo6El(Y`5DckT{_hUDr<^^(TD5jbJ)j3QLZ54iu(ou z0fB^-jk-Cu7kT^x0yMY|GTLhj-mH8D@W?FToc8{GZy8&dEBcJIv^3d{eSBukpXwsc zXMWwxPqF|ty>nBsvm;5A?eRIA!Aix!d&W(PI;d7T_7kJs-3K>Sj{J<~cDLk+lk}uc z(MmWcKcUo_Q(j*F#(C-%B^z5bqa-c0OyS8Cj|b-{DJfYWSgu(!F)@i)W%m?$C26Ea zefV%hLPA1N@OOQTa>}j9N+z`IvD$Y<4RLA<<1L4a9{cd1tpsrw?6r>PWBC@qsrE+s zz=yOt{_6wmf(%ucrB($kJH=a+6VcxOOk+(luXo?4+JE2xoBW$JARkPW-s}(3)dyA^ z!!`V0*b~j>&Ye4^fgzkSwpYgU|?FlXn(KfE$8#x%nh`sqn$kdZs zYW26WvGK=BCbW|jw}{YC9q16~4MbBH>2!?9%2IE9Q1b2DEAcnuHoTtu1dVGeUkfBT zvB|G))V$@#P5=PiOp6Xi%e-W}u|@y}Ou}XN2t~j-v<+^pjB(sl^QY@gd19nF2nWWr zhHF{Nlev~(J-0R4jelqM;G!#7o!9b9Qac*?@S7+9x7dq%d-YR-3$mFcoJ+SPsGf9~ zY#;36u1yiM8$IH22Y8_Q>fxi2PVWL_zYgjth#D!+@@Zy={CEMHK+uq&0}9ye0jf%e z&p&NXH&wfFq zxt|9fZQ8VH;NI;qr|>!U)r)du5du*w{+h2FS5>vE^DYIb#s%o)QSLu+MNN$k9Bb{Q zOX&bw#-Bfqh9NsgjZua+^qR0ub(R+JheW{23`cftbI*txS%m zc;YqFW3eG}o`TB7Qon~1N4pRh7|7u#`szwfvqeWvdQ4AY!QvpJciZysk;6ki6%`c% zOmsBHhO#=}0KmCuKgPwiW!pq+7cAEGBsz)S|MKa25m~lba+#kT0}Txgv-_VOtl@7S z4)#<#cLvOHL$-Ty-xiHvAq8wk+4%8LAMyBf+JJ z$$6NCo{1<&l`$sc-n<3Ri!^P=8s!y^8%W2nY2?@fg`s<9RVzo-m0m~%s?Ez2XZeN? z>d(Yl;{VhDge33B^UC*(%UjGwrykm_tvF?W{-krNO&Xs<8(LXG{OTWcY-X47 zBo|Pb&tuIsGP)oe6)XirTl$f5w8FXb=YumRjg5^@Sap96Tl>u>x@W%?r|m@RT#Hc{ zjkKV!@Y?hkkhN=flzi|_*Y5Y{yoD`0wRLpzO`CXld6SOom)MT{7y!-Q zO1b}IeEjcUzh2>9R-QJtuP$`0mvbhqKtqmCN(zHc3oacI5h27-g|cQ4T4tJ*AC>py z^{3R-;BTPfe^`eJMOG_wAu0WV6+f`*` z9`d&guYj#rzSf`q1WLa66HJ{J(=g%1(D<3|5=!FTzh=#mQ$=4 zi@w$6*ymePBExz4_m4}w>;pf4{HWXxC8sPO0)lF_KHokQQ&T!_&3@p#dfxE3xUt@1 zZ~3Q61-zv-HI}25HR7{A#e)Vfqoapf(@hh)7)dlgu*##{lQ*D z3k8Fyf@mgoz26?=gN)2MySTV`c&wL}ma?3ASemof1~>%NxFg@i0jMHB-wkbF#%JGj z?HQ_;)>c|t+Hzjo1Z_bP5r={DfY+~ItEi}iqN}uJPU%0BXaq_D4Dj^a%;Fgo6cq5d zpI0X*4ZZCCRw~^rtJBr%-Fud@k&$~w89@O`4KN^0*Z7D6D z1Seqn_wO{N8E94@^%rDib6_i=odSe-?PL_b_2USX8Seo@i96I^ySnt~4`_)7ggnrR z{wX9xnkFjEt)Eezo}MpX?)H`G>*+ZT1PnB){<`Ign8#XP$mg@_43967zj*ob(8)Jw zh%I>{dD+0W_?K(7@6JGx0ppf_gu)Lm2L1Z?SX0ePoEJDy3zboA_-R$LjXa#3CmnxX zLp!v$&oJZ1(`keH0UrcRthugk`i%(LDRXi*LgBx9mvZ{wum5oZ{_Btb|N8BhN&7Am zNnKT4J+ZdCy(4HQy@9sL%2WvXo8(W6I-5i;rZtDCX}s1-;buj=Y{g7gfe#Km#;Y|e_T z4wFh+DZzrTym4Z&sci_)t6%1wNpU_{he6fTeCAW0BbK_zC_I<^t5+pwkD`DGB7MP>py$&;DL~xzzm1El`SpV z;o;ZN62WFqVvWF!>%_B+jg5tL1ZZD6gL=Tl#U&_7xQXKv6G;naJ}Sr-{FP6`> zRVPn=oo`>F`;yZ+D!rhsCAw2Ft@{Uxc2H3j<~u;jHory+XQ+`}jt}T+1h(fq`rVh& zDDp#GT-@|6G*6SrU;Sl%fMDF6&#VK`Bk-+qk7a(Kn+(>3S+4TRDH~<<_7-;4j1SuV z__0ngC}sXJsjzcT2DfH{DtXWFFf>xf+^H=NLEz>CA`Z|*X)91JN;k5wu+VOgS5KA} zB?SJ;?fyYQoLg|wu9sA#?x&}!wuRGrtS@(Ud91s!c*;a%udgi^ef=tCJA4IY`pOln zl|R$y0{a%8H;&js2BxQ{hlVQs2%O7dxb}35DS!MYGxmC1`q!^t=&~m1tq}SA{r&OG z$ytcjuBw`0RKw&e^RqtRZLUErsbC3s6vPh?8;@U8hO3?Kz$)1Ui!g(iXO-lGlthq+Aa=IeTZ4fjn6bpiOWd_zgk0Fq_-1{p8PHR6^9>I4X()2}aE zWvEyPXPD;qfiX@`V{LRhQ-cjuswij6e^QovIh$M>vY-EU`E zTdeWWO!|?W2vKh~sqj{QWdL`VUC5tDv!C9HxY7tk$kUO!dA^I-+~N2T87U+vNY0|` z*t5mY;E?C1_;D zo#f;o{yx-)SXkb@fB!z_22AuE2z(ZERocE%wIMfDygafS9ISIoORR}%pmycTFIWfp zz@(tNz;Oab)hpBAnJuR#Hju31>f*Owc|Z`c2M%3s4)q@uzkKj_+w4Am@ z1O7e?2F>W78sekRJXZR+bC~t*wrr{5REsWjZYcTqtm?p-7*SEU;~y0x{MY61FK^wt zW!f0OSj4>c$$izCjxMOp;Teh4E{(cC^N-yFFkMks--oihXO93hLC}qJGBR&`UfAi+ z%tGB-nMDgHcJ93Xpnof)cc#Paf|b1RM4O0>^q%0rKXUa^kP%Ow)FK2w0sJsHWve5X zJt%hU$a9`E(bjGU;oeDB1Zk2FcSiJ4`z|gLDRVRC*1HPM5`z#2C+8*d+mM0*1t?fT zLPEeR@r)!uf*2Lp<5Uxz2VWccm|jX$Lp`zh(>!RigEFy3)UE1PvJ;QrvO zZ+tq*NOw%?hBjZ>4)~mwmiA0?$rwwzoIDS*If^e_2MFh~3JL{xf1+ACRELP-xj_G&Xkf){hTcw$oN(zK3hyg^Jm|5qGlI)NJL>cOsEAiRs$q;E2k+7JgQbSaQavEX1$RTUht_~{kHDhM3Avp)gi018#UI<5~X3<|{XA7<_!X7T52 z_Z@GK@kpEF7sk(cE0Sq986Ci$-$OWtYHDh;8~-YNTLQ&~P@AJbTXO9u_*-<*)T>0g zU`M&FEgm>oPLFbT`oxC%(l1}$VKD=zI!jA`MKR8GnBwt-U-uQXH*riHF5g8txxLb7 z&z_YFG0y(V0t^%voJ>gGP*LLj0 zui(L<*P56nNleJu5PCi6(>aALq^#6k3U`D*S5?XLG`F@gSWf+j^wvJc!I9z^p1ZuB zWN^Y}(5HAK*r@`zgnZ}Q=;-LVwk#g|Ru|Vyvzds$5_V}dfy46;W-jq+7o?@VUs1Zoe7!MnMBztBc8_ngv0!HbT*2&Lz>w)N z`bL$nSm=8D`sD4=7V?%xZ|E@;yTI{*bxXKhc^wSsGheQJy~7a#Yj+?>&mC1r2GEEQ89Y1y&+dQu)n-}f z1_^Jbs-dlIIr%%2faEB=4zsQTHx-oz=n@6kMxQ?MiB_rs5e0|~I8184J_SAHUG!UL zlRv+DuoMbG)WOU$&D(BTT6QJr6++nWtgHf(Sk+9(`1o-cmzkfQe*EN#l30reT0hKA z5u2G@&)wqC%G$2dFX~^@*N^ycn&s^E>pjrZ`e*zuOUC>hLA1(FOb65H<>jSR`ThHO z9>R6(PeYeJ-olTa15M1K)5&PMw;0|6!el?6vz)19493>F94yQYOBh73F?!_cdo@Q# zM`c5xc0FxvF3;K#TcGePQ#@oEc23T=JeYb_!Bq}(<1I)>g(4JH!YCmwu9~1Jq+j9# z?FRlsq-pN-!cPJrd7w_hK*+{U$S2rB{(`^GKl`=|3{n1}9~h`PU@5w{p`jsQYKd>I z6%d zUKfIoll$)acCf&zP{$Wg+8MRjG)P036SrKQFcNs|X5o zT5-(pD7n}C*FN7trDbJhRlRz()VG8;X?}iQYVA+o`r6dGJAhI-b()2R1vF}pfdI98 z_wM0iK&3w7$+z}ihXFpcc);{FZhQgpcOHuvP1+|{vaRcJ>d{V$nTEW7`32hT1~+dC z0|%wr>k;C{b|8!WeAiAXDJfZ7Tg%AEn42f;c*0t>7%vjfpA78`GZOjkT@R9egDwHu z27LSQAawgSJO%o9o*!=kZ9%>2Xlv85AF$+bcpXwD+p!BpAN1tvJ2X4|56ubSGdX!0 zVFT3p+xGU8KO!rHg0ueo`BNDt<$)BSJk6)fOplSsAYJE?NEBxvO?(250|O7JFB3M& zW%29NpC4?e6JFmXE-ualh=%Q%x9D{W*6z#4)v{iGQJozDEP23IaIOvpVY{WKq~J5* zLd1Xl{3^=JxtMqWa&R}JaBai{aU0*O)A0PjnH9oT$KR#8Jdyz<4eOTuW$?546 z5Yw={wN@&5{tj;UR#l`AHSd49wkG3ZbU{HOaC_A@Hr6?+l%X8zUPwdbGFX4ze@An( zdUj-Z_(9-e3|7{XKfkc>Bw;nkCef53vf?m>uptj_2RvDtyd|j2bGHIYE_fFi6;u2; z$FXA&z(BU(fWeAYR#13K{XQdO9=5nFd7ZH!_{8bcr%Q}%tgNnNkVun2cqH@Buv82y z4ngRcno9aPf~hpHq%b_#mvO0#3{xW?a4Yu4>5=`G$pZm!(TQQ;{2dGSDYKOB)3&%e zV;G5$FTk$moZVE3(Fmwwc(E^5IOfx*D=`W!@Jnv@EaU8 zrAGr!YC?kfsZ)2jJdh-5(dQ5py!tdq2S5AD6=GV(#!UI|36r`hWuOBGmhP%V4-qkJ zCSAUK`L&XP_VmJnrH#JC>V&07uz-;QTy`s~PsJO{_xg5rcCrfaJ$�TpKq$`T6c9 zZ04#dd0jkXEUr?66vydaEcA1#7j+6;MGqf7j94uk5&$`VfmdOwF%$I1Z+y$T-50NY zLG>}g10J)zy)rl!k48{drkHbt-N@U=NAA+4Qa-~CB16dE^(zqcm0~DlC zEY3!11)c0gCqrJMO*u-ASbg5!6>bY13y4m*LFMv|1XrU(igiu z6<2+vZ~ct)Paehu14`#oAHtMgWR)g|>eXkYSPdm--+=@9c4MlBUqd+6646veFx@C? zb^^>hNwxNhHXFlt{z1Q_p{I}e{Y%nm=EMG+a&1Gyfq&Y>AvHiO*eiw$+mOoHwd;DP z^u^VG85gKDb&;3oX7kZN7T1=i(6Q{X;edKp`Np4JC$ziRR>@FJG>MU_%*zDOO!s2?#WR^#x%8W@OL6 zfE`jITB20YPY%P6*@WGIgwy-Ht%y?@85kVXI4UHhLNC*DR$pKLCNTy9Jo{-aEiGGi z$d%CO>2);49#2ojINb`LlSzd2yLYR*sch?_(M^P*x_ zcq3g73+^x5aEU>+D5mR&q*9qsKtsj`TD7r%(PT3uE)GL}^M z78Niqsla)rAEiXbkbJ{ATidkh*zr^z&t`Z}PIhPQ?B)RPxPz+NY!Mw9fB^M1J~>b{ zQubSU`!DDb<w&Nz9ZUE_r}#&L_v|8!X>) zrwSNpY3JdlbL0Y!NIA_IMD_$aC0=1L)p)$W@6!5n@N6kT5Z1|R`Vqpc7} z&&3CleuSF)#?$Zj@87756_0Q9A1p=lKmtk%xJ+Na4=rP1q79xQ@@l+ena=q)^z}_m zO;J7c(8V(L8E-fh`p1Q$o3bCuv2{P7^7--xn3%ZWKdA-?{zG&$$ds1&r-A}WQDQhX z#Hv_(V`}&vEY?>{1SbT)R!(GYdZcoiKTWW2aNuiUgtyKw8L;r{1xgg*5EL5IExv=)S$hg1&}ht`ypu_JVJtLa127z9egpa95CSlEm9ri$pk78a9%+*w$8lTL_Ks z*2gIHZNZUmz1`i&I^e<~e^q_G0l0a&ux_tOx{!=-p(#2X0Fr1abj4g-Us6(01?y#L?zT3!wXIJ9vH>{$MnHxMTpM>3QRnI9MSwXd zgDHFRifJ`c2G2&+sfW4rqt94>`H<_qEpqT|4Ab#wAFg;g-$S0I6%|^kH+`v*w_HW) zBi*qZ@Tnggsb1_mFns*pr0_ND_|L-1-D zNjq9Py6DJAND@Rwfwj*KwLhYuz#Zv0>jvi{VbbThLm{L)?5K}WpLQ|`UPuI-4H0)z zDua#9PJ~~mkQ_!%5`##BX{Bi<@H42NJ?j}3rpGB!Bp)p9gwo+xm1$&TRN?GnI{#2D z=dso;p`g&vgx~A8%*@PUecSJD+S-5bw$M|4@uu_4ZfOW~LJLH}h2V-~Bp@Pc**PJ4 zN=Z%FzKmXe2(#6^I(|lDf;joXqXRpKmPh*p{0?YVNFyV1dtt=wNIqDmY!D;MH$t|9 zlZo&PV;DWjstB6B)lT%a>wgCpq`wwyBvfv|H0AHKczQH6K$OafIayg)NSqOOkqS-A z4r&D~`X`saS>P3UC%EL?=PQz?IRQzpGS%yc+d1+V!NI1t8~s7vkR3S9!${SDfKNCV z000JO)dW=a`}gjB{rc5bOb2GM0JA*Mur9a?H2=-6MgCgGk2`ul?uDkLVN zRadD?aOhwS zvt*l~)$ZN2tHQNE@A2pl3gS@Hyk0bACJe*AS5b?4M0aN-PTG%SuZ=S0muOB@kKN@mO{sS;dAde0MBcmz5`YhH$nA^NYcAiy_v|+ApQf6{8lo0GR%j%(| z4-Dtfa3CZCZM+E$otT{L%(gMY?t@yO5!wKYMI$~ow!Ht;z*{iQeCq*O7VKO$@Se6S z!4jk^+%Uz`__l8KLVhnYF73-T0=C}N019U3;jtKPM#F}NBl<9jC^tIx@jd-5Z+p0Q0wtGFMIyNC8AZ~Du zyZqOU@+&$eL7PbJJ8;|Wnhd_INX8#LcxpsX7m9-N6CY0WDYE-tdXvHVaTBw+6E3v_ z`}aen_SaCwVm7d}3`k8+KR`q{?rxILfQ7tu>(=O~D6-9)xir&^A@BmPKtP66GX3k< zFE9E!<-3AL@RNA+LOf+>FqRjy9 z|4X%+j0F>Z?-L;?zBCRaJR_1xSaP5abA-gHq0s_i3ciW&fs>8B$}33B5JEJPfm8>? z1YwmEUg7IAVSt4W(&Pv(5#8E=mJVORmqNh2!s>>uh!P?Z^Wz#UXTv1jRN=TD7+PKsja;>{Tac9zs{}o*+it$$;ik! zI>rtkI^?f5FIU=w3~5N>cBeuGl#Y)t^P7cbr!1k~5pKY2poy%ko& zUz&3>Q7coyVtRgFmGx88h%HD)!Yn!WM^p)Xdg!cc&^8S$Lx`q@h$nmZ)Rd+6ncjf` zLrY5(F%K+wEO8+6*CEjAAIP3}oqe;%u<^pI4(U!XG@e>ALOI2Z7YN*3H@*$U3P1t_ zI%=SnwT(@Hh9vYCC^~P%GT=GF?u7_vt@4^p(Y+qM&EDR;J8YY;3}FSKsi4cDJCtU% zXrbH7E1kvnGCTFZAkr;4InF@P_%=nE%{&|&nt6`X&{)~kaNwrW=={aYmxH#~hmbxe z)RL_=Y`6p53Q4c?c&bOc3A!2voKgQc6!sMTd>>B4&z$o+d+(NuKele@?36rm+qb86_?N#Y4EFc$JwOa@HQXyqfBp(EL|GJleY`WVdq{{e0VXQBFGZoNRCwihw{N9M}!0Dljk|u~iM}z58xExwxR7 zqIrYh5s`od=bt1iIwA$RN!Mv004=!FJk=wOl94g^^WOoLxttr?EZ`?{S4bAhAJ9@# z+Wi-Tun|uXEG(c#6ac!cEJo;UkIb!f&4Cyi@N3XXW1NpgIJuJQHL_>Kv6cH@_+0<| z8Xg`7pMqY=!99#qX^+8*&9f?u){W)i&Brx7WfW0JgpP_-2uGL=z74pucoVd#J%7hQ ztgo*;7)L;jM)c3uH5rN01S-MDffUa@xjLq|jxD`5nLEWGWp?Jwru?_0@N+2VX)4@8 z%U;M2fl&XL(2D+9ron$OJ1c7k0}{C5r>)Z=lmc0n4~0N>L`|4Jh3FmdK|nzI?LL|8 zb8!tE5qHm>J^N>IG5v=?+D9;xRfGyx@q^sg7KeO%d_c@zqE+ujaQfu8oi%H9wYATY zAPo$h$0>m_ZfxHYjR!BheX)We?_pGhh$>r@Ax2uwfwLL%2RL)zPW>N1k=gFTg-6I< znTi+5k3n_)r@phT{XEA+syzg>HYq0}uzVh=16^IF!*n8kz)eX>Q=Z%Q^qXjCyaqFq z8btsdI)V3AOy_tk%aUJypV8Z;{PPOYeZTbE0s8E3~jEw)$I1q{4tgN&j0;JD6#9__ME_wUqPJmT7 z82z!{-j^I5Phd$D=TUG%WQVYQu1J)2B(?HcY^T@5F zJ0}_Nxax-==DMj}y^0B%1z1C-7NbH1+W{mC^$+_4h@?Il#Q;Mq!%QR3F`1A8RCuL^ zkO63GZcb}FL;4&_+?YA(FkR;VEGM#DayaZ!Rh8~|4XX(!%5Y>$+s@jWV&_hk95N*C z#6(50&5)HM1Y7ZvG6W}XB~H^dpfMqgca-*~fdTUP!OVnSg6!R^SI5*K z4mmGOCdehf|2cvZt~s=Z$dj9^tLBg<$!ZqQsP)d#OMwXep0thpbnk_?mKUU4=&zup zN7SH70u3w@I;@;JQ3#Ixk3w+fZpV-7JBVm0Dy{{H2}pM4@NQD=J>uGAIZE1eeimIs zm}6m81k(s-9K}g%k#k-6OLa957uPrpF~2ypX4y!htP@Oe>;J(NJ9Dn@z*fK-Mvx0> zX({$nkuoXKn}W}(JsJA}c!MHOWZx4zKbICGVEj@`bAod{qZF!XQ3igAfI0Dm2((UF{&dUwlR`M~Tn*r5!dJ1#~M|7#RJr z|LBhUE6al}dN%v+gNgJtfRG||h?7U7@L^4}zi+$~c$Tmg9RE9BPY~5%uNQDu3dRH_ zoQkO1{C5z?-Yl|v#yjfy$gucWw>CBrryk#m+T4UenmJxZ>Mns3ly<$ky1GP%y}$Ni ziNs5l6V6In#0u0zvv?^` zY~LO%Y7^CNEfB$0MrXW8d0_9}Z>Ay1w9MJ+PswC#PU^F^ER#JvZfuf7-HTup*Rf+n zE+5BJE$db{ecsvKrAcz&=Zs)G%Eo47XecyC2}|Wa`I++n6oY}~wO_xiAgQQ2Kg{@S z(BI4Wr+huNMskl}X6oq8hsQq+M~Ar+fhCBH=d@0eMv>{TqB~A65ra4nio>nmu>Vg) zBfG}~Cj%Nxv_|M#?=h~GhYi{{6)*9Tfq=>TYR8vWR*p9%bpbws`&+KAZv32%iEGct zTp-wkF%YYpR8w88S}=7Pbd{a`)@U<;F9u*ysR|A`;OgIKWO#&2FtbGl`|jhtAcaUj zH;Kn)Wf6x|fpTwB>?N5X1$t+_rlf?Gj~G_}rRniIW39dna2Q5WLeI&{I+5L4;N~)L zK0JUEoa(`7y}T$07~(K)1(i346OhaJ?O7GrLInq{JLZs z9Vtx`twOMAXW75n?Gvi`n2!vXdiMqLiQJ70u zc>cwCrgwqc?oePdyU8@1yaFZ06XgB?xnO*PMU_1|ICwKe<=*}KuUCs+&Aspjb8EV( zpgIiH$iTpn`a~f*_x^dF>e5oak)6yyk&KKd4a@zZvUeorZoFd(5$=@taVIZ}HP0@H zZs5@v*6A_Z3RyaSl@=og1q~g;dXADQ`^3?s=Xlzonj!oPfj~c8fMmWAZNegeHtd&{ zdl&ph^krg&Y{Y|f0MkGo>o)8n=wX@5x}2L< zJaa&^ga(o{v~t;J@0@)QJWx+FuA3-8zGN4kQkFzz*E=Lr{pp6Gy~8|@ddkhigGK|Q zkvkBZnregOgR2)#ahsf(s!h26!h6HtgFSaN{{W&dEh)U>JDnI95KxwVaNwjZN}=)| zcQsX2hG*K9C)4b>C9S6s`#`Zt(=_LlKxQr*MLb*y*coyWPA6q)CnO|PK{Ak)6$4>J zxrRMyXO~{Q>Gf(~PFj6wF=Bq!yBQZnyCjNeH=Wu}%r0RDQxnP`jSvh67SDg%Ag3w% z!v}l@K|3?$RFBf8;m@B>Va{$E_V>b0_W&a#ny}Xkb60vA?d3Nj;QEizFx)E(pv;@u29X;dXOuf4%h_?tK?P`J(B#610 zP5TiQFw7kGwoQ%1P+y--)h;_kesL@oV;~mL7yxvui#}n=?GCysHIO+6|zm1AwMSzx_eFOkK)K!Rdhv#nhW>VfMgeqZmmg*}h2Ceib zT{)(Gwch;JEk%w1*+_(!3O^is^1ToEEVaSU_zf+^gSRKWbO~06S1Mo_`02Nb7~0V1 zo3{z4L|5=eEc;Zq7vd#LWcRNFju@oQ26zX^9(BTTB@inl$LrtOY$OfKk}HrtSA3$e zD`Q=FlA{`zPUwqyu3%gS&#oLhqE^$t<%R6^IJeeJ-6SKA8HX2(@^(drw#0Zb#dYZ)UT4B%(p@Q!uTs$c zab5d&g0to(u{BeYDT?@HBI;Vy1xY=L`i9;ylqx(t?Ci-MdGNc{!giAGv{7T1>FKK} zX2$Wq9IzHuVmCvY6$5hpBeOSEH?4<8Gh$9t@YA^c)vU$~DHZWMarHr`+GnlKQbwS@R79HK9ycyLd}Cao6;icbpS*{ONJCK!)pBCqyZALo@03rBt! z3r=jcXh|7B?%?1v08vPICJv|2nc7=Yfvn$cC_C zWI=&$q*8K_1KvH>a=8xt4r~^~z^`WLdvS4#aNQY>2i&>!2it10u_M#c>RX1))L7oU z{|Mdr>LdHoa|6BRZ5hz8%I1Ed2%4k2qqZBa8(}qCU^SAow^35Uc)yvdm$kmvJXsy- zDGZ!vDt!w*UU(unY6QPhh29q#{xu}0=-H~Fq?OnKj4unHIdkTsuiEa!Z^gyai>|~) zhUHSFb1CZ7Et)z3YHh_j%&8a=EcusToV$#!O?pXYfq z{JLuJFXqus8JU{m-TnB&ZqXc%G(>M%*z*Q^ZS3>g6?lTEaH#OiP_1c}y3l#N+pP!o zBVJA+W+G0ycRvL`?xB>f&{GKfvPeaf_01L=2*>yX~dH+3} z(W%)*zIpfKvxi~c4YTO$TMu!?@qD@$TB?(digR1)-PZ&6Cm!&cx_L9^ITY1_DZfA8io#->`G(nO7SEoQ9w z2g`=yo5r|ly^HSJS4=pH>)P51E{0tE*6Uj_p$a1jIXdDT##?gI9SM{uQvM?yj&sV& zezK7_;_hZ&i2O9RcHL#C?5EoxT zxjPhd8hQwm*xl+y_%2;Pl~H4MeAz}iHAO>B&G*;6Wb>1sjM+Pp^v*ojH{ zo2FfkJb?pk<_lDGF?;Phq^#&CHa`3A{TIM#6u64E=<~lWWs|eGPQQc9YZIJclpu_< zs8a_YJ$shPy@hu7?%sDcB+@ShV)f=VBTZi7LthjX-yEfMEI>A2cZ$ofc+N0Qn}_Oe z2XnlS8N73y;6eV9&<9qiurj!lM z_VxBQ`_sIZlUZ8Y(C~Ps7N5>JAMZfH%YVGbw^30UX|>b}9-yah>F`U6i4k>QIT4I> zPHhKq<$07Pay0S;yi1&iRxM^}p*38A${0kw(Q%1(dICnp8rqbXD zurKImDB@F%BMxSRyyy{mT7?1z9Q-04ElHtyk{JOrsg|PT<*nxwZ>AClj6_6&%$Amy zz0V%ny?Ai8?P-T*EHu?H)e-s$%6d}Sk*Vy+U$GYl>dT}p&)`FtznDU>AJK<-VX zS`}h~PHjz}!j`5__&I{`(E&;`3k%g_d(rC64$ZbbK$mL^^cpN^d}@8w+Sm2#5muk4 zPY0(R6T3Px9#lXbB~oM~dp44+wh+@`>x(y^wD||15EBJ~gNNbHiHen*!8FdeKdlqm zrr$GCfYylQtZ)YC_Fw7H&x~x#mWxMr3$vOre{6@U3n@9h?PsLXCek&kzgOLkE&f^# z<@3TO>f8$Bhu4u42=SeB9n1{6(L}rg!e_jvpfM4(C$XPl`RciI4_eG5UFPFcj~ehW z$ZsUslC5QJBHdw2EA#=UJrq>;A%6++_)uTFz(skAdl!$qL_n2jYQ}|)3TWL6N#H)0 zE}=$d+%Cl_FA#w+BEr~`3*&T+IMIXHUvs+OR?;an4wA%~V|($gDGU^{pgQ~4UE)l< zVq$O1;}w3UT{-r<{nl>_oND?jGnN^^%T5*zx$kr~LN-$Whn<8^BoM=u2$r)wsw%h^{4i&(-WKqdrt zmFTq&jEEnC!@mb7RX3ap`a3I~ck_K%gTb8*oRn_gK3>EJ@>iYs;lCGLBYo^_VdL6^ zfGa`7$q*dqx1Z{)8t8_G0TdS^W@iEj-yw0FG(1Zz*gI;3iUb1%{k8NaWIWA;o2l%0 zU^X_XMR&nX5$H-#B@_Ux`e2%Qi5*2&qWFvnFy+c#9(^Q73jcqKd-J%Q*S7!nGFvE9 zi!>M#C8U*PslN&Ef1x}Rt7 zXaDxU`}cGIbFcNX>bkz)^E}Su_)O=5k?bko4B}js0b$0CJLt4>!jN<;lcuGQ)UTpL zvRhX&b-bd7nZeCLva%)pw`3Fz36Uf>4L;as)eVRI2#@kliQ~HH`uK9SSxxPj@ruA~8->6}Z{EBaT=o2{T>12huRqScb8&L2 z1wPT%87Hc46K}~o0~q0k4NKkdLgCtqB}?Wwd^+wD19idNU2zv3P~nvEhYuY}GG6hA zgv7V!)3+USb{Qlkbsbg{@ftn~tGN7@x0gkybA`@S)c%4g%6W-7GEWT|iITfk@#b%C zZ_lgH{U4^`Rw1f(MVGoNRj}gXbYM#uAzhFOu zxU@O^_O}wkV;9|=sG*@j*_=2rAo?OY3_LTPJsD;6#ReXq+!4+wTv0pHcd~75Odu5w zE@cw}L>!IiD$<>_S6C0Hm!KNXz8{a)>O}O#L*$cq~+W)xHK_*~1fQBTZPVem0GpCOArGz`Y_ z)Tw!z=#uml26{T`DFk*WXZNj?NVKDm(1H5a{|N*$e7Fwd$h8#~77JW!=3fT-@2v0k z_M>u^qjFi`AKlsh8RywZ&`r!f^L)z%xSn?Lv)E>4VDJb__XU%vg_?-nG-QnD&;R4# z(pL^!*>ah?{+Bs;vEl65yDe2mOx+m1|5+E&X-{E~iK^#A^SP#4s-6lmj>})Uc=5gD zJ@muqdx{Dv;+@ihy-q!Q^;!^Ps%fjoHhmK zsL}X?+ETye*oi+iGt1YgO_S}EbfUjA(WEE$^3sxxLkACku{A4Pu-p>E)edb?_T`=E zc|zys%C9vw|6EhEROL_EIEosp?SEDda+rRKp&6V^z~` z(eq8>P0%>@|54dvVxUE@efp*ne2MJOqU9JQ)TD7L&C|pY{Yd`LRLk;^8EXtsiy6NihF=;tGt+jj*7oj^^?7`twHs6?3hu6gwQN5-!SId&71#NOUIj+ zynOl2<@AHK*!B4X4mi@Yuy{4+g#OvT6cl#KR6`}i|glV!YhhJ2W1(d5VhF@%zyJa(*KE!KP=q7Dg1yym-*L_fQVf79DY z(nGv3$-(Bz$%c`(Qx0%P39PF>*u^&`^O`C@x_YEt%&ppfaQ28OmlyRArR)=mh-k*6 ztwBMEOoSvWq%cHb`Dw5E>{ zHM&2V&i3;?vM4T}cDeRES_&tp32AV|evdG7{pWY=Hl8M68GzXyP(5@y$#xgIi^?UR z+~O0Q9*sCe5JCMp-u$ra)GHUNx{AEJ3k610ePC9{k6dl-9-?Z;tC#!oZ>T*z#7)II zo%ZHDq>hrq)GP4~X=@MVuWuLkGFjE%e&UErDtfABB$R&9j^V;jtNzP1HC$_vvXu5r z;h)5{SobJLWn~wO6GJ#(Cz}z0;X8awRfLD6wAx$LTdKQ*fCvNhGu55~K16yo%{N)t zUnJ5J21`-1v`!ST0+4`DTU`npq~8dg(t6+X=g&i}$K`*fF3^g6d!*P!ba$Qj)9)B$ zoL9LkGyFSA4o30jOQgPGdIY}R;kKT`VYB$dD+ zDH09eZ@Y??YSm@)8vOi%kPGR{SR@Dyoz59`tZ6OMno8wY)-3z`?=JSbk;jiWAvN9Z zTFig8M0lIJ1xYLUHXFV9)FmFv*mcSG7;h`SYy+NlUs1sEh{Td~gh9iHXLvMNjTli{ z)v$w2xHA5v^*_il{j4EuU*%2aBofa3Rf19!F%&*%<)VIl`uHC`3d5ALd;;Gd=(^=C zf)?muz%r;~?1Rv!D<_;3z=fF;=`o$7#`Kq8Vqx(_t)FZ_wc9Pm*_IV&78UN1d!fR6 z>bxUhOE=L-@n>Jg$!KxY#d_yX9lgaI!9tEIqf+~?UxnSCuHV9`RhKg$^gPIfvcIWH zFx{fCT4|Za`S0Mp;u(DRWpedQun%Vq(-kW|y+aGEVuGCS?l@sty@eCL#|M=5BTV0P z!Gb5Kf(3WbqtQn>`=)FCMte3Ya6V28#2?TfWnIpkIpgk$ZxCPKoW)X^Y7{{@KUF;; zpr`{!NZGflJdYJZ+n4L7K#_IQ+#)B-nMuO&Am*9ok>X#gOKb>-jo{WK#GWRb6$l+zn1l2X}{~If{3B4`Z<#6LC4_r!*^ofwRU{%?0Vybp)3A$~H zPHTcsyQ5G=w-|cQ28&y9K4y`!v>)R2t1IIzdm<%0@L+A|d5Rgp`Jo5s;>)xxM81;Z zh17zGf{NUu>pk4cf;f`4yzz%uli8d(nb;Z0&N_mo4f{+zrr;a$7foXy?V(qz-!v5k z4H8J1voB(G7-h)l+#I za3_cS?L=??6qcEE(O}2!Q!Ai{tV^ zZRN6v*h)OpTigPa%~&e(-MCvQ&9#5*LzX+6nRI{d&+p%bIM8jCf%Dk;bfb;9-x5s{ zv+JEs+z&JwJJajCn%-i=7+3Zp5U>yeL5f`At5?@Sc;VAFA)BGMme)RG_1H;se6XSB zzEueY788c3cI#~K=F6P1U#SvIgXGcb)J3Ev5-QxZ=Q`?th2eXl8T7Ie4@P-h(NNm0L8g{m(p<^{)x^yC{VSEK1xQ5)6APmKm>K? z%~ZzpCoF?+VOvuE?-Wg?jC1GqmQ8oB6=-e24LQl@$AVR>${;7n5uLiJ{1-35t*5<$EHl)1Qkd@8jUh427a6e&GtV%e@t zy&cbl3G)rtc`f_>7iM;jB}6v-H?@PlpxpF9K8KEGy`5c9hFrSRgZnn>LhjAEyaV^v zyq$(0)<7ZfYJ~n=g@JANC{%XhQn8Ef^w=XCrcgSbwL$ZjJ8iqGcG%hF8d=@A|6~I6 zBNnzYrVBsZY1rD=yv4}vA44#3Lcw00X{%nurx@pt?Zkfdn8VQ*$$&haHB<=fVW3U) z?mx^A{Pm+cr}S~#mFMydu-Vm>jp7&(KaXyxX27>V|wn{C7~zn_g9FKTrZaB@$Uc38@^ppRrN~y=P7)Lq2h0%t9pvw zGAQ~Y0W8%BR+798S7;Lz7_3GTsPLLlKiw>Q)-yc^-TQ|A$l@S_3>gycNQXxgM z=5@SX!~w54MNOQZ5+?}Pe} zyoalH6OUcDoi*;41GoLBPRdEsd7yGlS+?3cIQpVs|9F#8)>6a(kW#+mko7;D@CWCr zlXWvZ^n5qyO-&i=Q&DB+ozVW-fq@2lW7z^#cMoKNHoaJZ2%tK ziW|$=4#E_%s**`$>7L2wlpn86#K@&7#c8`fDdc@2dZ6UKX+~p-M)~|YcE`AqK3%}k zw6*0$)$hd*V~G&c+CWGHM;U9PSI?d=HBiHUkBl(w1ej(9FeE1LN_PYXNB1GDL$~G2 zmwV*CUSO{giq${6tPlNm#C~C|SOUwWQ-*f9bfxR4R0(mD`JV@Ea?ye16C`{0uC6cI zA#XbB^wq14C{1ghAt5a|JMCi~P()vGC#%4i9UZ+}LeF4j->gI+bPeOA9DQh2#Cx~4 zCYK*D>rMOqFDEcYDfZ9wo}N;Ae{}DYbJ!G?SG>`C)Xw%TvjoBF^=*Jt1THEJ$TvlF*cYSRh8UEaHw zC{p}vZY}%IQL|1%X1XNz8(GKMztE5$G2-Zm+7d?x2iRfGNo1BP!JlF1y9|-&%G(4jVCNW`(};z3vH>ou=Sjgu%LdHV*LSZnO8+yq z4n8mpzb8`j&z~Jq{Q7p^jaUxI7k2?>maBD^*H=!Iy;rZ*IOKt#UA%Nn9;Riw?ruFo^{QUWIGBsd^ zQJ;@hR6HHgSF~+lXA{hRU$J7C_@wrh4@73knJ%1`Mly`t-e6P?={qEdXDM@bPn36+J&CAS%@j2D_(}*q;-llmSn4|7Rhk znM|0oQJX`Xh!t+U*m^p-0evZ=IM73KPbZUJd3_5=ftVS*gXXz#(OzrmpzMD)uYpIg ze>g8X5HXuhl=zudzRqL8^2`9EdO5TanXE-F1%6{hmg*t>>MTX4JA}sdc~)R|$K+7J zYuWO1zc;!;R>2HYjr|Fnnej3(bgj~>yH^IqUgrtq&st{s5-Z? z3KxFDs6SrQ5{Ln`)P7REyQ27=LcE{Y!Op# zU}nB6rqsXv*JzWydCh-Sxa1Kd3P*WF36`>;piH&HX&<}=yrNT`hP9dmT;c8eP8Q1F z?ZXe&;-Yg>jy6-tSwA@|5n#%xuX~7*-%hV*$I?tt&w_O=Aoj!JgZB@w-xv-4UHe72 z5+f&cAMZ(2DquU+b^Le;#S;6uarv$jS_P<4R`!XXf`;+g5mA%rsfZyTKfZ=)N{QSU z+$yiOMig5&6JIgmT4(q)x}E&{R^Yow>JwkupQAv%&uw93Sey7yoyKnA(==tUWyOWy z9T)(8pIeQPy2kq^<>f?VWCmd@8$*>ux<10;dV2o~G0m$fWi3F1{_hL1p@g0_lsPZ0u&on+XY<=FZ(~recha z!q`}E-l-PYObfriW6fmx`9($!6gBq{-odSiA}4c|V5dHq8mqo1rQ^Q7nPc|@+Lsi-&*Cq!wOZ)iAs-q4wld@x+LOEwFdV24*1 zBO?Zhn&*puyw(3#aSRZiXmJK*5h8&Nxk2OB(vnkI?&_><4=vs{D*xsV{u5>HaqY?a~ zQMY?}FMFT5tZ+IWnN{-j2bXgt!nKb6IrsT?|VsII=DELkxV@ap8G)z3caB(_Ej2odQjv0{BQ7+aa#8g*&Jk;Qx5 z-noB^4?bU{Iwkhq+rNj8y5VgSrDN>ShBD)pMTrM}JyCe15{43V!CAp$S!lPzqZ4Be zJujerpdyQ<{vi-CK&EHoue z7V4im7kF`ofGcAGgu15$ap2K<(bzzj7{r*mxh3XRlt)PEN$~ooPhY_}b~-e@!419V zAWj;$!Bq3N58zlLNx^4mJm#b#)Flzo|TyyNlB&gb+U4DTC{|CaUnaE#*8_`@1(J9mz>;>Q*N51+xIDS8fz18S}64+ zi%xi=-+?TlbY#XOOGef^j&nqszl<$E4f^u%Al8;rf zjNe3}^3b7E=?pmnAAfM=d1emThX8D?w0?|fenG(&xkCwmiMI6>o}>3Ghi}B0lq7gV z{(}y^1R63J+)uN$ApXg;WM!2V=l88=10A;o7N$m$3;_F3R-8fk9CT1|%9O1TiL6Dd z{fF)^)*N$*eN^c>J>#M1?Zn1OT}0bLn9?U%BwyyBuSfy`W(?|V`z@`EM-KQmE2~6* z^oEC3hqG-=m8H;uqh?c?G|3|+c@tzSZUIYGN`bqfVV@bdt`}59yC^S~5PA1x3pI{~ zi~j+UT&*xmaq859Z{x4(EzY);O$GLX%wrB=jp zIrnu3c+)PYBd2`6&i9Ng!_(qt1L<7;i_h0zZ?ggtnjEPCXLr7a8 zzp6NC`2_Of0B}Zn7(C?&-bDgNt;3wq^TKcNy&E-K)Yz3npIHh$XE7)h_b1Wat&%I< zv}dsCc1_MzSYnRS%_+%(HBDHl&Tmj6oc+^pj^gC~Ict~2<;&G{KUS)gYPs;Z{vv+t z-BH2^pR#v$1=)|^K$irI4Nh$!XqIxjHJEf{l*LiC5F107p+(`A?U zYrUv#3HlCgky_i2$JdpVmOk~Kb~KAE3_@mXX!xHRdo1MrBF?UoeG%fW&DMIcfZ9$$Cph-zKnQ#noHm(=Hsfm1Wc$`l4^P0iO_&1G@nbBxfG^ja|F z{!%lu=H|o5Cxmk6eGg|o`|=iXee|b%IqaBOM3^_9E9ILJg2(N7LR96}l;j4^AY4Dj zdChdV>z{=mJfbQo{7xArYM^$oDt-NH<7!<~y^xjy3T_haOhFas+Asl4oA|uin_DFq z+3QPFwY3LcA>x%=(g=lLVP^JfS8)bQ1|S19nqaqQR=6f3cdh+{Rjiq7#|JBMK}mW6 z9m(E;|lhT2Za!~lA~;TEC%K#W6Ut-*0wJSt#4|_`8R#}GEB#7 z8z-czYZLX*K2+a)j%VlmxN7ddYW-SQ*E6TZn2wVTpuA7myM<$8JqTgHOq>|?V=m$9 zh8P1$yR~(r=xx3*J-VF&S@5%lApqp#@<$km(o1zURaH|RUMa_#g`PaANz}E?lxcH! zrrM=lMfO)eu3+|=V`g~@9Rd4XB+=|7C)qHu6#Wq5BAGC+^lIwt5em^lQq$lyZ+7V+ zQWcL7>U5tdCFw-i?Oej;!t3DdClO$#_5JRqB~!JuZijvRln>S5oOX3f$A`#HwRevj z8$$JiJnf|Ye&BY!WTw&yR!vhGIZ{_yidu3!hu6-$S0Ab%YF;dKS~Cv3YSG^OZmzlc zNltoQXOg}5A1}_}s9IoTgaD>KqiDyZvoAKt23#%3NckYvsXhvzbsRJ3kI>Q-uwtW) zLK17zOpm(a2euO|5)OOa2N>X=d86}ca2wpzfd(1~)By9rod&!;dR2Q+Q{bY#*L5nbZcNb7IN_$HSnzm* zYr`=VoGEh+u3jjpN%aW;Af6xnilGjB-oY?W(en?D4GN-d=_n2eD*#wSaDKya&)}pY z7KPwt6@25t!67;8DG>kNjD=1a&$?Duzt*?vwcW8}M~$tCLm+EJjLIJCv64WgW7#(R z@&TST^Uo}z$B_=u2PG&NzhL1)&U+fWUcE+>Bv3DiVQz;NISn~D_gsKSm1ic(^&wJH zh>GT(StJB-mTeTSJir?m2^2@m%9fOoktxpp(LBHMl&9+%XhCKM3k%s(yCer+77&Fj zF_6>p@PKKo+tiEal^qc9kVQ1;b8r{mDq)F6jI6dEe`yZCzCBM-Y#onf2M8XgF` ze-P?fu3qDI=?)k^Ty@AH>VOV~6ehwfa_gftYOKCvLu$nyfX#9T9Bow@W|45DHvFr0 z*gWpT&YE#rTH&vHLJ@Q|&btWcR4&S$Cz*~`L0LK7n#=_ zN5&!QW4NXz?0l8Ds##pEL4B^?P4(@{kh2J%O|^^4EKJU2d(WQTb-wk`PGU}Aq9W<@ zb`1KIXuS9p&0ylq5tih>_yB}rpLp2dLI=Q~& z(!p1P|^!u36#|Hs)!tEx*lxJCK z*MYZ#g99B&F3VOkA{$T)Oe)6gvK)_V6hh3c#wmUA{!QA8rKUi}A<=j}KGSrLsJe<` z=sWZ~e9lhfy(GWwUNA5piEn?S*r|@7lp8cEuP|(N&OYx85mmL=wZS94YT5d*r9KaX z@3bScgFHo|0->r-dXThyua4IU|4a5sd;R<}zP~$!GER#zThLtv23l)|MMw~y6&Y@~ zX^m!B_QZirPY?}5e_QrM*T#|(@3eCtroF02GuhzEWd=iNgr3)wsw7z1t^Ssy5cxZG6Q*S^QX0{@y(ui*F!eU=CLHJ{yY)d=yqre# z<2tDS8|Dc#*n3@WKl-y9D<^)sf5l9NC7YFIlH9ifOc7N^XlXZ*3+SjETr5r&F5`Jw z^KER9J%g-A>ECe|7f5cUc7t;mp>ATGfYnq`w)_IPWdy( zC%1WlPLAwF%bn#u?g-h0C;>|D99ejwB;7N`JZ|EoNv&VMa*NDzL&L|Hmd1t0ztlk_ zu>BC^KlB6{WZWVkWcHkzv|6%Xzrz6mL?_y!nXWN;4=4+9Xib1zwvCKVQqQQ~v?U?) z=l*TxwT(D*0Yt~fQgC+0QR{yY0l-;YCyoh^Dr!CGFyt5<2Cd~=pOe+};Dbj|T@B)Ra(3&1o3^a;CM2ieBg(^FVX6go9DJm1j0ChmUV_L7!2 zRy#uM60e3x?9?iM?w?fEU@>D`U1(s3SH!z7%Kn5EmooQ|K*`81XrIWYC~@2FXAL%% zpZG`f>>-Er5(=1-K)744cv0{b6M`H+@DMjfTx$K?Sf=D34TZdWY%8*g%s!2Gr2Wp= z7gwZ=!Q;R{<(`EZIk0;X;%C$NH%AO(6bfRy_3VfC#-@r^Y?ar-$N{7w`POr@zjI#O2}MGH{cBw@s1u(&^xuGwTFN zPR!5c^M3hANZ>EYw1cxM3Mu*W?OQvsrXO5{OEd`4c~?7H9w!d@e>F}z!d<)=casuL zV_#J$d)!g6w0x7kZ0bA7p&fr|-?>+Dm+%zgf4Q_P;Dl!J&CTn2m8Z9Yb5FyW!A5E( zbcr#-Wp(d=#o9Yqp%!jP8%{O`P9JS=mGlBu=T1_63amVx@9plfG3HF)?p>)rJXs;+ znt6vFkH5GoGx^#Nh}KVy(#fs|lV+_Z9PzI<1yr?l{jUbZ5nEjljL^foYkm0Lj%f(N zNHf~<{-K0_G{7MlIAHDMmPbZJc>n&jJt$iPT?11AnJCxh@T+bxy3oL<6Rrod&10fr zT^B{}fhAf4_A4qAcI$(+e-)YJX{?y?_vdz#P>t-Cl<`OGoKfJD z50v*l`xIu)db2js2^I|9Lvn|5@pO(=rm*~%FI~~B2(wGpd-|-X>0A7wFQRQ2tDx|X zkU6JWpAndNH)Shjuj4fyKmM+&>O$TpT7lJM-qzo_bn62@5NV~-miJ$OAPYB*MJOsr zEg45+Vr(&{BJ-+KvRtuZdtLDq|LDE5&lruBrMu_0k{27etLS2pQwG!r-g$rj#SL>nNGjX-vu2s(cY3{tTmRKMt-bRF%9m~DE!&z$;Xc)l5N1`WKG{id%I+;c!Ie>1A|(i_ zOQO5}WBcpe+yoGujhEqv6J~LbUq@lOJ&F;FMK}63gfE{!vY%8_El?wBkTq@C8B{&L zR#D5`*O7ykNM}c(;+N9h^k$@h&td>cS#!M3=`_n;Hi>$h%$>^~ndrsc< zq=Kq1+i0chvQ+U#$X$hePrJaSOY*H0Wr0c9F7f5i((=t4!soO9#VRyWcTwrUP&ez63bl}jT_vJ@>I9Bu) zj$GN1({^=Sij5sLwGRw|{Krkp<6jF6{o1)>gKqd}Dk<%^lpvODao3Q9539gHdnuR z_HwJG&(iR53k)iy!Vllfd{^mnqWml={?s-)0l`c)X3#<1yIUnDM&*vc34kMzOnslK z%1X;L_jr~;>kOJ$bZ2jNrJ2wuxTgMkF>i!$GcJ?cr`nG9?Ggf{E^%y2*N&}~j|RTn z)kfvqYW`il)1UxPVf_!ksW^BYvxO&Gkktr6Xg4rqpmEt_R_=x1Jv_$v&1*^>`Ly)9 zef9w{Jd<2tVxkoBQMh@4OWKp#yCA%GnWmGnCu!UJ={P<>u!~g%m-JSe@xBgt?tkyY*3Z==te#^`a=FrFU?&a ze9XfdQ*qN8=JIFdZY=tRRb``q=0+vX<-9x;r0xAvWBc1i8w%`I)Q~8rX4IUcdTcGp zA$1Q=#AhTL5Z2t^PG3$qedu57^tq%xsp($*&ZXc`(|RiXj{Y(+xN4K9^lIhG>8V~^ z+Fd&-t?aSGjgzNNCGGQ0O{qMqY>1+!udQjt)EP5myw&8G)f7W&6a~0RQRO449rrb5 z4jw*S`z!=i=c&J4SRtQYj%iv0cvPvn*HN-95Pr8-EJ;3p<;oGR^SG$2S$8fdX$D)H zVQOc|`}tokwoq|CzHXz+2Wmn+YIi;qf) zp52CV*3gx3ko@r4Noz+Zwv3xCu zyY$%AytRqwYxu)8r*Vux*PS;ME?phm(laM&*~EcUUM!2`;`jwM_jMpBmZ?w{M>=Q1t6kZoS;+>nloJ{nxTb0` z;p46G{0q9dQQ`Ygso#gvn_qo$llR<5?wcSjjtiLj#_KDFB&ZlTVh&g#l0QGO07KL0 z{>?kQv#idnD|D05{BbUwH{@Oyc1?-fMCrzMetfm$`u z_>Q%--vXDHAt$|HgKR>EG!3(ii;uTG)LpIL*}PAMB^y%+rOAHisbTDqZ@d_B`iBAG zcYV;8MQDA8rVwsybAS71Bw4pHrt=rS!n+uWzz%QPfd}vSINUL#G-b*S?{%*J(S>hx zkV$@MoWfe~Mtl60bxcghEVJrg%N|U*b1`x8iv6-jhqrw@H0fKf{$IwG zPhYU07L{_!_Q4GuCoc}9hsvYns&$=3ugj;;Xqi;<>XnU!@q)N~&8btjg!FG3cC*GM zTU{q=#L%Jh@|sD`d$hlF-}q*v$cKb8I@*2*0z)j{n2T{mhSYtYph`SI2Zj|>DLhAv z=n>2e-FZhE@%2nu1Nh#B8V zTty!u_+B}3>LhQv*U3!JQDy9)arr^Fg({!R!)zg)O0%v7-(X}c(Ak?tMdoD7+chq6 z+?eDC&NF{3tx#6psvi{9@K}CzP|i_X)3isG%v0OIMha_34#`xj{F|^E=lBbt+AY0i z?)B<<1k8?@)51mn#viv;hNKW3`Q{Y%XZ2EqB}N&4a$pMt#q_ggsQZgdp|_+jG!I+}Knoh#CX!i2598m*lWv!L~_G!aOqu1d#$%L6W_+{++4d0 z?KCze8BL5YdW%nF_PI@`R-atix6?|Av5yXA`^Trl>>|cg0C2!1MZrx2W&g!Z11W2h zK^OV_S_BjeE#t1WCQ%ZFa|Y|vwYXH#xPo279Nos6515Q0hnrgRm~k3z)u;|XDvb6p zE~IR7Uk{pt575qb^Qa-DvWL6n7T(-^wr(rWa}&NH%oEXzlop*xP@?5Dh@8r|@EY!tMK#mRHyOjYyFB7mc=Y z9^UK`+B>f2JJ?EpWLu)irNE6H_SOCA`meOx7lvAG8#QsFGk`Xs5K0|`E$MI=CTt8q zJ(wq*XF7lx^L&~(k;{Hwaytm#MV|RPT3so<74rEZ@d*h^#wx4Ud>v-+`G7^B(VdqS z$8BGA3|VyEUS@g?29mW_GVhZcGvyUK1QX6vFHQDX$f!WLV4eIR3!FRhCHp)U9zc5yw34P%1Yno8 z82#R}-}}R6Y*lGCjgxnpwa^e(>TB9Dc4 z%TaZ6Vd!m|;12A5RFm5NF))fePB{1$j=(~%&1YY11F7dIf`|h3m~PaY`^5-JYXdb` z_AkEUaONCdkI|2tcx$me?3Zr$pvDzBu~t8h*2&zc059_n{w+l3dHYlFhW z0mSb^;G`{v06rIUJ*l!7dkw$Fbm5G9y?XTmAQEikASwe(h5)O)=&#-Uyzivjx?@d> z%!=L}>E||N7vD{=TM=Ybn&F*><*sf5EN=)uo8g?@R~SJ1zeFWiSOcg=?!KN&LFW2* zc;OS!$(d83ZQ8kYfDl9>5e}la#6QzCwwa?&xO}|m{$ux2)=ealz_WaGPW8D%SOc7p zjidI#xzVbd4D+VCowr#b7Nq@*rvFV%4nuEby?LkGOQWo@n~hYghALzjpu8Wa}R-Bs+64jO6WQy1I9n znZJAKwC*bVvJ^aJuB)s1_FwpCHj`tX{C8|$dbhVPZ)J>d5Nkp4N;bPzSY1PrE{J#| z4*i#icXt)tJ681J!j6|LukX-Lo zGW_X~+TD_;CcN)dq-U$+>g?MgiPrXwAZELL7laZts}&(p611}5EnuHRorcRBy}s}* z_vanp^U8<4KTTaLsAig5TZbN;i=u3<0+kGTGFsc~f>ieQe8dU_iz2fm5>haWBFW+s z!)NvXFna~)2XP9Fp4@y+EPc%y8$iPu=3KiHnrg*)J+AkA@F4ZZF-&o%XA0zM0})}9 zDcyV)+K_MHAuKJSBVe->b(8k{6m)MsQs+ALiNrJ+>bPLxx;HkUGgd<^!LAlY^|ucM zv`Fxeyojdf!{^O?L^gJIDapwwm?9%r7C-0erLZSyc-Y|`5(^zc1(~rnA9h6i?p8tf$D)&TwPkJ{U zf%w(Bkeqz7bkL|52NqsfH*M$kM50fh;|7WdG832AR#lx9!M*Mj#J-z*FidV;PZ6C$svt*0P%Ff`2A6+ZgVb?ZEgM9YczMvVd>YiO3uj8;d3Me zg8g$Z%F9jtMOwPyqCZdtsB=dL!o25qiBC9m+YGf-#d+bQ5u<-xqKogF(F=VQe_*Mh z@>01s-A*A!h7Wj5fqRh{!^XTBAo}ji2RTfEVV+?`RHzb!NF^m-kFIH;Evl}*c=__L zv$9=9M@KF6wfQF`WSFe%_>Y@wLySHGKc`yOLzL3n`&dsEomTw!k7FWEE{t0bPRv&c zo-u1yM`OhtX341XJ|g9b9ZR|oI-Q#OgSw2v7#b@m`eS}__A_sXyh9*x$p&_!D=-mLNyI%$P z9?3S3h<{C!c)_Zd==+omL+NdPD=jRdcJvUvJvCjhOV1tv1EJgd9m4(If1`g9jXEGW zUAu3lChr|d_3$0=FZR0uyD6yu`~UJ@{|lst?ksX@p_%?ske;4ylGabl6H;m4otwmS zdfMrb$H`R@s18D&rgtNuaJ;2rSk7;c)4)+`jdZW`>icQU`ovPaP|i2Fn}yBWSNi|p hSNi|^$u{=)R(JckB`@m)FH + +
Leader
Leader
Follower
Follower
Follower
Follower
Client
Client
Request
Request
Response
ResponseViewer does not support full SVG 1.1 \ No newline at end of file diff --git a/media/replica-read.svg b/media/replica-read.svg new file mode 100644 index 00000000..1516ec0f --- /dev/null +++ b/media/replica-read.svg @@ -0,0 +1,3 @@ + + +
Leader
Leader
Follower
Follower
Follower
Follower
Client
Client
Request
Request
Read Index
Read Index
Confirm
Confirm
Response
ResponseViewer does not support full SVG 1.1 \ No newline at end of file diff --git a/media/txnkv-stale-read.svg b/media/txnkv-stale-read.svg new file mode 100644 index 00000000..6534bada --- /dev/null +++ b/media/txnkv-stale-read.svg @@ -0,0 +1,3 @@ + + +
Leader
Leader
Follower
Follower
Follower
Follower
Client
Client
Request
Request
Response
Response
Check Leader
Check Leader
Lastest Timestamp
Lastest TimestampViewer does not support full SVG 1.1 \ No newline at end of file diff --git a/text/96-rawkv-stale-read.md b/text/96-rawkv-stale-read.md new file mode 100644 index 00000000..90deca95 --- /dev/null +++ b/text/96-rawkv-stale-read.md @@ -0,0 +1,85 @@ +# RawKV Stale Read + +## Summary + +Allow eventually consistency on RawKV. This proposal is based on the https://github.com/andylokandy/rfcs/blob/staleread/text/0080-rawkv-stale-read.md. Thanks, @andylokandy's work!. + +## Motivation + +Eventually consistency is acceptable in some scenarios when using RawKV. In this case, RawKV is allowed to break linear consistency and therefore improve the cluster-wise throughput and latency. + +TiKV currently supports **three** features to process read-only queries more efficiently. + +1. Follower read. + + Follower read allows reading from the followers. Without breaking the linear consistency guarantee, the follower will send a read-index request to the leader. The leader will not respond with the actual value, instead, send a round of heartbeats to confirm its leadership and calculate the largest commit index (read index) across the cluster for the follower. After the follower advances its apply index to the read index, it is safe to get data from the local storage and respond to it to the client. This feature helps distribute the read stress on the leader but still increases the read latency. + +

+ +

+ +2. Lease. + + Lease use clocks instead of messages to handle read-only queries, the leader would use the normal heartbeat mechanism to maintain a lease (usually shorter than election timeout for the clock drift). This mechanism could improve the read latency dramatically. However, it still requires the client to read from the leader instead of other replicas. If the client and the leader are located in different regions, the read latency could be terribly high. + +

+ +

+ +3. Stale read for TxnKV only. + + Stale read on TxnKV allows to read outdated data from the replica other than the leader but still keeping snapshot isolation provided by TiKV transaction. Because the client tells the follower the timestamp of the stale snapshot it wants to read on (rather than read-the-latest), the follower can check if the data versions that are fresh enough via the `saft_ts` (the minimum timestamp of the on-the-fly transaction). Any read transaction with a smaller timestamp than `safe_ts` could be preserved the snapshot isolation. + +

+ +

+ +This RFC proposes stale read on RawKV which allows reading stale data from the not-leader replica with a lossy bound. Unlike the stale read on TxnKV, the stale read on RawKV breaks its original linear consistency guarantee -- downgrades to eventually consistency. TiKV only guarantees that the stale data read from followers are written before (read committed). + +The follower will read the local storage with a `read_ts` that reuses the mechanism from the stale read of TxnKV. Surprisingly, the leader can also stale read, and in this case, the leader can read locally without a lease. + +Note that the freshness of data may regress from the point of view of the client since the client may choose different followers to read from time to time. + +

+ +

+ + +## Detailed Design + +This RFC proposes stale read on RawKV which allows reading stale data from the not-leader replica with a lossy time-bound. Unlike the stale read on TxnKV, the stale read on RawKV breaks its original linear consistency guarantee -- downgrades to eventually consistency. TiKV only guarantees that the stale data read from followers are written before. + +### TiKV + +The replica should read the local storage with a `read_ts` that reuses the mechanism from the stale read of TxnKV. This requires the replica to check the `read_ts` against the `safe_ts` which is advaneced by `CheckLeader` message from the store of the leader. As long as the `safe_ts` is no less than `read_ts`, the replica is allowed to read the key from local storage. + +### Client + +The `read_ts` specified by the client could be acquired by the following ways: + +1. Calculate a timestamp from the physical time from the local. The `read_ts` might suffer from the clock drift and exceed the max timestamp allocated from TSO. The client will fail to read any data even if that target replica is the leader since the `safe_ts` of the replica don't catch up with the `read_ts`. **Deploying NTP services** in the cluster might mitigate this issue. + +2. Get the latest timestamp from TSO and rewind it. This might get a more accurate timestamp but requires an RPC before reading which will increase the latency of reading. Applying a local TSO cache might mitigate this issue but require more resources in the client. + +The client should create a strategy to select the replica for the stale read. If we select the target replica randomly, some cross-region replicas might be selected. In this case, the stale read loses its advantage. The client should maintain probe statistics tracking the fastest replicas (by latency). The client will try those replicas one by one until it read the value successfully. + +### Implementation details + +1. (TBD) While TiKV is handling raw read-related requests, construct a `SnapContext` with the `read_ts` before acquiring a snapshot from `storage`. + +```diff +fn future_raw_get(...) { + // ... + let snap_ctx = SnapContext { + pb_ctx: &ctx, ++ start_ts: ... + ..Default::default() + }; +} +``` + +2. (TBD) Client API interface; replica selection algorithm. + +```java +ByteString rawGet(ByteString key, readTs: Timestamp) +``` From e759b63d49e53d8cb28fc3a91416cae83d1bc51d Mon Sep 17 00:00:00 2001 From: iosmanthus Date: Tue, 7 Jun 2022 19:18:45 +0800 Subject: [PATCH 2/4] rename rfcs Signed-off-by: iosmanthus --- text/{96-rawkv-stale-read.md => 0096-rawkv-stale-read.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename text/{96-rawkv-stale-read.md => 0096-rawkv-stale-read.md} (100%) diff --git a/text/96-rawkv-stale-read.md b/text/0096-rawkv-stale-read.md similarity index 100% rename from text/96-rawkv-stale-read.md rename to text/0096-rawkv-stale-read.md From 6203c460ef32307a14c22fa5f122abf730e6055d Mon Sep 17 00:00:00 2001 From: iosmanthus Date: Wed, 8 Jun 2022 14:56:03 +0800 Subject: [PATCH 3/4] add more client defails Signed-off-by: iosmanthus --- text/0096-rawkv-stale-read.md | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-) diff --git a/text/0096-rawkv-stale-read.md b/text/0096-rawkv-stale-read.md index 90deca95..5d4bdcac 100644 --- a/text/0096-rawkv-stale-read.md +++ b/text/0096-rawkv-stale-read.md @@ -51,7 +51,7 @@ This RFC proposes stale read on RawKV which allows reading stale data from the n ### TiKV -The replica should read the local storage with a `read_ts` that reuses the mechanism from the stale read of TxnKV. This requires the replica to check the `read_ts` against the `safe_ts` which is advaneced by `CheckLeader` message from the store of the leader. As long as the `safe_ts` is no less than `read_ts`, the replica is allowed to read the key from local storage. + While trying to read data, clients should specify a timestamp which attachs to the request header as `read_ts`, typically a timestamp few seconds ago. The replica should read the local storage with the `read_ts` and reuses the mechanism from the stale read of TxnKV. This requires the replica to check the `read_ts` against the `safe_ts` which is advaneced by `CheckLeader` message from the store of the leader or `resolve-ts` worker. As long as the `safe_ts` is no less than `read_ts`, the replica is allowed to read the key from local storage. ### Client @@ -59,27 +59,39 @@ The `read_ts` specified by the client could be acquired by the following ways: 1. Calculate a timestamp from the physical time from the local. The `read_ts` might suffer from the clock drift and exceed the max timestamp allocated from TSO. The client will fail to read any data even if that target replica is the leader since the `safe_ts` of the replica don't catch up with the `read_ts`. **Deploying NTP services** in the cluster might mitigate this issue. -2. Get the latest timestamp from TSO and rewind it. This might get a more accurate timestamp but requires an RPC before reading which will increase the latency of reading. Applying a local TSO cache might mitigate this issue but require more resources in the client. +2. Get the latest timestamp from the TSO and rewind it. This might get a more accurate timestamp but requires an RPC before reading which will increase the latency of reading. Applying a local TSO cache might mitigate this issue but require more resources in the client. -The client should create a strategy to select the replica for the stale read. If we select the target replica randomly, some cross-region replicas might be selected. In this case, the stale read loses its advantage. The client should maintain probe statistics tracking the fastest replicas (by latency). The client will try those replicas one by one until it read the value successfully. +The client should also create a strategy to select the replica for the stale read. If we select the target replica randomly, some cross-region replicas might be selected. In this case, the stale read loses its advantage. The client should maintain probe statistics tracking the fastest replicas (by latency). The client should try those replicas one by one until it read the value successfully. ### Implementation details -1. (TBD) While TiKV is handling raw read-related requests, construct a `SnapContext` with the `read_ts` before acquiring a snapshot from `storage`. +1. While trying to read stale data, clients should set the appropriate `read_ts` to requests. +```diff +message RawGetRequest { + Context context = 1; + bytes key = 2; + string cf = 3; ++ uint64 read_ts = 4; +} +``` + +2. While TiKV is handling radw read-related requests, construct a `SnapContext` with the `read_ts` before acquiring a snapshot from `storage`. ```diff fn future_raw_get(...) { // ... let snap_ctx = SnapContext { pb_ctx: &ctx, -+ start_ts: ... ++ start_ts: req.get_read_ts() ..Default::default() }; } ``` -2. (TBD) Client API interface; replica selection algorithm. +2. Client API interface; replica selection algorithm. -```java -ByteString rawGet(ByteString key, readTs: Timestamp) +```diff +class RawKVClient { ++ ByteString rawGet(ByteString key, readTs: Timestamp) +} ``` From 5b588bb06495e309156413c4448a491eb799766f Mon Sep 17 00:00:00 2001 From: iosmanthus Date: Thu, 16 Jun 2022 14:42:53 +0800 Subject: [PATCH 4/4] fix typos and add the maintain logic of safe_ts Signed-off-by: iosmanthus --- text/0096-rawkv-stale-read.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0096-rawkv-stale-read.md b/text/0096-rawkv-stale-read.md index 5d4bdcac..9c30b778 100644 --- a/text/0096-rawkv-stale-read.md +++ b/text/0096-rawkv-stale-read.md @@ -51,7 +51,7 @@ This RFC proposes stale read on RawKV which allows reading stale data from the n ### TiKV - While trying to read data, clients should specify a timestamp which attachs to the request header as `read_ts`, typically a timestamp few seconds ago. The replica should read the local storage with the `read_ts` and reuses the mechanism from the stale read of TxnKV. This requires the replica to check the `read_ts` against the `safe_ts` which is advaneced by `CheckLeader` message from the store of the leader or `resolve-ts` worker. As long as the `safe_ts` is no less than `read_ts`, the replica is allowed to read the key from local storage. + While trying to read data, clients should specify a timestamp which attachs to the request header as `read_ts`, typically a timestamp few seconds ago. The replica should read the local storage with the `read_ts` and reuses the mechanism from the stale read of TxnKV. This requires the replica to check the `read_ts` against the `safe_ts` which is advaneced by `CheckLeader` message from the store of the leader (for follower) or `resolve-ts` worker (for leader). As long as the `safe_ts` is no less than `read_ts`, the replica is allowed to read the key from local storage. Notice that there is no lock for the RawKV regions, thus the `resolve-ts` worker advanced the `safe_ts` by requesting the TSO for the latest timestamp. ### Client @@ -75,7 +75,7 @@ message RawGetRequest { } ``` -2. While TiKV is handling radw read-related requests, construct a `SnapContext` with the `read_ts` before acquiring a snapshot from `storage`. +2. While TiKV is handling raw read-related requests, construct a `SnapContext` with the `read_ts` before acquiring a snapshot from `storage`. ```diff fn future_raw_get(...) {