From 0ea8a6aff4bb84b4c58d1149728fbc2e72e6a267 Mon Sep 17 00:00:00 2001 From: frantic Date: Wed, 22 Nov 2023 18:31:05 +0000 Subject: [PATCH] deploy: 27ea7812c115712ad62b24e446fd6d6bfd731c33 --- blog/index.html | 8 + blogpost-contexts/index.html | 8 + cdtmp/index.html | 8 + copy-with-syntax/index.html | 8 + ctrl-r/index.html | 8 + e2e-tests/index.html | 8 + feed.xml | 72 +++--- figma/og_watchdog.png | Bin 0 -> 27603 bytes good-errors-leave-trace/index.html | 8 + hacker-gifts/index.html | 8 + hello-world/index.html | 8 + how-not-to-flux-loops/index.html | 8 + how-not-to-flux-set-actions/index.html | 8 + .../index.html | 8 + keynote/index.html | 8 + macos-app-shortcuts/index.html | 8 + no-constraints-no-fun/index.html | 8 + notify-on-completion/index.html | 8 + octave/index.html | 8 + onityper/index.html | 8 + plotting-ideas/index.html | 8 + react-and-javascript-in-5-min/index.html | 8 + react-api-evolution/index.html | 8 + react-conf-2018/index.html | 8 + replacing-jekyll/index.html | 8 + side-projects-are-hard/index.html | 8 + test-plan/index.html | 8 + the-first-react-native-app/index.html | 8 + using-redux-with-flow/index.html | 8 + whos-watching-the-watchdog/index.html | 239 ++++++++++++++++++ 30 files changed, 484 insertions(+), 43 deletions(-) create mode 100644 figma/og_watchdog.png create mode 100644 whos-watching-the-watchdog/index.html diff --git a/blog/index.html b/blog/index.html index 353f26f..deb2350 100644 --- a/blog/index.html +++ b/blog/index.html @@ -200,6 +200,14 @@

2023

+

+ Who's watching the watchdog? + 11/22 +

+ + + +

A side project story: Hacker Gifts (2018-2024) 10/29 diff --git a/blogpost-contexts/index.html b/blogpost-contexts/index.html index 304d028..f12d83e 100644 --- a/blogpost-contexts/index.html +++ b/blogpost-contexts/index.html @@ -230,6 +230,14 @@

Related posts:

+ + + + + + + + diff --git a/cdtmp/index.html b/cdtmp/index.html index 4790854..7a2340b 100644 --- a/cdtmp/index.html +++ b/cdtmp/index.html @@ -213,6 +213,14 @@

Related posts:

+ + + + + + + + diff --git a/copy-with-syntax/index.html b/copy-with-syntax/index.html index e198bd8..99ec63c 100644 --- a/copy-with-syntax/index.html +++ b/copy-with-syntax/index.html @@ -217,6 +217,14 @@

Related posts:

+ + + + + + + + diff --git a/ctrl-r/index.html b/ctrl-r/index.html index 9823d74..1f937b0 100644 --- a/ctrl-r/index.html +++ b/ctrl-r/index.html @@ -212,6 +212,14 @@

Related posts:

+ + + + + + + + diff --git a/e2e-tests/index.html b/e2e-tests/index.html index 88c592c..ea49a7e 100644 --- a/e2e-tests/index.html +++ b/e2e-tests/index.html @@ -269,6 +269,14 @@

Related posts:

+ + + + + + + + diff --git a/feed.xml b/feed.xml index 227fd3d..ca68ea7 100644 --- a/feed.xml +++ b/feed.xml @@ -5,12 +5,40 @@ https://frantic.im/favicon.png Occasional posts on technology and stuff - 2023-10-24T19:13:00.962Z + 2023-11-22T18:31:03.008Z Alex Kotliarskyi + + https://frantic.im/whos-watching-the-watchdog + Who's watching the watchdog? + 2023-11-22T12:00:00+00:00 + + + Making reliable systems that expect things to go wrong + At my current company we have an automated pipeline for processing customer’s orders. It’s pretty complex — talking to multiple different services, training models, storing large files, updating the database, sending emails and push notifications.

+

Sometimes things get stuck because of a temporary 3rd party outage or a bug in our code.

+

So we built a watchdog service: it monitors the stream of orders and makes sure the orders get processed within reasonable timeframe (3 hours). The watchdog only looks at the final invariant — was the order fulfilled and delivered to the customer? It doesn’t care about any intermediary steps.

+

This system has saved us many times. When the watchdog finds a stuck order, it posts in our special channel in Slack. We investigate the problem and address the root cause, so hopefully we won’t see new orders stuck for the same reason.

+

But who’s watching the watchdog? What if it fails to run?

+

It actually happened to us once. The watchdog is running on the job scheduling system, and that system went down. That meant no orders were getting processed and watchdog also wasn’t running. The alerts channel in Slack was blissfully silent.

+

To address this case, we need a system that can watch the watchdog. We are using these two:

+ +

The idea behind both systems is the same: they expect a regular cron job to “check in” on a pre-defined schedule. If it misses a check-in, there’s likely a problem and we get an alert in Slack.

+

Complex systems always find surprising ways to fail. When adding an end-to-end quality watchdog (and ways to watch the watchdog) you can create a positive loop of detecting issues and hardening the system.

+ + ]]> + + https://frantic.im/hacker-gifts A side project story: Hacker Gifts (2018-2024) @@ -652,46 +680,4 @@ Things you can do: ]]> - - https://frantic.im/octave - A side project story: octave.im (2013-2016) - 2021-02-23T12:00:00+00:00 - - - A story about my attempt at SaaS - It all started around 2013: I was going through a course on Machine Learning by Andrew Ng.

-

The practical part of the course depended on GNU Octave (open source math toolkit), but installing it on a Mac was a huge pain. I did manage to do it, but noticed that many people on forums complanied about the same thing.

-

So I had a brilliant idea — wouldn’t it be great if Octave was available via SaaS model? With fancy features like built in code editor, command line and plots?

-

Node, React & Docker

-

I built the first prototype in one night on June 8, 2013. I used NodeJS 0.10-ish with socket.io on the server side and CodeMirror with some plugins on the frontend.

-

In October that year I rewrote the frontend in React — the experience of doing so was amazing! React was young (createClass/autobind/mixins) but its programming model “clicked” with me. I remember hanging out in their IRC channel looking for help with autoscrolling. I was really impressed at how quick and friendly the response was (thanks @sophiebits!).

-

The initial version of the backend would just run octave in a dedicated folder. My second iteration ued Docker, which at the time was very new and unproven. It all ran on a Digital Ocean 2GB RAM droplet.

-

The killer feature was displaying plots inline in a REPL. You can see it on this gif:

-

-

It worked through a clever hack: I pre-configured Octave to use gnuplot with special arguments that made it save the graph to a file (instead of showing it on the screen). My NodeJS backend listened to filesystem changes and notified the frontend when it detected the update.

-

Product market fit

-

I tried to promote octave.im for the students of the ML course. I posted the link on forums couple of times and added it to the course wiki page (that was surprisingly very hidden). The reception among students has been really positive, but the course moderators weren’t happy: they wanted some kind of validation that it’s a serious thing (which it wasn’t).

-

Overall I had more than 3500 people sign up over the course of several years. Unfortunately I didn’t keep any metrics screenshots. The twitter account, @OctaveCloud, got 57 followers (organically).

-

Speaking of which, I used Mixpanel and loved its simple API and dashboards. They even sent me a free T-shirt :)

-

Total profit: -$420

-

As every other hacker out there I also hoped to make it sustainable, so in October 2015 I added $4 monthly subscription with 2 weeks trial. To be honest I wasn’t very serious about it at that point. I just wanted to play with Stripe, see if people would actually pay. And they did! Overall I have collected about $300 in revenue.

-

An interesting thing that I noticed was that people subscribe and then stop using the product, without unsubscribing (I did have the unsubscribe button on the profile, no questions asked). I ended up manually cancelling a bunch of subscriptions on Stripe without updating the app DB, so people could still use the service (which they didn’t anyways).

-

In numbers

-
    -
  • 308 commits
    • -
  • 3,500 accounts created
    • -
  • 450,000 commands executed
    • -
  • $300 total revenue
    • -
  • $720 spent on hosting
    • -
-

Screenshot, for posterity:

-

- - ]]> - - \ No newline at end of file diff --git a/figma/og_watchdog.png b/figma/og_watchdog.png new file mode 100644 index 0000000000000000000000000000000000000000..371b05191b872ab3f407a39b9d17666451c83ecf GIT binary patch literal 27603 zcmeEu`8$;R|Nl^v8lg_M##X0J%39|%vV@b0HcK&zqp zfkePyFsZX=PF;e*Hq&4*aopA|;5XKP<6XeVw%{{Pp)i=_F6cioSVop2_@!9rB?~iH z*$3rG@WW=G6P71nu<{hiRZln!=6CJvsS|dQV%#A?Wk5vq$hvor)<01v8G^frtvk}? zBRWfR&c<}fESom2-&JqA`e*8u1Ia2%%F5gI-aiwcay63*-@A7|Y3^9}<1b6szxa(X zy1z^eho`8fJT-}_ z*Mfe1)iMs;6|9-E4LlnRmW~pWhdwJ0G>d}|kG{>0(2wsj{`VjM?+g3iAo<_K_#mNQu8q&$R$KL0nzW8JY!EPbD#ObkBBWt3WhYSLKZ8+Y&r)-LjMyihOzWDvCY=n7?r%OdW7UI_+=W6pde^g2HzhY=`~#bmfGoz zufqC>!DxShx8pJL)6}nVUBr{IK0fy5ocY}8%nR`5#SZ>7QF^4;o7b7EE|Lyprrn3~jvKO2^RWVX4i{Z-^Wn^@kO)}(MX zJ9ce(oXWd7VI%>A*+cKv;_+I~ceS-KwGM_o94#1%7^&@WR8D#S1&u z**<8$sn6$iRjj~0h4|RFIF^>Ri!TkW74>{=#CkG6V1w+7l&WPL3++LRZ`R$oFd=t+ zbz*$-)`)Oe&|x!qOF~sKcIbWVTiF9Dq3@1(JqYXA?X&%f%k?*k2Spu0s~x%~MWdu6 zuPoG}!~7%Uu7<0i8tMuIO0vVbbe8n7!Qi&qzreA?{tjTkRr30U98S4Nusil6}qWfVL8E?k2;A%T$Iv9+k$r z#JAc#-6yKHCJX&n%MWAKM`Zr{%B0CiH2o1>ID93)^a6j{cs#^Me!$#5i@?8~H@uFa(+3>DAW^@oDH zm6^zmT~l;hj7Y$AG+GjZ=oI8`t%j)(ZlN)sU(*z=9J={`u5$5S&p{9Mt=0VvFQ&B( zs5j_JTq<@s%IG=PNe;6((H4yz$qZ^V$T43Uy)Bb$M8zC!8u(+r4@wqLz;1afBsNU+Yi=3U_A?`<|VuI6Kg@H|G_CYj@hbB_0aU7IAQrTe%_!L# zkFdM(V(5Lh99%-$gC}1+k?vHBT@Bu^C10svFNcwH3#jvN?CiZZUOyt58GD|ngA8n# z>zE5`H5FExW^Sh)1*kU$dQ?Z}TWBsRkSKxs$74Lc-U;J&E&S_!^<=MG+)7ejFI6OZ z_|ePP+5umW_^yhdE?0p&5bTR`V^)?N5norLkw*TpeSc~P2kkY6!I~}rOik%cchN40 ze(s_6*6q7yAOyB?C3u`f#20;NdIlkCy`=1;$%C4$WqLbL(;cLqi zDZ~PExHTwuZ87nqx3Dg9ALKB`bFX%J=?hjkRGCbT2lb=zl6Sjw>ixHmCmYSwx+BKr z-OCnxutVL->JQ?Z!@i-Uan|L4*t;riC5qJ`OyCuN8y ztR&wm1D^**Omf$=eul6P`yIibxRig>$TwZig5D0~#5hP`w5lawqYhwx^iVr^rUbpn_M=KcP@{h{rYad1 z74IQcFtS7(Q9MZ!C%0yS(!4D^^ZWp*bGTw5z7A)_Jk06N+u~nEPLD(G99*zt zY?3=Chhc7++T6mnpU4XR5Dda~d*~%cyTcA`q~@~D;WDzN+-JHzqqF~hJhwIj`zSn@ z5YO$I+T0dTwlpKR(%dS&sn*@~TjXKQN5Oe0dm9?lMhgkkaEI*Q9q-YV2=-8Q(Klq3 z72%x2*~5dfis6{c=YKlCb_-to)uk(wNt(19AQ9$4pG>#RUaRR3VP3l|aOm+%Eq|5a zf?{FThT{d^EazZ!&wFu!sN`cspIe?EzIR3NgT7@0SBxI~DAP#LgWJOo5%v%zX9@$* z!r})c_;;xA2Y)`RPAuZSr^D@FAD=In3k=7o47bc=uAFjjHCDOe?~>*~jatmnvbt`J z8@HEzW#`Qnba#N5Qxk54sI1c?EA)W2Ob!HP!-|LC=c>m1&Cvs<&-@by0(P2DKW#ByVjQo`V(gRZG zYMxB=wVUtY=H#?)g`@uL@22 z$-cPU%*_r28@1I@i3C3?1Gg%=C7@${?8TnL{MJiCB1WPGqPHgy@TY{&To+E&dD!Sj zD9Y|I`G+XJTC;fxsz~b{gKr`SZ()_??j8NDU*-)#pSabE*!BKIN)4^->N##vO!LUu z0ZKUqW4(d1q-6(75tDTz;uSdOA>2TK*ElwnMw=}WUbCCc@GILr;Myg7PN#>yh}pe& z&>hJuMd>f9&hY#k+-`h-ta%a*8(rTC-os9tF49Mmy;Y03CE5I)bRgFM`JMuiA>k06 zs8jdzc}TZy>@lCjKPI(b@FdodHbkkw=bdg3dLu`_o@(h>0u5yf_46h5P!EUpX5RgF z=|r7y3I`fn+3Knr$D93_saUkbEB4SGH@Rm1nr6d{{a-s^ju0o5aMARm{G%|xlF+W# zO6Y-r0n%t;&n6)~brE52&yCEueR{qG0E?m^U7K5<3lTrrDCV2{BFuxZ;={#plxLJJ z7fNd^J9a5_M~HcHtL3ZIu-hwn?v?&e<7r+g#yF2yr!2$BgZ0ZbGO(aOKvU@VqL(5{ z5LiSR7AZG4T^+zI#uPo3W6NStzegTDYOr#P5%MM{g>4PZ6UpXqKNr-2{y$gP_Hc&; z#J_CgD#w3DMDppSN_!R<_~iPg_UOJG%nO{;zWWh_A)Qyq{-2sJ#~y0NOg#Gda^~5l zxR=n&RrSs75eY_2^=@LFQ5sbPyEY3wYkpXF_4}>Keyy6Hytcfzc6KInWc7zG##$8D zM`?;vK$YB}F+%>pRX3@|lA{+En?L*wIaDioDotob$db#(h>cS`8U> zO=g`ZiE7suCKuh+0le(n2U?%D<86X)TL->H9%l$LS%Eo zGv#HPL*kNx`fk;8Det$AtyRu!%>&hpVgU}c-bTMi-^J@PH&|moh%g5nomGglF8ayT z!EVFj(=cyM1ljfyq#K`WuXqk$4y%~J?lK;Z&;IL!zE>@T`CvgqJKx0T^RXbgqsP;~ z9vIIU3KG7;Y>5RULm4U~T|BY1Vj!p~`sO>%&$-^>%=yU&{$;W2^cm^J#$9e=+i7}G zkFz;}7CUS77@>pvOQ%Zz1Z}h++I|QTI4o!sNLi%Dc<5GXy!N|5d)ShFTo;amM{osL zy9z$j5FrI3BZ^+Y(Ce+ktPZ=8(^^YMmW*mPzcCuSh1I@ydI|I&Q#7kBK+ z;xHM$73QA+z#Z;cwWVcdVEyi0E?L(iAdr;D^exKPtEMJbn_G3CkgCeCAlxw8R)&~E zJkSA?NQgmn_i3ZXKE}E@S-hakD>YR`#^r;X5^2_dNT^M7PBNFjTsp8|ICf{(T~1kj z;P$u4)ertXIdEkpTmHYV+vsEb54MAm6qOAoiK4WNE$IB~Tr)#<@8OaF^aBBy$IDQ9 z;k47i==4y4Ta(3x4|Jct^(CNYNKcO%<*K)P#J?Nu6|e0RWEVU{SQ@erI7iaY_K<~S zH2-1B%`5)>DnzP5f_)tQ!)wYc0@eqjD^BI-W=8A3mJ4mz@!(@4H|O;YaYwmThBl_@ z3jI|DYXI&k4%lbTtdczYuCHM1p+UE^gIQ<#P9Z~EzAg5xapLlNT{@T&~@ zOf4H;j9*0ba+kZR#eUWhhv`-7EBKTW6E^t0eXVaj`e(KD-v4=Eo76)1An~^{@`IxC zx{dJ)JXg%8qlTE6xYB^W))k~f#85aZIi9TQmr}onI%G|M7^QRU2z;5guz7PahuH5edzq!b536r8XBD#@i7uN5!Q#S&6b^%mN zTvGT*_VdVz?1<#NFNXDux^DUIW53s>J_~%xcGJG6durN#7sXV743A}~U2zJ_4IfYnD`oK>TBz<;!id&DTQr!mq94!DHV^$7YOq*#Iv^##qo(qN3T`=x=h!Gevz&g{GzUBem=| zijFNkBWv7|WXoc=PRmXIl;x{96aD0|dO9zC&#+Mo#u&X=FpQ@VweaQADQ;Bp$&-gZ z4h0y=z=qEO*g!i;+)*vzr_qdkM}xia_}T0|K$b;@lf!!(i#mp}JLT?V1-EWexrvt861ntY5il9Q*iNTZ~Vp6arYxto*<$L%Ack>EC|aZwNwDYqk>_PNQLzF^ht`NeTxUzrs)tqaG64;ct$!z>;C73OP= zcN&TVC5=HX=#hGyEs>nP*#Z8$tHb$EN(x#Z2kz&j#AlX~JHtJedZT3rEsZ--qjl>Y zi`W?Mm5Wh$IvC=P%OOjRR+wuQt#8WkPdnDJmls9s`{BhlrL2OBZ-Q=)D~$_=(uQN< zPT~}Bn{|Pq)Q;z;g)o}QM!PVL>pPC!e$5GHms)tF&gzcle&3W&zaM4qKa5rOFJEB% z!Gog+jzfq~dcH+QGt!srS>-sxcw@Fe zi2rw^wGc5L?hh(6FSfL%K@7>#=O>K|iJX!XXr-x?wVx>+oxK98=OT5D*@VnIMR&$t zVFqd{51tLC>%)KfxeM7#7>d$YIekF={ozF}V^#TS%dYg8ih3UBo#x%@*d|%H!rqiy zS@xo^PoZ&^Q0`k}b+5X11mi$vA1^QqdAhtj7W)wL$IK0Fa3ogDczqea%3~HdN32!~Le{Hcaf{IKqk!0f z-HQ5(qqU|ZP2lH6v^twvn|o;ItH%)92#=LpFaIv|xC(-fgWX?huxpL%^u zzYBG}&S)koC?KjI25Wc>#n=->e%hqO7pR&J4%ylIGGBw!S42zA;;UJNeRsybM3?ZR|3jQp`Mq_#EBTMQlWR3>Kjw%5rbbBp(R zmK{8MT}B3A z_&;^!Qb2ZVqS>9>+VU@JPEIv{q9TuoVD-~}$Oz3>8np*c`F^8(=b$21VvM#Grybhw z>tiqYbd61lcEjfhvDDt*i^(mY7XeK!VkhAi>Fb3N5ogl!MAgkLk-HjBtM z5lvdBP~8PhlS@C$yzUcs_le_K^~3aEp?*{0|Bm+_yxd*6YpcMG_|_8lPVVw17zcta z@-8>$+RwKYw?4|)P5eit_GHNjD+)v_P-sA$-O!`G56LL*1x(Z?X^#-A1!MBVY3f!ix9F;GY@`0(84tHw2}Mke{oaFDC)Z zgr+8WS-tSLgB3gScD^)El-^E597AZZ6X;r_2jeDrlA(wr2psmaGIqiLTpqffj@6^v zHMUQOI_(+qI&Rl>?Qzb$_#MDIdbzkXy&0z6mU|vf^72Yb%(gfB6gs5uxwqG$@z-Oz zLTAy=4pm#vGgLfv=++5)HBCirS-}398dc91F66uf6k|jKEcYa5@meRwi^L(7)L^w_ zVM}OemW)U05ahbz|1lT$^OQrXD4tqa<~o9R7#-st;SpN6v(=odhu+!~NN#s45{3P8 z@by^-6Mk@=YSC{~hK=g8{&5#{TiLNlX! z#EYn=qh_`JPx=I!9-JdJJW_cJ(#JL`i7^bnjAwLDmr@9;3<)Kk>yw98u>LFn|EiyD zxw!XJ#o?*uU0IwkGY!JVM*X*^PH+4<>`Qt^vf;@%mB#2&a(iND}u zZ%M2K*@>?2S`I6I5d74wvSacZt5W(V6#@Z79P0>_bIFlW}=+ zj&lEoaT#mVM7P_mWXsHl!pV`}VahriuZDeuTjhHvz=>v)PqO(6QqrbC#epEz)mR84 zZ?Rcnr-kvZ+`{iYSER_NTi|yUb=r(NEZGG!=j&!dFYPLssI<4Go*WP7c4Z)*Ans!? z%N?k;iSF4`%ya(PPYOQuz_q6D(rhiwBu)0)%%}oIR%0(GTd;iZm&Iyz3m=RFYvcO z|C78Y@(U6FevLsYWaoECuvA=K^#MEBA=nS2Z28bibt-PC zWu!80AtF`~(J9!DC&yY7Bg^Usv>4_LJx0QRR*fF#+2r2W_W1!_t@q5*0d!e?f;mo5 zq#sG+Ue>Vj?Y1?Yl@F7BEgOUSTwP){>9;?ys6H$Cj%uvBeqS9{jagSDDKFhywem5+ z zHU2RyNpR;J{}KPbwJzf(@oZo7^{=DDn%5{hYbTr9k?9>0VjezF*!=;QARx1~`rm{~ zXr96<*lQ*Mz(hpb7o~VHsXd*ESJA#O0k~!@j}LosBtIVt*N(4V__-^TH4GlY8~Z97coO4 zWwrFKv;C^>yVbi<+d0xOe{;}HoqO-OX{URO*>kfXsOdkfT+DESD|I0+!4IJxAZQ2! zlcg3J>DaZ2SQCHJValw3S>j-QBgO&XK-LDr-!AM!AGBGrODzrdC=zpZ7v7S&>EYiV zy*8hltNdNFxmX)5S+v8iJcvXqv6Xzn{rfx%J<3cUvAg%luV-60orK^a-bTGgd{|j& zul3Mwil?w$EGl!xhsTYdFxK(!hE{~CUfmW={2$06a2m32r(Zn=459OuXk$X?=^MD_daNg_)SA;9jlaTwSC4QE?7`H`Q4!l{T zFnSkz4^Q3jy-9r#kp1apPNF!2?|?FN*i%LHc*YCuu{=g(*`#837AjLxSa&opta8Ep zHU>x*)xahZ|Hxqa=QCb-YMS;VQ^x=j+k=x6rSFjV$LbK`Ym}+LL~EMU#}n3@@J0uo zU+imo>^88Ly)Y;@Bez+*V=hW}VU8hPQ&)%yiCq`OHt|iW31y`m9m*CMZsSamO0Ln8 zc$^<4=O$-2;{*Sfp{`If%wg~lo|0s2ANNO9dn4Y^y2L3lkvxgf3_FRnAyT0wp0?s9ipzFzJB&Ku$eD_l)amb;dnsOI)}C>a`${7#BP!W% z{3z1!#es{$u9XjvjZ7KbSib?p>^QYpm>y?UVe~HbO-6~DhkWs<2l7T`g$v~| zkvhdB?@Nl8+6uN@x@8qmV0P#9vtS_muue9RBP}i`IBRFA@-~=;l<1*7&#q!p%D+$1 z9i;_l@reiQ^PJSXnfx!OJbyqdl#nCBPaa3)zZARkN-}&1IdJ>L@@DoK2K9-GVQ_(5 z0rL@7x#qx?0EW2$H{WAOE$e*Sv}UKjIiF46qP{2USkEYCG6If$ElsW zby8h-_T)qM^;Mbkl1kw`Na^5_5=6JWv^YH9o3T88KX^^ zV!7oHTQFklv}0ncenu(W3e3-DBm_1t{{VyV?Sb3XJ+m5I+sw9M1DHNr#5Pr($vFn} zM7UmGAb*xIzBDCmsQ$)wcLmFtfYy< zDhoO?5L`AnrE&-3kJueoipBX=(d7~4dP_}N&5b} z@{*V{)VXr4W=v9iDF)d@3{Z^{6D{9~trR zNANOZ8HU4v6ti@~>r3b-FHF`er}Gr~%bqUz#iAiY>WO`XXxF{=UbVIoFfCj(y{MGs zcWexuQnaV<&21P>1#0a3;Ep3JeKmI(_L+q$PAGFjLg^n|+p*Jm7JL7G{~&;aulczO zmWaxo(+F`rwC#)R6QQn=!(LjCII``Q&VLhhB#wPT7|JmJbiU(YVe1RT7=ACe79q&; z2})s_&6t{7_WixTtp^gUjTPXSZWH0O;(k3H=QFiaci!jJw-#sIXXp-`rE9(7hC*aX z$YA)FL(em>v0TQf3HTr-*tEmOO(qUmaAssQs(c422hgRdIM-6)$%U^H3qi5M0WW=; zzremOjj@%6kX^}Z8pWpez-4;9u9ZXs?kCgv9&=GU!@FMUn7n)`Dc?!V($hu^JvR9q z%@or3pT1U??kNhU2!A#rr6^9!maT+#MvVhJI(ii%SNiRRuhdQE&Z}u|1tlYk?Xy=k_DLkL&d#y_NZLIsmcV{j@uLe zl$V~?OFFbwSl=Y9pSzJcAhYf|01X^Td;6Txd6%i@1BzSw+=|-$(2n)};D2g|K29VbXuJr+P$~p`gd0uC`*<$;(@#ko?BQu5`Gong1;6$T8E-j6xP?Z_}N6S-> z3kB%VVCVI-gZ-mwqEBj~uh9SjL8~18ne|=H!YSSLU%H((lc<-fY^YXgnaM`4=*laCF4_0HD+1-*q zz7Svpf7`BVnW9#*d=zVYCSnSK$Mi>cY1bsRpuO%wLdGQ*xhtxZ>w}ohcG^VuJGO}< z0%ctsn(o0EeZc~ZTMN_mmUHL%5~4NU`kLaifai=81RvsdetMs; zt6Y}sJr}Ut9>3>cnG=Q`1`Obfa;IuaiCyt8p!$s3-jG~i)@t5`D2Q6^T0r6u0KPB3 zIg);XYEkr!o>{V^lGQj@m3=pvyr8UL&K%|3Ab$sdhi)nm2BKbpiabK4m#l=F>L7LX zr$;PKOuFT4sr%^n5no+?3g#i425hFWU2VwB%ZcO`%M(4nf1FGh$|imP^wUIHVi&hF(MGaw|BJ?%t|}z z5O)DvLCGmn4HfGnbq@Nb666zSH5eb@gdu=2Rs6i<1{b3xD^_wuFL&*e+@#P{__uF} z=7P%|yCFoOd=t*OsziEisU`LpD^*2sF6HG!W9P`yM2q_0fwn;Mz8;&GO6#t; zYYmo^ifM31f`KkTW{MwF?z=h;#Y)xSZZhUNlu~mY^ncBz#NPC0T29VIaKcH^Oo7fb zO~vWzrTTW!)$%7Ai4!x;e{)8I6Te3#8-DsFh~Wu3E)HA94_IaaeQlf6vBA;a6CgG~ zfat$nJURdMN^NV783hbEvkJl|(tRV^t;OE^lHeNj+w3vBxSpi}l6Su(;Sd~^UErwn z!u&I7=NpstADKa_n<~7eijaQXw{_qCJumHd_@fusVERfsy2un~37ESkgX*q6sfW@o zX&=VQx0c5UYhv;Km*w6D`XyEJTn{_la~hC_^&15V*Vm|GF#Rbxc{$QiO#eaKfE?Um z?Vk7<0E|;*Bx|rH(Q}FJ%A2yx3QlWYdmK+@+tbSspFm3SHBdj8XNG#i<&Vm4*TVU5 z=^&SS*OznGm*4(C8x=%R)Rvkh?p(CE72hV`BgoNu;3_{ob1IRiV0n_a3D)l{4DwQk z9p0h^NKEcG87AQ`iRv6gAm-1Qfq)vRSiVoJUh=-OeBOw3G4i1HUFu}Tq}eP#QZ7#0 zHgOtk;W7BIkYk9?QGbtm?iR7qL+;h+40 zp!LTo%r%YXsVHFs<8$a7Hz*db{M3bhlJcz7)dH+3$Ej7U^rnnk+JjW@{<9fI=*UH= zl4m%;f)yc}Hg`&b6aNCtZbc(X?tvzL%v{lHuOP@8Ev zkAO+5K}!?c><1BGKcy>j+;@XpOB(Ba{ww?mCKqfx+B!}6~m8mFLX=&rW}CnR=E(p|BQe3E#m?>DHw9ZB}NzAz=~ORm+8UGDPIue)#7 zYYnCvFOE$lP>MjSO@2v?4RtS5hTSDj;-4?gA3C8O%}ZFrxymI4IOn@^e}g!nPV?TD znWpu%5YLrraY{}~#~&vEX`6(uKzd|LakdL)$`x84qjIo-C7`yewrZpB_zD^KS zEDEd{yxtWI3J`({8{Gj|cwquugNco|spLufq|O@Waa#*jLi z?QYvuqSh}QYgA*)bk_Dls$2lIjoTnq`LcnG>FTV0E8P}FWk-aAMR~HI=A0)+z`TE3 zzM5xE+}YO(Nk%(tLZ7ApY4bS#Z=p0S`8X5?UIbl^o(0x4yaH1+y?8s?bW}~6+km|x z1pDJ7dZmRo1@$8;N_K?nnb?aW2s#`ohD-#|QE09$o32lc1EO(I!6K!Vjfu7efqS_O z-wc%VHPCRDBG=K|tdW#~NI;ZX0cBx%z@@nG+ITWS!7b9$-&_Uz6emUMSYyATC&wqW zlFVPRBbcB0XwOrYl{w_@n?L>XauSo`%K&5jhcnLaf3;XM_NxFYQXL%zcFjEJQmQA z;2d!=N7U$4?66$2DrK(EOshyny!jy{$cQd8wqm)2zE}L9CH@TQl>KMqevE@8)KL19 z*{{I29Z!Z^X)|L)i>yxG4KNv|NKwGI)B!I+XOBGIDw5-SoF;G2BY1oIwAio5z~c6u z2OTX)435sO#sLN)St>uq(ez(zbr{lvn`T+1P7Y~HwKAN1vewv#4j^M3AY+4mM&v>I zM?+s)aO>IIK>=__hD2exukHI6Z6%*tparo1Sru%Eum$_N0qDJ(w$kKd z8@bg39kV14|Hr2zSvh1d;*B7In?iQq?If|j-sk*`n+zX@HEi@X7 z2uh%#wnzrqr%!>ja7Q9Z%!&*3FA>E^_;E)f*(c}5Rey}Oe6t!U{4x74qIC>Z?5&Sv zVFzaWdVd$7xv$J@LHuv{Uq)$%(y>J!zaD*cn;yIdgpP_iH-Ei(A!hHs-n>aBsm5FQ zPnvJdMSL074bs|fQl|VJ$~>E8y7DPy%l|x-rPr+#)3v25LK4nyuWivbogP-Hk_)rW z5I7+^Pjex)Uw_3dZ={neeH`248USnNgBXizD~vWH!gOqfQx#o0N)|6yfVlaTti3B( z?`)_Pb+D$Ni*)`BO8mYis?&0j14ZrOMAMIre~7jnzul=E`fjNlT99w3o~KW>upk3X zbs41l6YE0}|LG3r4X_ zPZ(LgVG}F?!qNKvhzty=@%@W_xP$gJ-G&WoOtd-eKCaS|POZp%Xbn5sdJm4}Rb8w1 z=5b#d0=YU@{W8nY#e*pfVQwS9n#bmVHMQCE7Y8UAyq3ZMi6_O25!5GSYt&+$>SzB6 zxHcVzJ`)ZEH`4zpnlNcH#u3srDPGo=%tQRfPGtr>#mR|g6HJvm8$FaZL}cpOVV}RP zRYB2wvm01$lGkhgfCcxf#IDY@S%7mL0}d!Ps6mWLm@c)L^Y43l!O0yPN-;{T%8r^m50@J2sTME$ri&al< zG;0sJE84bkOtT2^d{D8eM5iA{nfEKQAfr)pHT@mqV(TG@qypAAH|QR#hk+T&N{{vn zvIo%uvSXa$Q1o}A76~(I5Q=S{RgAKCK%-c%3GV!$nu95YCPF@@^v6NI|n3Sal8H0 zJ^^*w@hw}r;*S=Z1!8Kb+S}F#s-q{L0-IXw5E=s305PY z{-Vfr4a|VMogG2)41c?)38O(<8jn8Yn8yBVp27+kXOZQuwgHF$jjO0%SJgzRW&+#E z;p4*T$~K5bJ}z45X{usPzj)$?Gy#S|X~=AmKrXAp4kqGi!|pYMPOskp^!iP|%EfD- zx9D}A@<0lJdFY$?2paM`rti)&Nuhcw}bp?cR@JZ+)k;J+iXchto^hw zMQ+d)YQX2mk@4Hb0R{q8k)yZV zmmMwBp+$)Dk?Hw#s`zU&vRfz80YZGAZLUAfz!B%^`#7NJ>-+ERgKB3MDDiE(zy&qc zG#&kV%hT)wv|%2+jq8$~!rSE>w4`X4@I^<=#QKiQEkG!2{6n$1_<$OEwzdRm4c2tA z$L>0y@`u;>jCJcT!+4^O4j@x|qDJI$5CG%{L4`lR74nHCW98(PDe~6BRP5UY17)Yg z)XkD)B!Frkj5>GtDMgi<3SM>-M|i+2d0fkqlBnAl`HLX{H7G@!l7X>RmlucC`&`@% z3$`l`u-$d_++H&;UxK=_0dXb#&u@@YZOnRo!8!qQ#W+h+<=F(L6(znrt)54h?}Wr) zscNwcpDYp}0VNE^-)N`U-d4jO>0W6ul(`D1KCV7)qn0;Y{j=Jt(0F*2=`Lah3%NBZ zMrC?Zva#dGV!Zs1yW+lhYg#L7$qJ_&nxY)dD6Kt^f2*O|+mJ|0l_eCxXc5Ewb&)%$ z^}ybt3Od~!8=lA~PRR5vqTcqOO;7OBRe&|f`GTysAhI2ZJFQTm z$nlk5Pn!I*IER-()jW2SUzw`{@FSQc;}*feN!6n(ryki_?T)aD{<jZ}d~(VsjA6!1xb zz$VdY)s&qt+2)XbLcdV}5c0Zm=LAyC#ph-jYg~mF*|~&VTPQMhT`>@em=5m7I;V8C zG>(#d(fa)P-1VE2tT0Pd6ZQm`$r+}Ht;JyjNEd*W98PndQE{$z%)F8(03c!4$H-R`pmZm4URGTQF2edDuNS;fN#&i4RklVAA!`(r8NaolR`aT-Agzbv6C zOMcU9QgS+##G(ydg`Z^n0FFHILX#$#{{S_5gxa~5&AaI{=~gEG66xq_`X4H6wAl86 z%`OfwRAA}^#e4F-2Ovn%@qHM1#Hh!rXoIhfM9-714%Df08IYU#Hd!Y6=gBtUk~PW- z?{gaWvp>iz<)L>}U7!vuPvowj7qe?J1>mCdLGcbh=4FA&@Yb(xcO2<7FfHc1;Fd2a z`mOJFhyTZ7Z1F<;U!13kV*qHib4mi@b7CwAuX+~P&O|0 z7MY-2fMmH1I*20+29Ejh5EY&llhRrwP6gnH4^Z}D606eBSAuPK4_ymFI65~#A{}lA zd96MlX>DZ$(mM@8lAe(jVTK9UO=I>lrsaH#_1L=!6T!=k z>GW;3tpq8+j@Gj@sJ1?*CIj?r37tn;#EVJ}TpH`;){Hb(y|Ih4w?Qw3&$1#+88ZY& zSB@R6PY|GPf8r7~miHv!XB09KDm3amkj<ZOW6{sqVHp19`;Kj#hhp&)`J+%(!hq{dgE_tnD zQH=;P09w!pP|Ir+_F-LK94_u)LY|$w4#f8X$Q{gN&6_TT^mM zXe!iIwzpR^Uuig!33UI>*XK+^<6toV6bL&{$zAEimE5=8@z;w7XT?Fo!^T=cV5%zg3 z=aRj)uR{SlMg)(QQ`nk=j)7c~Ef=Knhjb26w=p>vjix*Zf1~QLcsaw;OZS))izu>u zeFQhUK^U+ad<ch^*UqvqXM}Id#dX?>U844Ytk8=Y@fI-eeKi#H z2sB=;K+f^xf33K$rtl_!^%@sY?$`7ju;)E#tp1xK=&2A+iqQQvrQhdZv-Z$mo#8*m z9gx#G%BK69elioYkbEV19H~y>Rt}U`7?88=f5WlClEx9p!^~Bz7TqKG%Ni{s z4j6|C*?vHfwDI7FWPKY-?a`L`t!ak{(p*c7R;2sU3ci<;U-@B1beEFf>$1$)`u>(t0Zmd&C75!ZX8x;xZ{3vKdsRa}n=PkFR@3Kb07vG-_qjrjV;s1EBf}!at^WZ(oeI{yg#o9GC z&o+fB|HbMgWrp*`CpTNUOY30$j|{&pFpr2VjwEdQJbWKqlh{`3wX?74+bYz`j<2@6 zueK{*@V{<0lC`*0XqMJm2T@9G;Vjo`0e(&AgKI_D zBc+CN70m(IL+ctu?gsg))}ujqqD=>EpYX}dnBgOwuUwiX@siF!3LNGS=?|uziM0WB z`6bOHkyXt|jBqx0ttS?__RV_j3Mj>mhwnNO_sL(^8i=T}bUi7s{aFR90z5!6|C#-W zHA6FN+9c<{u`B7f%J{WWz}Ht;>P-T}l{M~jOyR@|^!Foh&spmm+Nv!q5)Mt}tgpDQ z`$3gQFD5|2bEH?b6o~Kl^HmsO7LJ?;WI=^VFosOIwEEw;_mty*pz)UDTgFW zt1H$wzwR9XB)>&Jrl5>O7Vz*>Vt0^h-oI2VTkaUTU zsp-+wu5`!MU8cS-G$Kc-NOlyO%fJB_2`#EUMl^OjMV$Wp^;cT7LW8V1*G}xb6e&;= zl_@VpX-6$goc?%mPY;jv<)s}Bl(Lj*{br3TzITG3s;(Q}vd#Zj`?PLx2BM7(1Z&;V zflD>({E@&xdCl5PL`hu-{{q*3@0TKz#gZdhlp{FWOw)^*%!6?*eMo1Bi{2PaqB%BF z6qCNd7m&5y{}S_^?+BWkm7R4a;@PIQlK<1*wZB81_wPZd5uJ9_5^Ybrc~Yphktmfm z)iOhkVH&3?QDzXEGvgE;R@fe`HZ7a2hB=u*Gb6{LjaqF=GtP&k!Z0WqBKf}Fv)}LY zKYXuif9i5wT{H9fd|vO@;lA(J{R!T3vhT0$6OPLZO`9t;+@Au`UGmUv6}P0(A1lFL zw4X>`w)Ex6&JcaWrpA#Yk&J?a6pt z!WMv53(!9POw__ypFRrjk$w)h*r<2m^K_Qq>8{QKk+CQ%@>_dKl1LF(i!4`K<^T3; zZg$(ixitzSk*W33XKh2hZb^8YjZ!B2kLuydd4jAa7b}b%52{#hilGp-AXl|@D#Kq6 z^(Gsb&eJ=lwO4y>9SZ~P;ky{8YCuOGzH^gCy79k#FR}2RK4O-)BOa2&S_BE4C{)93 ziN4Zfm4VrBhVZ`t|B#h&J94@&aX}DYwVY#3-rriFqFa2h{V>nU?+VtXkjyMwrJUof z00cDiD|SGisS;|fHM$AT3>Ig(VC2nr^|V>`Duj24~G(WTyVrouvJEdj4gcO zZELp;kxidk&(62n)Kkusjm`8F?9kaIl2-o|P^e^q^UM~L==ZPcPktThoxvdDg+>M4 zk&9&v8@xMpM$FNVyj4>pWGI+!m`(rv-cDPm?ww?ta}%F=Y4ul;v~su+>sT`@7i3EO zu`bv~@%^1_al5SOl!YKKKcxK}bw;#Bs)+I4DK%fDI9KR!8>Z^8cCluC&NC|pOD?5O ze_d}c8Q&(VY%%}>+;qWeJ@gfudF}36jDm(|Zd`T`nOl7KZOb*sVe$R1YhH3z)U(Gs zj-*$HA)6D@bmf@eaLnjA@QeW6#&*%8>Jf@0Sw2YAzL7rH{@08dX2JzwEj8lR8t%zW zvbIQ0-?dH~ZX|VzDY?I)WV!!x=e)+@4S1*>*Q-v6YL}UGbZzNsUUE>c{MxkZ_ z+qFpOFaM8(YdmdnK*JGPo%yMgt>9yAc$iz6q=r+bD^emsw(iT+^T$j?R& z6zzPn9mlX=#@R^960usZZP@1U3mg?1y?|ESd^5J1-hn8axW6AaJ$36%9Y_2*mg8c* zgV+f7P10PfKlUK2Vc43SGzCU8TjnWzwvhv9!Be+#BSY_I5xo8SDo47xI_`6oEFbQ} z{4%oPKo&{=TYU&jOH^J3fM5G(XD`;<45%iPpwSv)L^&=$O!eRwc6yN3JL`2zRuq4r z=%vIuH+@0l$0q3g$fBNP_Jzbn{OH>+mzAGxizr%AZxo@y^iC|gG(P( zZw(-i1W5FG&f2y&|6PudTrAA7U|w+Q$CLo_v~_K6J78o%EL z@nkBDAg&+X#*ejf&LC@I>>!sx#0QT_QC3K=&xvq(;u+!46LP(LHE#>jhOA+lhjw1B zFDL8u>#X*T^3g`oeZI*9$rTo+PI7+LK@}f z4k6Z>rOB%QDMITyS)bRVLlNcY1K#y`;thZuee@*xKoRu7(O5RU*kTfVv@GW5>VK;l z^vQfVRqX+SIICF8LZr>>2O1WmswSVP-{{n>hlkH37#+|yBPF}cajlog4cs58pw)$N z+TROcdAa;~Rmv@LR<%xR=~qQb!RAu)Duwtu*4m@zkl=yaM2-=a`E>i9)lfA44ytj! z(FU)mRAB=FBE+V9qeg21v1sJZgFMAMnUu!HsJw!Vq|9jj63AYq%F!k4ic)*9E@ ztpzzup8p;e8UAH?WySP_9s65jPwV6Amzft)4q5!p&Lec3N)-yZKl%L`(y-VU`l`nm zp6gKRD&iV3`z2@4-&~4(m=fOn@xVcB#T<6Z?kgKSEzsii!u7LqkKuVJljJD7P?AFM zzpA?a_nt8+CkBtBe({V4aE~yh7=8kr1eo2sUpHdYsIO`W-i(5dYak1q_t4}a@33fK z^{*!+{8dCfNz!YTu-h1l1gy|qDWxC%ygi*3B!^Ms?`}ViTc1jBTnsUxnNA)!qr3S| z;FZvhGv2J6^MV0@k}0#(-kxyDtF)p{LYA03HZUTk31~y0E&Mh93Udsh%^$miNOheu zdtu67c~_QO!ZWqJrJE%A7dpdmyVrY3lF7ep!^vbtojcapg5Q_Kl!hxMKuWzp1a5?{ zdqC8}%D+A~E=?+89!FFG^2H-2)J`_-J zjs6zb%`>Lq0w6y}uB1(U)Xi$O7!B8TZQY3auFiMb=a8ky z>q*MYhXl!njQiTjmztSpd&mz@g*9^9ZyHu(;GW?T}gbXrfAF!jZarg1^hD530KKqO;S zfQe-qfh6$k_p;#m#|E<;8pwUC{s#Ks4#5)~+mwn}nAeSwEmGJGaofy$d8((!OLrD5 z0PXHe*mGT~#|b?UoBpxH&P37TWp?ZdB;#9}X2kvkP`&M1qBrc$v%f0&YJ{eImmAxM zj#96>dFkbtK<9LjbUTLGSDzbIJ>vafBX$=YajF|>fAcl!e$h?%0~tz;%n5)Hwml+r zHQ$p@G>M06+gRHMOoOep*r3pHu}2z^OHD?>+{kNu+m7Y`cL6waXNX#2uH7@v-JT4s z5F-QrY;H08>O(>}lk2z)P`c@5UG8&q`l2!n<9Uer#CT1I2#lUCbs-#k{KPBGL>E4;s#R!8ufmqSuCGXtYvqV&}16E;EoD zZHk|$(>1F-J&xRUgAgg!KJ6|F!de*sbxQ37<>vlIRz*Jh*6opw@YGPUG3Tl{FE!|5 z%TvC;VS!I`TpJR8Sk71FpLaC20`~VoER(`8Z-0Q1{Hm+-i)N3Pt3=WaV=5VgJ0Gc*9 z&%0O-=`~S=yu4p#D+p1vsry{}L@Vlkl9DUqng7ZNi-SJSj#Ugq5Lk2&*JP&pOsX!x zCa7}4HuR6OpM=lrQYtqgKZMLaz5ng>`??&H@bVCpHllXt@5Duh6Ed_+IbVvOUFwx( z7N^|F-QQ)6`(?`Oi%&8)q-HYvt*g@mi97y${cThhBKw?g^<@ z_W1dn*S;X!)WQgN8Z3yNf!pU{80M#_r&A~50Cqp6#l7WM=$>I336+gxhO-R!)~1{V zkfkL{vY>G7^E0C^t##ILsUr_Wk_xD|V_?I~Ok8uFv1(5($!TbHLEo!cx-X2<-r1E^ zv6dkf!>q?wT`{z2@U=@~3cfl?Z*H;G=cS~xk(uHgAG_NS%Fu@x?I6-$w+!!$KU|WC zcuBLpt$~+F{p8M75da7n9BSdrduXTY))Z$r8U6|KMufW`;t_N(XAw&F(xSkBgaq{? zZ5teDm5p+vlr9V5q{&h~L}>EtPbKo^b=O&zFGU>GZQh$ghFZZafC3dgt^sb0M z`;xH`SLv%)Mlj=6&?1!UK0Z2RO*fgkpSCCO$LOycb%CIPYCTt6f?<^WHVX9VuR*?3 z_g91AX!}OHmMmS2%5DKzkI}d^H&3Yj#<^-1`n8G-u`~-i+L;FHu;WLJnAkbvbSyF*p@=0Z>Lv8*-bK>X8 z?C~t3YtV}Ms|mXS%cNblgQ_GCFu#;r$~h>rEQE8!dK#i~!qEs8Klm0V%NKVe!u9?F zMi;r@;8QCC5{r8Rt|*A4nn+`pzoq`9m}LZ#nWnEfszg1E;NFWcZ`Go9C6S`G_Xf1x&&_!!zUs6vw?EOEh_=q`EiA zEUBZ0vkSpD%Eu}3VEp#c*9*WGu%>I=I8Cu|Y{&_GK8+5*M-gHGg~nVXiik_cPK~%7 zZfUG&@W&Qog{;WLVA6(gi-5CtgiNnJzTTc~&n(&X`MZ>ta$}!j=PPz8Krk+kTu-tG<=N>jwUz1 z*_lXQTy}Hb%kC&mPU-0;WGoX;bGDWpeS}cC(io4SkD$qa^ePf~`%eN(m;-P=cmwM< zaAUn^r_1)(v-3$z8wzV8PzNNpY?fd+G|w`UZ)ETrZ`LDS^^H`_(` zsNdBp0J0OiAjCvsv|#?4eYDIuqX2am!)rdonq+bju}<#yxXs zEKI!DKhK~1BE-_J@NIx1g8NQ24ZCKWYgLHALgb?`eMPkso|)UeXtv8Tv}?+j+4z479dzcMwy2`hznG5j^i`8Byu@&w*xt8-Oump z0=0j#TT+0n0c!%yN~Fe|sI7rHd0HRqd4t{=Erm79OJQQauh~DS&e4WhCfzD z0pc)Qw=7{Rh*`ww&TY^KLigQWUsCk*nRetF9ORUIyzQfUG>z_mT>6y)_^vft*Zqxxw_NLgX4F%K z*fzwfcgMK<30ge`PW*D>+A2_QHRho&QO_xPYJ=bRss@egyMRV%3>}h^7U@}Fx3TK) z5H}-aLL#0sR|@GTKFRXA#gkyWIrnp|S^}VMPugTgrGU_{=_J@MMvp(`ww)vqm zuHapZ;gs>0i;`m0T^up)Hd04zdqiag5aNC}aJ$V`kqhTwE?(bgjW1P!3@8e~p*Fy< z?Uh2;F(jw)q%>{GJrKmY;N8?ysezdsSh%|gGiry=Ril6FxM8FN!!I#h=5cgxUq9s; zM!Q&e_N9-E)_`g0O9S3Pr%|Z$zbqX@(zd%z!y1^(0Cj}2ljZie{lOtPvq-B^m*IZ8G$NsizIvl)7@hBR?ZxU30f@I5rw!b#8?n@L*;|Jr+gTpiCG(% zQ?7-XBgPeD6LT^aMgB?K@2cUkAigN1ICJT?_?KpyKLNrWl&#mF1f7+x9;Nk;1%_ z*^Eh%$#Dx;tM77Q$qw(0oMM)XSMynDx~R(Z&jdwh*zQ-}Pp+BVmPsUQs;Ng2s(!(n@1L2-cJH=fq%k$0ioxsBO+{B%aG zRk^t|mrlpEFae>4G=Wj=aq8AcVvZo?N_itEOiDDZ0&R$ULc)HJbCGQ7PA&-lY7>qi)j!A^Jz(i5uu z>8jIeFDazbzAe~%uL7jh_XO}IYt{e3eUh_HibU7DwZlx9K4#yDtC&5oa?)-E{_%yG zQVk5vWgh9)(9(AQSmn4W-)A zLE}UwsHiWln;$yl*Q9`J?~8s#`-`jc5lKzpz8r@8vQRelated posts: + + + + + + + + diff --git a/hacker-gifts/index.html b/hacker-gifts/index.html index 7fc0ca8..cf9e9fb 100644 --- a/hacker-gifts/index.html +++ b/hacker-gifts/index.html @@ -257,6 +257,14 @@

Related posts:

+ + + + + + + + diff --git a/hello-world/index.html b/hello-world/index.html index 343ef7b..40d81b5 100644 --- a/hello-world/index.html +++ b/hello-world/index.html @@ -206,6 +206,14 @@

Related posts:

+ + + + + + + + diff --git a/how-not-to-flux-loops/index.html b/how-not-to-flux-loops/index.html index 7098177..676db3e 100644 --- a/how-not-to-flux-loops/index.html +++ b/how-not-to-flux-loops/index.html @@ -250,6 +250,14 @@

Related posts:

+ + + + + + + + diff --git a/how-not-to-flux-set-actions/index.html b/how-not-to-flux-set-actions/index.html index f02659b..097cac2 100644 --- a/how-not-to-flux-set-actions/index.html +++ b/how-not-to-flux-set-actions/index.html @@ -283,6 +283,14 @@

Related posts:

+ + + + + + + + diff --git a/how-to-convince-your-boss-to-use-react-native/index.html b/how-to-convince-your-boss-to-use-react-native/index.html index ac0df75..600d1db 100644 --- a/how-to-convince-your-boss-to-use-react-native/index.html +++ b/how-to-convince-your-boss-to-use-react-native/index.html @@ -253,6 +253,14 @@

Related posts:

+ + + + + + + + diff --git a/keynote/index.html b/keynote/index.html index 857ceda..3ffe5f7 100644 --- a/keynote/index.html +++ b/keynote/index.html @@ -248,6 +248,14 @@

Related posts:

+ + + + + + + + diff --git a/macos-app-shortcuts/index.html b/macos-app-shortcuts/index.html index a2d730a..108cf47 100644 --- a/macos-app-shortcuts/index.html +++ b/macos-app-shortcuts/index.html @@ -224,6 +224,14 @@

Related posts:

+ + + + + + + + diff --git a/no-constraints-no-fun/index.html b/no-constraints-no-fun/index.html index 231ae60..8d90c55 100644 --- a/no-constraints-no-fun/index.html +++ b/no-constraints-no-fun/index.html @@ -208,6 +208,14 @@

Related posts:

+ + + + + + + + diff --git a/notify-on-completion/index.html b/notify-on-completion/index.html index 9e4b07c..38c8451 100644 --- a/notify-on-completion/index.html +++ b/notify-on-completion/index.html @@ -244,6 +244,14 @@

Related posts:

+ + + + + + + + diff --git a/octave/index.html b/octave/index.html index f143c04..b14b553 100644 --- a/octave/index.html +++ b/octave/index.html @@ -230,6 +230,14 @@

Related posts:

+ + + + + + + + diff --git a/onityper/index.html b/onityper/index.html index a3aed3c..27d8a5f 100644 --- a/onityper/index.html +++ b/onityper/index.html @@ -292,6 +292,14 @@

Related posts:

+ + + + + + + + diff --git a/plotting-ideas/index.html b/plotting-ideas/index.html index 2bd2edc..d941e64 100644 --- a/plotting-ideas/index.html +++ b/plotting-ideas/index.html @@ -228,6 +228,14 @@

Related posts:

+ + + + + + + + diff --git a/react-and-javascript-in-5-min/index.html b/react-and-javascript-in-5-min/index.html index 6e131a8..776e97a 100644 --- a/react-and-javascript-in-5-min/index.html +++ b/react-and-javascript-in-5-min/index.html @@ -396,6 +396,14 @@

Related posts:

+ + + + + + + + diff --git a/react-api-evolution/index.html b/react-api-evolution/index.html index ee358c4..18b2354 100644 --- a/react-api-evolution/index.html +++ b/react-api-evolution/index.html @@ -459,6 +459,14 @@

Related posts:

+ + + + + + + + diff --git a/react-conf-2018/index.html b/react-conf-2018/index.html index 7b4d0c3..96db4cc 100644 --- a/react-conf-2018/index.html +++ b/react-conf-2018/index.html @@ -303,6 +303,14 @@

Related posts:

+ + + + + + + + diff --git a/replacing-jekyll/index.html b/replacing-jekyll/index.html index 4e7332e..30f7e44 100644 --- a/replacing-jekyll/index.html +++ b/replacing-jekyll/index.html @@ -239,6 +239,14 @@

Related posts:

+ + + + + + + + diff --git a/side-projects-are-hard/index.html b/side-projects-are-hard/index.html index 9d9ad11..7319eed 100644 --- a/side-projects-are-hard/index.html +++ b/side-projects-are-hard/index.html @@ -230,6 +230,14 @@

Related posts:

+ + + + + + + + diff --git a/test-plan/index.html b/test-plan/index.html index f78597c..372074a 100644 --- a/test-plan/index.html +++ b/test-plan/index.html @@ -216,6 +216,14 @@

Related posts:

+ + + + + + + + diff --git a/the-first-react-native-app/index.html b/the-first-react-native-app/index.html index 3262075..c1ef951 100644 --- a/the-first-react-native-app/index.html +++ b/the-first-react-native-app/index.html @@ -240,6 +240,14 @@

Related posts:

+ + + + + + + + diff --git a/using-redux-with-flow/index.html b/using-redux-with-flow/index.html index 1b4831f..7d15bf2 100644 --- a/using-redux-with-flow/index.html +++ b/using-redux-with-flow/index.html @@ -300,6 +300,14 @@

Related posts:

+ + + + + + + + diff --git a/whos-watching-the-watchdog/index.html b/whos-watching-the-watchdog/index.html new file mode 100644 index 0000000..f0ac6f7 --- /dev/null +++ b/whos-watching-the-watchdog/index.html @@ -0,0 +1,239 @@ + + + + + + + Who's watching the watchdog? / frantic.im + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ +

Who's watching the watchdog?

+
+

At my current company we have an automated pipeline for processing customer’s orders. It’s pretty complex — talking to multiple different services, training models, storing large files, updating the database, sending emails and push notifications.

+

Sometimes things get stuck because of a temporary 3rd party outage or a bug in our code.

+

So we built a watchdog service: it monitors the stream of orders and makes sure the orders get processed within reasonable timeframe (3 hours). The watchdog only looks at the final invariant — was the order fulfilled and delivered to the customer? It doesn’t care about any intermediary steps.

+

This system has saved us many times. When the watchdog finds a stuck order, it posts in our special channel in Slack. We investigate the problem and address the root cause, so hopefully we won’t see new orders stuck for the same reason.

+

But who’s watching the watchdog? What if it fails to run?

+

It actually happened to us once. The watchdog is running on the job scheduling system, and that system went down. That meant no orders were getting processed and watchdog also wasn’t running. The alerts channel in Slack was blissfully silent.

+

To address this case, we need a system that can watch the watchdog. We are using these two:

+ +

The idea behind both systems is the same: they expect a regular cron job to “check in” on a pre-defined schedule. If it misses a check-in, there’s likely a problem and we get an alert in Slack.

+

Complex systems always find surprising ways to fail. When adding an end-to-end quality watchdog (and ways to watch the watchdog) you can create a positive loop of detecting issues and hardening the system.

+ +
+ +
+ + + + + + + +
+
+

Hello! This text lives here to convince you to subscribe. If you are reading this, consider clicking that subscribe button for more details.

+

I write about programming, software design and side projects Subscribe

+
+
+ +
+ + + + +