deploy: 27ea781

blog/index.html

@@ -200,6 +200,14 @@ <h1>2023</h1>
 
 
 
+      <p class="post">
+        <a href="/whos-watching-the-watchdog">Who's watching the watchdog?</a>
+        <span class="date">11/22</span>
+      </p>
+
+
+
+
       <p class="post">
         <a href="/hacker-gifts">A side project story: Hacker Gifts (2018-2024)</a>
         <span class="date">10/29</span>

blogpost-contexts/index.html

@@ -230,6 +230,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+
 
 
 

cdtmp/index.html

@@ -213,6 +213,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+
 
 
 

copy-with-syntax/index.html

@@ -217,6 +217,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+
 
 
 

ctrl-r/index.html

@@ -212,6 +212,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+
 
 
 

e2e-tests/index.html

@@ -269,6 +269,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+
 
 
 

feed.xml

@@ -5,12 +5,40 @@
   <link rel="self" href="https://frantic.im/feed.xml" />
   <icon>https://frantic.im/favicon.png</icon>
   <subtitle>Occasional posts on technology and stuff</subtitle>
-  <updated>2023-10-24T19:13:00.962Z</updated>
+  <updated>2023-11-22T18:31:03.008Z</updated>
   <author>
     <name>Alex Kotliarskyi</name>
   </author>
 
 
+    <entry>
+      <id>https://frantic.im/whos-watching-the-watchdog</id>
+      <title>Who&#39;s watching the watchdog?</title>
+      <updated>2023-11-22T12:00:00+00:00</updated>
+
+      <link rel="alternate" href="https://frantic.im/whos-watching-the-watchdog" />
+      <summary>Making reliable systems that expect things to go wrong</summary>
+      <content type="html"><![CDATA[
+        
+        
+        
+        <p>At my current company we have an automated pipeline for processing customer’s orders. It’s pretty complex — talking to multiple different services, training models, storing large files, updating the database, sending emails and push notifications.</p>
+<p>Sometimes things get stuck because of a temporary 3rd party outage or a bug in our code.</p>
+<p>So we built a watchdog service: it monitors the stream of orders and makes sure the orders get processed within reasonable timeframe (3 hours). The watchdog only looks at the final invariant — was the order fulfilled and delivered to the customer? It doesn’t care about any intermediary steps.</p>
+<p>This system has saved us many times. When the watchdog finds a stuck order, it posts in our special channel in Slack. We investigate the problem and address the root cause, so hopefully we won’t see new orders stuck for the same reason.</p>
+<p>But who’s watching the watchdog? What if it fails to run?</p>
+<p>It actually happened to us once. The watchdog is running on the job scheduling system, and that system went down. That meant no orders were getting processed and watchdog also wasn’t running. The alerts channel in Slack was blissfully silent.</p>
+<p>To address this case, we need a system that can watch the watchdog. We are using these two:</p>
+<ul>
+<li><a href="https://docs.sentry.io/product/crons/">Sentry Cron</a></li>
+<li><a href="https://www.checklyhq.com/blog/heartbeat-monitoring-with-checkly/">Checkly Heartbeat</a></li>
+</ul>
+<p>The idea behind both systems is the same: they expect a regular cron job to “check in” on a pre-defined schedule. If it misses a check-in, there’s likely a problem and we get an alert in Slack.</p>
+<p>Complex systems always find surprising ways to fail. When adding an end-to-end quality watchdog (and ways to watch the watchdog) you can create a positive loop of detecting issues and hardening the system.</p>
+
+      ]]></content>
+    </entry>
+
     <entry>
       <id>https://frantic.im/hacker-gifts</id>
       <title>A side project story: Hacker Gifts (2018-2024)</title>
@@ -652,46 +680,4 @@ Things you can do:
       ]]></content>
     </entry>
 
-    <entry>
-      <id>https://frantic.im/octave</id>
-      <title>A side project story: octave.im (2013-2016)</title>
-      <updated>2021-02-23T12:00:00+00:00</updated>
-
-      <link rel="alternate" href="https://frantic.im/octave" />
-      <summary>A story about my attempt at SaaS</summary>
-      <content type="html"><![CDATA[
-        
-        
-        
-        <p>It all started around 2013: I was going through a course on <a href="https://www.coursera.org/learn/machine-learning">Machine Learning by Andrew Ng</a>.</p>
-<p>The practical part of the course depended on GNU Octave (open source math toolkit), but installing it on a Mac was a huge pain. I did manage to do it, but noticed that many people on forums complanied about the same thing.</p>
-<p>So I had a brilliant idea — wouldn’t it be great if Octave was available via SaaS model? With fancy features like built in code editor, command line and plots?</p>
-<h1>Node, React &amp; Docker</h1>
-<p>I built the first prototype in one night on June 8, 2013. I used NodeJS 0.10-ish with socket.io on the server side and CodeMirror with some plugins on the frontend.</p>
-<p>In October that year I rewrote the frontend in React — the experience of doing so was amazing! React was young (<code>createClass</code>/<code>autobind</code>/<code>mixins</code>) but its programming model “clicked” with me. I remember hanging out in their IRC channel looking for help with autoscrolling. I was really impressed at how quick and friendly the response was (thanks <a href="https://twitter.com/sophiebits">@sophiebits</a>!).</p>
-<p>The initial version of the backend would just run <code>octave</code> in a dedicated folder. My second iteration ued Docker, which at the time was very new and unproven. It all ran on a Digital Ocean 2GB RAM droplet.</p>
-<p>The killer feature was displaying plots inline in a REPL. You can see it on this gif:</p>
-<p><img src="https://frantic.im/assets/octave.im/octave-demo.gif" alt="" /></p>
-<p>It worked through a clever hack: I pre-configured Octave to use gnuplot with special arguments that made it save the graph to a file (instead of showing it on the screen). My NodeJS backend listened to filesystem changes and notified the frontend when it detected the update.</p>
-<h1>Product market fit</h1>
-<p>I tried to promote octave.im for the students of the ML course. I posted the link on forums couple of times and added it to the course wiki page (that was surprisingly very hidden). The reception among students has been really positive, but the course moderators weren’t happy: they wanted some kind of validation that it’s a serious thing (which it wasn’t).</p>
-<p>Overall I had more than 3500 people sign up over the course of several years. Unfortunately I didn’t keep any metrics screenshots. The twitter account, <a href="https://twitter.com/OctaveCloud">@OctaveCloud</a>, got 57 followers (organically).</p>
-<p>Speaking of which, I used Mixpanel and loved its simple API and dashboards. They even sent me a free T-shirt :)</p>
-<h1>Total profit: -$420</h1>
-<p>As every other hacker out there I also hoped to make it sustainable, so in October 2015 I added $4 monthly subscription with 2 weeks trial. To be honest I wasn’t very serious about it at that point. I just wanted to play with Stripe, see if people would actually pay. And they did! Overall I have collected about $300 in revenue.</p>
-<p>An interesting thing that I noticed was that people subscribe and then stop using the product, without unsubscribing (I did have the unsubscribe button on the profile, no questions asked). I ended up manually cancelling a bunch of subscriptions on Stripe without updating the app DB, so people could still use the service (which they didn’t anyways).</p>
-<h1>In numbers</h1>
-<ul>
-<li>308 commits</li>
-<li>3,500 accounts created</li>
-<li>450,000 commands executed</li>
-<li>$300 total revenue</li>
-<li>$720 spent on hosting</li>
-</ul>
-<p>Screenshot, for posterity:</p>
-<p><img src="https://frantic.im/assets/octave.im/screenshot.png" alt="" /></p>
-
-      ]]></content>
-    </entry>
-
 </feed>

good-errors-leave-trace/index.html

@@ -266,6 +266,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+
 
 
 

hacker-gifts/index.html

@@ -257,6 +257,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+
 
 
 

hello-world/index.html

@@ -206,6 +206,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+
 
 
 

how-not-to-flux-loops/index.html

@@ -250,6 +250,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+
 
 
 

how-not-to-flux-set-actions/index.html

@@ -283,6 +283,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+
 
 
 

how-to-convince-your-boss-to-use-react-native/index.html

@@ -253,6 +253,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+
 
 
 

keynote/index.html

@@ -248,6 +248,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+
 
 
 

macos-app-shortcuts/index.html

@@ -224,6 +224,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+
 
 
 

no-constraints-no-fun/index.html

@@ -208,6 +208,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+
 
 
 

notify-on-completion/index.html

@@ -244,6 +244,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+
 
 
 

octave/index.html

@@ -230,6 +230,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+
 
 
 

onityper/index.html

@@ -292,6 +292,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+
 
 
 

plotting-ideas/index.html

@@ -228,6 +228,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+
 
 
 

react-and-javascript-in-5-min/index.html

@@ -396,6 +396,14 @@ <h4>Related posts:</h4>
 
 
 
+
+
+
+
+
+
+
+