-
Notifications
You must be signed in to change notification settings - Fork 6
introduction
This document will help the curious decide if they want to try programming Matrix-powered front ends.
We describe the Matrix reactive engine just enough to motivate the developer experience we find so productive, and share a bit of code from a demo project, rxTrak.
Matrix began as Common Lisp Cells, and has now been ported to Clojure/ClojureScript, native JavaScript, and ClojureDart.
While Matrix offers a solution to complex state management in general, the big win comes when applied to GUI programming, where state management quickly scales to the unmanageable.
Programs are dominated by information processing, unless they have UIs, in which case there is this giant circle of information processing that makes the logic look like a dot. But I am not going to go there. I do not do that part.
Clojure/Conj 2017
Matrix has been integrated with several front-end frameworks:
-
Common Lisp: qooxdoo.js, Tcl/Tk, GTk, and OpenGL;
-
JavaScript: straight HTML/CSS;
-
ClojureScript HTML/CSS, ReactNative, and React for the web; and
-
ClojureDart-based mxFlutter is our current effort.
We stumbled onto Cells — re-inventing prior art — in the mid 1990s. The origin story will be a worthy read for those who do not enjoy reading unmotivated magic. More write-ups and links to live demos can be found in the bibliography.
Now let us look at the core Matrix reactive engine, and how it works with the CLJS+HTML/CSS stack.
Matrix GUI mechanisms will be familiar to GUI-savvy readers. Matrix just arranges the same things differently.
A Matrix app consists of a tree of objects whose properties are made reactive by structures we call cells. We call the objects models, in the sense of "working models" that react to change around them. We call the tree a matrix, for reasons explained below.
Developers use the Matrix API to generate Cells for any property they wish to be "reactive", meaning they can:
-
be computed from other properties, and kept up to date automatically;
-
used by other properties for their computation;
-
propagated as needed outside the emergent computed dataflow, such as when a fresh value for an HTML attribute is propagated to the actual DOM;
-
accept values from outside the Matrix universe, such as user mouse clicks or XHR responses; or
-
some combination of the above.
As soon as Cells are generated:
-
Matrix instance initialization "hides" them behind a getter/setter API; and
-
manages state change automatically.
Here is a snippet example from the TodoMVC classic, where the spec says not to show the "beef" of the to-dos list until at least one to-do has been entered:
(section {:class "main"
:hidden (cF (mget (mx-todos me) :empty?))}
...etc...hidden HTML attribute will track whether the mx-todos of the app are empty.
In general, Matrix models serve as proxies for any application state we wish to work reactively, such as DOM elements, localStorage, or XHRs.
In Depth: Implementations in different host languages
We want to stay out of the implementation weeds, but to ground things a bit, the model objects are implemented as:
-
maps in CLJ/S, with a type property that is a subtype of
:tiltontec.cell.base/model; -
CLOS instances in Common Lisp; or
-
class instances in JavaScript.
The properties are:
-
CLJ/S map key-value pairs;
-
CLOS slots; or
-
JS class members.
As mentioned, they get their reactive powers from structures called cells. As a shorthand, we refer to such properties themselves as cells.
Cells can get quite sophisticated in mediating reactive flow, but for our introductory purposes we just need to know that there are two fundamental kinds of cells:
-
ruled, or formulaic, cells which have a function that computes the value from other cells. The instance itself is available to the function asme, akin tothisorself; and -
inputcells, which can be assigned to by conventional imperative code, usually event handlers, be they mouse clicks or XHR responses.
In a more advanced write-up, we will talk about ephemeral cells for events, lazy cells, and even synapses, anonymous cells that live within a ruled cell formula.
Great. Now we have a declarative paradigm in which changes to one cell trigger a chain reaction of changes to other cells. And we have a problem. Suppose in our example above, the hidden formula of the SECTION proxy comes up with a new value. How do we tell the browser to change the hidden attribute of the actual SECTION element?
So-called observers can be defined for a property, such as the hidden attribute for a DIV, and cover all instances of that type, or they can be defined ad hoc when we create an ad hoc rule for a property.
|
Tip
|
Many reactive systems refer to derived properties as "observers". Perhaps because of the observer pattern. The dictionary disagrees.
— Merriam-Webster.com
Derived properties certainly do participate in the larger data flow. In Matrixese, then we use An exception that proves the rule: when coding complex UIs, we commonly want to help the user by automatically kicking off an application change that they might initiate themselves. Matrix lets observers sudo into participant mode to initiate such changes, but they must do so explicitly and their changes are deferred until the current change has been fully processed. |
Here is a heavily-annotated example that may help ground this abstract description of how Matrix apps work. It shows the implementation of the footer from the TodoMVC classic:
(defn dashboard-footer []
(footer {:class "footer"
:hidden (cF
;; we navigate to the app-global 'rxs' list,
;; then read its :empty? property, transparently establishing a dependency.
(mget (mx-rxs me) :empty?))}
(span {:class "todo-count"
:content (cF
;; 'mx-rx-items' likewise reads from app-global 'rxs' list,
;; then reads the actual :items property. The moral: Matrix "sees"
;; dependencies formed inside the call stack.
(pp/cl-format nil "<strong>~a</strong> item~:P remaining"
;; 'rx-completed' shows how we can hide `(mget rx :completed)`,
;; again because dependencies are "seen" inside function calls.
;; So here we end up with one dependency on the list itself, and
;; dependencies on the :completed property of each Rx.
(count (remove rx-completed (mx-rx-items me)))))})
(ul {:class "filters"}
(for [[label route] [["All", "#/"]
["Active", "#/active"]
["Completed", "#/completed"]]]
(li {} (a {:href route
:selector label
:class (cF
;; Note that an individual HTML attribute can have
;; its own formula. This kind of detailed semantic
;; relationship is lost with view functions having
;; piles of subscriptions to a Flux-pattern store.
;;
;; 'mx-route' tracks another property of the app
;; as a whole, the selected :route
(when (= (:selector @me) (mx-route me))
"selected"))}
label))))
(button {:class "clear-completed"
:hidden (cF
;; another DOM attribute formula. This fine granularity means
;; minimal workload to propagate a state change fully, as
;; well as making app behavior trivial to follow.
(empty? (mget (mx-rxs me) :items-completed)))
:onclick #(doseq [td (filter rx-completed (mx-rx-items))]
;; this loop sets the :deleted property of all completed items.
;; rx-delete! is just a function created to hide the generic setter 'mset!'
(rx-delete! td))}
"Clear completed")))We mentioned how properties get their reactive power. We did not mention models, for good reason: all data flow is between properties. As just described, a "ruled" property can be defined as a function of other properties. When one of those properties changes, the computed property will be re-computed. Without glitches, if you know your reactive art.
Models simply provide the infrastructure to support data flow, and an aggregation of things to be animated by that data flow. More on that next.
We said a Matrix app was a tree of models. Each model has two properties:
-
parent, not acell-mediated property. Not mutable during the life of a model. For life-cycle reasons, must be provided when instantiating a model; required for all but the root of a Matrix tree; and -
kids, the optional children of a model, and optionally a ruled cell.
Our tree, then, is just inter-connected parents and children.
The fact that kids is ruled means the matrix tree can change shape dynamically in response to events; the model population grows and shrinks. The largest matrix application starts out as a single model instance, then grows from that seed as the first kids rule gets evaluated and spawns descendants that recursively expand into the initial state of the app, perhaps a full-blown landing page.
Hence the name "matrix".
ma·trix | \ ˈmā-triks \: something in which something else develops or forms
Serious reactive engines must avoid so-called "glitches".
A glitch is a short-lived fault in a system, such as a transient fault that corrects itself
The problem arises when two paths in the dependency DAG lead from one node to another derived node; with the right combination of intervening dependencies, the derived node can compute twice, once with an out of date value because it is on the second path that will be refreshed.
This is indeed transient, but things can go seriously wrong before the state gets corrected. Not always, though. Matrix itself delivered complex enterprise software while still vulnerable to glitches, but eventually they broke an application and had to be resolved.
In depth: The Matrix Data Integrity contract
Matrix puts a positive spin on the issue of glitches by offering what we call a "Data Integrity" contract.
When application code assigns to some input cell X, the Cells engine guarantees:
-
recomputation exactly once of all and only state affected by the change to X, directly or indirectly through some intermediate datapoint. note that if A depends on B, and B depends on X, when B gets recalculated it may come up with the same value as before. In this case A is not considered to have been affected by the change to X and will not be recomputed.
-
recomputations, when they read other datapoints, must see only values current with the new value of X. Example: if A depends on B and X, and B depends on X, when X changes and A reads B and X to compute a new value, B must return a value recomputed from the new value of X.
-
similarly, client observer callbacks must see only values current with the new value of X; a corollary: should a client observer SETF a datapoint Y, all the above must happen with values current with not just X, but also with the value of Y prior to the change to Y; and finally
-
deferred “client” code must see only values current with X and not any values current with some subsequent change to Y queued by an observer
After program correctness and efficiency, what matters is programmer productivity, a function of the expressive power of our programming mechanisms. Matrix has been designed from the beginning to make it easy for developers to deliver functionality. Here are the key wins.
co-location seems to be the silly jargon for not scattering relevant program logic to the four winds in worship of the false god of "separation of concerns". [Discuss](https://news.ycombinator.com/item?id=30166318)
With Matrix, when working on X, we have everything related to X in front of us. With HTML, we can even have the CSS right in front of us, and with individual style properties with their own formulas. Of course the class attribute is supported, but both CLJS and JS have more expressive power.
Aside from CSS, the developer need not wrestle with Flux to create responsive U/X, they just need to know how the U/X they are coding should track specific other app state and code up the derivation. The uni-directional Grail of Flux external stores is automagically detected by Matrix internals. And now all the semantics of any given component will be right in front of us as we work on that part of the U/X.
Because a model knows its parent as well as its children, and because a rule has the model lexically available in me, rule code can navigate from me to any other object to get a value for its computation.
Likewise, an event handler can navigate anywhere in the application to mutate any input cell. Gasp.
In twenty-five years of practice, on very hard problems, this global reach has presented no problems. Consider that navigation is not a Hail Mary jump to a global reference. The "reach" navigates outwards from me. We can write our own traversal, or use a Matrix utility that has a bunch of options to constrain this search, which defaults to checking:
-
me; -
my children, depth first, left-to-right;
-
recursively checking my parent, which skips
mewhen checking its children, my siblings; and -
stopping when it finds the model whose value I want, which model is specified by a test I provide.
The consequence for the D/X is that the developer who needs a value to compute a derived value simply has to know where to find it—by name, type, or arbitrary test—given the inside-out search algorithm.
|
Note
|
The alternative to global reach is Flux, which fails the "expressive" test in several ways. More below. |
At the time we instantiate models, different instances of the same type can be provisioned with different cells for the same property. An example will make that clearer: two DIV proxies can have two different rules for deciding the class of the DIV. Obviously handy.
Even better, we can provide a new instance with an ad hoc property. So our two DIVs can have additional properties to get their job done. We will see an example of this in rxTrak where we have a DIV dispatch an XHR to look up adverse events in an NIH database and capture the results in a custom property.
The astute reader will notice that the Grail of OO re-use has been found, albeit by cheating: a Matrix type is reusable because, while the expression of a DIV class is still fixed, its derivation now can vary to suit a given use. Object types simply do not dictate as much as they once did or, put another way, are just more configurable.
A good example will be seen below when we get to our demo project, and endow a DIV with a system clock driven timer.
One difference with Matrix is that all long-lived application state consists of models and cells. View elements, domain objects, XHRs, localStorage, the system clock, and anything else we want to be reactive can be wrapped in proxy models. And because the reactive paradigm works so well, we want as much as possible to be reactive.
Reactivity is a contagious paradigm, and Matrix encourages its spread.
[WIP]
Our demo project for Matrix will be rxTrak, the TodoMVC classic converted to a personal medication tracking application.
We did rxTrak so we could include two async features:
-
a timer so we can keep an eye on refill dates; and
-
an XHR look-up of each medication on an NIH adverse events database.
By the way, our mxXHR add-on to Matrix is a good example of how to make reactive something that is not reactive, XHRs. This so-called "wrapping" process is straightforward once one is comfortable with Matrix.
We will be extending this section with code examples RSN.
In the meantime, here is another write-up involving the JS version, with live, working CodePen examples you can play with yourself.
[To be continued]
None of this will make much sense until one has a chance to use Matrix in anger, but it may help to point out contrasts with today’s conventional reactive front-end frameworks, with which we have had a few years experience.
With Matrix, no separate Flux-pattern store is needed to achieve uni-directional, coherent state flow. The application itself is the reactive DAG, automatically detected at run time.
Matrix manages state in place.
We avoid the burdens of:
-
architecting the store;
-
composing actions/reducers; and
-
mapping the application structure to the schema of the external store, now that the latter has been abstracted from the former.
In depth: A deeper look at the problem with Flux
Flux got invented because Facebook engineers could not figure out how to maintain a chat counter. Their very plausible solution was to:
-
Balkanize GUI application state out into an explicitly designed in-app database, the "store", with a schema abstracted from the noisy shape of the GUI application;
-
create an API, if you will, of allowed mutating "actions" or "events" to modify the store;
-
then let the noisy GUI structure work out how to live with the store, both:
-
discerning how to subscribe to specific elements in the store; and
-
work out how to use the mutating API to express user gestures.
-
Plausible, but exactly wrong.
Make everything as simple as possible, but not simpler.
ascribed
===== simpler considered harmful
Consider this. The developer is working on a GUI application. These are notoriously hard because of the many interdependencies of the presentation. GUI applications tend to have multiple ways for a user to accomplish the same thing. These apps also need to reflect any given state change in multiple ways. The very content presented by a dynamic GUI application must change over time.
The intimate blending of state with GUI components is a feature, not a bug.
Isolating state once owned by GUI components, and concocting a new "ideal" schema independent of the GUI application, just makes it harder to do what developers still have to do: make the GUI application work consistently as a whole. With a separate store, a developer working on a UI component now must work out how to get the state they need from that store. When they want a component event handler to alter the state seen by other UI components, they have to work out what abstract action/event will leave the store with the state where the other components will find it. They may even have to architect new store schema and new reducers when extending functionality.
Understandably frustrated by their chat counter, Facebook engineers made state management simpler than was possible, and Redux was the necessarily bloated outcome.
With Matrix, we do not publish or subscribe explicitly. Matrix automatically and transparently:
-
detects when formulas use other cells;
-
records those usages to form the implicit DAG; and
-
automatically propagates change through that DAG when application code, usually handlers, write to input cells.
Handlers do not dispatch actions/events that lead to complex state mutations coded elsewhere, they just change one cell of one model.
When it comes to rendering, we just write code that builds a proxy DOM, and observers transparently and efficiently maintain the actual DOM. There is no VDOM built or diffed to come up with DOM changes to be applied. e.g., When a proxy DIV decides on a new class for a DIV, an observer uses the DOM API to change that attribute.
Instead of wrestling with view lifecycle methods and artificially arranging for view functions to re-render, with Matrix we just write code to build a view and Matrix internals keep the DOM updated, efficiently and minimally.
Where Matrix wraps React, Matrix uses setState to trigger re-rendering only when necessary, and it automatically knows the minimum DOM that needs re-rendering.
Here are other Matrix write-ups and on-line working code using the JS version of Matrix, with the better write-ups first:
Obligatory TodoMVC, in CLJS
This is both a high quality write-up and a working TodoMVC implementation.
Scroll down to the end of the read-me to find a link to part II of the write-up, which then leads to part III, a crazy deep dive into the detailed flow of a state change. It reminds us of how many things we need to get right when changes happen—and TodoMVC is rather primitive when it comes to interdependency.
The evolution of a quotations carousel, SimpleJX in JS
This includes the live carousel grace a embedded CodeSandbox. The idea is to do great things with reactive programming without introducing a heavy mechanism such as Flux. The quotation carousel of "simplicity" quotations is crazy slick but uses just HTML.
Matrix, The Origin Story
No joke, this is exactly how I stumbled onto re-inventing Matrix. It happens also to be an ideal intro into how Matrix works, because you can see how we got from "pull" to "push", hence reactive data flow.
If, like me, reading about some magical library drives you nuts until you have some grasp of how it works, this is ten-minute read that will fill in that gap.
The Cells Manifesto classic
Surprisingly complete and comprehensive, given that I knocked it off in one sitting in about two hours, when I just meant to do a quick overview. Strongly recommended for the bit on "Data Integrity".
CodePen JS Matrix Intro
This is an application within an application, if you can beleive that. The CodePen code presents the evolution of maybe half of TodoMVC, which is also a working example. I scare myself sometimes.
OK, these are not write-ups, they are just showcase apps developed with Matrix:
-
A search tool for Hacker News’s monthly AskHN question, Who’s Hiring browser. This is the JS version, using HTML/CSS
mxWebas the front-end platform. -
Tilton’s Algebra, built with Allegro Common Lisp Cells driving the qooxdoo JS library.
For more information, DM the author @kennytilton on the #clojurians Slack, or visit the #matrix channel there.