Overhaul: new spec parser, refactor packages, add spatial support (#280)

* feat!: Add spec parser, refactor packages, and spatial support. WIP.

* chore: Fix lint errors, expand spec tests.

* fix: Update mosaic widget to use new spec parser.

* docs: Update package READMEs and descriptions.

* chore: Decouple Fixed attribute AST note.

* chore: Update eslint to allow imports used in jsdoc.

* test: Update JSON reference specs.

* fix: fix toJSON for mark data.

* fix: Ensure all ruleX/Y args are passed.

* fix: Handle null logger Coordinator constructor arg.

* fix: Fallback in consolidator if requestAnimationFrame is undefeind.

* docs: Add jsdoc for wasm connector.

* fix: Fix typo in InstantiateContext.

* feat: Add astToESM connector option, update namespace option.

* feat: Add OptionsNode filter method.

* fix: Fix JSON serialization of Fixed values.

* docs: Add jsdoc to parseSpec.

* feat: Rename to PlotFixedNode.

* feat: Export AST node implementations.

* test: Update spec tests.

* docs: Update core package README.

* chore: Update .gitignore for test output.

* fix: Fix mark directive argument handling.

* feat: Add sql loadExtension utility.

* feat: Add array support to exec.

* feat: Add extension loading, drop client-side data loading.

* fix: Fix margin attribute directive.

* test: Update example specs.

* test: Update test datasets.

* docs: Update prepare examples script.

* docs: Update docs website examples.

* test: Update dev example page.

* feat: Add cross option support to Selection specs.

* Update lockfile

* Use params from new astToDOM

* fix: don't execute empty queries in ast-to-dom

* fix: Update mosaic widget to use new spec parser.

* fix: Fix raster default color selection.

* test: Add NYC taxi spec.

* test: Expose vg api in dev tests.

* docs: Add NYC taxi example.

* chore: Update dev deps.

* fix: Do not apply M4 to unoriented marks.

* test: Update specs, add driving and airline travelers.

* docs: Add new examples, fix CSS for tips, use symlink to datasets.

* Rename selections -> params and use name-indexed dictionary

* selections -> params

* sql -> predicate and extend docs about how to use the widget

* docs: Update README files.

* feat: Update scripts, use Python duckdb-server.

* chore: Remove old specs from dev folder.

* feat: Rename expr to sql.

* fix: Remove outdated options.

* test: Update line spec.

* docs: Update docs to reflect overhaul.

* docs: Add API context docs.

* chore: Bump python packages to DuckDB 0.10.

* rename server to duckdb-server

* Make server executable

* document server API, simplify docs for users

* chore: Dead code removal.

* chore: Update dependencies.

---------

Co-authored-by: Dominik Moritz <[email protected]>

.eslintrc.json

@@ -12,8 +12,12 @@
     "es6": true
   },
   "rules": {
-    "no-unexpected-multiline": "off"
+    "no-unexpected-multiline": "off",
+    "jsdoc/no-undefined-types": 1
   },
+  "plugins": [
+    "jsdoc"
+  ],
   "globals": {
     "globalThis": false
   }

.gitignore

@@ -6,4 +6,4 @@ docs/.vitepress/cache
 docs/.vitepress/dist
 node_modules
 packages/*/dist
-
+packages/*/test-output

README.md

@@ -32,31 +32,46 @@ If referencing Mosaic, please use the following citation:
 
 ## Repository Structure
 
-This repository contains a set of related packages:
+This repository contains a set of related packages.
+
+_Note_: For convenience, the `vgplot` package re-exports much of the `mosaic-core`, `mosaic-sql`, `mosaic-plot`, and `mosaic-inputs` packages. For most applications, it is sufficient to either import `@uwdata/vgplot` alone or in conjunction with `@uwdata/mosaic-spec`.
+
+### Core Components
 
 - [`mosaic-core`](https://github.com/uwdata/mosaic/tree/main/packages/core): The core Mosaic components. A central coordinator, parameters and selections for linking scalar values or query predicates (respectively) across Mosaic clients, and filter groups with optimized index management. The Mosaic coordinator can send queries either over the network to a backing server (`socket` and `rest` clients) or to a client-side [DuckDB-WASM](https://github.com/duckdb/duckdb-wasm) instance (`wasm` client).
-- [`mosaic-duckdb`](https://github.com/uwdata/mosaic/tree/main/packages/duckdb): A Promise-based Node.js API to DuckDB, along with a data server that supports transfer of [Apache Arrow](https://arrow.apache.org/) and JSON data over either Web Sockets or HTTP.
 - [`mosaic-sql`](https://github.com/uwdata/mosaic/tree/main/packages/sql): An API for convenient construction and analysis of SQL queries. Query objects then coerce to SQL query strings.
 - [`mosaic-inputs`](https://github.com/uwdata/mosaic/tree/main/packages/inputs): Standalone data-driven components such as input menus, text search boxes, and sortable, load-on-scroll data tables.
-- [`vgplot`](https://github.com/uwdata/mosaic/tree/main/packages/vgplot): A prototype visualization grammar implemented on top of [Observable Plot](https://github.com/observablehq/plot), in which marks (plot layers) are individual Mosaic clients. These marks can push data processing (binning, hex binning, regression) and optimizations (such as M4 for line/area charts) down to the database.
-- [`widget`](https://github.com/uwdata/mosaic/tree/main/packages/widget): A Jupyter widget for Mosaic. Create interactive Mosaic plots over Pandas and Polars data frames or DuckDB connections.
-- [`vega-example`](https://github.com/uwdata/mosaic/tree/main/packages/vega-example): A preliminary example integrating Vega-Lite with Mosaic for data management and cross-view linking.
+- [`mosaic-plot`](https://github.com/uwdata/mosaic/tree/main/packages/plot): An interactive grammar of graphics implemented on top of [Observable Plot](https://github.com/observablehq/plot). Marks (plot layers) serve as individual Mosaic clients. These marks can push data processing (binning, hex binning, regression) and optimizations (such as M4 for line/area charts) down to the database. This package also provides interactors for linked selection, filtering, and highlighting using Mosaic Params and Selections.
+
+### Applications
+
+* [`vgplot`](https://github.com/uwdata/mosaic/tree/main/packages/vgplot): A visualization grammar API for building interactive Mosaic-powered visualizations and dashboards. This package provides convenient, composable methods that combine multiple Mosaic packages (core, inputs, plot, etc.) in an integrated API. This API re-exports much of the `mosaic-core`, `mosaic-sql`, `mosaic-plot`, and `mosaic-inputs` packages, enabling use in a stand-alone fashion.
+* [`mosaic-spec`](https://github.com/uwdata/mosaic/tree/main/packages/spec): Declarative specification of Mosaic-powered applications as JSON or YAML files. This package provides a parser and code generation framework for reading specifications in a JSON format and generating live Mosaic visualizations and dashboards using the [vgplot](https://github.com/uwdata/mosaic/tree/main/packages/vgplot) API.
+* [`duckdb-server`](https://github.com/uwdata/mosaic/tree/main/packages/duckdb-server): A Python-based server that runs a local DuckDB instance and support queries over Web Sockets or HTTP, returning data in either [Apache Arrow](https://arrow.apache.org/) or JSON format.
+* [`widget`](https://github.com/uwdata/mosaic/tree/main/packages/widget): A Jupyter widget for Mosaic. Given a declarative specification, will generate web-based visualizations while leveraging DuckDB in the Jupyter kernel. Create interactive Mosaic plots over Pandas and Polars data frames or DuckDB connections.
 
-_Note_: For convenience, `vgplot` re-exports much of the `mosaic-core`, `mosaic-sql`, and `mosaic-inputs` packages. For most applications, it is sufficient to import `@uwdata/vgplot` alone.
+### Miscellaneous
+
+* [`mosaic-duckdb`](https://github.com/uwdata/mosaic/tree/main/packages/duckdb): A Promise-based Node.js API to DuckDB, along with a data server that supports transfer of [Apache Arrow](https://arrow.apache.org/) and JSON data over either Web Sockets or HTTP. Due to persistent quality issues involving the Node.js DuckDB client and Arrow extension, we recommend using the Python-based `duckdb-server` package instead. However, we retain this package for both backwards compatibility and potential future use as quality issues improve.
+* [`vega-example`](https://github.com/uwdata/mosaic/tree/main/packages/vega-example): A proof-of-concept example integrating Vega-Lite with Mosaic for data management and cross-view linking.
 
 ## Build and Usage Instructions
 
 To build and develop Mosaic locally:
 
-1. Clone [https://github.com/uwdata/mosaic](https://github.com/uwdata/mosaic).
-2. Run `npm i` to install dependencies.
-3. Run `npm test` to run the test suite.
-4. Run `npm run build` to build client-side bundles.
+* Clone [https://github.com/uwdata/mosaic](https://github.com/uwdata/mosaic).
+* Run `npm i` to install dependencies.
+* Run `npm test` to run the test suite.
+* Run `npm run build` to build client-side bundles.
 
 To run local interactive examples:
 
-1. Run `npm run server` to launch a data server with default files loaded.
-2. Run `npm run dev` to launch a local web server and view examples.
+* Run `npm run dev` to launch a local web server and view examples. By default, the examples use DuckDB-WASM in the browser. For greater performance, launch and connect to a local DuckDB server as described below below.
+
+To launch a local DuckDB server:
+
+* Install [hatch](https://hatch.pypa.io/latest/install/), if not already present.
+* Run `npm run server` to launch the [`duckdb-server`](https://github.com/uwdata/mosaic/tree/main/packages/duckdb-server). This runs the server in development mode, so the server will restart if you change its code.
 
 To use Mosaic with DuckDB Python in Jupyter Notebooks:
 

bin/prepare-examples.js

@@ -1,24 +1,36 @@
 #! /usr/bin/env node
 import { basename, extname, join, resolve } from 'node:path';
-import { readdir, readFile, writeFile } from 'node:fs/promises';
-import { specToModule } from '@uwdata/vgplot';
+import { copyFile, readdir, readFile, writeFile } from 'node:fs/promises';
+import { parseSpec, astToESM } from '@uwdata/mosaic-spec';
 import { parse } from 'yaml';
 
-const baseDir = 'docs';
-const yamlDir = join(baseDir, 'public', 'specs', 'yaml');
-const jsonDir = join(baseDir, 'public', 'specs', 'json');
-const esmDir = join(baseDir, 'public', 'specs', 'esm');
-const exampleDir = join(baseDir, 'examples');
+const specDir = join('specs', 'yaml');
 
-const files = await Promise.allSettled((await readdir(yamlDir))
-  .filter(file => extname(file) === '.yaml')
-  .map(async file => {
-    const base = basename(file, '.yaml');
-    const text = await readFile(resolve(yamlDir, file), 'utf8');
+const docsDir = 'docs';
+const yamlDir = join(docsDir, 'public', 'specs', 'yaml');
+const jsonDir = join(docsDir, 'public', 'specs', 'json');
+const esmDir = join(docsDir, 'public', 'specs', 'esm');
+const exampleDir = join(docsDir, 'examples');
+
+const files = await Promise.allSettled((await readdir(specDir))
+  .filter(name => extname(name) === '.yaml')
+  .map(async name => {
+    const base = basename(name, '.yaml');
+    const file = resolve(specDir, name);
+    const text = await readFile(file, 'utf8');
+
+    // parse spec and perform code generation
+    // do this first to catch any errors
     const spec = parse(text);
-    const code = await specToModule(spec);
+    const code = astToESM(parseSpec(spec));
 
     try {
+      // copy YAML file
+      await copyFile(
+        file,
+        resolve(yamlDir, `${base}.yaml`)
+      );
+
       // write JSON spec
       await writeFile(
         resolve(jsonDir, `${base}.json`),
@@ -56,8 +68,8 @@ files
 
 function examplePage(spec, { title = spec, description, credit } = {}) {
   return `<script setup>
-  import { reset } from '@uwdata/vgplot';
-  reset();
+  import { coordinator } from '@uwdata/vgplot';
+  coordinator().clear();
 </script>
 
 # ${title}