From fa1f432571883a38deadbe4d5c9cf9d6823fdd74 Mon Sep 17 00:00:00 2001 From: Jon Anderson Date: Thu, 24 Oct 2024 21:42:41 -0700 Subject: [PATCH] Improve filtering docs (#214) * Cleaned up sections a bit. * Moved filtering into basic query section. * Added section about CLI. --- docs/index.html | 178 ++++++++++++++++++++++++++++++++++----------- docs/index.md | 189 +++++++++++++++++++++++++++++++++++++++--------- 2 files changed, 292 insertions(+), 75 deletions(-) diff --git a/docs/index.html b/docs/index.html index ea38bd3c..ffdddf06 100644 --- a/docs/index.html +++ b/docs/index.html @@ -38,22 +38,30 @@

FQL

id="toc-variables-schemas">Variables & Schemas
  • Space & Comments
    • -
  • Kinds of +
  • Basic Queries
    • +
  • Reads
  • Filtering
    • + +
  • Advanced + Queries +
    • +
  • Using FQL +
    • +
  • Project + Roadmap

    • Overview

    FQL is specified as a Space & Comments <uint>, % group ID <str>, % account name )=<int> % balance in USD -

    Kinds of Queries

    +

    Basic Queries

    FQL queries can write/clear a single key-value, read one or more key-values, or list directories. Throughout this section, snippets of Go code are included to show how the queries interact with the FDB API.

    Mutations

    -

    Queries lacking both variables and the +

    Queries lacking variables and the ... token perform mutations on the database by either - writing a key-value or clearing an existing one.

    + writing or clearing a key-value.

    Queries lacking a value altogether imply an empty variable as the value and should not be confused @@ -314,11 +322,20 @@

    Mutations

    tr.Clear(dir.Pack(tuple.Tuple{"hello", "world"})) return nil, nil }) -

    Single Reads

    -

    If the query has variables or the - ... token in its value (but not in its key) then it reads - a single key-value, if the key-value exists.

    - 
    /my/dir(99.8, 7dfb10d1-2493-4fb5-928e-889fdc6a7136)=<int|str>
    +

    Reads

    +

    TODO: Make sure all reads use ReadTransact().

    +

    Queries containing a variable or the + ... token read one or more key-values. The query defines + a schema which the returned key-values must conform to.

    +

    If the variable or ... token only appears in the + query’s value, then it returns a single key-value, if one matching the + schema exists.

    +
    +

    Queries lacking a value altogether imply an empty variable as the value, and are therefore read + queries.

    +
    + 
    /my/dir(99.8,7dfb10d1-2493-4fb5-928e-889fdc6a7136)=<int|str>
    db.Transact(func(tr fdb.Transaction) (interface{}, error) {
   dir, err := directory.Open(tr, []string{"my", "dir"}, nil)
   if err != nil {
@@ -362,11 +379,10 @@ 
    Single Reads
    
   // No value decoding...
   return tr.MustGet(dir.Pack(tuple.Tuple{10139})), nil
 })
    -

    Range Reads

    Queries with variables or the ... token in their key (and optionally in their value) result in a range of key-values being read.

    - 
    /people(3392,<str|int>,<>)=(<uint>,...)
    + 
    /people("coders",...)
    db.ReadTransact(func(tr fdb.ReadTransaction) (interface{}, error) {
   dir, err := directory.Open(tr, []string{"people"}, nil)
   if err != nil {
@@ -376,6 +392,54 @@ 
    Range Reads
    
     return nil, err
   }
 
+  rng, err := fdb.PrefixRange(dir.Pack(tuple.Tuple{"coders"}))
+  if err != nil {
+    return nil, err
+  }
+
+  var results []fdb.KeyValue
+  iter := tr.GetRange(rng, fdb.RangeOptions{}).Iterator()
+  for iter.Advance() {
+    kv := iter.MustGet()
+
+    tup, err := dir.Unpack(kv.Key)
+    if err != nil {
+      return nil, err
+    }
+
+    results = append(results, kv)
+  }
+  return results, nil
+})
    +

    Filtering

    +

    Read queries define a schema to which key-values may or may-not + conform. In the Go snippets above, non-conformant key-values were + being filtered out of the results.

    +

    Alternatively, FQL can throw an error when encountering + non-conformant key-values. This may help enforce the assumption that + all key-values within a directory conform to a certain schema.

    +

    TODO: Link to FQL options.

    +

    Because filtering is performed on the client side, range reads may + stream a lot of data to the client while the client filters most of it + away. For example, consider the following query:

    + 
    /people(3392,<str|int>,<>)=(<uint>,...)
    +

    In the key, the location of the first variable or ... + token determines the range read prefix used by FQL. For this + particular query, the prefix would be as follows:

    + 
    /people(3392)
    +

    Foundation DB will stream all key-values with this prefix to the + client. As they are received, the client will filter out key-values + which don’t match the query’s schema. Below you can see a Go + implementation of how this filtering would work.

    + 
    db.ReadTransact(func(tr fdb.ReadTransaction) (interface{}, error) {
+  dir, err := directory.Open(tr, []string{"people"}, nil)
+  if err != nil {
+    if errors.Is(err, directory.ErrDirNotExists) {
+      return nil, nil
+    }
+    return nil, err
+  }
+
   rng, err := fdb.PrefixRange(dir.Pack(tuple.Tuple{3392}))
   if err != nil {
     return nil, err
@@ -428,25 +492,20 @@ 
    Range Reads
    
   }
   return results, nil
 })
    -

    The actual implementation pipelines the reading, filtering, and - value decoding across multiple threads.

    -

    Filtering

    -

    Read queries define a schema to which key-values may or may-not - conform. In the Go snippets above, non-conformant key-values were - being filtered out of the results.

    -
    -

    Filtering is performed on the client-side and may result in lots of - data being transferred to the client machine.

    -
    -

    Alternatively, FQL can throw an error when encountering - non-conformant key-values. This helps enforce the assumption that all - key-values within a directory conform to a certain schema.

    -

    Indirection

    +

    Advanced Queries

    +

    Besides basic CRUD + operations, FQL is capable of performing indirection and aggregation + queries.

    +

    Indirection

    +

    Indirection queries are similar to SQL joins. They associate + different groups of key-values via some shared data element.

    In Foundation DB, indexes are implemented by having one key-value (the index) point at another key-value. This is also called “indirection”.

    -

    Indirection is not yet implemented.

    +

    Indirection is not yet included in the grammar, nor is it + implemented. The design of this feature is somewhat finalized.

    Suppose we have a large list of people, one key-value for each person.

    @@ -471,7 +530,7 @@

    Indirection

    /index/last_name("Johnson",23)=nil
 /index/last_name("Johnson",348)=nil
 /index/last_name("Johnson",2003)=nil
    -

    Aggregation

    +

    Aggregation

    The design of aggregation queries is not complete. This section describes the general idea. Exact syntax may change. This feature is @@ -512,11 +571,47 @@

    Aggregation

    /deltas("group A",3)=nil 
    /deltas("group A",<sum>)
    /deltas("group A",5)=<>
    -

    Transactions

    -

    TODO: Finish section.

    -

    Design Recipes

    -

    TODO: Finish section.

    -

    As a Layer

    +

    Using FQL

    +

    FQL can be used for exploring a Foundation DB cluster in a CLI + environment or programmatically as a Foundation DB layer.

    +

    CLI

    +

    Headless

    +

    FQL provides a CLI for performing queries from the command line. To + execute a query in “headless” mode (without fullscreen), you can use + the -q flag.

    + 
    ᐅ fql -q '/my/dir("hello","world")'
+/my/dir("hello","world")=nil
    +

    When using BASH (or a BASH-like shell), The queries must be wrapped + in single quotes to avoid mangling.

    +

    The -q flag may be provided multiple times. When + invoking FQL in this manner, all queries are performed in the same + transaction.

    + 
    ᐅ fql -q '/my/dir("hello",<var:str>)' -q '/other(22,...)'
+/my/dir("hello","world")=nil
+/other(22,"1")=0xa8
+/other(22,"2")=0xf3
    +

    Fullscreen

    +

    If the CLI is executed without the -q flag, a + fullscreen environment is started up. In this case, the connection to + Foundation DB is maintained for the lifetime of the application. + Single queries may be executed in their own transactions and the + results are displayed in a scrollable list.

    +

    +

    Currently, this environment is not very useful, but it lays the + groundwork for a fully-featured FQL frontend (accidental + alliteration). The final version of this environment will include the + following features:

    +
      +
    • Syntax highlighting
      • +
    • Context-aware autocompletion
      • +
    • Querying of data cached on the client
      • +
    • Importing & exporting subspaces to disk
      • +
    • Customizable formatting of key-values
      • +
    • Restoring a session after restart
      • +
    +

    API (Layer)

    +

    TODO: Review this section.

    When integrating SQL into other languages, there are usually two choices each with their own drawbacks:

      @@ -568,6 +663,7 @@

      As a Layer

      panic(err) } } +

      Project Roadmap