diff --git a/docs/img/demo.gif b/docs/img/demo.gif new file mode 100644 index 00000000..dda4aff8 Binary files /dev/null and b/docs/img/demo.gif differ diff --git a/docs/index.html b/docs/index.html index de66af34..6b494947 100644 --- a/docs/index.html +++ b/docs/index.html @@ -16,9 +16,9 @@

FQL

-
/user/index/surname("Johnson",<userID:int>)
+    
/user/index/surname("Johnson",<userID:int>)
 /user(:userID,...)
-
/user(9323,"Timothy","Johnson",37)=nil
+    
/user(9323,"Timothy","Johnson",37)=nil
 /user(24335,"Andrew","Johnson",42)=nil
 /user(33423,"Ryan","Johnson",0x0ffa83,42.2)=nil

FQL is an open @@ -54,11 +54,8 @@

FQL

  • Using FQL
  • Roadmap
  • @@ -68,12 +65,12 @@

    Overview

    href="https://github.com/janderland/fql/blob/main/syntax.ebnf">context-free grammar. The queries look like key-values encoded using the directory & tuple layers.

    -
    /my/directory("my","tuple")=4000
    +
    /my/directory("my","tuple")=4000

    FQL queries may define a single key-value to be written, as shown above, or may define a set of key-values to be read, as shown below.

    -
    /my/directory("my","tuple")=<int>
    -
    /my/directory("my","tuple")=4000
    +
    /my/directory("my","tuple")=<int>
    +
    /my/directory("my","tuple")=4000

    The query above has a variable <int> as its value. Variables act as placeholders for any of the supported data elements. In this case, the variable @@ -82,33 +79,33 @@

    Overview

    FQL queries can also perform range reads & filtering by including a variable in the key’s tuple. The query below will return all key-values which conform to the schema defined by the query.

    -
    /my/directory(<>,"tuple")=nil
    -
    /my/directory("your","tuple")=nil
    +  
    /my/directory(<>,"tuple")=nil
    +
    /my/directory("your","tuple")=nil
     /my/directory(42,"tuple")=nil

    All key-values with a certain key prefix can be range read by ending the key’s tuple with ....

    -
    /my/directory("my","tuple",...)=<>
    -
    /my/directory("my","tuple")=0x0fa0
    +  
    /my/directory("my","tuple",...)=<>
    +
    /my/directory("my","tuple")=0x0fa0
     /my/directory("my","tuple",47.3)=0x8f3a
     /my/directory("my","tuple",false,0xff9a853c12)=nil

    A query’s value may be omitted to imply a variable, meaning the following query is semantically identical to the one above.

    -
    /my/directory("my","tuple",...)
    -
    /my/directory("my","tuple")=0x0fa0
    +  
    /my/directory("my","tuple",...)
    +
    /my/directory("my","tuple")=0x0fa0
     /my/directory("my","tuple",47.3)=0x8f3a
     /my/directory("my","tuple",false,0xff9a853c12)=nil

    Including a variable in the directory tells FQL to perform the read on all directory paths matching the schema.

    -
    /<>/directory("my","tuple")
    -
    /my/directory("my","tuple")=0x0fa0
    +  
    /<>/directory("my","tuple")
    +
    /my/directory("my","tuple")=0x0fa0
     /your/directory("my","tuple")=nil

    Key-values can be cleared by using the special clear token as the value.

    -
    /my/directory("my","tuple")=clear
    +
    /my/directory("my","tuple")=clear

    The directory layer can be queried by only including a directory path.

    -
    /my/<>
    -
    /my/directory
    +
    /my/<>
    +
    /my/directory

    Data Elements

    An FQL query contains instances of data elements. These are the same types of elements found in the Data Elements

    bint support is not yet implemented.

    Tuples & values may contain any of the data elements.

    -
    /region/north_america(22.3,-8)=("rain","fog")
    +  
    /region/north_america(22.3,-8)=("rain","fog")
     /region/east_asia("japan",nil)=0xff

    Strings are the only data element allowed in directories. If a directory string only contains alphanumericals, underscores, dashes, and periods then the quotes don’t need to be included.

    -
    /quoteless-string_in.dir(true)=false
    +  
    /quoteless-string_in.dir(true)=false
     /"other ch@r@cters must be quoted!"(20)=32.3

    Quoted strings may contain quotes via backslash escapes.

    -
    /my/dir("I said \"hello\"")=nil
    +
    /my/dir("I said \"hello\"")=nil

    Value Encoding

    The directory and tuple layers are responsible for encoding the data elements in the key. As for the value, FDB doesn’t provide a @@ -245,33 +242,33 @@

    Variables & Schemas

    href="#data-elements">data element
    may be represented with a variable. Variables are specified as a list of element types, separated by |, wrapped in angled braces.

    -
    <uint|str|uuid|bytes>
    +
    <uint|str|uuid|bytes>

    The variable’s type list describes which data elements are allowed at the variable’s position. A variable may be empty, including no element types, meaning it represents all element types.

    -
    /user(<int>,<str>,<>)=<>
    -
    /user(0,"jon",0xffab0c)=nil
    +  
    /user(<int>,<str>,<>)=<>
    +
    /user(0,"jon",0xffab0c)=nil
     /user(20,"roger",22.3)=0xff
     /user(21,"",nil)="nothing"

    Before the type list, a variable can be given a name. This name is used to reference the variable in subsequent queries, allowing for index indirection.

    -
    /index("cars",<varName:int>)
    +  
    /index("cars",<varName:int>)
     /data(:varName,...)
    -
    /user(33,"mazda")=nil
    +  
    /user(33,"mazda")=nil
     /user(320,"ford")=nil
     /user(411,"chevy")=nil

    Space & Comments

    Whitespace and newlines are allowed within a tuple, between its elements.

    -
    /account/private(
    +  
    /account/private(
       <uint>,
       <uint>,
       <str>,
     )=<int>

    Comments start with a % and continue until the end of the line. They can be used to describe a tuple’s elements.

    -
    % private account balances
    +  
    % private account balances
     /account/private(
       <uint>, % user ID
       <uint>, % group ID
    @@ -293,7 +290,7 @@ 

    Mutations

    Mutation queries with a data element as their value perform a write operation.

    -
    /my/dir("hello","world")=42
    +
    /my/dir("hello","world")=42
    db.Transact(func(tr fdb.Transaction) (interface{}, error) {
       dir, err := directory.CreateOrOpen(tr, []string{"my", "dir"}, nil)
       if err != nil {
    @@ -309,7 +306,7 @@ 

    Mutations

    })

    Mutation queries with the clear token as their value perform a clear operation.

    -
    /my/dir("hello","world")=clear
    +
    /my/dir("hello","world")=clear
    db.Transact(func(tr fdb.Transaction) (interface{}, error) {
       dir, err := directory.Open(tr, []string{"my", "dir"}, nil)
       if err != nil {
    @@ -334,7 +331,7 @@ 

    Reads

    href="#variables">variable as the value, and are therefore read queries.

    -
    /my/dir(99.8,7dfb10d1-2493-4fb5-928e-889fdc6a7136)=<int|str>
    +
    /my/dir(99.8,7dfb10d1-2493-4fb5-928e-889fdc6a7136)=<int|str>
    db.ReadTransact(func(tr fdb.ReadTransaction) (interface{}, error) {
       dir, err := directory.Open(tr, []string{"my", "dir"}, nil)
       if err != nil {
    @@ -365,7 +362,7 @@ 

    Reads

    the key-value does not match the schema.

    If the value is specified as an empty variable, then the raw bytes are returned.

    -
    /some/data(10139)=<>
    +
    /some/data(10139)=<>
    db.ReadTransact(func(tr fdb.ReadTransaction) (interface{}, error) {
       dir, err := directory.Open(tr, []string{"some", "data"}, nil)
       if err != nil {
    @@ -381,7 +378,7 @@ 

    Reads

    Queries with variables or the ... token in their key (and optionally in their value) result in a range of key-values being read.

    -
    /people("coders",...)
    +
    /people("coders",...)
    db.ReadTransact(func(tr fdb.ReadTransaction) (interface{}, error) {
       dir, err := directory.Open(tr, []string{"people"}, nil)
       if err != nil {
    @@ -415,7 +412,7 @@ 

    Directories

    directory as a query. These queries can only perform reads. If the directory path contains no variables, the query will read that single directory.

    -
    /root/<>/items
    +
    /root/<>/items
     root, err := directory.Open(tr, []string{"root"}, nil)
       if err != nil {
         if errors.Is(err, directory.ErrDirNotExists) {
    @@ -452,11 +449,11 @@ 

    Filtering

    Because filtering is performed on the client side, range reads may stream a lot of data to the client while the client filters most of it away. For example, consider the following query:

    -
    /people(3392,<str|int>,<>)=(<uint>,...)
    +
    /people(3392,<str|int>,<>)=(<uint>,...)

    In the key, the location of the first variable or ... token determines the range read prefix used by FQL. For this particular query, the prefix would be as follows:

    -
    /people(3392)
    +
    /people(3392)

    Foundation DB will stream all key-values with this prefix to the client. As they are received, the client will filter out key-values which don’t match the query’s schema. Below you can see a Go @@ -539,25 +536,25 @@

    Indirection

    Suppose we have a large list of people, one key-value for each person.

    -
    /people(<id:uint>,<firstName:str>,<lastName:str>,<age:int>)=nil
    +
    /people(<id:uint>,<firstName:str>,<lastName:str>,<age:int>)=nil

    If we wanted to read all records with the last name of “Johnson”, we’d have to perform a linear search across the entire “people” directory. To make this kind of search more efficient, we can store an index of last names in a separate directory.

    -
    /index/last_name(<lastName:str>,<id:uint>)=nil
    +
    /index/last_name(<lastName:str>,<id:uint>)=nil

    FQL can forward the observed values of named variables from one query to the next, allowing us to efficiently query for all people with the last name of “Johnson”.

    -
    /index/last_name("Johnson",<id:uint>)
    +  
    /index/last_name("Johnson",<id:uint>)
     /people(:id,...)
    -
    /people(23,"Lenny","Johnson",22,"Mechanic")=nil
    +  
    /people(23,"Lenny","Johnson",22,"Mechanic")=nil
     /people(348,"Roger","Johnson",54,"Engineer")=nil
     /people(2003,"Larry","Johnson",8,"N/A")=nil

    The first query returned 3 key-values containing the IDs of 23, 348, & 2003 which were then fed into the second query resulting in 3 individual single reads.

    -
    /index/last_name("Johnson",<id:uint>)
    -
    /index/last_name("Johnson",23)=nil
    +  
    /index/last_name("Johnson",<id:uint>)
    +
    /index/last_name("Johnson",23)=nil
     /index/last_name("Johnson",348)=nil
     /index/last_name("Johnson",2003)=nil

    Aggregation

    @@ -570,11 +567,11 @@

    Aggregation

    href="https://apple.github.io/foundationdb/blob.html">storing large blobs, the data is usually split into 10 kB chunks stored in the value. The respective key contain the byte offset of the chunk.

    -
    /blob(
    +  
    /blob(
       "my file",    % The identifier of the blob.
       <offset:int>, % The byte offset within the blob.
     )=<chunk:bytes> % A chunk of the blob.
    -
    /blob("my file",0)=10e3_bytes
    +  
    /blob("my file",0)=10e3_bytes
     /blob("my file",10000)=10e3_bytes
     /blob("my file",20000)=2.7e3_bytes
    @@ -587,25 +584,26 @@

    Aggregation

    themselves. This can be done using aggregation queries.

    FQL provides a pseudo data type named agg which performs the aggregation.

    -
    /blob("my file",...)=<blob:agg>
    -
    /blob("my file",...)=22.7e3_bytes
    +
    /blob("my file",...)=<blob:agg>
    +
    /blob("my file",...)=22.7e3_bytes

    Aggregation queries always result in a single key-value. With non-aggregation queries, variables & the ... token are resolved as actual data elements in the query results. For aggregation queries, only aggregation variables are resolved.

    A similar pseudo data type for summing integers could be provided as well.

    -
    /deltas("group A",<int>)
    -
    /deltas("group A",20)=nil
    +  
    /deltas("group A",<int>)
    +
    /deltas("group A",20)=nil
     /deltas("group A",-18)=nil
     /deltas("group A",3)=nil
    -
    /deltas("group A",<sum>)
    -
    /deltas("group A",5)=<>
    +
    /deltas("group A",<sum>)
    +
    /deltas("group A",5)=<>

    Using FQL

    FQL can be used for exploring a Foundation DB cluster in a CLI environment or programmatically as a Foundation DB layer.

    Command Line

    +

    Headless

    FQL provides a CLI for performing queries from the command line. To execute a query in “headless” mode (without fullscreen), you can use @@ -627,7 +625,7 @@

    Fullscreen

    Foundation DB is maintained for the lifetime of the application. Single queries may be executed in their own transactions and the results are displayed in a scrollable list.

    -

    +

    Currently, this environment is not very useful, but it lays the groundwork for a fully-featured FQL frontend (accidental alliteration). The final version of this environment will include the @@ -640,6 +638,7 @@

    Fullscreen

  • Customizable formatting of key-values
  • Restoring a session after restart
  • +

    API (Layer)

    TODO: Review this section.

    When integrating SQL into other languages, there are usually two @@ -726,7 +725,16 @@

    Roadmap

    diff --git a/docs/index.md b/docs/index.md index 891670e4..af2fb9b9 100644 --- a/docs/index.md +++ b/docs/index.md @@ -4,11 +4,11 @@ title: FQL # We include this intro via the 'include-before' # metadata field so it's placed before the TOC. include-before: | - ```lang-fql {.query} + ```language-fql {.query} /user/index/surname("Johnson",) /user(:userID,...) ``` - ```lang-fql {.result} + ```language-fql {.result} /user(9323,"Timothy","Johnson",37)=nil /user(24335,"Andrew","Johnson",42)=nil /user(33423,"Ryan","Johnson",0x0ffa83,42.2)=nil @@ -29,7 +29,7 @@ grammar](https://github.com/janderland/fql/blob/main/syntax.ebnf). The queries look like key-values encoded using the directory & tuple layers. -```lang-fql {.query} +```language-fql {.query} /my/directory("my","tuple")=4000 ``` @@ -37,11 +37,11 @@ FQL queries may define a single key-value to be written, as shown above, or may define a set of key-values to be read, as shown below. -```lang-fql {.query} +```language-fql {.query} /my/directory("my","tuple")= ``` -```lang-fql {.result} +```language-fql {.result} /my/directory("my","tuple")=4000 ``` @@ -57,11 +57,11 @@ including a variable in the key's tuple. The query below will return all key-values which conform to the schema defined by the query. -```lang-fql {.query} +```language-fql {.query} /my/directory(<>,"tuple")=nil ``` -```lang-fql {.result} +```language-fql {.result} /my/directory("your","tuple")=nil /my/directory(42,"tuple")=nil ``` @@ -69,11 +69,11 @@ defined by the query. All key-values with a certain key prefix can be range read by ending the key's tuple with `...`. -```lang-fql {.query} +```language-fql {.query} /my/directory("my","tuple",...)=<> ``` -```lang-fql {.result} +```language-fql {.result} /my/directory("my","tuple")=0x0fa0 /my/directory("my","tuple",47.3)=0x8f3a /my/directory("my","tuple",false,0xff9a853c12)=nil @@ -83,11 +83,11 @@ A query's value may be omitted to imply a variable, meaning the following query is semantically identical to the one above. -```lang-fql {.query} +```language-fql {.query} /my/directory("my","tuple",...) ``` -```lang-fql {.result} +```language-fql {.result} /my/directory("my","tuple")=0x0fa0 /my/directory("my","tuple",47.3)=0x8f3a /my/directory("my","tuple",false,0xff9a853c12)=nil @@ -96,11 +96,11 @@ above. Including a variable in the directory tells FQL to perform the read on all directory paths matching the schema. -```lang-fql {.query} +```language-fql {.query} /<>/directory("my","tuple") ``` -```lang-fql {.result} +```language-fql {.result} /my/directory("my","tuple")=0x0fa0 /your/directory("my","tuple")=nil ``` @@ -108,18 +108,18 @@ the read on all directory paths matching the schema. Key-values can be cleared by using the special `clear` token as the value. -```lang-fql {.query} +```language-fql {.query} /my/directory("my","tuple")=clear ``` The directory layer can be queried by only including a directory path. -```lang-fql {.query} +```language-fql {.query} /my/<> ``` -```lang-fql {.result} +```language-fql {.result} /my/directory ``` @@ -151,7 +151,7 @@ Descriptions of these elements can be seen below. Tuples & values may contain any of the data elements. -```lang-fql {.query} +```language-fql {.query} /region/north_america(22.3,-8)=("rain","fog") /region/east_asia("japan",nil)=0xff ``` @@ -161,14 +161,14 @@ a directory string only contains alphanumericals, underscores, dashes, and periods then the quotes don't need to be included. -```lang-fql {.query} +```language-fql {.query} /quoteless-string_in.dir(true)=false /"other ch@r@cters must be quoted!"(20)=32.3 ``` Quoted strings may contain quotes via backslash escapes. -```lang-fql {.query} +```language-fql {.query} /my/dir("I said \"hello\"")=nil ``` @@ -205,7 +205,7 @@ element](#data-elements) may be represented with a variable. Variables are specified as a list of element types, separated by `|`, wrapped in angled braces. -```lang-fql +```language-fql ``` @@ -214,11 +214,11 @@ allowed at the variable's position. A variable may be empty, including no element types, meaning it represents all element types. -```lang-fql {.query} +```language-fql {.query} /user(,,<>)=<> ``` -```lang-fql {.result} +```language-fql {.result} /user(0,"jon",0xffab0c)=nil /user(20,"roger",22.3)=0xff /user(21,"",nil)="nothing" @@ -229,12 +229,12 @@ name is used to reference the variable in subsequent queries, allowing for [index indirection](#index-indirection). -```lang-fql {.query} +```language-fql {.query} /index("cars",) /data(:varName,...) ``` -```lang-fql {.result} +```language-fql {.result} /user(33,"mazda")=nil /user(320,"ford")=nil /user(411,"chevy")=nil @@ -245,7 +245,7 @@ indirection](#index-indirection). Whitespace and newlines are allowed within a tuple, between its elements. -```lang-fql {.query} +```language-fql {.query} /account/private( , , @@ -256,7 +256,7 @@ its elements. Comments start with a `%` and continue until the end of the line. They can be used to describe a tuple's elements. -```lang-fql +```language-fql % private account balances /account/private( , % user ID @@ -285,7 +285,7 @@ clearing a key-value. Mutation queries with a [data element](#data-elements) as their value perform a write operation. -```lang-fql {.query} +```language-fql {.query} /my/dir("hello","world")=42 ``` @@ -308,7 +308,7 @@ db.Transact(func(tr fdb.Transaction) (interface{}, error) { Mutation queries with the `clear` token as their value perform a clear operation. -```lang-fql {.query} +```language-fql {.query} /my/dir("hello","world")=clear ``` @@ -341,7 +341,7 @@ the schema exists. > [variable](#variables) as the value, and are therefore > read queries. -```lang-fql {.query} +```language-fql {.query} /my/dir(99.8,7dfb10d1-2493-4fb5-928e-889fdc6a7136)= ``` @@ -380,7 +380,7 @@ cannot be decoded, the key-value does not match the schema. If the value is specified as an empty variable, then the raw bytes are returned. -```lang-fql {.query} +```language-fql {.query} /some/data(10139)=<> ``` @@ -403,7 +403,7 @@ Queries with [variables](#variables) or the `...` token in their key (and optionally in their value) result in a range of key-values being read. -```lang-fql {.query} +```language-fql {.query} /people("coders",...) ``` @@ -445,7 +445,7 @@ a lone directory as a query. These queries can only perform reads. If the directory path contains no variables, the query will read that single directory. -```lang-fql {.query} +```language-fql {.query} /root/<>/items ``` @@ -495,7 +495,7 @@ reads may stream a lot of data to the client while the client filters most of it away. For example, consider the following query: -```lang-fql {.query} +```language-fql {.query} /people(3392,,<>)=(,...) ``` @@ -503,7 +503,7 @@ In the key, the location of the first variable or `...` token determines the range read prefix used by FQL. For this particular query, the prefix would be as follows: -```lang-fql {.query} +```language-fql {.query} /people(3392) ``` @@ -600,7 +600,7 @@ also called "indirection". Suppose we have a large list of people, one key-value for each person. -```lang-fql {.query} +```language-fql {.query} /people(,,,)=nil ``` @@ -610,7 +610,7 @@ entire "people" directory. To make this kind of search more efficient, we can store an index of last names in a separate directory. -```lang-fql {.query} +```language-fql {.query} /index/last_name(,)=nil ``` @@ -618,11 +618,11 @@ FQL can forward the observed values of named variables from one query to the next, allowing us to efficiently query for all people with the last name of "Johnson". -```lang-fql {.query} +```language-fql {.query} /index/last_name("Johnson",) /people(:id,...) ``` -```lang-fql {.result} +```language-fql {.result} /people(23,"Lenny","Johnson",22,"Mechanic")=nil /people(348,"Roger","Johnson",54,"Engineer")=nil /people(2003,"Larry","Johnson",8,"N/A")=nil @@ -632,10 +632,10 @@ The first query returned 3 key-values containing the IDs of 23, 348, & 2003 which were then fed into the second query resulting in 3 individual [single reads](#single-reads). -```lang-fql {.query} +```language-fql {.query} /index/last_name("Johnson",) ``` -```lang-fql {.result} +```language-fql {.result} /index/last_name("Johnson",23)=nil /index/last_name("Johnson",348)=nil /index/last_name("Johnson",2003)=nil @@ -654,14 +654,14 @@ blobs](https://apple.github.io/foundationdb/blob.html), the data is usually split into 10 kB chunks stored in the value. The respective key contain the byte offset of the chunk. -```lang-fql {.query} +```language-fql {.query} /blob( "my file", % The identifier of the blob. , % The byte offset within the blob. )= % A chunk of the blob. ``` -```lang-fql {.result} +```language-fql {.result} /blob("my file",0)=10e3_bytes /blob("my file",10000)=10e3_bytes /blob("my file",20000)=2.7e3_bytes @@ -679,11 +679,11 @@ queries. FQL provides a pseudo data type named `agg` which performs the aggregation. -```lang-fql {.query} +```language-fql {.query} /blob("my file",...)= ``` -```lang-fql {.result} +```language-fql {.result} /blob("my file",...)=22.7e3_bytes ``` @@ -696,21 +696,21 @@ resolved. A similar pseudo data type for summing integers could be provided as well. -```lang-fql {.query} +```language-fql {.query} /deltas("group A",) ``` -```lang-fql {.result} +```language-fql {.result} /deltas("group A",20)=nil /deltas("group A",-18)=nil /deltas("group A",3)=nil ``` -```lang-fql {.query} +```language-fql {.query} /deltas("group A",) ``` -```lang-fql {.result} +```language-fql {.result} /deltas("group A",5)=<> ``` @@ -722,6 +722,8 @@ a CLI environment or programmatically as a Foundation DB ## Command Line +
    + ### Headless FQL provides a CLI for performing queries from the command @@ -756,7 +758,7 @@ application. Single queries may be executed in their own transactions and the results are displayed in a scrollable list. -![](../vhs/demo.gif) +![](img/demo.gif) Currently, this environment is not very useful, but it lays the groundwork for a fully-featured FQL frontend (accidental @@ -770,6 +772,8 @@ include the following features: - Customizable formatting of key-values - Restoring a session after restart +
    + ## API (Layer) TODO: Review this section. diff --git a/docs/index.tmpl b/docs/index.tmpl index e9ef3150..416944c5 100644 --- a/docs/index.tmpl +++ b/docs/index.tmpl @@ -24,7 +24,16 @@ diff --git a/docs/js/fql.js b/docs/js/fql.js index f6b2ad73..f6df45ad 100644 --- a/docs/js/fql.js +++ b/docs/js/fql.js @@ -195,7 +195,7 @@ NUMBER, { // Highlight lone bar for inline text. scope: 'variable', - begin: /|/, + begin: /\|/, }, ], }));