require.q

Require - Code Loading Library

This library is inspired by RequireJS and other import systems within JavaScript applications. It allows the loading of libraries into a kdb+ process and manages dependencies between them to aid separation of code into small, reusable components.

What is a Library?

We define a library to be a collection of q-script files that are related by a common prefix. In the simplest case, a library can be a single q-script with the name equalling the library name (e.g. log.q provides the log library). We also assume that the library name matches the namespace used for that library (e.g. log library uses .log namespace).

To allow code within a single library to also be split into separate files, files with the same common prefix are assumed to be the same library. The suffixes used to determine this are: .q, .k, .*.q, .*.k, .q_, .*.q_. Note that with the prefixes .*.q, .*.k and .*.q_, log.q and, for example, log.extra.q would be classified as the same library and would both be loaded as part of the log library.

Initialising Require

To start using Require within your application, you must load and initialise the library manually:

q) system "l /path/to/kdb-common/src/require.q"

The library must then be initialised. There are a number of options, depending on the customisation required:

• .require.init[]: Uses the current folder as the root to discover files
• .require.init *path*: Uses the specified path as the root folder to discover files
• Setting .require.location.discovered: Manually define list of files to use with require

The scanning of files is only done on initialisation, any added or removed files will not be reflected in this list. You should use .require.rescanRoot to update the list before attempting to load a newly defined library.

Current Folder as Root

By calling .require.init with no arguments, it will take the current working directory (equivalent of pwd) as the root folder and scan for all files underneath this folder.

> cd /home/jas/git/kdb-common
> rlwrap $QHOME/l64/q src/require.q

q) .require.init[]
Library root location refreshed [ File Count: 51 ]
q) .require.location.root
`:/home/jas/git/kdb-common

Specific Folder as Root

By calling .require.init with a single folder-path argument, it will use the specified folder as the root folder and scan for all files underneath this folder.

NOTE: It is recommended to pass an absoluate path to .require.init to allow it to continue working even if the process working directory is later changed (e.g. system "l").

> rlwrap $QHOME/l64/q /home/jas/git/kdb-common/src/require.q

/ Equivalent to the "current folder as root" example
q).require.init `$":/home/jas/git/kdb-common"
Library root location refreshed [ File Count: 51 ]
q) .require.location.root
`:/home/jas/git/kdb-common

Custom File List

In the case where there are files from multiple distinct locations on the filesystem that should be exposed to 'require', override the default .require.rescanRoot function to build the .require.location.discovered list of files manually.

Notes:

• .require.i.tree can be used to descend within a list of folders.
• .require.location.root will be set to the current working directory regardless of the list of files present, so should be ignored

> rlwrap $QHOME/l64/q /home/jas/git/kdb-common/src/require.q

/ Need kdb-common and kdb-systemd here
.require.rescanRoot:{ .require.location.discovered:raze .require.i.tree each `$(":/home/jas/git/kdb-common"; ":/home/jas/git/kdb-systemd"); };
.require.init[];

Loading and Initialising Libraries

Once Require is loaded and initialised, you can then use .require.lib to load any other library. This function will search for a library with the specified name and load all matching files into the current process. If there is a function *lib*.init, it will be executed with a single dictionary argument after all the files are loaded.

If you need to perform some other set up or configuration after the files are loaded but before the initialisation function is run, you should use .require.libNoInit to load the library, run you custom code and then run .require.lib to initialise it.

Initialisation Function Argument

If the function *lib*.init exists, it is called with a single dictionary argument. The dictionary contains the following keys:

• reinit: Boolean set to false on first initialisation (via .require.lib or .require.libForce) or true if re-initialised via .require.libForce

Defining Dependencies

.require.lib can also be used to define dependencies between libraries that you write. Use it at the top of your library.

For example, if you need to use .type.isInteger from the type library in some new code, you can do the following:

// your-lib.q

/ Loads the "type" library (if not already loaded)
.require.lib `type;

.lib.intSum:{[x;y]
    if[(not .type.isInteger x) | not .type.isInteger y];
        '"IllegalArgumentException";
    ];

    :x + y;
}

Currently Loaded Libraries

The Require library keeps track of all libraries that are loaded (and initialised). This is used internally to know which libraries to load from disk but can also be used to manually see which libraries are within the process.

The table .require.loadedLibs provides this detail:

q) .require.loadedLibs
q).require.loadedLibs
lib    | loaded loadedTime                    initExists inited initedTime                    forced files
-------| ------------------------------------------------------------------------------------------------------------------------------
require| 1      2024.04.17D21:54:46.715152900 1          1      2024.04.17D21:54:46.715163600 0      ()
type   | 1      2024.04.17D21:54:46.725174900 1          1      2024.04.17D21:54:46.725710400 0      ,`:./src/type.q
if     | 1      2024.04.17D21:54:46.726011100 1          1      2024.04.17D21:54:46.726160900 0      ,`:./src/if.q
...
cron   | 1      2024.04.17D21:54:46.777817700 1          1      2024.04.17D21:54:46.778018800 0      ,`:./src/cron.q

Forced Reload of Libraries

There is also the option to force a reload and re-initialisation of a library via .require.libForce. This can be useful in situations where a library has been modified and can be safely reloaded within a process without having to restart it.

The require library explicitly logs if a forced reload is performed on an already loaded library and updates the forced column in .require.loadedLibs to true.

Notes:

1. Any dependent libraries are not reloaded
2. If the library is not yet loaded, .require.libForce behaves identically to .require.lib
3. If the library is loaded but not yet initialised, .require.libForce will call the *lib*.init function with force set to false

/ Load 'ns' normally
q) .require.lib`ns;
...

/ Now force a reload
q) .require.libForce`ns;
2023.10.31 17:23:53.848 INFO  pid-1875 jas 0 Force reloading library [ Library: ns ] [ Already Loaded: yes ] [ Already Initialised: yes ]
...

q) .require.loadedLibs[`ns]`forced
1b

Exceptions

The following exceptions can be thrown when using the Require library:

• LibraryDoesNotExistException : If there are no files that match the library requested
• LibraryLoadException : If any of the matched files fail to load into the process
  • A back trace will be printed if running kdb+ 3.5 or later
• LibraryInitFailedException : If a initialisation function exists for the library and it does not execute successfully
  • A back trace will be printed if running kdb+ 3.5 or later
• UnknownLibraryException : If you attempt to initialise a library that has not been loaded by Require
• RequireReinitialiseAssertionError: If a library re-initialisation is attempted, but the re-initialise parameter of .require.i.init is false

Default Logging

The Require library provides very basic logging for the .log.if interfaces that all libraries within kdb-common use to print information.

These are defined by .require.i.log (prints to standard out) and .require.i.logE (prints to standard error).

These logging functions also support slf4j-style parameterised logging (as does the full logging library) which can be accessed for other use via .require.i.parameterisedLog.