JGNN
+JGNN
Graph Neural Networks (GNNs) are getting more and more popular, for example to make predictions based on relational information, or to perform inference on small datasets. JGNN provides cross-platform implementations of this machine learning paradigm that do not require dedicated hardware or firmware. While reading this guidebook, keep in mind that this is not a library for running computationally intensive architectures; it has no GPU support and does not - plan to add any (unless the latter becomes integrated in the base Java virtual + plan to add any (unless such support becomes integrated in the Java virtual machine). Instead, the goal is to provide highly portable solutions that can run under smaller compute and available memory. So, while complex architectures like gated attention networks with many layers and hidden dimensions @@ -179,107 +221,133 @@
1. Setup
2. Quickstart
-Here we demonstrate JGNN on a node classification - task. Graph classification is also supported, but we choose the simpler - concept for a first taste. For this task, GNN architectures are typically written - as message-passing mechanisms that diffuse node representations across graphs, - and our demonstration focuses on this scenario. In particular, +
Here we demonstrate usage of JGNN for node classification, that is, the inductive learning + task of inferring node labels given a graph's structure, node features, and a few node labels. + Classifying graphs is also supported, but it is a harder task to explain and set up. + GNN architectures for the chosen node classification task are typically written + as message-passing mechanisms; these diffuse node representations across edges, where node neighbors pick up, aggregate (e.g., average), and transform incoming representations to update theirs. Alternatives that boast higher - expressive power do exist and are also supported, but simple architectures + expressive power also exist and are supported, but simple architectures may be just as good or better than complex alternatives in solving - practical problems - [Krasanakis et al., 2024]. - Simple architectures also enjoy reduced resource consumption. Before delving deeper - into JGNN's functionalities, you are encouraged to work with - the boilerplate code snippets provided here.
- + practical problems [Krasanakis et al., 2024]. + Simple architectures also enjoy reduced resource consumption. -Before starting with Java code, below we build some neural components
- in a scripting language called Neurallang. JGNN can parse strings containing such definitions
- to create trainable models with little effort. Other ways to declare
- architectures exist too, but Neuralang declarations fully decouple GNN architectures from Java code.
- Notice that the language is function-based, where double dots :
- indicate set or default hyperparameter values. Furthermore, the Neuralang keyword
- extern indicates a value declared by the parsing Java code,
- whereas values prepended with ! are broadcasted everywhere.
- Matrices and vectors are declared as required based on their dimensions and
- regulariation hyperparameter, where ? indicates hyperparameters
- that are dynamically determined when parsing the code with Java.
- More details on Neuralang are presented later.
fn classify(nodes, h, epochs: !3000, patience: !100, lr: !0.01) {
- return softmax(h[nodes], dim: "row");
-}
-fn gcnlayer(A, h, hidden: 16, reg: 0.005) {
- return A@h@matrix(?, hidden, reg) + vector(hidden);
-}
-fn gcn(A, h, classes: extern) {
- h = gcnlayer(A, h);
- h = dropout(relu(h), 0.5);
- return gcnlayer(A, h, hidden: classes);
-}
-fn ngcn(A, h, nodes) { // a node classification gcn
- return classify(nodes, gcn(A,h));
-}Let us now move to the GNN code.
- We first load the Cora dataset from those shipped
- with the library for usage in prototyping and testing. The first time this dataset
- runs, it automatically downloads some data and stores them in a local downloads/
- folder. The data are then loaded into a graph adjacency matrix, node features,
- and node labels. Before doing anything else, we apply the renormalization trick,
- which makes GNN computations numerically stable by adding self-loops to all nodes.
- We also create a symmetric normalization of the adjacency matrix. which is required
- by the model we impelement next. Both operations use in-place arithmetics to
- directly alter raw data.
The demonstration starts by loading the Cora dataset from those shipped
+ with the library for out-of-the-box testing. The first time this dataset is
+ constructed, it automatically downloads some data and stores them in a local downloads/
+ folder. The data are then loaded into a sparse graph adjacency matrix, a dense node feature matrix,
+ and a dense node label matrix. Sparse and dense representations are interchangeable in terms of operations,
+ but sparse matrices can store graphs with many nodes but relatively smaller degrees with greater memory efficiency.
+ In the loaded matrices, each row contains the corresponding node's
+ neighbors, features, or one-hot encoding of labels. We also apply the renormalization trick and symmetric normalization on
+ the dataset using in-place operations. The first of these makes GNN computations numerically stable by adding self-loops
+ to all nodes, whereas the second is required by the model we impelement next.
Dataset dataset = new Cora();
dataset.graph().setMainDiagonal(1).setToSymmetricNormalization();We now need to generate a trainable model. In GNNs, models are generated from model builders.
- These differ by which Neuralang capabilities they can parse - mainly whether they go beyond
- support of simple expressions involving operators only. The builder chosen here supports the whole
- language, but in turn misses out on some Java-side methods that help fill parts of the architecture
- with boilerplate patterns. To use the selected builder, save the first Neuralang snippet
- to a file and retrieve it with the
- expression String architecture = Paths.get("filename.nn");.
- (Use a Rust language highlighter for visual assistance when writting in Neuralang.)
- Alternatively, avoid external files by inlining the definition within Java code through
- a multiline string per String architecture = """ ... """;.
- This string is parsed within a functional programming chain, where
- each method call returns the modelBuilder instance to continue calling more methods.
-
For our model builder, we set remaining hyperparameters and overwrite the default value
- for "hidden" using the
- .config(String, double) method. We also determine
- which variables are constants, namely the adjacency matrix A and node
- representation h, and that node identifiers is a variable that serves
- as the architecture's inputs. There could be multiple inputs, so this distinction of what
- is a constant and what is a variable depends mostly on the which quantities change
- during training. In the case of node classification, both the adjacency matrix and
- node features remain constant, as we work in one graph. Finally, the definition
- sets an Neuralang expression as the architecture's output
- by calling the .out(String) method,
- and applies the .autosize(Tensor...) method to infer hyperparameter
- values denoted with ? from an example input (for faster inference, we
- provide dataless list of node identifiers as input).
We now incrementally construct a trainable model using symbolic definitions resembling math
+ notation. Symbolic expressions are part of a scripting-like language, called Neuralang,
+ that is covered in section 4. However, for faster onboarding we stick to
+ the FastBuilder class for creating models. This class's constructor
+ creates two constants A and h0 from its two arguments to. Other constants
+ and input variables can be set too, but more on this later. After instantiation, we use the
+ following model builder methods to constuct a model. Some of these methods parse symbolic expressions
+ to fastly declare machine learning components.
+
-
+
config- Configures hyperparameter values. These can be used in all subsequent function and layer declarations.
+ function- Declares a Neuralang function, in this case with inputsAandh.
+ layer- Declares a layer that can use builtin and Neuralang functions. In this, the symbols{l}and{l+1}specifically are replaced by a layer counter.
+ classify- Adds a softmax layer tailored to classification. This also silently declares an inputnodesthat represents a list of node indices where the outputs should be computed.
+ autosize- Automatically sizes matrix and vector dimensions filled by?. This requires some input example, and here we provide a list of node identifiers, which we also make dataless (have only the correct dimensions without allocating memory). This method also checks for integrity errors in the declared architecture, such as computational paths that do not lead to an output.
+
matrix and vector builtins to inline declarations of learnable parameter for
+ given dimensions and regularization.
+ The builder stores internally a constructed model, obtained through modelBuilder.getModel().
+
long numSamples = dataset.samples().getSlice().size();
long numClasses = dataset.labels().getCols();
-ModelBuilder modelBuilder = new Neuralang()
- .parse(architecture)
- .constant("A", dataset.graph())
- .constant("h", dataset.features())
- .var("nodes")
+ModelBuilder modelBuilder = new FastBuilder(dataset.graph(), dataset.features())
+ .config("reg", 0.005)
.config("classes", numClasses)
- .config("hidden", numClasses+2)
- .out("ngcn(A, h, nodes)")
+ .config("hidden", 64)
+ .function("gcnlayer", "(A,h){Adrop = dropout(A, 0.5); return Adrop@(h@matrix(?, hidden, reg))+vector(?);}")
+ .layer("h{l+1}=relu(gcnlayer(A, h{l}))")
+ .config("hidden", "classes") // reassigns the output gcnlayer's "hidden" to be the number of "classes"
+ .layer("h{l+1}=gcnlayer(A, h{l})")
+ .classify()
.autosize(new EmptyTensor(numSamples));Training epochs for the constructed model can be implemented
+ manually, by passing inputs, obtaining outputs, computing losses, and triggering backpropagation
+ on an optimizer. However, this may be complicated to write properly, so JGNN automates common
+ training patterns with a ModelTraining class. Instances of this class
+ accept a method chain notation to set their parameters, like the number of epochs, patience
+ for early stopping, the employed optimizer, and loss functions. An example is presented below,
+ where Adam optimization with learning rate 0.01 is performed, and a verbose
+ variation of a validation loss keeps track of training progress. To run a full training process,
+ the defined strategy is passed to the model alongside input data, corresponding output data, as well
+ as training and validation slices.
Notice how, before training starts, a parameter initializer is also applied on the model for a cold
+ start. Selecting an initilizer is not part of training strategies
+ to signify its model-dependent nature; dense layers should maintain the expected
+ input variances in the output before the first epoch. Moreover,
+ the graph's adjacency matrix and node features are already declared as constants by the
+ FastBuilder constructor, as node classification takes place on the same graph
+ with fully known node features. Instead, what is considered as inputs and outputs in this case
+ are the node identifiers (which in the classify method above are used to gather
+ the outputs of respective nodes) and corresponding labels. Labels that are not known (e.g., because
+ they refer to test data) still need to have some value, so as a convention if your are working
+ with your own data leave the one-hot label encoding of test nodes as zeroes. Doing so in this
+ example would not affect the outcome either. To recap, our full dataset consists of all node
+ identifiers and corresponding labels. The last two arguments of the train method
+ then accept training and validation data slices. Slices are effectively lists of integer entries
+ pointing to rows of the datasets - find more later.
+
ModelTraining trainer = new ModelTraining()
+ .setOptimizer(new Adam(0.01))
+ .setEpochs(3000)
+ .setPatience(100)
+ .setLoss(new CategoricalCrossEntropy())
+ .setValidationLoss(new VerboseLoss(new Accuracy()).setInterval(10)); // print validation every 10 epochs
+
+Slice nodes = dataset.samples().getSlice().shuffle(); // a permutation of node identifiers
+Matrix inputData = Tensor.fromRange(nodes.size()).asColumn(); // each node has its identifier as an input
+Model model = modelBuilder.getModel()
+ .init(new XavierNormal())
+ .train(trainer,
+ inputData,
+ dataset.labels(),
+ nodes.range(0, 0.6), // training slice
+ nodes.range(0.6, 0.8) // validation slice
+ );Trained models and their generating builders can be saved and loaded. These next snippet demonstrates + how raw predictions can be made too. During this process, + some matrix manipulation operations are employed to obtain transparent access to parts of input and output data + of the dataset. +
+ +modelBuilder.save(Paths.get("gcn_cora.jgnn")); // needs a Path as an input
+Model loadedModel = ModelBuilder.load(Paths.get("gcn_cora.jgnn")).getModel(); // loading creates an intermediate modelbuilder
+
+Matrix output = loadedModel.predict(Tensor.fromRange(0, nodes.size()).asColumn()).get(0).cast(Matrix.class);
+double acc = 0;
+for(Long node : nodes.range(0.8, 1)) {
+ Matrix nodeLabels = dataset.labels().accessRow(node).asRow();
+ Tensor nodeOutput = output.accessRow(node).asRow();
+ acc += nodeOutput.argmax()==nodeLabels.argmax()?1:0;
+}
+System.out.println("Acc\t "+acc/nodes.range(0.8, 1).size());3. GNN Builders
3.1. ModelBuilder
[Under construction] -So far, we have discussed the propagation mechanisms of - GNNs, which consider the features of all nodes. However, - in a node classification setting, training data labels - are typically available only for certain nodes. We thus - need a mechanism to retrieve the predictions of the top - neural layer for certain nodes and pass them through a - softmax activation.
- -This can be achieved with normal neural model definitions - using the gather bracket operation after declaring a - variable of which nodes to retrieve:
- -.var("nodes")
-.layer("h{l} = softmax(h{l})")
-.operation("output = h{l}[nodes]")Recall that h{l} always points to the top
- layer when writing a new layer. This way, the built
- model takes as inputs a set of nodes, performs the
- forward pass of the architecture, and then selects the
- provided nodes to use as outputs (and backpropagate
- from). All nodes are needed for
- training because they are made aware of each other via
- the graph's structure.
- To simplify how node classification architectures are
- defined, the above symbolic snippet is automatically
- generated and applied by calling the
- .classify() method of the
- FastBuilder instead.
-
x and y with any expression. Sometimes, y needs to be a constant defined either by presenting a number, calling ModelBuilder.config(y, double), or calling ModelBuilder.constant(y, double) to set the numbers as hyperparameters.
+
+ | Symbol | +Type | +Number of inputs | +
|---|---|---|
x = y |
+ Operator | +Assign to variable x the outcome of executing y. |
+
x + y |
+ Operator | +Element-by-element addition. | +
x * y |
+ Operator | +Element-by-element multiplication. | +
x - y |
+ Operator | +Element-by-element subtraction. | +
x @ y |
+ Operator | +Matrix multiplication. | +
x | y |
+ Operator | +Row-wise concatenation of x and y. |
+
x [y] |
+ Operator | +Gathers the rows of x with indexes y. |
+
transpose(x) |
+ Function | +Transposes matrix x. |
+
log(x) |
+ Function | +Apply logarithm on each tensor element. | +
relu(x) |
+ Function | +Apply relu on each tensor element. | +
tanh(x) |
+ Function | +Apply a tanh activation on each tensor element. | +
sigmoid(x) |
+ Function | +Apply a sigmoid activation on each tensor element. | +
dropout(x, rate) |
+ Function | +Apply training dropout on tensor x with constant dropout rate hyperparameter rate. |
+
lrelu(x, slope) |
+ Function | +Leaky relu on tensor x with constant negative slope hyperparameter slope. |
+
prelu(x) |
+ Function | +Leaky relu on tensor x with learnable negative slope. |
+
softmax(x, dim) |
+ Function | +Apply a softmax reduction on x, where dim is either dim:'row' (default) or dim:'col'. |
+
sum(x, dim) |
+ Function | +Apply a sum reduction on x, where dim is either dim:'row' (default) or dim:'col'. |
+
sum(x, dim) |
+ Function | +Apply a sum reduction on x, where dim is either dim:'row' (default) or dim:'col'. |
+
max(x, dim) |
+ Function | +Apply a max reduction on x, where dim is either dim:'row' (default) or dim:'col'. |
+
min(x, dim) |
+ Function | +Apply a min reduction on x, where dim is either dim:'row' (default) or dim:'col'. |
+
matrix(rows, cols) |
+ Function | +Generate a matrix parameter with respective hyperparameter dimensions. | +
matrix(rows, cols, reg) |
+ Function | +Generate a matrix parameter with respective hyperparameter dimensions, and L2 regularization hyperparameter reg. |
+
vector(len) |
+ Function | +Generate a vector with size hyperparameter size len. |
+
vector(len, reg) |
+ Function | +Generate a vector with size hyperparameter len, and L2 regularization hyperparameter reg. |
+
Prefer using hyperparameters (set via .config) for matrix and vector creation, as these transfer their names to respective dimensions for error checking. For dropout, matrix, and vector you can also use the short names drop, mat, vec.
3.2. FastBuilder
The FastBuilder class for building GNN architectures extends the generic
LayerBuilder with common graph neural network operations. The main difference
@@ -379,8 +553,39 @@
3.2. FastBuilder
if 1-2% accuracy gains are worth the loss of portability. -3.3. Neuralang
- [Under construction] + +So far, we have discussed the propagation mechanisms of + GNNs, which consider the features of all nodes. However, + in a node classification setting, training data labels + are typically available only for certain nodes. We thus + need a mechanism to retrieve the predictions of the top + neural layer for certain nodes and pass them through a + softmax activation.
+ +This can be achieved with normal neural model definitions + using the gather bracket operation after declaring a + variable of which nodes to retrieve:
+ +.var("nodes")
+.layer("h{l} = softmax(h{l})")
+.operation("output = h{l}[nodes]")Recall that h{l} always points to the top
+ layer when writing a new layer. This way, the built
+ model takes as inputs a set of nodes, performs the
+ forward pass of the architecture, and then selects the
+ provided nodes to use as outputs (and backpropagate
+ from). All nodes are needed for
+ training because they are made aware of each other via
+ the graph's structure.
+ To simplify how node classification architectures are
+ defined, the above symbolic snippet is automatically
+ generated and applied by calling the
+ .classify() method of the
+ FastBuilder instead.
+
3.4. Debugging
JGNN offers high-level tools for debugging @@ -444,7 +649,70 @@
3.4. Debugging
at mklab.JGNN.nn.ModelBuilder.assertBackwardValidity(ModelBuilder.java:504) at nodeClassification.APPNP.main(APPNP.java:45) - + +In addition to other operations, some methods
+ do not affect tensor or matrix values but
+ are only responsible for naming dimensions.
+ Functionally, these are decorative and aim
+ to improve debugging by throwing errors for
+ incompatible non-null names. For example,
+ adding two matrices with different dimension
+ names will result in an error. Likewise, the
+ inner dimension names during matrix
+ multiplication should agree.
+ Arithmetic operations, including
+ matrix multiplication and copying,
+ automatically infer dimension names in the
+ result to ensure only compatible data types
+ are compared. Dimension names can be freely
+ changed for any Tensor without
+ backtracking changes (even for see-through
+ data types, such as the outcome of
+ asTransposed()).
+ Matrices effectively have three
+ dimension names: for their rows, columns,
+ and inner data as long as they are treated
+ as tensors.
| Operation | +Type | +Comments | +
|---|---|---|
Tensor setDimensionName(String name) |
+ arithmetic | +For naming tensor dimensions (of + the 1D space tensors lie + in). | +
Tensor setRowName(String rowName) |
+ arithmetic | +For naming what kind of + information matrix rows hold + (e.g., "samples"). | +
Tensor setColName(String colName) |
+ arithmetic | +For naming what kind of + information matrix columns hold + (e.g., "features"). | +
Tensor setDimensionName(String rowName, String colName) |
+ arithmetic | +A shorthand of calling
+ setRowName(rowName).setColName(colName).
+ |
+
There are two main mechanisms for identifying logically erroneous architectures: a) mismatched @@ -464,10 +732,10 @@
3.4. Debugging
that declare hyperparameters:new ModelBuilder()
-.config("features", 7)
-.config("hidden", 64)
-.var("x")
-.operation("h = x@matrix(features, hidden)");instead of the simpler:
@@ -500,42 +768,157 @@3.4. Debugging
easy—the inner matrix dimensions of matrix multiplication do not agree. However, we need to find the error within our architecture to fix - it. - -To do this, continue reading and see the message
- During the forward pass of class mklab.JGNN.nn.operations.MatMul: _tmp4 = null. This tells us that the problem
+ it. To do this, the error message message states
+ that the error occures.
+ During the forward pass of class mklab.JGNN.nn.operations.MatMul: _tmp4 = null. This tells us that the problem
occurs when trying to calculate
- _tmp4, which is currently assigned
+ _tmp4, which is currently assigned
a null tensor as value. Some more
information is available to see what the
operation's inputs are like—in this case, they
coincide with the multiplication's inputs, but
this will not always be the case.
-
The important point is to go back to the + The important point is to go back to the execution tree and see during which exact operation this variable is defined. There, we will undoubtedly find that some dimension had 64 instead of 32 elements or conversely.
-Monitoring Operations
-In addition to all other debugging mechanisms,
-JGNN presents a way to show when forward and
-backward operations of specific code components
-are executed and with what kinds of arguments.
-This can be particularly useful when testing new
-components in real (complex) architectures.
-The practice consists of calling a
-monitor(...) function within
-operations. This does not affect what
-expressions do and only enables printing
-execution tree operations on operation
-components. For example, the next snippet
-monitors the outcome of matrix multiplication:
-
builder.operation("h = relu(monitor(x@matrix(features, 64)) + vector(64))")In addition to all other debugging mechanisms,
+ JGNN presents a way to show when forward and
+ backward operations of specific code components
+ are executed and with what kinds of arguments.
+ This can be particularly useful when testing new
+ components in real (complex) architectures.
+ The practice consists of calling a
+ monitor(...) function within
+ operations. This does not affect what
+ expressions do and only enables printing
+ execution tree operations on operation
+ components. For example, the next snippet
+ monitors the outcome of matrix multiplication:
+
builder.operation("h = relu(monitor(x@matrix(features, 64)) + vector(64))")4. Neuralang
+ +The language
+Neuralang scripts consist of functions that declare machine learning
+ components and their interactions using a syntax inspired by the Mojo.
+ language. Use a Rust highlighter to cover all keywords.
+ Before explaining how to use the Neuralang model builder,
+ we first analyse some of the functions in the Quickstart section code
+ to explain the lenguage's syntax. First, let us look at classify
+ function that takes
+ two inputs: nodes that correspond to node identifiers
+ for classification, and a node feature matrix h.
+ A softmax is returned for the features of the specified nodes. The function's
+ signature also has several configuration values, whose defaults
+ are indicated by a colon :. The same notation is used to
+ set/overwrite configurations when calling functions, as we do for softmax
+ to apply it row-wise. Think of configurations as keyword
+ arguments of typical programming languages, with the difference that
+ they control hyperparameters, like dimension sizes or regularization.
+ Write exact values for configurations, as for now there no
+ arithmetics take place for them. For example, a configuration
+ patience:2*50 creates an error.
fn classify(nodes, h, epochs: !3000, patience: !100, lr: !0.01) {
+ return softmax(h[nodes], dim: "row");
+}Exclamation marks ! before numbers broadcast values
+ to all subsequent function calls that have configurations with the same
+ name. The broadcasted defaults overwrite default values. Importantly,
+ broadcasted values are also retrievable from JGNN's Neuralang model
+ builder too; which is useful for Java integration. In this case, values
+ for the training process are obtained (see below). Configuration
+ values have the priority:
+ 1. function call arguments
+ 2. broacasted configurations (the last broadcasted value, including configurations set by Java)
+ 3. function signature defaults
+
Next, let us look at the gcnlayer function. This accepts
+ two parameters: an adjacency matrix A and input feature matrix h.
+ The configuration hidden: 64 in the functions's signature
+ specifies the deafult number of hidden units,
+ whereas reg: 0.005 is the L2 regularization applied
+ during machine learning. The questionmark ?
+ in matrix definitions lets the autosize feature of JGNN determine
+ dimension sizes based on a test run - if possible.
+ Finally, the function returns the activated output of a
+ GCN layer.
+
+
fn gcnlayer(A, h, hidden: 64, reg: 0.005) {
+ return A@h@matrix(?, hidden, reg) + vector(hidden);
+}The last of the functions we tackle, the gcn function declares the popular Graph Convoluational Network (GCN)
+ architecture and has as configuration the number of output classes.
+ The function first applies a gcnlayer, and then applies another layer
+ of the same type with the hidden units configuration set to the value of classes.
+ Thus the output matches the number of classes, which is set as externally declared
+ with the extern keyword; this signifies that the configuration has no default
+ and should be provided by Java's side of the implementation.
Java-side integration
+ + + +We now need generate a trainable model by incrementally constructing model builders.
+ These differ by which Neuralang capabilities they can parse - mainly whether they go beyond
+ support of simple expressions involving operators only. The builder chosen here supports the whole
+ language, but in turn misses out on some Java-side methods that help fill parts of the architecture
+ with boilerplate patterns. To use the selected builder, save the first Neuralang snippet
+ to a file and retrieve it with the
+ expression String architecture = Paths.get("filename.nn");.
+ (Use a Rust language highlighter for visual assistance when writting in Neuralang.)
+ Alternatively, avoid external files by inlining the definition within Java code through
+ a multiline string per String architecture = """ ... """;.
+ This string is parsed within a functional programming chain, where
+ each method call returns the modelBuilder instance to continue calling more methods.
+
For our model builder, we set remaining hyperparameters and overwrite the default value
+ for "hidden" using the
+ .config(String, double) method. We also determine
+ which variables are constants, namely the adjacency matrix A and node
+ representation h, and that node identifiers is a variable that serves
+ as the architecture's inputs. There could be multiple inputs, so this distinction of what
+ is a constant and what is a variable depends mostly on the which quantities change
+ during training. In the case of node classification, both the adjacency matrix and
+ node features remain constant, as we work in one graph. Finally, the definition
+ sets an Neuralang expression as the architecture's output
+ by calling the .out(String) method,
+ and applies the .autosize(Tensor...) method to infer hyperparameter
+ values denoted with ? from an example input (for faster inference, we
+ provide dataless list of node identifiers as input).
long numSamples = dataset.samples().getSlice().size();
+long numClasses = dataset.labels().getCols();
+ModelBuilder modelBuilder = new Neuralang()
+ .parse(architecture)
+ .constant("A", dataset.graph())
+ .constant("h", dataset.features())
+ .var("nodes")
+ .config("classes", numClasses)
+ .config("hidden", numClasses+2)
+ .out("ngcn(A, h, nodes)")
+ .autosize(new EmptyTensor(numSamples));3.4. Generalized message passing
Similarly, JGNN also supports the the fully generized message-passing scheme between node neighbors to support @@ -586,16 +969,15 @@
3.4. Generalized message passing
reduction, given that this has the same theoretical expressive power as maximum-based reduction but is easier to backpropagate - through. Perform this like so: + through. Perform this like below. + The sum is weighted per the values of + the adjacency matrixA. Thus,
+ perform adjacency matrix normalization only if
+ you want such weighting to occur.
modelBuilder
.operation("received{l}=reduce(transformed_message{l}, A)")Note that the sum is weighted per the values of
- the adjacency matrix A. Thus,
- perform adjacency matrix normalization only if
- you want such weighting to occur.
You can finally define layers that transform node representations while accounting for received messages:
@@ -613,7 +995,7 @@3.4. Generalized message passing
A common realization of message-passing GNNs is
via sparse-dense matrix multiplication to
emulate neighbor attention per
- A.(hTh), where
+ A.(hTh), where
A is a sparse adjacency matrix,
. is the Hadamard product
(element-by-element multiplication), and
@@ -708,11 +1090,11 @@
3.4. Generalized message passing
4. Create Data
-If you have been following the tutorial, we have only used - automatically downloaded datasets till now. In practice, you - will want to use your own data. Here, we cover how to manually - fill in data, as well as which operations are provided to manipulate +
5. Create Data
+JGNN provides automatically downloaded dataset classes that automatically + download their data and are ready to test with. + In practice, though, you will want to use your own data. We now cover how to + manually fill in data, as well as which operations are provided to manipulate those data. Read more of these operations to learn how to process neural inputs, post-process learning outcomes, create custom parameters, contribute to the library with more components, @@ -722,7 +1104,7 @@
4. Create Data
too deep in the first read of this guidebook. -4.1. Fill in data
+5.1. Fill in data
In the simplest case, both the number of nodes or data samples, and the number of feature dimensions are known beforehand. If so, create dense feature matrices with the following code. This uses the @@ -779,7 +1161,7 @@
4.1. Fill in data
.put(edge.getKey(), edge.getValue(), 1) .put(edge.getValue(), edge.getKey(), 1); -4.2. Identifiers
+5.2. Identifiers
The above snippets all make use of numerical node identifiers. To
manage these, JGNN provides an IdConverter class.
You can convert hashable
@@ -812,7 +1194,7 @@
4.2. Identifiers
System.out.println(classIds.get(predictedClassId));4.3. Tensor Operations
+5.3. Tensor Operations
Tensor operations are performed element-by-element and can be split into the following categories:
@@ -999,8 +1381,9 @@4.3. Tensor Operations
of elements. If there are many zero elements expected, prefer using a sparse tensor. -Tensor denseTensor = new mklab.JGNN.tensor.DenseTensor(long size);
-Tensor sparseTensor = new mklab.JGNN.tensor.SparseTensor(long size);long size = ...;
+Tensor denseTensor = new mklab.JGNN.tensor.DenseTensor(size);
+Tensor sparseTensor = new mklab.JGNN.tensor.SparseTensor(size);For example, one-hot encodings for classification problems can be generated with the following @@ -1123,76 +1506,10 @@
4.3. Tensor Operations
- -4.4. Named Dimensions
-In addition to other operations, some methods
- do not affect tensor or matrix values but
- are only responsible for naming dimensions.
- Functionally, these are decorative and aim
- to improve debugging by throwing errors for
- incompatible non-null names. For example,
- adding two matrices with different dimension
- names will result in an error. Likewise, the
- inner dimension names during matrix
- multiplication should agree.
- Arithmetic operations, including
- matrix multiplication and copying,
- automatically infer dimension names in the
- result to ensure only compatible data types
- are compared. Dimension names can be freely
- changed for any Tensor without
- backtracking changes (even for see-through
- data types, such as the outcome of
- asTransposed()).
- Matrices effectively have three
- dimension names: for their rows, columns,
- and inner data as long as they are treated
- as tensors.
| Operation | -Type | -Comments | -
|---|---|---|
Tensor setDimensionName(String name) |
- arithmetic | -For naming tensor dimensions (of - the 1D space tensors lie - in). | -
Tensor setRowName(String rowName) |
- arithmetic | -For naming what kind of - information matrix rows hold - (e.g., "samples"). | -
Tensor setColName(String colName) |
- arithmetic | -For naming what kind of - information matrix columns hold - (e.g., "features"). | -
Tensor setDimensionName(String rowName, String colName) |
- arithmetic | -A shorthand of calling
- setRowName(rowName).setColName(colName).
- |
-
5. Training
+6. Training
Most neural network architectures are designed with the idea of learning to classify nodes or samples. However, GNNs also provide the capability to classify entire graphs based on @@ -1240,49 +1557,46 @@
5. Training
.layer("h{l+1}=softmax(mean(h{l}, dim: 'row'))") .out("h{l}"); -Training the Architecture
-For the time being, training architectures like the above - on prepared data requires manually calling the - backpropagation for each epoch and each graph in the - training batch. To do this, first retrieve the model and - initialize its parameters:
- -Model model = builder.getModel().init(new XavierNormal());Next, define a loss function and set up a batch
- optimization strategy wrapping any base optimizer and
- accumulating parameter updates until
- BatchOptimizer.updateAll() is called later
- on:
-
For the time being, training architectures like the above + on prepared data requires manually calling the + backpropagation for each epoch and each graph in the + training batch. To do this, first retrieve the model and + initialize its parameters:
+ +Model model = builder.getModel().init(new XavierNormal());Loss loss = new CategoricalCrossEntropy();
+ Next, define a loss function and set up a batch
+ optimization strategy wrapping any base optimizer and
+ accumulating parameter updates until
+ BatchOptimizer.updateAll() is called later
+ on:
+
+
+ Loss loss = new CategoricalCrossEntropy();
BatchOptimizer optimizer = new BatchOptimizer(new Adam(0.01));
- Finally, training can be conducted by iterating through
- epochs and training samples and appropriately calling
- the Model.train for combinations of node
- features and graph adjacency matrix inputs and graph
- label outputs. At the end of each batch (e.g., each
- epoch), don't forget to call the
- optimizer.updateAll() method to apply the
- accumulated gradients. This process can be realized with
- the following code:
-
+ Finally, training can be conducted by iterating through
+ epochs and training samples and appropriately calling
+ the Model.train for combinations of node
+ features and graph adjacency matrix inputs and graph
+ label outputs. At the end of each batch (e.g., each
+ epoch), don't forget to call the
+ optimizer.updateAll() method to apply the
+ accumulated gradients. This process can be realized with
+ the following code:
+
- for(int epoch=0; epoch<300; epoch++) {
- for(int graphId=0; graphId<graphLabels.size(); graphId++) {
- Matrix adjacency = adjacencyMatrices.get(graphId);
- Matrix features = nodeFeatures.get(graphId);
- Tensor label = graphLabels.get(graphId);
- model.train(loss, optimizer,
- Arrays.asList(features, adjacency),
- Arrays.asList(label));
- }
- optimizer.updateAll();
+ for(int epoch=0; epoch<300; epoch++) {
+ for(int graphId=0; graphId<graphLabels.size(); graphId++) {
+ Matrix adjacency = adjacencyMatrices.get(graphId);
+ Matrix features = nodeFeatures.get(graphId);
+ Tensor label = graphLabels.get(graphId);
+ model.train(loss, optimizer,
+ Arrays.asList(features, adjacency),
+ Arrays.asList(label));
+ }
+ optimizer.updateAll();
}
-