Specification Document

For this project I will implement a small neural network library in order to understand its functionality better. I will mostly use matrixes, which means mostly using lists in python. I might implement my own matrix class.

Matrix multiplication consumes the most computation time in neural networks. Unfortunately their optimization is often done using parallel computation and hierarchial memory. And apparently the divide and conquer method (Strassen) hardly ever works better than the naïve version in practice. I still need to do more research on the subject.

Estimating the computational complexity of a neural network is not trivial. The amount of required computations depends on the amount of layers, neurons and the size of the output. I will start by implementing a fully connected network, which requires multiplying the activation of each neuron of the previous layer with the appropriate weight for each neuron. The naïve matrix multiplication is O(n³).

I will use the gradient descent algorithm with backpropagation to minimize the loss and optimize the network.

The library will probably be used the same way as Pytorch: One has to define the layers used in the constructor and the feedforward function in order to train the model. I will provide some examples of its use.