Scaling n_neurons past ~200 does not improve accuracy

[arvoelke]

There seems to be a critical noise floor (~5% error) that can't be overcome by adding more neurons to a single layer encode-decode. The floor is already hit around 200 neurons. This could have to do with synchrony in the spikes (i.e., a lack of heterogeneity, causing two neurons to be no different than one with twice the spike height)? This synchrony could be coming from the discretization of the neuron model in time, or the quantization of its variables and time-constants in space? Or systematic differences between loihi_rates and the actual spike rates?

I've tested this (on master) against a wide range of L2-regularization constants (from 1e-5 to 1e0). With standard Nengo, you can always continue to improve performance by increasing n_neurons and dropping reg according to the spike-noise (O(1/sqrt(n))), but it doesn't seem to be the case here.

Is there any way to break the ~5% floor?

Of relevance to #126: uncommenting the configuration to the max_rates and intercepts defaults below makes everything worse by raising the noise floor to ~17% (not shown). I believe this is because it increases the level of spike synchrony by narrowing the range of the response curves at 1. And so further investigation might be required to determine the best defaults.

On the other hand, lowering dt does seem to make things slightly better.

I've also tried partitioning up to 2000 neurons across many sub-ensembles, and solving the decoder optimization problem across all ensembles simultaneously using my VirtualEnsemble network, and that actually shows performance degrade past 1000 neurons (code omitted, for brevity).

from collections import defaultdict

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from pandas import DataFrame

import nengo
from nengo.utils.numpy import rmse

import nengo_loihi

# nengo.Ensemble.max_rates.default = nengo.dists.Uniform(100, 120)
# nengo.Ensemble.intercepts.default = nengo.dists.Uniform(-1, 0.5)

def go(n_neurons, solver=nengo.solvers.LstsqL2(),
       tau_probe=0.005, sim_t=1.0, dt=0.001,
       simulator=nengo_loihi.Simulator):

    with nengo.Network() as model:
        u = nengo.Node(1)
        x = nengo.Ensemble(n_neurons, 1)
        
        nengo.Connection(u, x, synapse=None)
        
        p = nengo.Probe(x, synapse=tau_probe, solver=solver)
        p_ideal = nengo.Probe(u, synapse=tau_probe)
        
    with simulator(model, dt=dt) as sim:
        sim.run(sim_t)
    
    return rmse(sim.data[p], sim.data[p_ideal])

data = defaultdict(list)
for trial in range(3):
    print("Trial #", trial)
    for reg in ['1e-5', '1e-4', '1e-3', '1e-2', '1e-1', '1e0']:
        for n_neurons in np.arange(100, 1100, 100, dtype=int):
            data['Trial'].append(trial)
            data['reg'].append('=%s' % reg)  # work-around for bug related to hues that can be casted to float
            data['# Neurons'].append(n_neurons)
            data['RMSE'].append(go(n_neurons, solver=nengo.solvers.LstsqL2(reg=float(reg))))

df = DataFrame(data)

plt.figure(figsize=(14, 6))
sns.lineplot(data=df, x='# Neurons', y='RMSE', hue='reg')
plt.show()

Code for dt plot:

data = defaultdict(list)
for trial in range(10):
    print("Trial #", trial)
    for dt in ['1e-4', '5e-4', '1e-3']:
        for n_neurons in np.arange(100, 1100, 100, dtype=int):
            data['Trial'].append(trial)
            data['dt'].append('=%s' % dt)  # work-around for bug related to hues that can be casted to float
            data['# Neurons'].append(n_neurons)
            data['RMSE'].append(go(n_neurons, dt=float(dt)))

df = DataFrame(data)

plt.figure(figsize=(14, 6))
sns.lineplot(data=df, x='# Neurons', y='RMSE', hue='dt')
plt.show()

[arvoelke]

FYI, I've now tried the L2-optimization directly on the filtered spike data (as I've been doing for my Reservoir Computing experiments), but that doesn't seem to improve things. I've tried this in conjunction with lowering the regularization, and using the VirtualEnsemble to scale up to 3000 neurons, but still can't break the floor. This means loihi_rates should not be the main issue here, but rather most of the neurons must be doing nearly the same thing. I think decreasing the time-step helps a little bit, but doesn't overcome the limited heterogeneity.

Any examples of using random sampling to break the synchrony without increasing noise (as suggested in https://www.nengo.ai/nengo-loihi/examples/communication_channel.html)? I tried to do this by starting the simulation with a brief pulse of random current to each neuron, but this ended up raising an error for blowing through the memory. Could each post-neuron on the the host -> chip stimulus connection be assigned dedicated heterogeneous spike-generators?

[arvoelke]

I've been trying to dig into quantifying and separating out potential sources of error. @celiasmith mentioned that @hunse has a way of improving the emulator accuracy by more accurately modelling the voltage overshoot? We could try this on the above code?

I've since tried a number of other things:

• Looking at the distribution of distinct ISIs (from nengo.utils.neurons import rates_isi), within each neuron, and across all neurons
• Looking at how often the ISI changes over time
• Plotting the "actual" tuning curves (input versus ISI)
• Looking at the profile of pairwise Victor-Purpura spike distances, and its mean

In all cases things look really similar between nengo and nengo_loihi on a sinusoidal encode-decode with set_defaults, and so I've been unable to tease apart any differences here.

By the way, dropping dt further to 1e-5 raises an exception:

~/CTN/nengo-loihi/nengo_loihi/loihi_cx.py in discretize(self)
    263                     break
    264             else:
--> 265                 raise BuildError("Could not find appropriate wgtExp")
    266         elif b_max > 1e-8:
    267             b_scale = BIAS_MAX / b_max

BuildError: Could not find appropriate wgtExp

[arvoelke]

After some more investigation, the dominant source of error was found to be coming from the on/off NIF generators that encode the input signal into spikes. One way around this is to use Principle 3, configure the inter_tau accordingly, and set the input connection synapse to None (see #115 (comment)) -- but this is not a good solution at the moment because all of the interneurons and spike generators share the same tau, and special care is needed to handle the pre-synaptic and post-synaptic radii. Another way around this is to use a 2*k-bit binary code to transmit the input signal at each time-step with 2^k precision. A proof-of-concept is provided in #158. This reveals a new noise floor, and weirdness at 600-800 neurons, that both require further investigation.

[arvoelke]

This is what the picture looks like, comparing the two simulators, on branch binary-encoding (#158), using neuron_type=nengo_loihi.neurons.LoihiSpikingRectifiedLinear(), the stimulus np.sin(2*np.pi*t/sim_t), and reporting the nengolib.signal.nrmse across 10 trials. The two are getting closer, but nengo_loihi still bottoms out at around 300 neurons. Edit: It should be even better, but I think there are issues with the branch I need to sort out.

For Principle 3, the tricks from #115 (comment) work with the same ReLU model but on integrator-accuracy (#124) and without any need for binary spike encoding. However this also plateaus at around 300 neurons.

(Code available on request.)

[hunse]

Since we quantize everything when putting it on Loihi, it's not unlikely for some neurons to end up with exactly the same parameters, and thus be redundant. I've seen this happen with at least one pair even in the case of a 100-neuron population. So this could be another effect adding to this issue. One way around it could be to add a little bit of noise to all neuron voltages, just to help them do different things. The first step, though, would be to quantify how big of an issue this really is.

[arvoelke]

Some progress was made in the investigation that I don't believe was sync'd back to this issue. In particular, all of the spike-times tend to be synchronized with one another until a burst of noise is injected into the network (at 20 seconds in the plot below). In general, the dependent spike-variability has an enormous effect on the error for large numbers of neurons.

The first step, though, would be to quantify how big of an issue this really is.

What kind of quantification are you looking for? I've done some pretty thorough experimentation above and with the emulator for my thesis. In particular, there is a noise floor of 5% that Nengo-Loihi can never scale below, even with continuous binary encoding with 8 bits of precision (0.4% error; #158), and no decode neurons. That's just a normal NEF encode-decode and spiking ReLU neurons and no extra filtering. With 1000 neurons the error is around 300% worse relative to vanilla Nengo. Furthermore, 100 Nengo neurons are the same as 1000 Nengo-Loihi neurons (see plot from previous post).

To me this is quite alarming. It should in theory continue to scale to 0% as more neurons are added, but instead is bottoms-out quite rapidly. As an engineer I would like to be able to add neurons until my function falls within spec, and right now I can barely meet spec for 100 Nengo neurons. Theorem 3.2.1 in my thesis proves rigorous criteria for ensuring the error continues to scale away even with arbitrarily high input frequencies and arbitrarily small taus, regardless of firing rates. The natural next step IMO is to see why this criteria is being violated.

[hunse]

The quantification I'm looking for is regarding neurons with the same parameters. If by 200-300 neurons, most new neurons generated have the exact same parameters as an existing neuron in the population, then it's unsurprising that adding more doesn't help. But if new neurons still have new parameters, then the cause could be something else.

I made a basic script to try this, and it's not finding much redundancy between neurons. However, this will largely depend on the input connections to an ensemble, and thus how the weights/biases get quantized. If you ever see a warning about "lost bits" in the weights, be very wary. Even without such a warning, there is quantization in the weights and biases that could be making a lot of neurons redundant. But based on this basic exploration, this doesn't seem to be the main factor driving what you see in your scripts above. It seems that even neurons with slightly different parameters are still having a lot of redundancy.

The other thing is that you'll likely see a lot more redunancy for higher firing rates (because of the aliasing discussed above), which means for values near +/- 1. Of course, though, if you just lower the rates, then that adds its own "noise" since the spikes are more sparse. So just lowering the max firing rates of the population doesn't seem to help much.

I tried my trick of subtracting the voltage threshold on spike, to get rid of the aliasing problem, but it doesn't appear to help (code is here). So it seems that the aliasing isn't the main cause of neuron synchrony. In general, it's not terribly surprising that synchrony is more of a problem on Loihi then in core nengo, since with discretized values there's fewer ways for neurons to become desynchronized.

I just realized, what we really need to be looking at is the diversity in the firing rates at your particular input value of 1. Since you jump straight to that input value, neurons with the same max firing rate but different gain/bias don't have time to desynchronize. I played around a bit with ramping up to an input of 1 over e.g. 0.1 seconds, and it looks like it might help a bit, but I don't have any conclusive results since changing the input signal changes the RMSE of it as well (we should maybe be using relative RMSE---dividing by the reference signal RMS---rather than absolute).

One other thing to be aware of is that the maximum firing rate of a neuron is different on Loihi, because we can't discount the overshoot from the refractory period like we do in core nengo. So for example, if the refractory period is two steps, then that means the voltage is zero for two steps, and then can fire the third step (on Loihi). So if your tau_ref=0.002, that means the maximum firing rate is 333 Hz on Loihi, whereas it's 500 Hz in core Nengo.

Anyway, this needs more research, because there's a lot of ideas here but nothing conclusive. I think your approach of adding noise is the best one, since I'm not convinced we can get adequate heterogeneity to properly desynchronize neurons through heterogeneity alone. I'd also suggest using more diverse input signals, since I'm not sure how well just using an input of 1 will generalize (though we do often have sudden changes in signals either coming into or within our networks, which will have a synchronizing effect on neurons). I think we need some more practical examples on which to test adding some noise, maybe some of the benchmarks.