sim.predict make GPU full memory

[kebitmatf]

Hello,

After traning the nengodl model, I evaluate the accuracy of the model by two ways:

1. using sim.evaluation: sim.evaluate(test_images, {out_p_filt: test_labels}, the result is fine.
2. using: pred = sim.predict(test_images), the GPU memory get full. Why this happened and how to solve?

Thank you!

[drasmuss]

Hello @kebitmatf,

I haven't seen that behaviour before, do you have a minimal example to help debug the issue?

[kebitmatf]

Dear Drasmuss,
Thank you for your comment.

Here is my code:

with nengo.Network(seed=42) as net:
net.config[nengo.Ensemble].max_rates = nengo.dists.Choice([100])
net.config[nengo.Ensemble].intercepts = nengo.dists.Choice([0])
net.config[nengo.Connection].synapse = None
neuron_type = nengo.LIF(amplitude=0.01)

nengo_dl.configure_settings(stateful=False)

input = nengo.Node(np.zeros(80*56))
x = nengo_dl.Layer(tf.keras.layers.Conv2D(32,3,2))(input,shape_in=(80,56,1))
x = nengo_dl.Layer(neuron_type)(x)

x = nengo_dl.Layer(tf.keras.layers.Conv2D(64,3,2))(x,shape_in=(39, 27, 32))
x = nengo_dl.Layer(neuron_type)(x)

x = nengo_dl.Layer(tf.keras.layers.Conv2D(128,3,2))(x,shape_in=(19, 13, 64))
x = nengo_dl.Layer(neuron_type)(x)

x = nengo_dl.Layer(tf.keras.layers.Conv2D(256,3,2))(x,shape_in=(9, 6, 128))
x = nengo_dl.Layer(neuron_type)(x)

output = nengo_dl.Layer(tf.keras.layers.Dense(num_classes))(x)
out_p = nengo.Probe(output,label='out_p')
out_p_filt = nengo.Probe(output,synapse=0.1,label='out_p_filt')

def classification_accuracy(y_true, y_pred):
return tf.metrics.sparse_categorical_accuracy(y_true[:, -1], y_pred[:, -1])

minibatch_size = 100
sim = nengo_dl.Simulator(net, minibatch_size=minibatch_size)
sim.load_params("models/model_snn")

#%% ACC vs Timesteps------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

sim.compile(loss={out_p_filt: classification_accuracy})

images = X_valid.reshape((X_valid.shape[0], -1))
ylabels = Y_valid

ACC = []
trange = range(1,31)

for n_steps in trange:
test_images = np.tile(images[:, None, :], (1, n_steps, 1))
test_labels = np.tile(ylabels[:, None, None], (1, n_steps, 1))

acc = sim.evaluate(test_images, {out_p_filt: test_labels}, verbose=1)["loss"]
print('Timesteps = ', n_steps, ' -> Accuracy = ', acc)
ACC.append(acc)

plt.figure(figsize=(8, 4))
plt.plot(trange, ACC,linewidth=2)
plt.legend(labels, loc="right")
plt.xlabel("Timesteps")
plt.ylabel("Prediction Accuracy")
plt.tight_layout()
plt.rc('font', size=14)

[drasmuss]

I tried running a memory profiler on your code, but I wasn't seeing the effect you describe (in fact, predict was using less memory than evaluate). However, one thing to keep in mind is that evaluate and predict require TensorFlow to build slightly different graphs, each of which will consume memory. So if you were calling evaluate and predict in your loop that would use more memory than just calling evaluate alone (but it's just because you're doing two things now instead of one, it doesn't have anything to do with predict vs evaluate specifically). So that might explain the effect you were seeing.

[kebitmatf]

Dear Drasmuss,

Thank you for checking the issue.
There are two issues:

1. Attaching probe after each layer will lead to full memory if calling sim.predict
   For example:
   nengo_dl.configure_settings(stateful=False)
   input = nengo.Node(np.zeros(80*56))
   x = nengo_dl.Layer(tf.keras.layers.Conv2D(32,3,2))(input,shape_in=(80,56,1))
   x = nengo_dl.Layer(neuron_type)(x)
   xp1 = nengo.Probe(x)

x = nengo_dl.Layer(tf.keras.layers.Conv2D(64,3,2))(x,shape_in=(39, 27, 32))
x = nengo_dl.Layer(neuron_type)(x)
xp2 = nengo.Probe(x)

x = nengo_dl.Layer(tf.keras.layers.Conv2D(128,3,2))(x,shape_in=(19, 13, 64))
x = nengo_dl.Layer(neuron_type)(x)
xp3 = nengo.Probe(x)

x = nengo_dl.Layer(tf.keras.layers.Conv2D(256,3,2))(x,shape_in=(9, 6, 128))
x = nengo_dl.Layer(neuron_type)(x)

output = nengo_dl.Layer(tf.keras.layers.Dense(num_classes))(x)
out_p = nengo.Probe(output,label='out_p')
out_p_filt = nengo.Probe(output,synapse=0.1,label='out_p_filt')

Then call: sim.predict(X_valid) will make full GPU memory. It's seem that the nengo.Probe make trouble?. My X_valid has size of 5400x80x56x1 and nsteps = 40, mini_batchsize = 100. I have GPU with RAM = 24GB (NVIDIA A5000).

2. The GPU will get full memory if I repeatly call sim.predict in a loop as below code:
   _for n_steps in [30,31,32,33,34,35]:
   test_images = np.tile(images[:, None, :], (1, n_steps, 1))
   test_labels = np.tile(ylabels[:, None, None], (1, n_steps, 1))

   acc = sim.evaluate(test_images, {out_p_filt: test_labels}, verbose=1)["loss"]
   print('Timesteps = ', n_steps, ' -> Accuracy = ', acc)
   ACC.append(acc)_
   I think the sim.predict make a new copy of data to GPU memory, so the memory is getting double after a call sim.predict

Thank you!

[kebitmatf]

Here I produce two codes for the two errors.
https://drive.google.com/drive/folders/1jsdqj9F-FHQD9xh8kne1CI0PIbUQpJ2D?usp=sharing
Thank you very much!!!!

[drasmuss]

1. Attaching probe after each layer will lead to full memory

This is expected. The probe data needs to be stored in memory, so the more probes you have the more memory you will require.

2. The GPU will get full memory if I repeatly call sim.predict in a loop

This is also expected. TensorFlow retains information in memory throughout the lifetime of the process, so repeatedly calling predict/evaluate will increase the memory usage over time. You would see this same effect using a regular, non-NengoDL Keras model (although memory would accumulate more slowly, since the non-neuromorphic models are simpler). You could try using tf.keras.backend.clear_session https://www.tensorflow.org/api_docs/python/tf/keras/backend/clear_session to free up resources.

[kebitmatf]

Dear Drasmuss,

Problem 1 is understood.
But, still "tf.keras.backend.clear_session"does not solve the second problem.
Event I added more: "delete sim" and "tf.compat.v1.reset_default_graph()"