How to fix the high float of the loss during training？

[modestlyh]

the code :

class e2esystem():
....
 if training:
            bce = tf.nn.sigmoid_cross_entropy_with_logits(c, llr)
            bce = tf.reduce_mean(bce)

            return bce, llr, c
# training
ebno_db_min = 2.0
ebno_db_max = 15.0
    for i in range(num_training_iterations):

        # Sampling a batch of SNRs
        ebno_db = tf.random.uniform(shape=[], minval=ebno_db_min, maxval=ebno_db_max)
        # Forward pass
        with tf.GradientTape() as tape:
            bce, llr, c = model(training_batch_size, ebno_db, training=True)

            loss = bce
        # Computing and applying gradients
        weights = model.trainable_weights
        grads = tape.gradient(loss, weights)

        optimizer.apply_gradients(zip(grads, weights))

This is how the loss（BCE） changes。 How do I get the best weight?

[SebastianCa]

Hi @modestlyh,

It's difficult to give detailed advice without knowing your exact setup. However, it appears that you are currently sampling a single SNR value per batch, and that SNR may vary from iteration to iteration. In the extreme case, this might mean that one iteration uses an SNR of 15 dB while the next uses 2 dB. Even without any training progress, this can introduce significant variation in the observed BCE loss across iterations.

There are a couple of ways you might mitigate this issue:

1. Sample a random SNR for each sample in the batch instead of a single SNR for the entire batch. In other words, ensure that ebno_db has a shape of [batchsize] rather than [].

2. Evaluate the loss at a fixed SNR for the evaluation phase. For example, run the model with a fixed validation SNR every N iterations to get a more stable loss measurement.

I'm moving this thread to discussions as it is not an issue with Sionna.