Can't reproduce the result of Synthetic_Data.ipynb

[fanyimeng0]

Hello, When trying to reproduce the results in Synthetic_Data.ipynb, I encountered significant discrepancies. @tf.function, raises the following error:

TypeError: Input 'e' of 'SelectV2' Op has type float32 that does not match type int32 of argument 't'.

Consequently, I commented out #@tf.function and obtained some results, but the displayed materials do not match the real coefficients at all.
Synthetic_Data.zip

Environment Details

• Operating System: Debian 10
• Python Version: 3.8
• Sionna Version: 0.17.0
• TensorFlow Version: 2.13.0

[jhoydis]

Hi @fanyimeng0,

I was able to reproduce your issue. It seems to come from the Sionna 0.17 version. Could you try to downgrade to Sionna v0.16.0 and upgrade Tensorflow to 2.15.0 give it a try?

pip install sionna==0.16.2
pip install tensorflow==2.15.0

[fanyimeng0]

Hello, I downgrade my code version and ran it again, and I upgrade python to 3.10 since python3.8 does not support tensorflow 2.15. But this time I couldn't run it successfully. The issue is very similar to Issue4; more detail in the jupyter notebook I uploaded. Additionally, I don't quite understand why the version issue would cause such a significant difference in results. Although raytracing does not have convex function properties for material coefficients, the error between versions should not significantly affect the algorithm itself. I noticed this because I am trying to use RIS which is in Sionna 0.18.
Synthetic_Data.zip

[jhoydis]

The issue arises because we introduced a change in the way scene objects obtain ids. In 0.16, the start at 0, in 0.17 and later, they start at 1.

There is another bug in 0.17 which was fixed in 0.18.

If you use Sionna 0.16.2 and TF 2.15, it should work without any problem. I will push a fix for Sionna 0.18 later today.

[jhoydis]

I have pushed a fix that should ensure that it runs with Sionna v0.18. Please let me know if this solves the issue for you.

[fanyimeng0]

I have pushed a fix that should ensure that it runs with Sionna v0.18. Please let me know if this solves the issue for you.

After using Sionna version 0.18, the results seem to be more in line with expectations, as shown in the attached file. However, it still does not correspond to the results in your paper. I am trying to increase the number of training iterations to observe the effect. Another noteworthy point is that the code seems to run only on the GPU. When I run the code on the host machine (Sionna 0.18 TensorFlow CPU), I encounter the previous issues, but it works well in Docker (Sionna 0.18 TensorFlow GPU). I would like to know if this part of the code specifically needs to run on the GPU. Thank you.
Synthetic_Data.zip

[jhoydis]

I think that the discrepancy you observe is due to the random seed. But globally, the results look good as they should.
I just tried running the code on CPU and it seems to work (although significantly slower).

Which error do you get specifically on the CPU? Please copy and paste the full error trace.

[fanyimeng0]

Hello, this time my attachment includes the results of running on the CPU, as well as a YAML file that includes the environment. The "result" shows the results of running on the GPU. In my results, it seems that the "ceiling" and "walls" results got mixed up, and some other training results are also not right.
Test_018.zip

[jhoydis]

I have fixed the issue with the mixed-up objects in the plots.

Concerning the results, I think that it is not possible to regenerate exactly the same learning curves as in the paper using Sionna v0.18. However, the results should all be fairly similar. If the XPD coefficient of the floor has not converged after 3k iterations, you can simply train for little bit longer. The reason why this value converges slower is because the floor has a rather weak scattering coefficient and the impact of the KPD is very small.

[fanyimeng0]

Thanks for your response, it works well!