Linux and MacOS ARM64 Release Mode issues

[NoelFB]

Merging of #23 and #69
Probably related to the issues seen in #14

• Game will start but stop rendering on the titlescreen
• Overworld menu cards will stop rendering once idle
• Certain collision stops working (ex. Glass object)
• Only appears to happen in Release Mode with optimizations on.

[NoelFB]

I think to fully understand this we'd need to be able to debug the build and see what the actual values are. Potentially try compiling with .NET7 and see if the same issues occur (can be done by making Foster a local ProjectReference and changing both to .NET7).

[NoelFB]

I have made progress on this but it makes zero sense. Basically at some point Vector math breaks.

If I log out this:

Console.WriteLine($"{Vector2.One * scale}");

In Game.cs when it draws the texture to the screen, it will log <4,4> for a time, but after a few seconds it changes to log <4,0>.

If I then change the log to

Console.WriteLine($"{Vector2.One} * {scale} = {Vector2.One * scale}");

Then it will never break and always log <1,1> * 4 = <4,4>.

If I write new Vector2(scale, scale) instead of Vector2.One * scale it will also never break.

If I write Vector2.One * 4 instead of using the scale variable, it will also never break.

Basically, somehow using Vector2.One * scale will eventually explode and set the Y component to zero. If I use scale at all before this it behaves normally. Like some weird quantum bug lol

[NoelFB]

Alright, in a brand new console application with no dependencies on ARM64, this will eventually break:

using System.Numerics;

while (true)
{
    Thread.Sleep(10);
    Step(4.0f);
}

static void Step(float number)
{
    var variable = Vector2.One * number;
    if (variable.X != variable.Y)
        throw new Exception("ouch");
    Console.WriteLine($"{variable}");
}

[andymandias]

Can confirm the console application dies the same way (after ~80 iterations with Optimize set to true, appears to run indefinitely with Optimize false) on ARM64 Linux (Fedora Asahi Remix).

[NoelFB]

Yeah this appears to be a bug in the .NET 8 runtime. I submitted an issue there, will see what happens! I'm not sure there's much we can do on our end to work around it in the meantime.

[origamiswami]

Whenever I try to run the game on my Mac, it just spits out this error log. I'm on MacOS Big Sur v11.7.10, M1 chip. Not sure if this is a new issue or not so I'm just commenting here
ErrorLog.txt

[theofficialgman]

Potentially try compiling with .NET7 and see if the same issues occur (can be done by making Foster a local ProjectReference and changing both to .NET7).

tried that. fosterframework seems to use multiple features not available in C# 11
eg:

error CS9058: Feature 'primary constructors' is not available in C# 11.0. Please use language version 12.0 or greater.
error CS0246: The type or namespace name 'InlineArrayAttribute' could not be found (are you missing a using directive or an assembly reference?)
error CS0246: The type or namespace name 'InlineArray' could not be found (are you missing a using directive or an assembly reference?)

[theofficialgman]

Yeah this appears to be a bug in the .NET 8 runtime. I submitted an issue there, will see what happens! I'm not sure there's much we can do on our end to work around it in the meantime.

@NoelFB if you know where exactly each function could possibly trigger the bug you can disable jit optimization for that function with [MethodImpl(MethodImplOptions.NoOptimization)] before the function

I put it infront of every function in Source/Helpers/Utils.cs and that solved madeline phasing through glass as well as walls (only one wall on the far side of the map had I experienced that on before).

edit: I take that back, that should work if you do it on the proper functions. Those don't seem to be it though.

[theofficialgman]

also for users that have powerful enough CPUs to simply brute force it, you can disable the jit entirely with COMPlus_JITMinOpts=1 when launching any build of the game

it is possible there are other variables you could set to only disable that optimization https://github.com/dotnet/runtime/blob/512bcaf1b198a7234fad4d1791c558f698b1eb15/src/coreclr/inc/clrconfigvalues.h#L317

[theofficialgman]

only ARM64 simd can be disabled (which is much better than disabling the jit outright) with
DOTNET_EnableArm64AdvSimd=0

I have tried to set this somewhere in code but, as a non developer, I can not seem to get it to work. it works when setting it in terminal when running
@NoelFB maybe you can find a way to set that in code

[NoelFB]

I'm also not super sure how to set that and don't have access to a ARM64 pc again right now. I wonder if just setting an environment variable for that would work? Something like

Environment.SetEnvironmentVariable("DOTNET_EnableArm64AdvSimd", "0");

However looking up the docs for that again it seems like as of modern dotnet that may intentionally not work on non-windows environments... calling SDL_SetHint might make sense?

[DllImport("FosterFramework")]
public static extern void SDL_SetHint(string variable, string value);
// ... in progam.cs:
SDL_SetHint("DOTNET_EnableArm64AdvSimd", "0");

But I'm not sure if setting environment variables through SDL like that will work or not.

[theofficialgman]

I'm also not super sure how to set that and don't have access to a ARM64 pc again right now. I wonder if just setting an environment variable for that would work? Something like

Environment.SetEnvironmentVariable("DOTNET_EnableArm64AdvSimd", "0");

However looking up the docs for that again it seems like as of modern dotnet that may intentionally not work on non-windows environments

I tried that already before posting in Program.cs but that doesn't work. The problem is that this environment variable needs to be set before the dotnet process even starts up it seems otherwise the jit does not see it. For that reason I think it may not be possible besides actually setting it from terminal/system environment before running the binary (which is what I am doing now).

[NoelFB]

I suppose for ARM64 there could be a bash script to launch the game that does that, until dotnet gets updated?

[theofficialgman]

I suppose for ARM64 there could be a bash script to launch the game that does that, until dotnet gets updated?

Yeah that's what I do currently.
Since Linux users are already launching through the command line I think it's enough to just document the issue on the release notes for now.

[theofficialgman]

.NET 8.0.2 was released but the patch didn't make the cut https://github.com/dotnet/runtime/releases/tag/v8.0.2

.NET 9 Preview 1 was released also, maybe this could be used? https://github.com/dotnet/runtime/releases/tag/v9.0.0-preview.1.24080.9

Releases are made approximately every month, so it will be another 30 days before a release gets made.

[Alexa314]

I suppose for ARM64 there could be a bash script to launch the game that does that, until dotnet gets updated?

Yeah that's what I do currently. Since Linux users are already launching through the command line I think it's enough to just document the issue on the release notes for now.

Hi! So I don't quite know how to computer and I'm on mac; if there exists a simple bash script that would fix this, it would be really appreciated if that could be posted alongside the releases. I can't quite figure out how to set the environment variable properly. I can imagine that linux would require that level of knowhow but I've never actually made a bash script etc and so I think having an existing solution shared (or even just posted in here) would be really nice.

[theofficialgman]

@Alexa314 you can use the build from here https://github.com/theofficialgman/Celeste64/releases/tag/test4 which uses .NET 9 Preview 1 which has the bug fixed.

[theofficialgman]

.NET 8.0.3 came out a week ago. Can we get a new tag on foster (for the latest fixes there) and then update the implementation here and have a new release published? That should close this issue.

[NoelFB]

Yeah, that sounds great, I'll have a chance to do that in the next few days. Apologies for the slow response to various pull requests and issues - I've been traveling and not working my normal hours!