Hang when compilation fails after certain types of invalidation

[dsouzai]

It is possible for a hang to occur when a compilation consistently fails after a method is invalidated in specific circumstances. This does not apply to invalidation due to HCR. At present there are two situations when this can occur:

1. invalidation due to prex
2. invalidation when -Xnojit -Xnoaot is specified post-restore; this is especially noticeable under -XX:+DebugOnRestore (see cmdLineTester_criu_jitPostRestore Test -Xnojit -Xnoaot hang #20663).

The sequence of events is as follows:

1. A method (foo) gets compiled.
2. foo gets recompiled. This patches the startPC of the old body to jump to a helper that forwards execution to the new body.
3. foo gets invalidated (because of prex or -Xnojit -Xnoaot post-restore), resulting in the startPC of the recompiled foo getting patched to trigger a sync recompilation
4. The sync recompilation fails; this results in the startPC of the recompiled foo (from step 2) getting patched to revert to the interpreter.
5. Once in the interpreter, the extra field of the J9Method of foo gets reset to 0x1 (J9_STARTPC_NOT_TRANSLATED)
6. On the next (interpreted) invocation of foo, the interpreter triggers a compilation request (because the count is technically 0)
7. This compilation also fails, which results in the extra getting set to -3 (J9_JIT_NEVER_TRANSLATE)
8. Some compiled method calls the old foo (step 1) because it has a direct call to the original body of foo (step 1);
9. The helper cannot determine the new body to forward execution to (because the extra of the J9method of foo is set to -3); so it requests a compilation.
10. The compilation request sequence has an early return in j9jit_testarossa_err, because compilation happens asynchronously and because linkageInfo->isBeingCompiled() returns true; this flag never gets reset even after the recompilation succeeds in step 2.
11. Execution returns back to the startPC; return to step 9.

[github-actions]

Issue Number: 20750
Status: Open
Recommended Components: comp:jit, comp:vm, comp:jitserver

[mpirvu]

Once in the interpreter, the extra field of the J9Method of foo gets reset to 0x1 (J9_STARTPC_NOT_TRANSLATED)

I remember that for failed sync compilations due to PREX we do not touch the extra field. I remember it because I thought it was odd to have a method being interpreted while the extra field said it's compiled. Maybe things have changed at some point.

[mpirvu]

Looking at the code, on failure we call

#define J9JIT_REVERT_METHOD_TO_INTERPRETED(javaVM, method) \
	(javaVM)->internalVMFunctions->initializeMethodRunAddressNoHook((javaVM), (method))

and initializeMethodRunAddressNoHook() only touches j9method->methodRunAddress and leaves j9method->extra unchanged.
There is another function initializeMethodRunAddress that changes j9method->extra as well.

[dsouzai]

Looking at the code, on failure we call

#define J9JIT_REVERT_METHOD_TO_INTERPRETED(javaVM, method) \
	(javaVM)->internalVMFunctions->initializeMethodRunAddressNoHook((javaVM), (method))

and initializeMethodRunAddressNoHook() only touches j9method->methodRunAddress and leaves j9method->extra unchanged. There is another function initializeMethodRunAddress that changes j9method->extra as well.

This was very recently changed in #20387; initializeMethodRunAddressNoHook now does modify the j9method->extra, which is what exposed this hang.

[github-actions]

Issue Number: 20750
Status: Closed
Actual Components: bug, comp:jit
Actual Assignees: No one :(
PR Assignees: dsouzai