Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UI tests extremely flaky on macOS in GitHub Actions #3057

Closed
LorenzoBettini opened this issue May 21, 2024 · 5 comments · Fixed by #3063
Closed

UI tests extremely flaky on macOS in GitHub Actions #3057

LorenzoBettini opened this issue May 21, 2024 · 5 comments · Fixed by #3063
Milestone

Comments

@LorenzoBettini
Copy link
Contributor

Some examples:

Failures: 
org.eclipse.xtend.ide.tests.compiler.ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_4
  Run 1: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_4:295->testResolvingXFunctionTypeRef:447->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 Couldn't find the mandatory library 'org.eclipse.xtext.xbase.lib' 2.8.0 or higher on the project's classpath.
  Run 2: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_4:295->testResolvingXFunctionTypeRef:421->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 org.eclipse.emf.ecore.EObject cannot be resolved to a type.
  Run 3: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_4:295->testResolvingXFunctionTypeRef:421->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 org.eclipse.emf.ecore.EObject cannot be resolved to a type.
  Run 4: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_4:295->testResolvingXFunctionTypeRef:447->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 myannotation.MyAnnotation cannot be resolved to a type.

Flakes: 
org.eclipse.xtend.ide.tests.compiler.ActiveAnnotationsInSameProjectTest.testActiveAnnotationInSameProject
  Run 1: ActiveAnnotationsInSameProjectTest.testActiveAnnotationInSameProject:144->assertHasErrors:529->Assert.fail:89 Expected an error marker containing 'same project' on /test.project/src/mypack/Client.xtend but found Couldn't find the mandatory library 'org.eclipse.xtext.xbase.lib' 2.8.0 or higher on the project's classpath.
  Run 2: PASS

org.eclipse.xtend.ide.tests.compiler.CircularDepsBetweenJavaAndXtendTest.testJavaSignatureDependsOnXtend
  Run 1: CircularDepsBetweenJavaAndXtendTest.testJavaSignatureDependsOnXtend:242->Assert.assertEquals:633->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 expected:<0> but was:<3>
  Run 2: PASS

org.eclipse.xtend.ide.tests.compiler.ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef
  Run 1: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef:261->testResolvingXFunctionTypeRef:421->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 org.eclipse.emf.ecore.EObject cannot be resolved to a type.
  Run 2: PASS

org.eclipse.xtend.ide.tests.compiler.ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_3
  Run 1: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_3:283->testResolvingXFunctionTypeRef:447->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 myannotation.MyAnnotation cannot be resolved to a type.
  Run 2: PASS

org.eclipse.xtend.ide.tests.compiler.ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_5
  Run 1: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_5:307->testResolvingXFunctionTypeRef:421->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 org.eclipse.emf.ecore.EObject cannot be resolved to a type.
  Run 2: PASS

org.eclipse.xtend.ide.tests.compiler.ResolvingCrossReferenceDuringIndexingTest.testResolvingXtendAnnotationReference
  Run 1: ResolvingCrossReferenceDuringIndexingTest.testResolvingXtendAnnotationReference:246->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 myannotation.MyAnnotation cannot be resolved to a type.
  Run 2: ResolvingCrossReferenceDuringIndexingTest.testResolvingXtendAnnotationReference:246->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 myannotation.MyAnnotation cannot be resolved to a type.
  Run 3: ResolvingCrossReferenceDuringIndexingTest.testResolvingXtendAnnotationReference:225->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 The method or field Assert is undefined
  Run 4: PASS

org.eclipse.xtend.ide.tests.macros.JdtBasedProcessorProviderTest.testClassLoaderSeesAllUpstreamProjects
  Run 1: JdtBasedProcessorProviderTest.testClassLoaderSeesAllUpstreamProjects:137 Workspace contained errors: 
 - Lib.xtend:1 - Couldn't find the mandatory library 'org.eclipse.xtext.xbase.lib' 2.8.0 or higher on the project's classpath.(org.eclipse.xtend.ide.xtend.check.fast)
 - A.xtend:1 - Couldn't find the mandatory library 'org.eclipse.xtext.xbase.lib' 2.8.0 or higher on the project's classpath.(org.eclipse.xtend.ide.xtend.check.fast)
 - A.xtend:3 - annotation.MyAA cannot be resolved to an annotation type.(org.eclipse.xtend.ide.xtend.check.fast)
  Run 2: PASS


Tests run: 3121, Failures: 1, Errors: 0, Skipped: 52, Flakes: 7

Maybe there's some missed synchronization with project refresh and building.
Even when the build succeeds, there are lots of flakes.

I can try increase the number of retries.

@LorenzoBettini
Copy link
Contributor Author

Xtend IDE tests are the favorite, but sometimes it happens much before, in Xtext UI tests:

Results:

Failures: 
org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testBug463215_01
  Run 1: XtextContentAssistTest.testBug463215_01:114->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 2: XtextContentAssistTest.testBug463215_01:114->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 3: XtextContentAssistTest.testBug463215_01:114->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 4: XtextContentAssistTest.testBug463215_01:114->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.common.Terminals]> but was:<[]>

org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testBug463215_02
  Run 1: XtextContentAssistTest.testBug463215_02:125->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 2: XtextContentAssistTest.testBug463215_02:125->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 3: XtextContentAssistTest.testBug463215_02:125->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 4: XtextContentAssistTest.testBug463215_02:125->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>

org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testCompletionOnDatatypeReference_01
  Run 1: XtextContentAssistTest.testCompletionOnDatatypeReference_01:136->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 2: XtextContentAssistTest.testCompletionOnDatatypeReference_01:136->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 3: XtextContentAssistTest.testCompletionOnDatatypeReference_01:136->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 4: XtextContentAssistTest.testCompletionOnDatatypeReference_01:136->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>

org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testCompletionOnDatatypeReference_02
  Run 1: XtextContentAssistTest.testCompletionOnDatatypeReference_02:147->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 2: XtextContentAssistTest.testCompletionOnDatatypeReference_02:147->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 3: XtextContentAssistTest.testCompletionOnDatatypeReference_02:147->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 4: XtextContentAssistTest.testCompletionOnDatatypeReference_02:147->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>

org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testCompletionOnDatatypeReference_03
  Run 1: XtextContentAssistTest.testCompletionOnDatatypeReference_03:155->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 2: XtextContentAssistTest.testCompletionOnDatatypeReference_03:155->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 3: XtextContentAssistTest.testCompletionOnDatatypeReference_03:155->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 4: XtextContentAssistTest.testCompletionOnDatatypeReference_03:155->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>

org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testCompletionOnDatatypeReference_04
  Run 1: XtextContentAssistTest.testCompletionOnDatatypeReference_04:161->Assert.fail:89 No such proposal: Terminals - org.eclipse.xtext.common.Terminals Found: [@, @Deprecated, @Exported, @Final, @Override]
  Run 2: XtextContentAssistTest.testCompletionOnDatatypeReference_04:161->Assert.fail:89 No such proposal: Terminals - org.eclipse.xtext.common.Terminals Found: [@, @Deprecated, @Exported, @Final, @Override]
  Run 3: XtextContentAssistTest.testCompletionOnDatatypeReference_04:161->Assert.fail:89 No such proposal: Terminals - org.eclipse.xtext.common.Terminals Found: [@, @Deprecated, @Exported, @Final, @Override]
  Run 4: XtextContentAssistTest.testCompletionOnDatatypeReference_04:161->Assert.fail:89 No such proposal: Terminals - org.eclipse.xtext.common.Terminals Found: [@, @Deprecated, @Exported, @Final, @Override]


Tests run: 267, Failures: 6, Errors: 0, Skipped: 1

@cdietrich
Copy link
Contributor

See also eclipse-mwe/mwe#270

@LorenzoBettini
Copy link
Contributor Author

I guess yes...

for Xtend UI tests I can increase the test retries.

For XtextContentAssistTest maybe that won't help, because the project is created only once and I guess once the build has been done it won't be retried:

	@Before
	public void doSetupProject() throws Exception{
		if(project == null){
			with(doGetSetup());
			PluginProjectFactory projectFactory = getInjector().getInstance(PluginProjectFactory.class);
			projectFactory.setProjectName(TEST_PROJECT);
			projectFactory.addFolders(Collections.singletonList("src"));
			projectFactory.addBuilderIds(XtextProjectHelper.BUILDER_ID, JavaCore.BUILDER_ID, "org.eclipse.pde.ManifestBuilder",
					"org.eclipse.pde.SchemaBuilder");
			projectFactory.addProjectNatures(
					XtextProjectHelper.NATURE_ID, JavaCore.NATURE_ID, "org.eclipse.pde.PluginNature");
			projectFactory.addRequiredBundles(Lists.newArrayList("org.eclipse.xtext"));
			project = projectFactory.createProject(new NullProgressMonitor(), null);
			get(SyncUtil.class).waitForBuild(new NullProgressMonitor());
		}
	}

maybe moving the waitForBuild outside the if?

On a side note, in my projects things improved a lot concerning flakiness in these situations, if not completely fixed, by stopping creating projects programmatically and by importing existing ad-hoc projects for testing, e.g., https://github.com/LorenzoBettini/edelta/blob/master/edelta.parent/edelta.ui.tests/src/edelta/ui/tests/EdeltaContentAssistTest.java#L57, using https://github.com/LorenzoBettini/edelta/blob/master/edelta.parent/edelta.ui.tests/src/edelta/ui/tests/utils/ProjectImportUtil.java

Moreover, I wonder whether project creation in projectFactory happen in a workspace operation in a sort of synchronized way.

@LorenzoBettini
Copy link
Contributor Author

I'm re-opening it: there are still failures on macOS :'(

let's see whether increasing the retries makes things better, and then we'll investigate further...

@LorenzoBettini
Copy link
Contributor Author

I'm closing this, since I don't see that anymore

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants