-Dsurefire.rerunFailingTestsCount=5

[LorenzoBettini]

For #3057

Increasing retries from 3 to 5 gives more chances, as shown in the following.

For the time being, I'd switch to such a solution, waiting to understand how to remove flakiness.

Results:

Flakes: 
org.eclipse.xtend.ide.tests.compiler.ActiveAnnotationsInSameProjectTest.testActiveAnnotationInSameProjectInJar
  Run 1: ActiveAnnotationsInSameProjectTest.testActiveAnnotationInSameProjectInJar:479->assertNoErrorsInWorkspace:503->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 Couldn't find the mandatory library 'org.eclipse.xtext.xbase.lib' 2.8.0 or higher on the project's classpath.
  Run 2: PASS

org.eclipse.xtend.ide.tests.compiler.QuickDebugSourceInstallingCompilationParticipantTest.testIfThereIsAnyStatum
  Run 1: QuickDebugSourceInstallingCompilationParticipantTest.testIfThereIsAnyStatum:92 » NullPointer Cannot invoke "String.replace(java.lang.CharSequence, java.lang.CharSequence)" because "debug" is null
  Run 2: QuickDebugSourceInstallingCompilationParticipantTest.testIfThereIsAnyStatum:92 » NullPointer Cannot invoke "String.replace(java.lang.CharSequence, java.lang.CharSequence)" because "debug" is null
  Run 3: PASS

org.eclipse.xtend.ide.tests.compiler.ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef
  Run 1: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef:261->testResolvingXFunctionTypeRef:447->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 myannotation.MyAnnotation cannot be resolved to a type.
  Run 2: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef:261->testResolvingXFunctionTypeRef:421->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 org.junit.Assert cannot be resolved to a type.
  Run 3: PASS

org.eclipse.xtend.ide.tests.compiler.ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_2
  Run 1: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_2:271->testResolvingXFunctionTypeRef:421->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 org.eclipse.xtext.common.types.JvmGenericArrayTypeReference cannot be resolved to a type.
  Run 2: PASS

org.eclipse.xtend.ide.tests.compiler.ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_4
  Run 1: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_4:295->testResolvingXFunctionTypeRef:447->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 myannotation.MyAnnotation cannot be resolved to a type.
  Run 2: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_4:295->testResolvingXFunctionTypeRef:447->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 myannotation.MyAnnotation cannot be resolved to a type.
  Run 3: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_4:295->testResolvingXFunctionTypeRef:447->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 Couldn't find the mandatory library 'org.eclipse.xtext.xbase.lib' 2.8.0 or higher on the project's classpath.
  Run 4: PASS

org.eclipse.xtend.ide.tests.compiler.ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_5
  Run 1: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_5:307->testResolvingXFunctionTypeRef:447->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 myannotation.MyAnnotation cannot be resolved to a type.
  Run 2: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_5:307->testResolvingXFunctionTypeRef:421->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 org.eclipse.emf.ecore.EObject cannot be resolved to a type.
  Run 3: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_5:307->testResolvingXFunctionTypeRef:421->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 org.eclipse.emf.ecore.EObject cannot be resolved to a type.
  Run 4: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_5:307->testResolvingXFunctionTypeRef:447->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 myannotation.MyAnnotation cannot be resolved to a type.
  Run 5: PASS

org.eclipse.xtend.ide.tests.macros.Bug457681Test.testDaBug_1
  Run 1: Bug457681Test.testDaBug_1:133 Workspace contained errors: 
 - A.xtend:1 - Couldn't find the mandatory library 'org.eclipse.xtext.xbase.lib' 2.8.0 or higher on the project's classpath.(org.eclipse.xtend.ide.xtend.check.fast)
 - B.xtend:3 - annotation.MyAA cannot be resolved to a type.(org.eclipse.xtend.ide.xtend.check.fast)
 - B.xtend:6 - MyAA cannot be resolved to an annotation type.(org.eclipse.xtend.ide.xtend.check.fast)
 - B.xtend:1 - Couldn't find the mandatory library 'org.eclipse.xtext.xbase.lib' 2.8.0 or higher on the project's classpath.(org.eclipse.xtend.ide.xtend.check.fast)
  Run 2: PASS

org.eclipse.xtend.ide.tests.macros.MoreActiveAnnotationsTest.testBug461761_02
  Run 1: MoreActiveAnnotationsTest.testBug461761_02:325 Workspace contained errors: 
 - UserCode.xtend:4 - annotation.MyEnum cannot be resolved to a type.(org.eclipse.xtend.ide.xtend.check.fast)
 - UserCode.xtend:6 - DItemMini cannot be resolved to an annotation type.(org.eclipse.xtend.ide.xtend.check.fast)
 - UserCode.xtend:11 - DItemMini cannot be resolved to an annotation type.(org.eclipse.xtend.ide.xtend.check.fast)
 - UserCode.xtend:16 - DItemMini cannot be resolved to an annotation type.(org.eclipse.xtend.ide.xtend.check.fast)
 - UserCode.xtend:8 - DefaultsMyEnumVAL1 cannot be resolved to a type.(org.eclipse.xtend.ide.xtend.check.fast)
 - UserCode.xtend:13 - ExplicitDefaultsVAL2VAL1 cannot be resolved to a type.(org.eclipse.xtend.ide.xtend.check.fast)
 - UserCode.xtend:11 - The method or field VAL2 is undefined(org.eclipse.xtend.ide.xtend.check.fast)
 - UserCode.xtend:11 - The method or field MyEnum is undefined(org.eclipse.xtend.ide.xtend.check.fast)
 - UserCode.xtend:18 - MixedObjectVAL1 cannot be resolved to a type.(org.eclipse.xtend.ide.xtend.check.fast)
 - UserCode.xtend:1 - Couldn't find the mandatory library 'org.eclipse.xtext.xbase.lib' 2.8.0 or higher on the project's classpath.(org.eclipse.xtend.ide.xtend.check.fast)
  Run 2: PASS


Tests run: 3121, Failures: 0, Errors: 0, Skipped: 52, Flakes: 8

[cdietrich]

At other places we use the Flaky rule

[LorenzoBettini]

At other places we use the Flaky rule

But using the Maven options makes it easy to experiment with different counts without touching the code

[szarnekow]

Should we deprecated Flaky in favor of mvn capabiliities?

[LorenzoBettini]

Should we deprecated Flaky in favor of mvn capabiliities?

I think Flaky is still valuable when flakiness happens in the IDE as well :)
Or when one uses the "elephant" ;)

[LorenzoBettini]

I'd also like to experiment with the WorkspaceOperation: I wonder whether we don't properly create the projects for tests; if I understand correctly, we don't create them inside a WorkspaceOperation, is that right?

See also #3057 (comment)

[LorenzoBettini]

another new flake:

Flakes: 
org.eclipse.xtext.builder.impl.DisabledBuilderTest.testDisabledProject
   Run 1: DisabledBuilderTest.testDisabledProject:51
   Run 2: DisabledBuilderTest.testDisabledProject:51
   Run 3: PASS

again, due to a project created programmatically...

can we merge this in the meantime?

[LorenzoBettini]

Superseded by #3063