Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallel + DataProvider: Only the first parameter set gets executed #501

Closed
mischcon opened this issue Oct 5, 2018 · 2 comments
Closed

Parallel + DataProvider: Only the first parameter set gets executed #501

mischcon opened this issue Oct 5, 2018 · 2 comments
Labels
State: Duplicate State: Won't fix For issues that will not be fixed

Comments

@mischcon
Copy link

mischcon commented Oct 5, 2018

Description
Assume you have a TestNG DataProvider that returns n parameter sets.

If you have a parameterized test and are running the test / DataProvider sequential the test gets executed with each and every of the n parameter sets provided by the DataProvider (expected behavior).

If you have the same setup, but are running the test / DataProvider in parallel only the first parameter sets gets executed, but n times (definitely not the expected behavior).

Citrus Version
2.7.7

Expected behavior
Test should be executed with each and every parameter set provided by the DataProvider (similar to sequential execution).

Actual behavior
Only the first parameter set gets executed, but n times.

Test case sample

public class TestIT extends TestNGCitrusTestRunner {

    @DataProvider(name = "dp", parallel = true)
    public static Object[][] dataProvider() {
        return Stream.of("a", "b", "c").map(s -> new Object[]{s}).toArray(Object[][]::new);
    }

    @Parameters("dataProviderValue")
    @CitrusTest
    @Test(dataProvider = "dp")
    public void testIT(String dataProviderValue) {
        echo("DATAPROVIDER VALUE: " + dataProviderValue);
    }
}

results in the output:

14:10:28,852 DEBUG        citrus.Citrus| Loading Citrus application properties
14:10:28,870 DEBUG BeanDefinitionReader| Loaded 0 bean definitions from location pattern [classpath*:citrus-context.xml]
14:10:29,332 INFO  port.LoggingReporter| 
14:10:29,332 INFO  port.LoggingReporter| ------------------------------------------------------------------------
14:10:29,332 INFO  port.LoggingReporter|        .__  __                       
14:10:29,332 INFO  port.LoggingReporter|   ____ |__|/  |________ __ __  ______
14:10:29,332 INFO  port.LoggingReporter| _/ ___\|  \   __\_  __ \  |  \/  ___/
14:10:29,332 INFO  port.LoggingReporter| \  \___|  ||  |  |  | \/  |  /\___ \ 
14:10:29,332 INFO  port.LoggingReporter|  \___  >__||__|  |__|  |____//____  >
14:10:29,332 INFO  port.LoggingReporter|      \/                           \/
14:10:29,332 INFO  port.LoggingReporter| 
14:10:29,332 INFO  port.LoggingReporter| C I T R U S  T E S T S  2.7.7
14:10:29,332 INFO  port.LoggingReporter| 
14:10:29,332 INFO  port.LoggingReporter| ------------------------------------------------------------------------
14:10:29,332 DEBUG port.LoggingReporter| BEFORE TEST SUITE
14:10:29,332 INFO  port.LoggingReporter| 
14:10:29,333 INFO  port.LoggingReporter| 
14:10:29,333 INFO  port.LoggingReporter| BEFORE TEST SUITE: SUCCESS
14:10:29,333 INFO  port.LoggingReporter| ------------------------------------------------------------------------
14:10:29,333 INFO  port.LoggingReporter| 
14:10:29,355 DEBUG t.TestContextFactory| Created new test context - using global variables: '{}'
14:10:29,355 DEBUG t.TestContextFactory| Created new test context - using global variables: '{}'
14:10:29,355 DEBUG t.TestContextFactory| Created new test context - using global variables: '{}'
14:10:29,363 INFO  port.LoggingReporter| 
14:10:29,363 INFO  port.LoggingReporter| ------------------------------------------------------------------------
14:10:29,363 INFO  port.LoggingReporter| 
14:10:29,364 INFO  port.LoggingReporter| ------------------------------------------------------------------------
14:10:29,363 INFO  port.LoggingReporter| 
14:10:29,364 INFO  port.LoggingReporter| ------------------------------------------------------------------------
14:10:29,364 DEBUG port.LoggingReporter| STARTING TEST TestIT.testIT <test.citrus.DataProviderParallel>
14:10:29,364 INFO  port.LoggingReporter| 
14:10:29,364 DEBUG port.LoggingReporter| STARTING TEST TestIT.testIT <test.citrus.DataProviderParallel>
14:10:29,364 INFO  port.LoggingReporter| 
14:10:29,364 DEBUG      citrus.TestCase| Initializing test case
14:10:29,364 DEBUG      citrus.TestCase| Initializing test case
14:10:29,364 DEBUG port.LoggingReporter| STARTING TEST TestIT.testIT <test.citrus.DataProviderParallel>
14:10:29,364 INFO  port.LoggingReporter| 
14:10:29,364 DEBUG      citrus.TestCase| Initializing test case
14:10:29,365 DEBUG  context.TestContext| Setting variable: citrus.test.name with value: 'TestIT.testIT'
14:10:29,365 DEBUG  context.TestContext| Setting variable: citrus.test.name with value: 'TestIT.testIT'
14:10:29,365 DEBUG  context.TestContext| Setting variable: citrus.test.name with value: 'TestIT.testIT'
14:10:29,365 DEBUG  context.TestContext| Setting variable: citrus.test.package with value: 'test.citrus.DataProviderParallel'
14:10:29,365 DEBUG  context.TestContext| Setting variable: citrus.test.package with value: 'test.citrus.DataProviderParallel'
14:10:29,365 DEBUG  context.TestContext| Setting variable: citrus.test.package with value: 'test.citrus.DataProviderParallel'
14:10:29,365 DEBUG      citrus.TestCase| Initializing test parameter 'dataProviderValue' as variable
14:10:29,365 DEBUG      citrus.TestCase| Initializing test parameter 'dataProviderValue' as variable
14:10:29,366 DEBUG      citrus.TestCase| Initializing test parameter 'dataProviderValue' as variable
14:10:29,366 DEBUG  context.TestContext| Setting variable: dataProviderValue with value: 'a'
14:10:29,366 DEBUG  context.TestContext| Setting variable: dataProviderValue with value: 'a'
14:10:29,366 DEBUG  context.TestContext| Setting variable: dataProviderValue with value: 'a'
14:10:29,366 DEBUG      citrus.TestCase| Test variables:
14:10:29,366 DEBUG      citrus.TestCase| Test variables:
14:10:29,366 DEBUG      citrus.TestCase| Test variables:
14:10:29,366 DEBUG      citrus.TestCase| citrus.test.name = TestIT.testIT
14:10:29,366 DEBUG      citrus.TestCase| dataProviderValue = a
14:10:29,366 DEBUG      citrus.TestCase| citrus.test.package = test.citrus.DataProviderParallel
14:10:29,366 DEBUG      citrus.TestCase| citrus.test.name = TestIT.testIT
14:10:29,366 DEBUG      citrus.TestCase| citrus.test.name = TestIT.testIT
14:10:29,366 DEBUG      citrus.TestCase| dataProviderValue = a
14:10:29,367 DEBUG      citrus.TestCase| citrus.test.package = test.citrus.DataProviderParallel
14:10:29,366 DEBUG      citrus.TestCase| dataProviderValue = a
14:10:29,367 INFO  port.LoggingReporter| 
14:10:29,366 INFO  port.LoggingReporter| 
14:10:29,367 DEBUG port.LoggingReporter| TEST STEP 1: echo
14:10:29,367 DEBUG      citrus.TestCase| citrus.test.package = test.citrus.DataProviderParallel
14:10:29,367 DEBUG port.LoggingReporter| TEST STEP 1: echo
14:10:29,367 INFO  port.LoggingReporter| 
14:10:29,367 DEBUG port.LoggingReporter| TEST STEP 1: echo
14:10:29,368 INFO    actions.EchoAction| DATAPROVIDER VALUE: a
14:10:29,368 INFO    actions.EchoAction| DATAPROVIDER VALUE: a
14:10:29,368 INFO  port.LoggingReporter| 
14:10:29,368 INFO    actions.EchoAction| DATAPROVIDER VALUE: a
14:10:29,368 DEBUG port.LoggingReporter| TEST STEP 1 SUCCESS
14:10:29,368 INFO  port.LoggingReporter| 
14:10:29,368 INFO  port.LoggingReporter| 
14:10:29,368 DEBUG port.LoggingReporter| TEST STEP 1 SUCCESS
14:10:29,368 DEBUG port.LoggingReporter| TEST STEP 1 SUCCESS
14:10:29,371 INFO  port.LoggingReporter| 
14:10:29,371 INFO  port.LoggingReporter| 
14:10:29,371 INFO  port.LoggingReporter| TEST SUCCESS TestIT.testIT (test.citrus.DataProviderParallel)
14:10:29,371 INFO  port.LoggingReporter| ------------------------------------------------------------------------
14:10:29,371 INFO  port.LoggingReporter| 
14:10:29,371 INFO  port.LoggingReporter| 
14:10:29,371 INFO  port.LoggingReporter| TEST SUCCESS TestIT.testIT (test.citrus.DataProviderParallel)
14:10:29,371 INFO  port.LoggingReporter| ------------------------------------------------------------------------
14:10:29,371 INFO  port.LoggingReporter| 
14:10:29,371 INFO  port.LoggingReporter| TEST SUCCESS TestIT.testIT (test.citrus.DataProviderParallel)
14:10:29,372 INFO  port.LoggingReporter| ------------------------------------------------------------------------
14:10:29,372 INFO  port.LoggingReporter| 
14:10:29,375 INFO  port.LoggingReporter| 
14:10:29,375 INFO  port.LoggingReporter| ------------------------------------------------------------------------
14:10:29,375 DEBUG port.LoggingReporter| AFTER TEST SUITE
14:10:29,375 INFO  port.LoggingReporter| 
14:10:29,375 INFO  port.LoggingReporter| 
14:10:29,376 INFO  port.LoggingReporter| AFTER TEST SUITE: SUCCESS
14:10:29,376 INFO  port.LoggingReporter| ------------------------------------------------------------------------
14:10:29,376 INFO  port.LoggingReporter| 
14:10:29,376 INFO  port.LoggingReporter| ------------------------------------------------------------------------
14:10:29,376 INFO  port.LoggingReporter| 
14:10:29,376 INFO  port.LoggingReporter| CITRUS TEST RESULTS
14:10:29,376 INFO  port.LoggingReporter| 
14:10:29,377 INFO  port.LoggingReporter|  TestIT.testIT .................................................. SUCCESS
14:10:29,377 INFO  port.LoggingReporter|  TestIT.testIT .................................................. SUCCESS
14:10:29,377 INFO  port.LoggingReporter|  TestIT.testIT .................................................. SUCCESS
14:10:29,377 INFO  port.LoggingReporter| 
14:10:29,377 INFO  port.LoggingReporter| TOTAL:	3
14:10:29,377 DEBUG port.LoggingReporter| SKIPPED:	0 (0.0%)
14:10:29,377 INFO  port.LoggingReporter| FAILED:	0 (0.0%)
14:10:29,377 INFO  port.LoggingReporter| SUCCESS:	3 (100.0%)
14:10:29,377 INFO  port.LoggingReporter| 
14:10:29,377 INFO  port.LoggingReporter| ------------------------------------------------------------------------
14:10:29,377 DEBUG  report.HtmlReporter| Generating HTML test report
14:10:29,378 DEBUG       util.FileUtils| Reading file resource: 'test-detail.html' (encoding is 'UTF-8')
14:10:29,388 DEBUG       util.FileUtils| Reading file resource: 'test-report.html' (encoding is 'UTF-8')
14:10:29,389 INFO  ctOutputFileReporter| Generated test report: target/citrus-reports/citrus-test-results.html
14:10:29,389 DEBUG report.JUnitReporter| Generating JUnit test report
14:10:29,390 DEBUG       util.FileUtils| Reading file resource: 'junit-test.xml' (encoding is 'UTF-8')
14:10:29,394 DEBUG       util.FileUtils| Reading file resource: 'junit-report.xml' (encoding is 'UTF-8')

===============================================
Default Suite
Total tests run: 3, Failures: 0, Skips: 0
===============================================


Process finished with exit code 0

As you can see the echo output is always

DATAPROVIDER VALUE: a

while it should be

DATAPROVIDER VALUE: a
DATAPROVIDER VALUE: b
DATAPROVIDER VALUE: c

I tried several things (running the test via an testng.xml file, running the test from my IDE, adding 'threadPoolSize' to the @test annotation, adding the @test annotation to the class rather than to the method) - everything failed.

It seems as if DataProvider + Parameterized Tests work only as expected when executed in sequential mode.

Is there anything that I forget to take into account when trying to run parameterized tests with Citrus in parallel?
Is there any workaround available?

Thanks!
@svettwer
Copy link
Contributor

svettwer commented Oct 6, 2018 •
edited
Loading

Hi!

At first: Thx for this awesome issue! 👏 🥇
Nevertheless, this is a duplicate of #350. 😉

Parallel data provider is not supported yet

#EDIT
Will close this issue to keep the timeline correct. Reference is provided, so we're good to go.

BR,
Sven

@svettwer svettwer closed this as completed Oct 6, 2018
@svettwer
Copy link
Contributor

svettwer commented Oct 6, 2018

Hi @mischcon!

Please vote for #350, so that we're able to prioritize.

BR,
Sven

@svettwer svettwer mentioned this issue Oct 6, 2018
Open
@svettwer svettwer added State: Won't fix For issues that will not be fixed State: Duplicate labels Oct 6, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
State: Duplicate State: Won't fix For issues that will not be fixed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mischcon @svettwer