Enable verbose testing

[politas]

Change to core.testing to allow a choice to get more detailed testing results, using a temporary file to store test results.

Calling core.testing.run_test() with 'verbosity=` as a parameter will now open a temporary file, redirect the TextTestRunner output to it, and then feed the test results to the log. If no verbosity is supplied, the result is unchanged.

Sample testing results before:

2021-02-22 12:10:35.410 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - Running tests: 'CoreUtilsTestCase'
2021-02-22 12:10:39.020 [WARN ] [jsr223.jython.core.utils            ] - 'DogeDoesNotExist' is not in the ItemRegistry
2021-02-22 12:10:43.604 [WARN ] [jsr223.jython.core.utils            ] - 'DogeDoesNotExist' is not in the ItemRegistry
2021-02-22 12:10:43.609 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - {
  "run": 17,
  "errors": ,
  "failures": ,
  "skipped": []
}

Sample test results after:

2021-02-22 12:56:04.743 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - Running tests: 'CoreUtilsTestCase'
2021-02-22 12:56:08.352 [WARN ] [jsr223.jython.core.utils            ] - 'DogeDoesNotExist' is not in the ItemRegistry
2021-02-22 12:56:13.345 [WARN ] [jsr223.jython.core.utils            ] - 'DogeDoesNotExist' is not in the ItemRegistry
2021-02-22 12:56:13.352 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_ColorType_for_color_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 12:56:13.354 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_DateTimeType_for_date_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 12:56:13.355 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_DecimalType_for_number_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 12:56:13.357 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_PercentType_for_dimmer_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 12:56:13.358 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_StringType_for_string_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 12:56:13.359 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_ZonedDateTime_for_date_item_with_return_type (__main__.CoreUtilsTestCase) ... ok
2021-02-22 12:56:13.360 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_datetime_for_date_item_with_return_type (__main__.CoreUtilsTestCase) ... ok
2021-02-22 12:56:13.361 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_default_for_non_existing_item_with_default (__main__.CoreUtilsTestCase) ... ok
2021-02-22 12:56:13.362 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_expected_color_for_color_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 12:56:13.364 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_expected_datetime_for_datetime_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 12:56:13.365 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_expected_int_for_number_item_with_return_type (__main__.CoreUtilsTestCase) ... ok
2021-02-22 12:56:13.366 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_expected_number_for_number_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 12:56:13.367 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_expected_percentage_for_dimmer_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 12:56:13.368 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_expected_string_for_number_item_with_return_type (__main__.CoreUtilsTestCase) ... ok
2021-02-22 12:56:13.369 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_expected_string_for_string_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 12:56:13.371 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_farenheit_for_number_item_with_unit (__main__.CoreUtilsTestCase) ... ok
2021-02-22 12:56:13.372 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_none_for_non_existing_item_with_no_default (__main__.CoreUtilsTestCase) ... ok
2021-02-22 12:56:13.373 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - 
2021-02-22 12:56:13.374 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - ----------------------------------------------------------------------
2021-02-22 12:56:13.375 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - Ran 17 tests in 8.595s
2021-02-22 12:56:13.376 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - 
2021-02-22 12:56:13.377 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - OK
2021-02-22 12:56:13.379 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - {
  "run": 17,
  "errors": ,
  "failures": ,
  "skipped": []
}

[politas]

Here's an induced failure showing the verbose results with the non-verbose result format at the bottom. I'm not sure there is any value to including the JSON results along with the verbose results

2021-02-22 13:37:17.320 [INFO ] [jsr223.jython.OHL-test-core.utils   ] - Running tests: 'CoreUtilsTestCase'
2021-02-22 13:37:20.942 [WARN ] [jsr223.jython.core.utils            ] - 'DogeDoesNotExist' is not in the ItemRegistry
2021-02-22 13:37:25.048 [WARN ] [jsr223.jython.core.utils            ] - 'DogeDoesNotExist' is not in the ItemRegistry
2021-02-22 13:37:25.054 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_ColorType_for_color_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 13:37:25.055 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_DateTimeType_for_date_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 13:37:25.057 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_DecimalType_for_number_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 13:37:25.058 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_PercentType_for_dimmer_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 13:37:25.059 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_StringType_for_string_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 13:37:25.060 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_ZonedDateTime_for_date_item_with_return_type (__main__.CoreUtilsTestCase) ... ok
2021-02-22 13:37:25.061 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_datetime_for_date_item_with_return_type (__main__.CoreUtilsTestCase) ... ok
2021-02-22 13:37:25.062 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_default_for_non_existing_item_with_default (__main__.CoreUtilsTestCase) ... ok
2021-02-22 13:37:25.063 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_expected_color_for_color_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 13:37:25.064 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_expected_datetime_for_datetime_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 13:37:25.066 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_expected_int_for_number_item_with_return_type (__main__.CoreUtilsTestCase) ... FAIL
2021-02-22 13:37:25.067 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_expected_number_for_number_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 13:37:25.068 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_expected_percentage_for_dimmer_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 13:37:25.069 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_expected_string_for_number_item_with_return_type (__main__.CoreUtilsTestCase) ... ok
2021-02-22 13:37:25.070 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_expected_string_for_string_item (__main__.CoreUtilsTestCase) ... ok
2021-02-22 13:37:25.072 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_farenheit_for_number_item_with_unit (__main__.CoreUtilsTestCase) ... ok
2021-02-22 13:37:25.073 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - test_get_item_state_returns_none_for_non_existing_item_with_no_default (__main__.CoreUtilsTestCase) ... ok
2021-02-22 13:37:25.074 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - 
2021-02-22 13:37:25.075 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - ======================================================================
2021-02-22 13:37:25.076 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - FAIL: test_get_item_state_returns_expected_int_for_number_item_with_return_type (__main__.CoreUtilsTestCase)
2021-02-22 13:37:25.077 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - ----------------------------------------------------------------------
2021-02-22 13:37:25.078 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - Traceback (most recent call last):
2021-02-22 13:37:25.079 [ERROR] [jsr223.jython.OHL-test-core.utils   ] -   File "<script>", line 98, in test_get_item_state_returns_expected_int_for_number_item_with_return_type
2021-02-22 13:37:25.081 [ERROR] [jsr223.jython.OHL-test-core.utils   ] -   File "<script>", line 30, in get_item_state_returns_equality
2021-02-22 13:37:25.082 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - AssertionError: Function get_item_state on created 'Number' item should return '56' but returned '55'
2021-02-22 13:37:25.083 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - 
2021-02-22 13:37:25.084 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - ----------------------------------------------------------------------
2021-02-22 13:37:25.086 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - Ran 17 tests in 7.722s
2021-02-22 13:37:25.087 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - 
2021-02-22 13:37:25.088 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - FAILED (failures=1)
2021-02-22 13:37:25.089 [ERROR] [jsr223.jython.OHL-test-core.utils   ] - {
  "run": 17,
  "errors": ,
  "failures": {"name":"__main__.CoreUtilsTestCase.test_get_item_state_returns_expected_int_for_number_item_with_return_type", "stack":"Traceback (most recent call last):
  File \"<script>\", line 98, in test_get_item_state_returns_expected_int_for_number_item_with_return_type
  File \"<script>\", line 30, in get_item_state_returns_equality
AssertionError: Function get_item_state on created 'Number' item should return '56' but returned '55'
"},
  "skipped": []
}

[politas]

I might add a docstring to run_test() while I'm working on this.

[CrazyIvan359]

Interesting work, seems you know more about unittest than me as I've never used it!

Does the call unittest.TextTestRunner(None, verbosity=0) give the same result as unittest.TextTestRunner()? If so you don't need to be checking the value of out or verbosity in _run_test. In run_test set the default verbosity to 0 and just check if the value is > 1 and set out to be the temp file if it is. out should be defined with a value of None.

[politas]

Interesting work, seems you know more about unittest than me as I've never used it!

I haven't actually used it before, but I have used an equivalent in a C# project I was working on for a while, so I'm familiar with unit test reporting and what it looks like.

Does the call unittest.TextTestRunner(None, verbosity=0) give the same result as unittest.TextTestRunner()? If so you don't need to be checking the value of out or verbosity in _run_test. In run_test set the default verbosity to 0 and just check if the value is > 1 and set out to be the temp file if it is. out should be defined with a value of None.

Unfortunately, unittest.TextTestRunner(None) throws an AttributeError: 'NoneType' object has no attribute 'write', so we can't do that.