Skip to content

Commit

Permalink
CI/utils.py: Better diagnostic output when tests fail
Browse files Browse the repository at this point in the history 
Improve the developer experience when files do no show up
in the bugball output archive and the associated test fails.

Signed-off-by: Bernhard Kaindl <[email protected]>
  • Loading branch information
@bernhardkaindl
bernhardkaindl committed Jan 12, 2024
1 parent 1aae840 commit 57f4b55
Showing 1 changed file with 8 additions and 4 deletions.
12 changes: 8 additions & 4 deletions tests/integration/utils.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,19 +51,23 @@ def check_file(path):


def assert_content_from_dom0_template(path, control_path=None):
"""Compare the contents of output directories or files with the test's Dom0 template directories"""
assert path[0] != "/"
"""Check the given path against the files from the test's Dom0 template"""

assert path[0] != "/" # We expect a relative path in the report archive
control = BUGTOOL_DOM0_TEMPL + (control_path or path)
print(control)
if os.path.isdir(path):
# path is a directory, compare it recursively using dircmp():
result = filecmp.dircmp(path, control)
if result.diff_files or result.right_only: # pragma: no cover
print(result.report)
raise RuntimeError("Missing or Differing files found in " + path)
else:
if not os.path.exists(path):
raise AssertionError("/%s is missing in the report archive" % path)
if not filecmp.cmp(path, control):
os.system("cat " + path) # pragma: no cover
raise RuntimeError(control)
os.system("diff -u %s %s" % (path, control)) # pragma: no cover
raise AssertionError("/%s from report has different content" % path)
# Remove verified output files/directories. Untested files will remain and cause the testcase to FAIL:
try:
os.unlink(path)
Expand Down

2 comments on commit 57f4b55

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coverage

https://github.com/marketplace/actions/pytest-coverage-comment
FileStmtsMissCoverMissing
__init__.py00100% 
conftest.py300100% 
test_dir_list.py110100% 
test_fs_funcs.py100100% 
test_load_plugins.py70100% 
test_output.py430100% 
test_process_output.py80100% 
test_xapidb_filter.py170100% 
xen-bugtool152171652%95, 100–101, 410, 490–491, 493–494, 511, 517, 540, 551–552, 555–556, 602, 606–608, 612, 614–618, 620–623, 632–634, 637, 656–657, 660, 662–666, 668–669, 672–673, 677–678, 680–681, 685–688, 690–693, 696–698, 700–701, 705–707, 710–711, 720, 722, 724–727, 729–731, 733–734, 738–739, 741, 743–749, 753–754, 756–760, 765–773, 775–777, 779–781, 783–784, 786, 801, 803–807, 809–810, 812, 814, 816–839, 842–846, 848, 850–871, 874–875, 877, 879, 882, 884–893, 895, 897–900, 902–912, 914–930, 932–936, 938–944, 947–976, 978–988, 990–993, 995–1015, 1017–1020, 1022, 1024, 1026–1027, 1029–1030, 1032, 1047–1048, 1050–1051, 1055–1056, 1058–1059, 1061, 1063, 1065–1066, 1068–1069, 1071–1080, 1083–1089, 1092–1093, 1095–1096, 1105–1106, 1108–1109, 1113, 1135–1138, 1140–1144, 1146–1153, 1155, 1157–1160, 1162, 1165–1167, 1169–1175, 1178–1180, 1184–1185, 1187–1188, 1190–1192, 1195–1196, 1199, 1201–1202, 1204, 1206, 1208–1209, 1213–1216, 1219, 1222, 1225–1227, 1230–1234, 1237–1238, 1241–1244, 1250–1256, 1259–1263, 1265–1281, 1284–1292, 1295–1296, 1298, 1300–1301, 1303–1311, 1314, 1316–1319, 1321, 1323–1324, 1358, 1386, 1401, 1409–1410, 1414–1416, 1418, 1421, 1423–1425, 1427–1431, 1433, 1437, 1440, 1443–1449, 1451, 1453–1457, 1463–1465, 1467–1472, 1475–1479, 1483–1484, 1486, 1503–1504, 1506, 1508–1509, 1511–1513, 1515–1522, 1524–1529, 1533–1534, 1536, 1538, 1540, 1556–1557, 1559, 1561–1563, 1566–1567, 1569, 1571–1573, 1576–1577, 1579, 1581–1584, 1587–1597, 1600–1604, 1614, 1626, 1636, 1640, 1651, 1675–1677, 1689–1692, 1700, 1715, 1719–1723, 1728–1730, 1743, 1749, 1759–1765, 1782, 1803–1806, 1808–1809, 1821, 1881, 1900, 1902, 1904, 1907–1915, 1918–1919, 1922–1929, 1931–1932, 1934, 1938–1939, 1943, 1947–1949, 1953–1954, 1957, 1961, 1966, 1972–1973, 1975, 1979, 1981–1984, 1987–1997, 2001, 2003, 2088–2090, 2093–2095, 2107, 2126–2128, 2168–2172, 2179, 2181–2184, 2188, 2202
TOTAL164771656% 

Tests Skipped Failures Errors Time
11 0 💤 0 ❌ 0 🔥 0.832s ⏱️

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coverage

https://github.com/marketplace/actions/pytest-coverage-comment
FileStmtsMissCoverMissing
__init__.py00100% 
conftest.py300100% 
test_dir_list.py110100% 
test_fs_funcs.py100100% 
test_load_plugins.py70100% 
test_output.py430100% 
test_process_output.py80100% 
test_xapidb_filter.py170100% 
xen-bugtool152171652%95, 100–101, 410, 490–491, 493–494, 511, 517, 540, 551–552, 555–556, 602, 606–608, 612, 614–618, 620–623, 632–634, 637, 656–657, 660, 662–666, 668–669, 672–673, 677–678, 680–681, 685–688, 690–693, 696–698, 700–701, 705–707, 710–711, 720, 722, 724–727, 729–731, 733–734, 738–739, 741, 743–749, 753–754, 756–760, 765–773, 775–777, 779–781, 783–784, 786, 801, 803–807, 809–810, 812, 814, 816–839, 842–846, 848, 850–871, 874–875, 877, 879, 882, 884–893, 895, 897–900, 902–912, 914–930, 932–936, 938–944, 947–976, 978–988, 990–993, 995–1015, 1017–1020, 1022, 1024, 1026–1027, 1029–1030, 1032, 1047–1048, 1050–1051, 1055–1056, 1058–1059, 1061, 1063, 1065–1066, 1068–1069, 1071–1080, 1083–1089, 1092–1093, 1095–1096, 1105–1106, 1108–1109, 1113, 1135–1138, 1140–1144, 1146–1153, 1155, 1157–1160, 1162, 1165–1167, 1169–1175, 1178–1180, 1184–1185, 1187–1188, 1190–1192, 1195–1196, 1199, 1201–1202, 1204, 1206, 1208–1209, 1213–1216, 1219, 1222, 1225–1227, 1230–1234, 1237–1238, 1241–1244, 1250–1256, 1259–1263, 1265–1281, 1284–1292, 1295–1296, 1298, 1300–1301, 1303–1311, 1314, 1316–1319, 1321, 1323–1324, 1358, 1386, 1401, 1409–1410, 1414–1416, 1418, 1421, 1423–1425, 1427–1431, 1433, 1437, 1440, 1443–1449, 1451, 1453–1457, 1463–1465, 1467–1472, 1475–1479, 1483–1484, 1486, 1503–1504, 1506, 1508–1509, 1511–1513, 1515–1522, 1524–1529, 1533–1534, 1536, 1538, 1540, 1556–1557, 1559, 1561–1563, 1566–1567, 1569, 1571–1573, 1576–1577, 1579, 1581–1584, 1587–1597, 1600–1604, 1614, 1626, 1636, 1640, 1651, 1675–1677, 1689–1692, 1700, 1715, 1719–1723, 1728–1730, 1743, 1749, 1759–1765, 1782, 1803–1806, 1808–1809, 1821, 1881, 1900, 1902, 1904, 1907–1915, 1918–1919, 1922–1929, 1931–1932, 1934, 1938–1939, 1943, 1947–1949, 1953–1954, 1957, 1961, 1966, 1972–1973, 1975, 1979, 1981–1984, 1987–1997, 2001, 2003, 2088–2090, 2093–2095, 2107, 2126–2128, 2168–2172, 2179, 2181–2184, 2188, 2202
TOTAL164771656% 

Tests Skipped Failures Errors Time
11 0 💤 0 ❌ 0 🔥 0.831s ⏱️

Please sign in to comment.