Add stability tests

[trungleduc]

References

Add Playwright tests to test the outputs of notebook with bugged widgets.

Code changes

• Add a library of bugged widgets to tests/widget_lib
• Separate notebook rendering tests and stability tests in two Playwright projects.
• Update github action configuration.

[github-actions]

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

Results table

Test file	basics.ipynb	bqplot.ipynb	dashboard.ipynb	gridspecLayout.ipynb	interactive.ipynb	ipympl.ipynb	ipyvolume.ipynb	multiple_widgets.ipynb	query-strings.ipynb	reveal.ipynb
Render
chromium
actual	2356 <- [2383 - 2400 - 2498] -> 2821	2297 <- [2307 - 2385 - 2407] -> 2665	2468 <- [2508 - 2564 - 2615] -> 2816	3019 <- [3028 - 3078 - 3148] -> 3244	1878 <- [1914 - 1922 - 2003] -> 2129	3365 <- [3377 - 3390 - 3539] -> 3924	7111 <- [8080 - 8418 - 8581] -> 8974	9690 <- [9704 - 9733 - 9754] -> 9795	1249 <- [1270 - 1281 - 1325] -> 1343	2322 <- [2332 - 2341 - 2355] -> 2375
expected	3379 <- [3442 - 3517 - 3701] -> 3876	2976 <- [3227 - 3321 - 3421] -> 3604	3608 <- [3623 - 3709 - 3793] -> 3825	4453 <- [4453 - 4523 - 4661] -> 4748	2559 <- [2655 - 2656 - 2660] -> 2674	3982 <- [4079 - 4213 - 4356] -> 4743	12183 <- [18509 - 19553 - 20811] -> 21515	15319 <- [15660 - 15796 - 15912] -> 16056	1517 <- [1920 - 1997 - 2103] -> 2113

❗ Test metadata have changed

--- /dev/fd/63	2021-12-15 13:17:39.487296171 +0000
+++ /dev/fd/62	2021-12-15 13:17:39.487296171 +0000
@@ -4,37 +4,37 @@
     "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
-    "chromium": "97.0.4666.0"
+    "chromium": "94.0.4595.0"
   },
   "systemInformation": {
     "cpu": {
-      "brand": "Xeon® Platinum 8272CL",
+      "brand": "Xeon® E5-2673 v3",
       "cache": {
         "l1d": 65536,
         "l1i": 65536,
-        "l2": 2097152,
-        "l3": 36700160
+        "l2": 524288,
+        "l3": 31457280
       },
       "cores": 2,
       "family": "6",
-      "flags": "fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear",
+      "flags": "fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt md_clear",
       "governor": "",
       "manufacturer": "Intel®",
-      "model": "85",
+      "model": "63",
       "physicalCores": 2,
       "processors": 1,
       "revision": "",
       "socket": "",
-      "speed": 2.6,
+      "speed": 2.4,
       "speedMax": null,
       "speedMin": null,
-      "stepping": "7",
+      "stepping": "2",
       "vendor": "GenuineIntel",
       "virtualization": false,
       "voltage": ""
     },
     "mem": {
-      "total": 7289614336
+      "total": 7291699200
     },
     "osInfo": {
       "arch": "x64",
@@ -42,11 +42,11 @@
       "codename": "Focal Fossa",
       "codepage": "UTF-8",
       "distro": "Ubuntu",
-      "kernel": "5.11.0-1022-azure",
+      "kernel": "5.8.0-1040-azure",
       "logofile": "ubuntu",
       "platform": "linux",
       "release": "20.04.3 LTS",
-      "serial": "176d176198ea43028bfd21edea1d38d5",
+      "serial": "cfc067bfcb844f35865e279a1b0e66c5",
       "servicepack": "",
       "uefi": false
     }

[jtpio]

Thanks @trungleduc!

Looks like there is a small conflict to fix (probably we just need to replace the screenshot).

[trungleduc]

@jtpio updated

[jtpio]

Looks like the command might be outdated? Should we keep 1 retry?

[jtpio]

Looks good!

Left one comment above. Also there is one more conflict now that #1022 has been merged.

[jtpio]

Also there is one more conflict now that #1022 has been merged.

Just rebased to fix the conflict.

[trungleduc]

Thank @jtpio, I'm waiting for #984 to be merged before re-working on this PR since it will introduce more conflicts to #984. But it seems that we changed the calendar for #984 so this can be merged before.

[jtpio]

we changed the calendar for #984

What does that concretely mean? Would you mind (or @davidbrochart) posting a quick update in #984 or #977 if that refers to offline discussions? Thanks!

[davidbrochart]

I think #984 is not ready to be merged, but #1025 is.

[trungleduc]

This PR is ready again.

[jtpio]

Just to be sure we don't forget later, where is this issue being tracked?

[trungleduc]

Sorry, I forgot about it, where should I put the issue?