Fix #5486 & part of #5343: Introducing new wiki page for code coverage usage and limitations #5483
Conversation
APK & AAB differences analysisNote that this is a summarized snapshot. See the CI artifacts for detailed differences. DevExpand to see flavor specificsUniversal APKAPK file size: 19 MiB (old), 19 MiB (new), 0 bytes (No change) APK download size (estimated): 17 MiB (old), 17 MiB (new), 1 bytes (Added) Method count: 259155 (old), 259155 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 6806 (old), 6806 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 18 MiB (old), 18 MiB (new), 0 bytes (No change) Configuration hdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 49 KiB (old), 49 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 45 KiB (old), 45 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 86 KiB (old), 86 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 57 KiB (old), 57 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 63 KiB (old), 63 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 63 KiB (old), 63 KiB (new), 0 bytes (No change) AlphaExpand to see flavor specificsUniversal APKAPK file size: 11 MiB (old), 11 MiB (new), 0 bytes (No change) APK download size (estimated): 10 MiB (old), 10 MiB (new), 10 bytes (Added) Method count: 115689 (old), 115689 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 5774 (old), 5774 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 11 MiB (old), 11 MiB (new), 4 bytes (Removed) Configuration hdpiAPK file size: 43 KiB (old), 43 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 44 KiB (old), 44 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 38 KiB (old), 38 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 73 KiB (old), 73 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) BetaExpand to see flavor specificsUniversal APKAPK file size: 11 MiB (old), 11 MiB (new), 4 bytes (Removed) APK download size (estimated): 10 MiB (old), 10 MiB (new), 18 bytes (Added) Method count: 115695 (old), 115695 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 5774 (old), 5774 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 10 MiB (old), 10 MiB (new), 0 bytes (No change) Configuration hdpiAPK file size: 43 KiB (old), 43 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 44 KiB (old), 44 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 38 KiB (old), 38 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 73 KiB (old), 73 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) GaExpand to see flavor specificsUniversal APKAPK file size: 11 MiB (old), 11 MiB (new), 0 bytes (No change) APK download size (estimated): 10 MiB (old), 10 MiB (new), 34 bytes (Added) Method count: 115695 (old), 115695 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 5774 (old), 5774 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 10 MiB (old), 10 MiB (new), 0 bytes (No change) Configuration hdpiAPK file size: 43 KiB (old), 43 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 44 KiB (old), 44 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 38 KiB (old), 38 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 73 KiB (old), 73 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) |
1 similar comment
APK & AAB differences analysisNote that this is a summarized snapshot. See the CI artifacts for detailed differences. DevExpand to see flavor specificsUniversal APKAPK file size: 19 MiB (old), 19 MiB (new), 0 bytes (No change) APK download size (estimated): 17 MiB (old), 17 MiB (new), 1 bytes (Added) Method count: 259155 (old), 259155 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 6806 (old), 6806 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 18 MiB (old), 18 MiB (new), 0 bytes (No change) Configuration hdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 49 KiB (old), 49 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 45 KiB (old), 45 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 86 KiB (old), 86 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 57 KiB (old), 57 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 63 KiB (old), 63 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 63 KiB (old), 63 KiB (new), 0 bytes (No change) AlphaExpand to see flavor specificsUniversal APKAPK file size: 11 MiB (old), 11 MiB (new), 0 bytes (No change) APK download size (estimated): 10 MiB (old), 10 MiB (new), 10 bytes (Added) Method count: 115689 (old), 115689 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 5774 (old), 5774 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 11 MiB (old), 11 MiB (new), 4 bytes (Removed) Configuration hdpiAPK file size: 43 KiB (old), 43 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 44 KiB (old), 44 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 38 KiB (old), 38 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 73 KiB (old), 73 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) BetaExpand to see flavor specificsUniversal APKAPK file size: 11 MiB (old), 11 MiB (new), 4 bytes (Removed) APK download size (estimated): 10 MiB (old), 10 MiB (new), 18 bytes (Added) Method count: 115695 (old), 115695 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 5774 (old), 5774 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 10 MiB (old), 10 MiB (new), 0 bytes (No change) Configuration hdpiAPK file size: 43 KiB (old), 43 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 44 KiB (old), 44 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 38 KiB (old), 38 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 73 KiB (old), 73 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) GaExpand to see flavor specificsUniversal APKAPK file size: 11 MiB (old), 11 MiB (new), 0 bytes (No change) APK download size (estimated): 10 MiB (old), 10 MiB (new), 34 bytes (Added) Method count: 115695 (old), 115695 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 5774 (old), 5774 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 10 MiB (old), 10 MiB (new), 0 bytes (No change) Configuration hdpiAPK file size: 43 KiB (old), 43 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 44 KiB (old), 44 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 38 KiB (old), 38 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 73 KiB (old), 73 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) |
|
@BenHenning, the current update includes the new wiki page titled "Oppia-Android Code Coverage." The page on "Writing Effective Tests / Writing Tests with Good Behavioral Coverage" has not been added yet. Could you PTAL at the first wiki page and provide feedback on the flow and structure? |
There was a problem hiding this comment.
Oops I think I accidentally submitted the review halfway through.
Thanks @Rd4dev! I think this is an excellent start. I have a bunch of comments, but most are just direct wording change suggestions to improve clarity. I don't have many content suggestions for the code coverage page since I think it's really well put together. I have a bunch more thoughts on the writing good tests page, and probably will have more after the next pass as well.
APK & AAB differences analysisNote that this is a summarized snapshot. See the CI artifacts for detailed differences. DevExpand to see flavor specificsUniversal APKAPK file size: 19 MiB (old), 19 MiB (new), 0 bytes (No change) APK download size (estimated): 17 MiB (old), 17 MiB (new), 21 bytes (Removed) Method count: 259155 (old), 259155 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 6806 (old), 6806 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 18 MiB (old), 18 MiB (new), 4 bytes (Removed) Configuration hdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 49 KiB (old), 49 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 45 KiB (old), 45 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 86 KiB (old), 86 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 57 KiB (old), 57 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 63 KiB (old), 63 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 63 KiB (old), 63 KiB (new), 0 bytes (No change) AlphaExpand to see flavor specificsUniversal APKAPK file size: 11 MiB (old), 11 MiB (new), 0 bytes (No change) APK download size (estimated): 10 MiB (old), 10 MiB (new), 8 bytes (Removed) Method count: 115689 (old), 115689 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 5774 (old), 5774 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 11 MiB (old), 11 MiB (new), 0 bytes (No change) Configuration hdpiAPK file size: 43 KiB (old), 43 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 44 KiB (old), 44 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 38 KiB (old), 38 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 73 KiB (old), 73 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) BetaExpand to see flavor specificsUniversal APKAPK file size: 11 MiB (old), 11 MiB (new), 4 bytes (Removed) APK download size (estimated): 10 MiB (old), 10 MiB (new), 4 bytes (Removed) Method count: 115695 (old), 115695 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 5774 (old), 5774 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 10 MiB (old), 10 MiB (new), 4 bytes (Removed) Configuration hdpiAPK file size: 43 KiB (old), 43 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 44 KiB (old), 44 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 38 KiB (old), 38 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 73 KiB (old), 73 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) GaExpand to see flavor specificsUniversal APKAPK file size: 11 MiB (old), 11 MiB (new), 0 bytes (No change) APK download size (estimated): 10 MiB (old), 10 MiB (new), 3 bytes (Added) Method count: 115695 (old), 115695 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 5774 (old), 5774 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 10 MiB (old), 10 MiB (new), 4 bytes (Removed) Configuration hdpiAPK file size: 43 KiB (old), 43 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 44 KiB (old), 44 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 38 KiB (old), 38 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 73 KiB (old), 73 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) |
APK & AAB differences analysisNote that this is a summarized snapshot. See the CI artifacts for detailed differences. DevExpand to see flavor specificsUniversal APKAPK file size: 19 MiB (old), 19 MiB (new), 4 bytes (Added) APK download size (estimated): 17 MiB (old), 17 MiB (new), 1 bytes (Added) Method count: 259155 (old), 259155 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 6806 (old), 6806 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 18 MiB (old), 18 MiB (new), 0 bytes (No change) Configuration hdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 49 KiB (old), 49 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 45 KiB (old), 45 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 86 KiB (old), 86 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 57 KiB (old), 57 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 63 KiB (old), 63 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 63 KiB (old), 63 KiB (new), 0 bytes (No change) AlphaExpand to see flavor specificsUniversal APKAPK file size: 11 MiB (old), 11 MiB (new), 0 bytes (No change) APK download size (estimated): 10 MiB (old), 10 MiB (new), 13 bytes (Removed) Method count: 115689 (old), 115689 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 5774 (old), 5774 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 11 MiB (old), 11 MiB (new), 0 bytes (No change) Configuration hdpiAPK file size: 43 KiB (old), 43 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 44 KiB (old), 44 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 38 KiB (old), 38 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 73 KiB (old), 73 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) BetaExpand to see flavor specificsUniversal APKAPK file size: 11 MiB (old), 11 MiB (new), 4 bytes (Removed) APK download size (estimated): 10 MiB (old), 10 MiB (new), 27 bytes (Removed) Method count: 115695 (old), 115695 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 5774 (old), 5774 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 10 MiB (old), 10 MiB (new), 4 bytes (Removed) Configuration hdpiAPK file size: 43 KiB (old), 43 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 44 KiB (old), 44 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 38 KiB (old), 38 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 73 KiB (old), 73 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) GaExpand to see flavor specificsUniversal APKAPK file size: 11 MiB (old), 11 MiB (new), 4 bytes (Removed) APK download size (estimated): 10 MiB (old), 10 MiB (new), 20 bytes (Removed) Method count: 115695 (old), 115695 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 5774 (old), 5774 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 10 MiB (old), 10 MiB (new), 8 bytes (Removed) Configuration hdpiAPK file size: 43 KiB (old), 43 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 44 KiB (old), 44 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 38 KiB (old), 38 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 73 KiB (old), 73 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) |
As download artifact was accessed even without anything being uploaded causing an error
… centric philosophy sections
|
Thank you so much, @BenHenning, for the detailed instructions that provided clear direction. I’ve revamped the 'Writing Tests with Good Behavioral Coverage' section and incorporated all the suggested changes to the Oppia Android code coverage page. I’ve addressed all the review comments; could you PTAL? Additionally, the code coverage check required an additional skip files condition check to skip evaluations and upload of comments when no files are computed, so incorporated them to this PR as well. |
There was a problem hiding this comment.
Thanks @Rd4dev! I have only nits left in my comments code-wise.
Re: the 'working on develop' bit, what I'm actually referring to is how your previous PR broke the develop branch on Oppia Android. See this CI run: https://github.com/oppia/oppia-android/actions/runs/10375275721/job/28844581024 (accessible from the CI run on f9106d9).
Looking at your fork, it seems that merging in the example PR is still having some sort of issue (though now a different one): https://github.com/Rd4dev/Oppia-Android-Fork-from-Fork/actions/runs/10493357497/job/29067124928. When a PR is merged, it kicks off a set of CI runs (including running tests and coverage for the whole codebase). We need to make sure that these CI workflows run and pass, and that for code coverage its "add comment to pull request" workflow is correctly ignored (since that's what's currently failing for Oppia Android).
Does this help clarify?
Re: the filed issues, just to confirm: are there any other JaCoCo gap issues that we need to keep track of? Or, put another way, will we as a team be able to start the process of moving all of our tests toward 100% code coverage once all of #5481, #5501, and #5503 are addressed? Or, is there anything else that will need to be done (that we know now or can at least anticipate) before that can happen?
If you have any follow-up discussions let's try to address them over chat, if possible, to speed up the last remnants of this PR (since it seems like it should be mergeable on the next review).
|
@BenHenning, ahh actually thank you so much, for the reviews and also for pointing out to the exact failure check, only then I realized the workflow had a potential issue as it was including failure cases with the code coverage checks. :| So, to fix that I made the coverage_report workflow to only allow success conclusions from the code_coverage workflow. Reg. the confirmation on 'working on develop,' it has been observed that workflows using With both fixes applied, I ran two merge checks to confirm the 'working on develop',
Edit: Reassigning this issue for review as the remaining task is to confirm the functionality of Additional ContextTo give a bit more context on why the previous Merge with the fork reported as fail is, So, when I initially cloned the repo (not fork -- cloning it as a separate repo so I can't directly update it with upstream), I did not have the link to footnote wiki sections updated to the reports. But recently while having it run on [RunAllTests], it triggered the CoverageReportTest and caused unit tests to fail. and also in CI, But it should have even failed for Check Code Coverage Result!! And that is where I messed up keeping the Why was allowed_conclusions initially introduced? So initially I did only have comment upload functionality to the 2nd split workflow, (having evaluation and code coverage check result in the 1st workflow) while doing that it was necessary to even allow failures (as min threshold would fail the check) to let it complete the uploading job. But then I moved the evaluation and check coverage result to the 2nd workflow too (I will add the reasons to the description for later reference), while still having the As earlier it was allowing even failure cases, when unit tests failed (CoverageReporterTest), the code coverage wait action too failed, skipping all coverage runs (no pb files uploaded), but as the 2nd workflow (that does the final check for coverage result) still triggered with failure To fix this I had to remove allow_conclusions to only allow success cases: - allowed-conclusions: |
- success
- failureTo again test these:
To confirm the develop works:
Updating here as the coverage runs would still take some time to report, It ran its set of merge develop ci run checks - https://github.com/Rd4dev/Oppia-Android-Fork-from-Fork/actions/runs/10499496692, and it seems that only the workflows on Also, for the above run, I have only triggered the unit tests and code coverage checks to speed up testing. I ran another set of ci runs - [RunAllTests] with auto enable merge on while having 'unit test (gradle)', 'unit test(bazel)', coverage_report, oppiabot, code_coverage, repository_messaging workflows available and the same behavior can be seem as it ignored all the
As it can be seem from the actions above for the PR checks [RunAllTests] it does trigger
But with the base branch merge, it only triggers
The repository_messaging failed stack trace, and I hope that has nothing to do with this, so I ignored it. (or does it?) Also, I was unaware of the 2nd set of Merge base branch (develop) checks earlier, and with the behavior of the checks ignoring the |
APK & AAB differences analysisNote that this is a summarized snapshot. See the CI artifacts for detailed differences. DevExpand to see flavor specificsUniversal APKAPK file size: 19 MiB (old), 19 MiB (new), 4 bytes (Added) APK download size (estimated): 17 MiB (old), 17 MiB (new), 44 bytes (Added) Method count: 259155 (old), 259155 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 6806 (old), 6806 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 18 MiB (old), 18 MiB (new), 4 bytes (Added) Configuration hdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 49 KiB (old), 49 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 45 KiB (old), 45 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 86 KiB (old), 86 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 57 KiB (old), 57 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 63 KiB (old), 63 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 63 KiB (old), 63 KiB (new), 0 bytes (No change) AlphaExpand to see flavor specificsUniversal APKAPK file size: 11 MiB (old), 11 MiB (new), 0 bytes (No change) APK download size (estimated): 10 MiB (old), 10 MiB (new), 48 bytes (Added) Method count: 115689 (old), 115689 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 5774 (old), 5774 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 11 MiB (old), 11 MiB (new), 0 bytes (No change) Configuration hdpiAPK file size: 43 KiB (old), 43 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 44 KiB (old), 44 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 38 KiB (old), 38 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 73 KiB (old), 73 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) BetaExpand to see flavor specificsUniversal APKAPK file size: 11 MiB (old), 11 MiB (new), 0 bytes (No change) APK download size (estimated): 10 MiB (old), 10 MiB (new), 8 bytes (Removed) Method count: 115695 (old), 115695 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 5774 (old), 5774 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 10 MiB (old), 10 MiB (new), 4 bytes (Added) Configuration hdpiAPK file size: 43 KiB (old), 43 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 44 KiB (old), 44 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 38 KiB (old), 38 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 73 KiB (old), 73 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) GaExpand to see flavor specificsUniversal APKAPK file size: 11 MiB (old), 11 MiB (new), 4 bytes (Removed) APK download size (estimated): 10 MiB (old), 10 MiB (new), 9 bytes (Removed) Method count: 115695 (old), 115695 (new), 0 (No change) Features: 2 (old), 2 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 5774 (old), 5774 (new), 0 (No change)
Lesson assets: 111 (old), 111 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 10 MiB (old), 10 MiB (new), 8 bytes (Removed) Configuration hdpiAPK file size: 43 KiB (old), 43 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 44 KiB (old), 44 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 38 KiB (old), 38 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 73 KiB (old), 73 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) |
|
@BenHenning, Reg.
I'm currently trying to generate HTML reports for a bunch of files to figure out if they miss reporting anything inspite of being hit, few findings are:
Edit: Filed the issue - #5506 When the flow of execution is aborted by the termination statements, the line of code is not marked as covered. This can be seen with the use of RetrieveChangedFiles.kt fun main(args: Array<String>) {
if (args.size < 5) {
println(
"Usage: bazel run //scripts:retrieve_changed_files --" +
" <encoded_proto_in_base64> <path_to_bucket_name_output_file>" +
" <path_to_file_list_output_file> <path_to_test_target_list_output_file>"
)
println("Exiting...")
exitProcess(1)
}
...
}The generated report shows coverage as:
and also this seems to be a known issue with jacoco:
I think this could be worth filing too, will go ahead and file this and update this comment with any other findings.
While I believe this issue may be related to the same problem as described earlier #5506, I want to confirm this. In the DataProviderTestMonitor.kt file, we have the following lines: fun <T> ensureDataProviderExecutes(dataProvider: DataProvider<T>) {
// Waiting for a result is the same as ensuring the conditions are right for the provider to
// execute (since it must return a result if it's executed, even if it's pending).
val monitor = createMonitor(dataProvider)
monitor.waitForNextResult().also {
monitor.stopObservingDataProvider()
}.also {
// There must be an actual result for the provider to be successful.
println("Asserting...")
assertThat(it).isNotPending()
println("After Asserting...")
}
}The generated HTML report:
The coverage report indicates that the lines with If the assertion fails and throws an And also, if this is considered to be filed, should it be added to the existing termination issue #5506 or as a separate issue? (will add others as I find them) |
There was a problem hiding this comment.
Thanks @Rd4dev! Code looks good minus the final change to finalize the workflows to work correctly on both develop and forks (per our discussion today).
|
@BenHenning, moved back the evaluation, check jobs, fixed a new issue with merge concurrent cancellations and added the proof of runs below, can you PTAL?
and to fix this I updated the concurrency grouping with pr number and a fall back run id value: Available references for develop and pr branchesDevelop:GitHub href: PR Number: GitHub href: pr_number_group1
Proof of ci check runs for every mentioned caseFORK
UPSTREAM
Concurrency within PR
Concurrency in develop merge on: push
Filing coverage gaps and related issues
and reg. other coverage gaps, I haven't yet explored them completely... |
…s and fixing concurrency cancellations with merge develop
There was a problem hiding this comment.
Thanks @Rd4dev for the follow-up and extremely thorough verification analysis. This LGTM!
|
Unassigning @BenHenning since they have already approved the PR. |
|
Hi @Rd4dev, this PR is ready to be merged. Please address any remaining comments prior to merging, and feel free to merge this PR once the CI checks pass and you're happy with it. Thanks! |
…de coverage usage and limitations (oppia#5483) ## Explanation Fixes part of oppia#5343 Fixes oppia#5486 ### Project [PR 2.6 of Project 4.1] ### Changes Made - This PR introduces 2 new wiki pages: - Oppia Android Code Coverage - Writing tests with Good Behavioural Coverage - Fix to the comment upload feature permission issues with PRs created from the forked branches. - Split the code coverage workflow into 2 1. Core code coverage workflow to handle - Collection of changed files - Bucket partitioning - Run coverage in matrices 2. Comment upload workflow to handle - evaluation of reports - generation of md reports - uploading comments - Coverage Status Checks (as the later required `pull_request_target`) - Fix to oppia#5486 - The issue should have arisen as the pr got merged with the branch being deleted while the publish comment job still running, finding it hard to fetch the pr-issue number to publish a comment. - Now the Coverage Check Status was made to be dependent on the comment uploader ie. the final Coverage check job occurs only after the Evaluation and Comment jobs are done, so it will always have the pr reference) ``` check_coverage_results: name: Check Code Coverage Results needs: [ evaluate-code-coverage-reports, comment_coverage_report ] ``` # ### Reasons for splitting the code_coverage workflow The single code_coverage workflow was split into 1. **code_coverage** (to run coverages) 2. **coverage_report** (to generate and publish reports) ### Separating the comment upload job - The primary reason is the need to have ability to upload comments from PRs opened from a fork branch. - [`pull_request_target`](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#pull_request_target): For workflows that are triggered by the pull_request_target event, the GITHUB_TOKEN is granted read/write repository permission, even when it is triggered from a fork. - Workflows triggered by `pull_request_target` events are run in the context of the base branch. Since the base branch is considered trusted, workflows triggered by these events will always run, regardless of approval settings. [[GitHub Docs](https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-workflow-runs/approving-workflow-runs-from-public-forks#about-workflow-runs-from-public-forks)] ### Separating the evaluation / generation of report job - While initially it was split to help with 'skip files' - no files changed conditional check, with the introduction to 'SKIP' status check, it doesn't continue to serve the mentioned purpose. - But if we still have it as one workflow then the flow will work as such: Workflow 1: **code_coverage** - compute changed files - run coverage (needs compute changed files) - evaluate / generate md - code coverage check result Workflow 2: **coverage_report** - comment publication If no `.kt` files changes are detected - compute changed files - skips run coverage - skips evaluate / generate md - pass code coverage check result As the workflow was concluded as success, the 2nd workflow runs as, - failed comment publication (as no report is generation due to skip) But expectation is to still produce a pass check for the comment publication. (either to at least skip or upload a skip status as coverage comment report) - With moving it to a separate workflow allows us to not make the evaluation / generation jobs rely on the Run coverage job making it independently behave once the code_coverage workflows are completed successfully. - And it checks if pb files are generated and based on that it decides whether to generate PASS, FAIL or SKIP status checks. ### Separating the coverage status check result job There are 2 main reasons to moving it to new workflow. While it would still make sense to have it with the 1st workflow itself after Run coverage, the drawbacks are, - If the check coverage status result was left with the 1st workflow, then when the Run coverage job completes in the 1st workflow the coverage status check result is set to true on success even before the sibling part of upload comment is done, making it an incomplete status result. - oppia#5486 occurred as it lost its reference to the pr number, so making the coverage check status result dependent on (needs) the comment upload job should resolve this by only allowing the PR to be closed once the comment is uploaded. # **Todo:** - **[Done]** Add a new wiki page for "Writing effective tests / Writing test with good behavioural coverage" ## Essential Checklist - [x] The PR title and explanation each start with "Fix #bugnum: " (If this PR fixes part of an issue, prefix the title with "Fix part of #bugnum: ...".) - [x] Any changes to [scripts/assets](https://github.com/oppia/oppia-android/tree/develop/scripts/assets) files have their rationale included in the PR explanation. - [x] The PR follows the [style guide](https://github.com/oppia/oppia-android/wiki/Coding-style-guide). - [x] The PR does not contain any unnecessary code changes from Android Studio ([reference](https://github.com/oppia/oppia-android/wiki/Guidance-on-submitting-a-PR#undo-unnecessary-changes)). - [x] The PR is made from a branch that's **not** called "develop" and is up-to-date with "develop". - [x] The PR is **assigned** to the appropriate reviewers ([reference](https://github.com/oppia/oppia-android/wiki/Guidance-on-submitting-a-PR#clarification-regarding-assignees-and-reviewers-section)). ## For UI-specific PRs only If your PR includes UI-related changes, then: - Add screenshots for portrait/landscape for both a tablet & phone of the before & after UI changes - For the screenshots above, include both English and pseudo-localized (RTL) screenshots (see [RTL guide](https://github.com/oppia/oppia-android/wiki/RTL-Guidelines)) - Add a video showing the full UX flow with a screen reader enabled (see [accessibility guide](https://github.com/oppia/oppia-android/wiki/Accessibility-A11y-Guide)) - For PRs introducing new UI elements or color changes, both light and dark mode screenshots must be included - Add a screenshot demonstrating that you ran affected Espresso tests locally & that they're passing --------- Co-authored-by: Ben Henning <[email protected]>
…overage Wiki Page (#5511) <!-- READ ME FIRST: Please fill in the explanation section below and check off every point from the Essential Checklist! --> ## Explanation <!-- - Explain what your PR does. If this PR fixes an existing bug, please include - "Fixes #bugnum:" in the explanation so that GitHub can auto-close the issue - when this PR is merged. --> Fixes Part of #5343 ### This PR includes - Updated the incorrect link reference in the #5483 for the Oppia Android Code Coverage Page - Current Broken link: https://github.com/oppia/oppia-android-workflow/wiki/Oppia-Android-Code-Coverage - Correct link to wiki: https://github.com/oppia/oppia-android/wiki/Oppia-Android-Code-Coverage - Fixed an incorrect SKIP status that triggered even after 'Unit Tests - Bazel' failure causing it to miscalculate the pb file list as zero thereby posting a skip comment. - Solved by adding an additional condition for the evaluation job. ``` if: ${{ !cancelled() && needs.check_unit_tests_completed.result == 'success'}} ``` - The code coverage comment got triggered even after assignment changes due to the presence of 'assigned' in the on `pull_request_target` triggered, fixed it by removing the 'assigned' trigger. - Utilized 'HtmlEscapers' to escape HTML characters from source code lines in CoverageReporter.kt as strings those already included html tags overlapped / conflicted with the report html templates. - Added limitations section to the 'Oppia-Android-Code-Coverage' wiki page - Filed all possible coverage gaps that exist right now based on the available pass cases. ## Essential Checklist <!-- Please tick the relevant boxes by putting an "x" in them. --> - [x] The PR title and explanation each start with "Fix #bugnum: " (If this PR fixes part of an issue, prefix the title with "Fix part of #bugnum: ...".) - [x] Any changes to [scripts/assets](https://github.com/oppia/oppia-android/tree/develop/scripts/assets) files have their rationale included in the PR explanation. - [x] The PR follows the [style guide](https://github.com/oppia/oppia-android/wiki/Coding-style-guide). - [x] The PR does not contain any unnecessary code changes from Android Studio ([reference](https://github.com/oppia/oppia-android/wiki/Guidance-on-submitting-a-PR#undo-unnecessary-changes)). - [x] The PR is made from a branch that's **not** called "develop" and is up-to-date with "develop". - [x] The PR is **assigned** to the appropriate reviewers ([reference](https://github.com/oppia/oppia-android/wiki/Guidance-on-submitting-a-PR#clarification-regarding-assignees-and-reviewers-section)). ## For UI-specific PRs only <!-- Delete these section if this PR does not include UI-related changes. --> If your PR includes UI-related changes, then: - Add screenshots for portrait/landscape for both a tablet & phone of the before & after UI changes - For the screenshots above, include both English and pseudo-localized (RTL) screenshots (see [RTL guide](https://github.com/oppia/oppia-android/wiki/RTL-Guidelines)) - Add a video showing the full UX flow with a screen reader enabled (see [accessibility guide](https://github.com/oppia/oppia-android/wiki/Accessibility-A11y-Guide)) - For PRs introducing new UI elements or color changes, both light and dark mode screenshots must be included - Add a screenshot demonstrating that you ran affected Espresso tests locally & that they're passing
Explanation
Fixes part of #5343
Fixes #5486
Project
[PR 2.6 of Project 4.1]
Changes Made
(as the later required
pull_request_target)Reasons for splitting the code_coverage workflow
The single code_coverage workflow was split into
Separating the comment upload job
pull_request_target: For workflows that are triggered by the pull_request_target event, the GITHUB_TOKEN is granted read/write repository permission, even when it is triggered from a fork.pull_request_targetevents are run in the context of the base branch. Since the base branch is considered trusted, workflows triggered by these events will always run, regardless of approval settings. [GitHub Docs]Separating the evaluation / generation of report job
Workflow 1: code_coverage
Workflow 2: coverage_report
If no
.ktfiles changes are detectedAs the workflow was concluded as success, the 2nd workflow runs as,
But expectation is to still produce a pass check for the comment publication. (either to at least skip or upload a skip status as coverage comment report)
Separating the coverage status check result job
There are 2 main reasons to moving it to new workflow. While it would still make sense to have it with the 1st workflow itself after Run coverage, the drawbacks are,
Todo:
Essential Checklist
For UI-specific PRs only
If your PR includes UI-related changes, then: