Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: fix tests for lower API levels #145

Open
wants to merge 68 commits into
base: test/active-session
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
5dd75ff
fix bottom bar test
matthiasemde Dec 15, 2024
c34d15f
fix all instrumented tests for lower api levels
matthiasemde Dec 15, 2024
d784a1a
enable all api levels in CI
matthiasemde Dec 15, 2024
5f27da4
fix matrix range
matthiasemde Dec 15, 2024
2b81d6d
always upload test results
matthiasemde Dec 15, 2024
9e200f4
add await idle in flaky test
matthiasemde Dec 15, 2024
02fd1fb
improve build command so test can run immediately
matthiasemde Dec 15, 2024
a049416
add another awaitIdle() to library folder test
matthiasemde Dec 15, 2024
dc8229b
fix pipeline file
matthiasemde Dec 15, 2024
bf6b289
fix pipeline file
matthiasemde Dec 15, 2024
1de1e3f
resolve flaky test behavior by waiting until assertion runs or timeou…
matthiasemde Dec 17, 2024
90039f5
downgrade android-emulator-runner
matthiasemde Dec 17, 2024
1528bd0
try different gradle task for testing
matthiasemde Dec 17, 2024
40c3393
make ci more like android examples
matthiasemde Dec 18, 2024
2ec9a16
fix ci
matthiasemde Dec 18, 2024
aa961ee
fix ci
matthiasemde Dec 18, 2024
b4360cb
fix ci
matthiasemde Dec 18, 2024
d05300a
fix deprecations
matthiasemde Dec 18, 2024
d76828d
update dependencies
matthiasemde Dec 18, 2024
54b94a5
Simplify ci
matthiasemde Dec 19, 2024
a40747f
Revert "update dependencies"
matthiasemde Dec 21, 2024
e68b70c
Revert "fix deprecations"
matthiasemde Dec 21, 2024
1e8154a
Revert "chore(deps): update dependencies (#128)"
matthiasemde Dec 21, 2024
c38fe0a
test config from reactivecircus
matthiasemde Dec 21, 2024
d2edd50
fixup! fix all instrumented tests for lower api levels This includes …
matthiasemde Dec 22, 2024
88d16d0
use test orchestrator
matthiasemde Dec 22, 2024
918249a
wip
matthiasemde Dec 23, 2024
e42105b
store screenshots in additional_test_output dir so they get synced au…
matthiasemde Dec 30, 2024
8d72994
enable all gradle managed devices
matthiasemde Dec 30, 2024
e54f69c
revert ci back to runner
matthiasemde Dec 30, 2024
44417de
improve ci with caching
matthiasemde Dec 30, 2024
ab44a89
swap order of build and lint
matthiasemde Dec 30, 2024
1786e0b
minor improvements
matthiasemde Dec 30, 2024
20c03e4
embed screenshots into test report
matthiasemde Dec 30, 2024
6d5a29c
make sure screenshot embedding is only run for actual tests
matthiasemde Dec 30, 2024
2c93afd
test
matthiasemde Dec 30, 2024
7c07484
test
matthiasemde Dec 30, 2024
be85a07
test
matthiasemde Dec 30, 2024
1e155be
test
matthiasemde Dec 30, 2024
86edba8
test
matthiasemde Dec 30, 2024
18fb683
test
matthiasemde Dec 30, 2024
b17e8a1
test
matthiasemde Dec 30, 2024
2c51f1d
sabotage test to induce failure in ci
matthiasemde Dec 30, 2024
e229946
do not show snackbar on section delete
mipro98 Dec 30, 2024
282b10a
disable test for undoing sections for now
mipro98 Dec 30, 2024
f6dffef
fix directory
matthiasemde Dec 30, 2024
a42425b
fix directory
matthiasemde Dec 31, 2024
2fc211a
fix active session screen test
matthiasemde Dec 31, 2024
ac5a972
remove caching
matthiasemde Dec 31, 2024
df43a5b
run all tests
matthiasemde Dec 31, 2024
4e5f79a
improve test stability
matthiasemde Jan 1, 2025
b3f8be7
improve assertWithLease functionality
matthiasemde Jan 1, 2025
d0ed410
fix errors
matthiasemde Jan 1, 2025
fa44c74
try alternate way to register devices
matthiasemde Jan 1, 2025
1b655c9
disable api28
matthiasemde Jan 1, 2025
6ce7433
fix
matthiasemde Jan 1, 2025
77c23d5
fix
matthiasemde Jan 1, 2025
ef413a9
fix
matthiasemde Jan 1, 2025
d7307d5
add api27 and improve embedding job
matthiasemde Jan 1, 2025
ce6ab34
add api27 and improve embedding job
matthiasemde Jan 1, 2025
b00a8fc
remove api27
matthiasemde Jan 1, 2025
6ecd367
fix test by adding new vertical ordering assertion
matthiasemde Jan 1, 2025
aa694ee
revert some unnecessary changes back to main
matthiasemde Jan 2, 2025
1b266e3
revert some unnecessary changes back to main
matthiasemde Jan 2, 2025
63295f7
re-revert some changes which were actually necessary
matthiasemde Jan 2, 2025
b1ed939
adapt ci to new build caching
matthiasemde Jan 3, 2025
978fe15
remove obsolete code
matthiasemde Jan 3, 2025
2fdae4c
fixup! adapt ci to new build caching
matthiasemde Jan 3, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 0 additions & 22 deletions .github/actions/get-avd-info/action.yml

This file was deleted.

102 changes: 50 additions & 52 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,23 +20,23 @@ jobs:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with:
java-version: ${{ env.JAVA_VERSION }}
distribution: ${{ env.JAVA_DISTRO }}

# See https://community.gradle.org/github-actions/docs/setup-gradle/ for more information
- uses: gradle/actions/setup-gradle@v4 # creates build cache when on main branch
with:
build-scan-publish: true
build-scan-terms-of-use-url: "https://gradle.com/terms-of-service"
build-scan-terms-of-use-agree: "yes"
cache-encryption-key: ${{ secrets.GRADLE_CACHE_ENCRYPTION_KEY }}
dependency-graph: generate-and-submit # submit Github Dependency Graph info

- name: Build
run: ./gradlew --build-cache --configuration-cache --configuration-cache-problems=warn app:assembleDebug
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with:
java-version: ${{ env.JAVA_VERSION }}
distribution: ${{ env.JAVA_DISTRO }}

# See https://community.gradle.org/github-actions/docs/setup-gradle/ for more information
- uses: gradle/actions/setup-gradle@v4 # creates build cache when on main branch
with:
build-scan-publish: true
build-scan-terms-of-use-url: "https://gradle.com/terms-of-service"
build-scan-terms-of-use-agree: "yes"
cache-encryption-key: ${{ secrets.GRADLE_CACHE_ENCRYPTION_KEY }}
dependency-graph: generate-and-submit # submit Github Dependency Graph info

- name: Build
run: ./gradlew --build-cache --configuration-cache --configuration-cache-problems=warn app:assembleDebug

checks:
name: Checks
Expand Down Expand Up @@ -74,26 +74,26 @@ jobs:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with:
java-version: ${{ env.JAVA_VERSION }}
distribution: ${{ env.JAVA_DISTRO }}

- uses: gradle/actions/setup-gradle@v4
with:
cache-encryption-key: ${{ secrets.GRADLE_CACHE_ENCRYPTION_KEY }}
cache-read-only: true

- name: Run unit tests
run: ./gradlew --build-cache --configuration-cache --configuration-cache-problems=warn app:testDebugUnitTest

- name: Upload test results
if: always()
uses: actions/upload-artifact@v4
with:
name: unit-test-results
path: app/build/reports/tests/testDebugUnitTest
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with:
java-version: ${{ env.JAVA_VERSION }}
distribution: ${{ env.JAVA_DISTRO }}

- uses: gradle/actions/setup-gradle@v4
with:
cache-encryption-key: ${{ secrets.GRADLE_CACHE_ENCRYPTION_KEY }}
cache-read-only: true

- name: Run unit tests
run: ./gradlew --build-cache --configuration-cache --configuration-cache-problems=warn app:testDebugUnitTest

- name: Upload test results
if: always()
uses: actions/upload-artifact@v4
with:
name: unit-test-results
path: app/build/reports/tests/testDebugUnitTest

instrumentation-tests:
name: Instrumentation tests
Expand All @@ -102,7 +102,8 @@ jobs:
timeout-minutes: 60
strategy:
matrix:
api-level: [34]
api-level: [29, 30, 31, 33, 35]
fail-fast: false # make sure all tests finish, even if some fail

steps:
- uses: actions/checkout@v4
Expand All @@ -116,26 +117,23 @@ jobs:
cache-encryption-key: ${{ secrets.GRADLE_CACHE_ENCRYPTION_KEY }}
cache-read-only: true

# API 30+ emulators only have x86_64 system images.
- name: Get AVD info
uses: ./.github/actions/get-avd-info
id: avd-info
with:
api-level: ${{ matrix.api-level }}

- name: Accept licenses
run: yes | "$ANDROID_HOME"/cmdline-tools/latest/bin/sdkmanager --licenses

- name: Enable KVM group perms
run: |
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
sudo udevadm control --reload-rules
sudo udevadm trigger --name-match=kvm
- name: Instrumentation tests
uses: reactivecircus/[email protected]

- name: Cache AVD
uses: actions/cache@v4
with:
api-level: ${{ matrix.api-level }}
arch: ${{ steps.avd-info.outputs.arch }}
target: ${{ steps.avd-info.outputs.target }}
script: ./gradlew --build-cache --configuration-cache --configuration-cache-problems=warn app:connectedDebugAndroidTest
path: ~/.config/.android/avd
key: avd-${{ hashFiles('app/build.gradle.kts') }} # gradle-managed devices are defined there

- name: Run device tests
run: ./gradlew --build-cache --configuration-cache --configuration-cache-problems=warn app:api${{ matrix.api-level }}DebugAndroidTest --stacktrace

- name: Upload test results
if: always()
Expand Down
110 changes: 109 additions & 1 deletion app/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -79,10 +79,51 @@ android {
targetCompatibility = javaVersion
}

// needed for mockk
testOptions {
animationsDisabled = true

execution = "ANDROIDX_TEST_ORCHESTRATOR"

managedDevices {
localDevices {
create("api29") {
device = "Pixel 6a"
apiLevel = 29
systemImageSource = "aosp"
}
create("api30") {
device = "Pixel 6a"
apiLevel = 30
systemImageSource = "aosp-atd"
}
create("api31") {
device = "Pixel 6a"
apiLevel = 31
systemImageSource = "aosp-atd"
}
create("api33") {
device = "Pixel 6a"
apiLevel = 33
systemImageSource = "aosp-atd"
}
create("api35") {
device = "Pixel 6a"
apiLevel = 35
systemImageSource = "aosp-atd"
}
}
groups {
create("all") {
targetDevices.add(devices["api29"])
targetDevices.add(devices["api30"])
targetDevices.add(devices["api31"])
targetDevices.add(devices["api33"])
targetDevices.add(devices["api35"])
}
}
}

// needed for mockk
packaging {
jniLibs { useLegacyPackaging = true }
}
Expand Down Expand Up @@ -258,6 +299,72 @@ tasks.register("fixLicense") {
}
}

val embedScreenshotsTask = tasks.register("embedScreenshots") {
doFirst {
println("Embedding screenshots into JUnit reports...")

val additionalTestOutputDir = file(
"$projectDir/build/intermediates/managed_device_android_test_additional_output/debugAndroidTest"
)

val deviceDirectory = additionalTestOutputDir.listFiles()?.firstOrNull()

if (deviceDirectory == null) {
println("No device directory found in '$additionalTestOutputDir'")
return@doFirst
}
val deviceName = deviceDirectory.name.replace("DebugAndroidTest", "")

println("Processing screenshots for device '$deviceName'...")

val managedDeviceReportDirectory = File("$reportsPath/androidTests/managedDevice/debug")
val screenshotDirectory = File("$managedDeviceReportDirectory/$deviceName", "screenshots")

println("Copying screenshots to '$screenshotDirectory'...")

copy {
from(deviceDirectory)
into(screenshotDirectory)
}

screenshotDirectory.listFiles()?.forEach { failedTestClassDirectory ->
println("Processing screenshots for test class '${failedTestClassDirectory.name}'...")

val failedTestClassName = failedTestClassDirectory.name

failedTestClassDirectory.listFiles()?.forEach filesLoop@{ failedTestFile ->
println("Embedding screenshot for test '$failedTestFile'...")
val failedTestName = failedTestFile.name
val failedTestNameWithoutExtension = failedTestName.substringBeforeLast('.')
val failedTestClassJunitReportFile = File(
"$managedDeviceReportDirectory/$deviceName",
"$failedTestClassName.html"
)

if (!failedTestClassJunitReportFile.exists()) {
println("Could not find JUnit report file for test class '$failedTestClassJunitReportFile'")
return@filesLoop
}

var failedTestJunitReportContent = failedTestClassJunitReportFile.readText()

val patternToFind = "<h3 class=\"failures\">$failedTestNameWithoutExtension</h3>"
val patternToReplace = "$patternToFind <img src=\"screenshots/$failedTestClassName/$failedTestName\" width=\"360\" /></br>"

failedTestJunitReportContent = failedTestJunitReportContent.replace(patternToFind, patternToReplace)

failedTestClassJunitReportFile.writeText(failedTestJunitReportContent)
}
}
}
}

tasks.whenTaskAdded {
if (name.matches(regex = Regex("api[0-9]{2}DebugAndroidTest"))) {
finalizedBy(embedScreenshotsTask)
}
}

dependencies {
detektPlugins(libs.detekt.formatting)

Expand Down Expand Up @@ -356,6 +463,7 @@ dependencies {
debugImplementation(libs.androidx.compose.ui.test.manifest)

// Instrumentation tests
androidTestUtil(libs.androidx.test.orchestrator)
androidTestImplementation(libs.hilt.android.testing)
kspAndroidTest(libs.hilt.android.compiler)
androidTestImplementation(libs.junit)
Expand Down
76 changes: 76 additions & 0 deletions app/src/androidTest/java/app/ComposeRule.kt
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
/*
* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at https://mozilla.org/MPL/2.0/.
*
* Copyright (c) 2024 Matthias Emde
*/

package app

import androidx.compose.ui.test.SemanticsNodeInteraction
import androidx.compose.ui.test.getBoundsInRoot
import androidx.compose.ui.test.junit4.AndroidComposeTestRule
import kotlin.time.Duration.Companion.milliseconds

val LeaseSleepDuration = 500.milliseconds
const val LeaseDefaultAttempts = 10

fun AndroidComposeTestRule<*, *>.assertWithLease(
attempts: Int = LeaseDefaultAttempts,
assertion: () -> Unit
) {
try {
assertion()
} catch (e: Throwable) {
if (attempts > 0) {
mainClock.advanceTimeBy(LeaseSleepDuration.inWholeMilliseconds)
assertWithLease(attempts - 1, assertion)
} else {
throw e
}
}
}

fun SemanticsNodeInteraction.assertWithLease(
attempts: Int = LeaseDefaultAttempts,
assertion: SemanticsNodeInteraction.() -> Unit
): SemanticsNodeInteraction {
try {
assertion()
} catch (e: Throwable) {
if (attempts > 0) {
Thread.sleep(LeaseSleepDuration.inWholeMilliseconds)
assertWithLease(attempts - 1, assertion)
} else {
throw e
}
}

return this
}

/**
* Asserts that the given SemanticsNodeInteractions are vertically ordered on the screen.
*
* This function iterates through the provided nodes and verifies that each node is
* positioned above the subsequent node in the list. It checks if the bottom bound
* of the current node is less than or equal to the top bound of the next node.
*
* If any two consecutive nodes violate this vertical order, an assertion error is thrown.
*
* @param nodes The SemanticsNodeInteractions to be checked for vertical order.
*
* @throws AssertionError If any two consecutive nodes are not vertically ordered as expected.
*/
fun assertNodesInVerticalOrder(vararg nodes: SemanticsNodeInteraction) {
for (i in 0 until nodes.size - 1) {
val currentBounds = nodes[i].getBoundsInRoot()
val nextBounds = nodes[i + 1].getBoundsInRoot()

check(currentBounds.bottom <= nextBounds.top) {
"Expected node ${i + 1} to be above node ${i + 2}, but was not. " +
"Bounds: current = $currentBounds, next = $nextBounds"
}
}
}
Loading
Loading