Preview - Keycloak Test Framework #488
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lhanusov
force-pushed
the
32399
branch
3 times, most recently
from
2d89793
to
46b3979
Compare
vaceksimon
suggested changes
Aug 29, 2024
vaceksimon
approved these changes
Sep 2, 2024
stianst
reviewed
Sep 27, 2024
|
|
||
| == Brighter Future? | ||
|
|
||
| Keycloak team had started a https://github.com/orgs/keycloak/projects/42[joint effort] between the engineering and quality engineering people to design a new test framework (May 2024). It had been initiated with a prototype to verify if our thoughts were feasible. The prototype was based on the _JUnit5_ testing framework, specifically to implement _JUnit5_ callback classes which extend the default test lifecycle functionality and provide custom inject annotations, like _@KeycloakIntegrationTest_, _@InjectWebDriver_ or _@InjectRealm_. |
There was a problem hiding this comment.
Just say Keycloak team - engineering and quality engineering is a RH thing, and we're all just the Keycloak team :)
stianst
reviewed
Sep 27, 2024
| == Are you curious about where to start? | ||
| We suggest reading the https://github.com/keycloak/keycloak/blob/main/test-framework/README.md[user guide], which will provide a basic overview of how the framework works and should be used. If this is not enough, you can also check https://github.com/keycloak/keycloak/tree/main/test-poc/examples[examples module]. | ||
|
|
||
| For extension developers we recommend to look into the showcase example on how to start _Keycloak_ with their custom provider - link. |
There was a problem hiding this comment.
Just a note that we need to update this before publishing the blog
stianst
reviewed
Sep 27, 2024
|
|
||
| For extension developers we recommend to look into the showcase example on how to start _Keycloak_ with their custom provider - link. | ||
|
|
||
| Once you find a bug, want to discuss something, or propose a new enhancement, please follow this GitHub feedback https://www.github.com/discussion#blabla[discussion] link. |
There was a problem hiding this comment.
Just a note that we need to update this before publishing the blog
stianst
reviewed
Sep 27, 2024
|
|
||
| == Coming Soon | ||
|
|
||
| Preview - https://github.com/keycloak/keycloak/tree/main/test-framework[_Keycloak Test Framework_] will be introduced in _Keycloak Nightly_ builds. We plan to add more features, which will cover 80% of functionalities currently available in the https://github.com/keycloak/keycloak/tree/main/testsuite[integration test suite]. We also plan to introduce some new concepts, e.g. _@KeycloakTest_ annotation, which will add an option to test providers without the _Keycloak_ server running. It should be based on the _KeycloakSession_ class. Other interesting features should be parallel execution on a GitHub runner, adding a custom extension to a Keycloak server, or enabling a Keycloak cluster setup. |
There was a problem hiding this comment.
Is available, not will be ;)
There was a problem hiding this comment.
Remove 80%, and say something like cover enough for most tests...
stianst
reviewed
Sep 27, 2024
|
|
||
| == Brighter Future? | ||
|
|
||
| Keycloak team had started a https://github.com/orgs/keycloak/projects/42[joint effort] between the engineering and quality engineering people to design a new test framework (May 2024). It had been initiated with a prototype to verify if our thoughts were feasible. The prototype was based on the _JUnit5_ testing framework, specifically to implement _JUnit5_ callback classes which extend the default test lifecycle functionality and provide custom inject annotations, like _@KeycloakIntegrationTest_, _@InjectWebDriver_ or _@InjectRealm_. |
There was a problem hiding this comment.
The board is private, and not visible to everyone. So we either remove the link, or cleanup the board and make it public
stianst
reviewed
Sep 27, 2024
|
|
||
| For extension developers we recommend to look into the showcase example on how to start _Keycloak_ with their custom provider - link. | ||
|
|
||
| Once you find a bug, want to discuss something, or propose a new enhancement, please follow this GitHub feedback https://www.github.com/discussion#blabla[discussion] link. |
There was a problem hiding this comment.
Suggested change
| Once you find a bug, want to discuss something, or propose a new enhancement, please follow this GitHub feedback https://www.github.com/discussion#blabla[discussion] link. | |
| If you find a bug, want to discuss something, or propose a new enhancement, please follow this GitHub feedback https://www.github.com/discussion#blabla[discussion] link. |
lhanusov
force-pushed
the
32399
branch
4 times, most recently
from
a496d31
to
9b8325f
Compare
stianst
reviewed
Nov 14, 2024
|
|
||
| The idea to replace the current https://github.com/keycloak/keycloak/tree/main/testsuite[test suite] has been on the table for multiple years. Initially, it was meant to be only a refactoring of the current approach on how to write tests, but after a few internal discussions and refactor updates it turned out a new test suite, based on a https://github.com/keycloak/keycloak/tree/main/test-framework[new framework] would be a better solution. | ||
|
|
||
| It would be good to mention a few drawbacks, that stand out when working with the current https://github.com/keycloak/keycloak/tree/main/testsuite[test suite]. First of all, is the complexity of various configurations and additions made on top of the https://arquillian.org/[_Arquillian framework_]. These changes make the test suite powerful, but the cons is that without proper documentation for beginners is almost unreadable. The second thing has the same importance, the _Arquillian framework_ is not fully supported anymore. Other things to mention are a complicated execution system, where you want to specify what exactly should be tested, then abstract classes with shared configurations, missing the option to add a custom extension, and last but not least, the massive amount of tests without knowledge if they are needed anymore. |
There was a problem hiding this comment.
Not convinced about the statement the massive amount of tests without knowledge if they are needed anymore. There may be some tests not needed, but probably not a massive amount. Perhaps just drop this?
stianst
reviewed
Nov 14, 2024
|
|
||
| == Brighter Future? | ||
|
|
||
| The Keycloak team began an effort to design a new test framework in May 2024. It started with a prototype to verify if our ideas were feasible. The prototype was based on the https://junit.org/junit5/[_JUnit5_ testing framework], specifically to implement https://junit.org/junit5/docs/current/user-guide/#extensions-lifecycle-callbacks[_JUnit5_ callback classes] which extend the default test lifecycle functionality and provide custom inject annotations, like _@KeycloakIntegrationTest_, _@InjectWebDriver_ or _@InjectRealm_. |
There was a problem hiding this comment.
Suggested change
| The Keycloak team began an effort to design a new test framework in May 2024. It started with a prototype to verify if our ideas were feasible. The prototype was based on the https://junit.org/junit5/[_JUnit5_ testing framework], specifically to implement https://junit.org/junit5/docs/current/user-guide/#extensions-lifecycle-callbacks[_JUnit5_ callback classes] which extend the default test lifecycle functionality and provide custom inject annotations, like _@KeycloakIntegrationTest_, _@InjectWebDriver_ or _@InjectRealm_. | |
| The Keycloak team began an effort to design a new test framework in May 2024. It started with a prototype to verify if our ideas were feasible. The prototype was based on the https://junit.org/junit5/[_JUnit5_ testing framework], specifically to implement https://junit.org/junit5/docs/current/user-guide/#extensions-lifecycle-callbacks[_JUnit5_ callback classes] which extend the default test lifecycle functionality and provide custom inject annotations, like _@KeycloakIntegrationTest_, _@InjectWebDriver_ or _@InjectRealm_. |
Perhaps writing something like The prototype is an JUnit 5 Extension...
stianst
reviewed
Nov 14, 2024
Comment on lines
16
to
21
| For our purpose, we have https://github.com/keycloak/keycloak/blob/main/test-framework/core/src/main/java/org/keycloak/test/framework/KeycloakIntegrationTestExtension.java[implemented] these three callback classes: | ||
| ---- | ||
| BeforeEachCallback | ||
| AfterEachCallback | ||
| AfterAllCallback | ||
| ---- |
There was a problem hiding this comment.
Would drop this, folks just need to know it's a JUnit 5 extension that is automatically added when you add the dependency; they don't need to know the internals.
stianst
reviewed
Nov 14, 2024
| AfterAllCallback | ||
| ---- | ||
|
|
||
| After a successful test round, we've continued with a proof of concept extending features list to support multiple server modes, different databases and _WebDrivers_, clients and users setup, _SmallRye_ configuration support, _OAuthClient_ based on https://connect2id.com/products/nimbus-oauth-openid-connect-sdk[_Nimbus SDK_] (this feature is a preview only) etc. The full list of currently implemented features can be found here in the https://github.com/keycloak/keycloak/tree/main/test-framework/core/src/main/java/org/keycloak/test/framework/annotations[annotations package]. It is already present in the main branch and _Keycloak_ nightly builds. |
There was a problem hiding this comment.
The full list of features in the annotations package isn't the best way to list the features; and now that we split it up into multiple modules it doesn't list all annotations.
There was a problem hiding this comment.
Maybe just do a bullet point list of features?
stianst
reviewed
Nov 14, 2024
Comment on lines
32
to
34
| == Coming Soon | ||
|
|
||
| Main plan is to cover enough features for most functionalities currently available in the https://github.com/keycloak/keycloak/tree/main/testsuite[integration test suite]. Next future step shall be to introduce some new concepts, e.g. _@KeycloakTest_ annotation, which will add an option to test providers without the _Keycloak_ server running. It should be based on the _KeycloakSession_ class. Other interesting features should be https://junit.org/junit5/docs/current/user-guide/#junit-platform-suite-engine[_JUnit5_ test suites], parallel execution on a GitHub runner or enabling a Keycloak cluster setup. |
There was a problem hiding this comment.
Suggested change
| == Coming Soon | |
| Main plan is to cover enough features for most functionalities currently available in the https://github.com/keycloak/keycloak/tree/main/testsuite[integration test suite]. Next future step shall be to introduce some new concepts, e.g. _@KeycloakTest_ annotation, which will add an option to test providers without the _Keycloak_ server running. It should be based on the _KeycloakSession_ class. Other interesting features should be https://junit.org/junit5/docs/current/user-guide/#junit-platform-suite-engine[_JUnit5_ test suites], parallel execution on a GitHub runner or enabling a Keycloak cluster setup. | |
| == Next steps | |
| We already have enough capabilities in the new test framework to start migrating some tests from the old testsuite; and in fact already have our very first test migrated. We plan to migrate one package at a time from the old testsuite starting with `admin` tests, then moving on to `forms` and `oauth` packages. As we are doing this we will expand on the capabilities of the test framework. | |
| Some features we know will be coming soon include: | |
| * An easier way to deploy custom providers, not requiring a Maven build of the provider first | |
| * Improved logging, making it easy to configure logging from tests as well as Keycloak | |
| * Easy testing of OAuth and OpenID Connect, including a mock application | |
| * Extension to allow running code on the tested server when it's not possible to easily test through only remote interfaces |
We also have some more long term plans to deliver:
- Provider tests that can be used to easily test a provider by invoking the provider directly
- Parallel execution of tests, to take full advantage of multiple cores to reduce test execution time
lhanusov
force-pushed
the
32399
branch
2 times, most recently
from
ac1034f
to
71366f2
Compare
stianst
reviewed
Nov 14, 2024
| @@ -0,0 +1,67 @@ | |||
| :title: "Writing new tests has never been easier than before." Keycloak Test Framework | |||
There was a problem hiding this comment.
This title won't show very nicely on the website or feeds
Suggested change
| :title: "Writing new tests has never been easier than before." Keycloak Test Framework | |
| :title: Introducing the Keycloak Test Framework |
stianst
reviewed
Nov 14, 2024
| :title: "Writing new tests has never been easier than before." Keycloak Test Framework | ||
| :date: 2024-11-14 | ||
| :publish: true | ||
| :author: Lukas Hanusovsky, Stian Thorgersen |
There was a problem hiding this comment.
You get the full creds for this one; I just suggested some minor changes ;)
Suggested change
| :author: Lukas Hanusovsky, Stian Thorgersen | |
| :author: Lukas Hanusovsky |
Signed-off-by: Lukas Hanusovsky <[email protected]> Co-authored-by: Stian Thorgersen <[email protected]>
stianst
approved these changes
Nov 14, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #32399