#9 Add design for integration test framework

.github/workflows/ci-build.yml

@@ -86,7 +86,7 @@ jobs:
 
       - name: Integration tests
         run: |
-          # -p 1 tells go not to run the tests in parallel. See the developers_guide.
+          # -p 1 tells go not to run the tests in parallel. See the developer guide.
           go test -v -p 1 -count 1 -coverprofile=coverage.out ./...
 
       - name: Prepare testing extension

.github/workflows/render-design.yml

@@ -23,6 +23,7 @@ jobs:
           mkdir gh-pages
           cd gh-pages
           pandoc ../doc/design.md -o design.html --filter pandoc-plantuml
+          pandoc ../doc/system_requirements.md -o system_requirements.html --filter pandoc-plantuml
       - name: Deploy 🚀
         uses: JamesIves/[email protected]
         with:

doc/changes/changes_0.5.2.md

@@ -4,12 +4,16 @@ Code name:
 
 ## Summary
 
-This release updates the upload process for the extension registry to verify that the extension URLs are valid.
+This release updates the upload process for the extension registry to verify that the extension URLs are valid. It also adds design, requirements and user guide for the integration testing framework.
 
 ## Features
 
 * #129: Added verification for extension URLs before uploading to registry
 
+## Documentation
+
+* #9: Add design, requirements and user guide for integration testing framework
+
 ## Refactoring
 
 * #139: Converted `extensionForTesting` to JavaScript to speedup tests

doc/design.md

@@ -173,9 +173,18 @@ Needs: impl, utest, itest
 ### Extension Definitions
 `dsn~extension-definition~1`
 
-Each extension might be implemented in an arbitrary programming language and typically are based on a so-called [user defined function](system_requirements.md#terms-and-abbreviations). In order to allow EM to manage multiple heterogenous extensions in a unique way, each extension is represented by small wrapper implementing a uniform interface. This wrapper is called an "extension definition".
+Each extension might be implemented in an arbitrary programming language and typically is based on a so-called [user defined function](system_requirements.md#terms-and-abbreviations). In order to allow EM to manage multiple heterogenous extensions in a unique way, each extension is represented by small wrapper implementing a uniform interface. This wrapper is called an "extension definition".
 
-The interface is defined in [`extension-manager-interface`](https://github.com/exasol/extension-manager-interface/):
+Covers:
+
+* [`req~extension~1`](system_requirements.md#install-required-artifacts)
+
+Needs: impl, utest, itest
+
+#### Extension API Interface
+`dsn~extension-api~1`
+
+Each [extension definition](#extension-definitions) implements the TypeScript interface defined in [`extension-manager-interface`](https://github.com/exasol/extension-manager-interface/). This allows EM to uniformly manage all extensions that implement this interface.
 
 ```plantuml
 @startuml
@@ -190,9 +199,9 @@ ExasolExtension <-- "mysql-virtual-schema (repo).MySqlExtensionDefinition"
 
 Covers:
 
-* [`req~extension~1`](system_requirements.md#install-required-artifacts)
+* [`req~extension-api~1`](system_requirements.md#extension-api)
 
-Needs: impl, utest, itest
+Needs: impl, utest
 
 #### Storage for Extension Definitions
 `dsn~extension-definitions-storage~1`
@@ -323,7 +332,7 @@ Covers:
 
 Each parameter definition is attached to a specific version of the extension.
 
-Rationale: Parameters can change over time, see [Updates](#updates).
+Rationale: Parameters can change over time, see [Upgrades](#upgrades).
 
 Covers:
 * [`req~define-configuration-parameters~1`](system_requirements.md#parameter-types)
@@ -384,6 +393,46 @@ See design decision [against a callback for the client side validation](#callbac
 Covers:
 * [`req~validate-parameter-values~1`](system_requirements.md#ui-languages)
 
+### Extension Integration Testing Framework
+
+The Extension Integration Testing Framework for Java (EITFJ) allows [extension developers](system_requirements.md#extension-developers) to create integration tests for an extension involving its definition](#extension-definitions) and implementation.
+
+The EITFJ is written in Java as most extensions like virtual schemas are also written in Java. This way it's easy to add integration tests for an extension into the existing Maven build.
+
+#### Starting Extension Manager during Integration Tests
+`dsn~eitfj-start-extension-manager~1`
+
+The EITFJ provides a method for installing and starting an [Extension Manager REST interface](#em-provides-a-rest-interface).
+
+Covers:
+* [`req~extension-testing-framework~1`](system_requirements.md#integration-test-framework-for-extensions)
+
+Needs: impl, itest, doc
+
+#### Accessing the Extension Manager REST Interface
+`dsn~eitfj-access-extension-manager-rest-interface~1`
+
+The EITFJ provides a Java API for accessing the EM REST interface.
+
+Rationale:
+This simplifies integration tests and avoids code duplication.
+
+Covers:
+* [`req~extension-testing-framework~1`](system_requirements.md#integration-test-framework-for-extensions)
+
+Needs: impl, utest, itest, doc
+
+#### Preparing Previous Extension Versions
+`dsn~eitfj-prepare-previous-extension-version~1`
+
+The EITFJ provides a Java API for preparing previous versions of an extension. This allows writing integration tests for upgrading from a previous version to the current version of an extension.
+
+Covers:
+* [`req~extension-testing-framework~1`](system_requirements.md#integration-test-framework-for-extensions)
+* [`req~upgrade-extension~1`](system_requirements.md#upgrade-extension)
+
+Needs: impl, utest
+
 ## Runtime
 
 ### Listing Extensions
@@ -475,7 +524,7 @@ Notes:
 * See details about [BucketsFS Buckets in Exasol SaaS](#bucketsfs-buckets-in-exasol-saas) in the next section.
 
 Covers:
-* [`req~install-extension-database-objects~1`](system_requirements.md#update-extension)
+* [`req~install-extension-database-objects~1`](system_requirements.md#install-database-objects)
 * [`req~define-configuration-parameters~1`](system_requirements.md#parameter-types)
 * [`req~uninstall-extension~1`](system_requirements.md#uninstalling-extensions)
 
@@ -567,7 +616,7 @@ Installation "1" o-- "*" Instance
 ```
 
 Covers:
-* [`req~install-extension-database-objects~1`](system_requirements.md#update-extension)
+* [`req~install-extension-database-objects~1`](system_requirements.md#install-database-objects)
 
 
 #### Installation Metadata
@@ -578,7 +627,7 @@ Extensions don't store their own metadata. Instead they read information about e
 However, for example for reading back the credentials stored in a connection, EM uses a temporary UDF that reads back the secret value.
 
 Covers:
-* [`req~install-extension-database-objects~1`](system_requirements.md#update-extension)
+* [`req~install-extension-database-objects~1`](system_requirements.md#install-database-objects)
 
 Needs: impl, utest, itest
 
@@ -627,7 +676,7 @@ database Database {
 ```
 
 Covers:
-* [`req~install-extension-database-objects~1`](system_requirements.md#update-extension)
+* [`req~install-extension-database-objects~1`](system_requirements.md#install-database-objects)
 
 Needs: impl, utest, itest
 
@@ -638,7 +687,7 @@ EM can upgrade an installed extensions and its instances to the latest version.
 
 Covers:
 * [`req~upgrade-extension~1`](system_requirements.md#uninstalling-extensions)
-* [`req~install-extension-database-objects~1`](system_requirements.md#update-extension)
+* [`req~install-extension-database-objects~1`](system_requirements.md#install-database-objects)
 
 Needs: impl, utest, itest
 
@@ -658,7 +707,7 @@ Rationale:
 * While EM also provides [access to metadata](#extension-context-metadata) via the context, this information may not be sufficient. Executing arbitrary queries ensures maximum flexibility for extensions.
 
 Covers:
-* [`req~install-extension-database-objects~1`](system_requirements.md#update-extension)
+* [`req~install-extension-database-objects~1`](system_requirements.md#install-database-objects)
 
 Needs: impl, utest, itest
 
@@ -668,7 +717,7 @@ Needs: impl, utest, itest
 The BucketFS client in the extension context allows the extension definition to interact with BucketFS.
 
 Covers:
-* [`req~install-extension-database-objects~1`](system_requirements.md#update-extension)
+* [`req~install-extension-database-objects~1`](system_requirements.md#install-database-objects)
 
 Needs: impl, utest, itest
 
@@ -697,7 +746,7 @@ Rationale:
   * **Note:** EM will only need to work with Exasol v8, so support for v7 is actually not necessary. However the Docker container for v8 currently does not support running Python and Java UDFs. Until this is fixed we still need v7 for integration testing.
 
 Covers:
-* [`req~install-extension-database-objects~1`](system_requirements.md#update-extension)
+* [`req~install-extension-database-objects~1`](system_requirements.md#install-database-objects)
 
 Needs: impl, utest, itest
 

doc/developer_guide.md

@@ -1,4 +1,6 @@
-# Developers Guide
+# Developer Guide
+
+This guide describes how to develop, test and build the Extension Manager.
 
 ## Building
 
@@ -36,42 +38,17 @@ If tracing fails with a `org.xml.sax.SAXParseException` you might need to run `m
 
 ## Testing
 
-The different components of the project are responsible for testing different things.
-
-### extension-manager
-
 The extension-manager project contains unit and integration tests that verify
 * Loading and executing of JavaScript extensions
 * Database interactions
 * REST API interface
 * Server-side parameter validation using `extension-parameter-validator`
-* ...
 
 Tests use dummy extensions, no real extensions.
 
-### Extensions
-
-Extensions are located in the repositories of the virtual schema implementations, e.g. `s3-document-files-virtual-schema`.
-
-Tests for extensions are:
-* Verify correct implementation of a specific version of the `extension-manager-interface` using the TypeScript compiler
-* Unit tests written in TypeScript verify all execution paths of the extension
-* Integration tests written in Java use a specific version of the `extension-manager` to verify that the extension
-  * can be loaded
-  * can install a virtual schema and check that it works
-  * can update parameters of an existing virtual schema
-  * can upgrade a virtual schema created with an older version
-  * ...
-
-### Restrictions as Document Virtual Schemas Only Support a Single Version
-
-Document virtual schemas like `s3-document-files-virtual-schema` require a `SET SCRIPT` that must have a specific name. As this script references a specific virtual schema JAR archive, it is not possible to install multiple version of the same virtual schema in the same database `SCHEMA`.
-
-This means that in order to test a new version of a virtual schema, you need to create a new `SCHEMA` with the required database objects.
-
 ### Non-Parallel Tests
 
-The tests of this project use the exasol-test-setup-abstraction-server. There the tests connect to an Exasol database running in a docker container. For performance reasons the test-setup-abstraction reuses that container.  This feature is not compatible with running tests in parallel.
+The tests of this project use [`exasol-test-setup-abstraction-server`](https://github.com/exasol/exasol-test-setup-abstraction-server/). There the tests connect to an Exasol database running in a docker container. For performance reasons the test-setup-abstraction reuses that container. This feature is not compatible with running tests in parallel.
 
 Problems would be:
 
@@ -93,6 +70,8 @@ To run only tests without the database use:
 go test -p 1 -short ./...
 ```
 
+Please note that also `-short` tests need `-p 1` because extension integration tests share a directory for building a test extension. Tests will fail randomly without `-p 1`.
+
 ## Static Code Analysis
 
 ### Go Linter

doc/extension_developer_guide.md

@@ -0,0 +1,85 @@
+# Extension Developer Guide
+
+This guide describes how to create and test an extension definition for the Extension Manager.
+
+## Extensions
+
+Definition and implementation of each extension are located in a common repository, e.g. `s3-document-files-virtual-schema`.
+
+The tests for an extensions usually will include
+* Using the TypeScript compiler to verify correct implementation of a specific version of the `extension-manager-interface`
+* Unit tests written in TypeScript to verify all execution paths of the extension's implementation.
+* Integration tests written in Java using a specific version of the `extension-manager` to verify that the extension
+  * can be loaded
+  * can install a virtual schema and check that it works
+  * can update parameters of an existing virtual schema
+  * can upgrade a virtual schema created with an older version
+  * ...
+
+### Restrictions as Document-based Virtual Schemas Only Support a Single Version
+
+Document-based virtual schemas like `s3-document-files-virtual-schema` require a `SET SCRIPT` that must have a specific name. As this script references a specific virtual schema JAR archive, it is not possible to install multiple version of the same virtual schema in the same database schema.
+
+This means that in order to test a new version of a virtual schema, you need to create a new database schema with the required database objects.
+
+## Extension Manager Interface
+
+Extension definitions are written in TypeScript and compiled to a single JavaScript file. They implement the [extension-manager-interface](https://github.com/exasol/extension-manager-interface/). See [testing-extension](../extension-manager-integration-test-java/testing-extension) for an example including build scripts.
+
+## Extension Integration Test Framework for Java
+
+The Extension Integration Test Framework for Java (EITFJ) allows writing integration tests for extensions and their extension definitions.
+
+### Preconditions
+
+We assume your extension definition project is located in folder `$EXTENSION`. `$EXTENSION_ID` is the filename of your JavaScript extension definition.
+
+The project in `$EXTENSION` must fulfill the following preconditions:
+* NPM modules are already installed to `node_modules` before running integration tests.
+* `package.json` is configured to build the extension definition with `npm run build`.
+* The build process writes the JavaScript file to `$EXTENSION/dist/$EXTENSION_ID`
+
+If your extension definition uses a different build process you can create a custom `ExtensionBuilder`.
+
+### Using EITFJ
+
+The EITFJ library is published to [Maven Central](https://central.sonatype.com/artifact/com.exasol/extension-manager-integration-test-java), so you can add it to your project as follows:
+
+```xml
+<dependency>
+    <groupId>com.exasol</groupId>
+    <artifactId>extension-manager-integration-test-java</artifactId>
+    <version>$VERSION</version>
+    <scope>test</scope>
+</dependency>
+```
+
+See [`ExampleIT.java`](../extension-manager-integration-test-java/src/test/java/com/exasol/extensionmanager/ExampleIT.java) for an example of how to use EITFJ in your integration tests. Adapt the following constants depending to your own extension definition:
+
+* `EXTENSION_SOURCE_DIR`: relative path to the directory containing the extension definition sources (`$EXTENSION`)
+* `EXTENSION_ID`: file name of the built JavaScript file (`$EXTENSION_ID`)
+
+Depending on the requirements of your extension you might also need to upload the adapter JAR or a JDBC driver to BucketFS in `@BeforeAll`.
+
+#### Features of class `ExtensionManagerSetup`
+
+Class `ExtensionManagerSetup` offers the following useful features:
+
+* `ExtensionManagerSetup.create()` downloads and starts the EM REST interface, builds your extension definition and adds it to EM's extension registry.
+* Call `setup.client()` to get a client for the EM's REST interface. It allows you to install your extension, create a new instance etc.
+* Call `setup.client().assertRequestFails()` to verify that a REST call fails with an expected status code and error message. This allows testing that your extension definitions throws an expected error.
+* Call `setup.previousVersionManager()` to prepare a previous version of your extension. This is useful for testing the upgrade process.
+* Call `setup.exasolMetadata()` to verify that expected database objects like `SCRIPT`, `CONNECTION` or `VIRTUAL SCHEMA` were created.
+* Call `setup.addVirtualSchemaToCleanupQueue()` and `setup.addConnectionToCleanupQueue()` to delete a `CONNECTION` or `VIRTUAL SCHEMA` after a test.
+
+### EITFJ Configuration
+
+EITFJ works without additional configuration. During development you can however create file `extension-test.properties` to simplify local testing. We recommend adding this file to `.gitignore` to avoid accidentally committing it.
+
+`extension-test.properties` supports the following optional settings:
+
+* `localExtensionManager`: Path to a local clone of the `extension-manager` repository. This allows testing against a local version of extension manager that was not yet released. By default EITFJ will install extension manager using `go install`.
+* `buildExtension`: Set this to `false` in order to skip building the extension definition before the tests. Use this to speedup tests when the extension definition is not modified.
+* `buildExtensionManager`: Set this to `false` to skip building/installing the extension manager binary. Use this to speedup tests when extension manager is not modified.
+* `extensionManagerVersion`: Version of EM to use during tests. By default EITFJ uses the same version as the version defined in `pom.xml` for `extension-manager-integration-test-java`. Changing this is not recommended.
+