Skip to content

CITE Test Upgrade

Jump to bottom
Jody Garnett edited this page Nov 13, 2019 · 10 revisions

The OGC Compliance Program provides the CITE tool for testing standards compliance. Our build process makes heavy use of this tool ... ensuring that each and every release of GeoServer correctly implements WMS, WFS, and WCS.

There are four problems with this:

  • We last updated our CITE build tool 2012 (so our testing is quite out of date).
  • geoserver.org website can only say we are a "compliant implementation", rather than a "certified implementation" of the WCS, WMS and WFS standards.
  • New services such as WPS and CSW are not "blackbox" tested.
  • Finally the transition to build.geoserver.org and subsequent rebuild means all CITE tests are currently failing

The OGC provide the CITE Team Engine both as a web service and as an open source project that can be used to check a single instance of GeoServer. This is a little awkward as we would like to integrate these tests into our build server; as such our build server stands up a GeoServer instance, performance the tests, and publishes the results here. These tests also require a database - and modify that database over the course of the tests (so our build server needs a PostGIS running).

Armed with a positive test result we can as an OSGeo Project ask for OGC certification, allowing us to update our website. Open source projects also have the option to be listed as a "reference implementation" (providing an instance of the server is available online for potential clients to test against).

Activities:

  • 2017 Code sprint - design and approached discussed proof of concept of git submodule approach
  • FOSS4G NA 2014 CITE Sprint - foss4gna activity to get developers set up to run cite tests locally

Resources:

How to fix this?

This activity is difficult enough that it has not attracted a contributor, and our previous 2012 CITE Test automation was good enough that no funding a replacement has been forthcoming.

This is intended as a sponsored activity:

Propose the following:

  1. A short term contract to setup the test harness and build scripts to automat cite tests on build.geosever.org.

  1. A team code sprint to pass the tests once infrastructure support is in place

  2. Work with OSGeo to achieve certification once tests results are available.

  3. Work with OSGeo SAC or OGC to host GeoServer configurations to act as a reference implementation.

Test Harness and Build Automation

  • Use git submodule to to manage CITE data directory configurations

  • Setup CITE TEAM Engine as a submodule

    Control which revision exactly we run against using this approach. Avoid making our own fork and contribute any and all test fixes to the upstream project.

  • Build support scripts

    • scripts to manage temporary GeoServer in build directory
    • scripts to manage CITE Test Database (assume PostGIS but hsql possible)

  • Alternatives for running tests:

    • Option: Take advantage of the CITE TEAM Engine's new REST API to run tests

    • Option: CITE TEAM Engine may be available to run as a Docker container

    • Option: Run using a script (same as 2012 approach)

CITE-Automation Contract

Setup a short-term contract, to set up the following:

  • Upgrade the test harness

  • Set up build jobs to test existing tests

    • WCS 1.0 - previously certified
    • WCS 1.1
    • WFS 1.0 - previously certified
    • WFS 1.1
    • WMS 1.1 - previously certified
    • WMS 1.3

Code Sprint on Certification

In the latter part of the year set up a code sprint to work on passing the tests:

  • Initial focus on passing build jobs above

    • WCS 1.0
    • WCS 1.1
    • WFS 1.0
    • WFS 1.1
    • WMS 1.1
    • WMS 1.3

  • Add new test that we are missing

    • WFS 2.0 - passed in isolation
    • WCS 2.0
    • CSW
    • WMTS 1.0 - passed in isolation
    • WPS 1.0
    • Maybe GML and KML and GeoPackage

  • OGC Certification

    • Submit required paperwork, and test results, update website

  • Reference Implementation

    • Stand up a reference server for OGC to use (if OSGeo or OGC provide hardware) with one test per workspace/virtual service?

©2020 Open Source Geospatial Foundation

Home
Proposals
Release Schedule

Clone this wiki locally