CITE Test Upgrade

The OGC Compliance Program provides the CITE tool for testing standards compliance. Our build process makes heavy use of this tool ... ensuring that each and every release of GeoServer correctly implements WMS, WFS, and WCS.

There are four problems with this:

• We last updated our CITE build tool 2012 (so our testing is quite out of date).
• geoserver.org website can only say we are a "compliant implementation", rather than a "certified implementation" of the WCS, WMS and WFS standards.
• New services such as WPS and CSW are not "blackbox" tested.
• Finally the transition to build.geoserver.org and subsequent rebuild means all CITE tests are currently failing

The OGC provide the CITE Team Engine both as a web service and as an open source project that can be used to check a single instance of GeoServer. This is a little awkward as we would like to integrate these tests into our build server; as such our build server stands up a GeoServer instance, performance the tests, and publishes the results here. These tests also require a database - and modify that database over the course of the tests (so our build server needs a PostGIS running).

Armed with a positive test result we can as an OSGeo Project ask for OGC certification, allowing us to update our website. Open source projects also have the option to be listed as a "reference implementation" (providing an instance of the server is available online for potential clients to test against).

This activity is difficult enough that it has not attracted a contributor, and our previous 2012 CITE Test automation was good enough that no funding a replacement has been forthcoming.

Resource Planning:

• 2019 Contract + Code sprint
• 2017 Code sprint - design and approached discussed
• FOSS4G NA 2014 CITE Sprint - foss4gna activity to get developers set up to run cite tests locally

Approach

• Setup CITE TEAM Engine as a submodule

  We can control which revision exactly we run against using this approach.

  We want to avoid making our own fork and contribute any and all test fixes to the upstream project.

• Start up a local geoserver to test against

• Take advantage of the CITE TEAM Engine's new REST API to run tests

CITE-Automation Contract

Setup a short-term contract, to set up the following:

• Upgrade the test harness

  • Use https://github.com/opengeospatial/teamengine as a git submodule

• Set up build jobs to test existing tests

  • WCS 1.0
  • WCS 1.1
  • WFS 1.0
  • WFS 1.1
  • WMS 1.1
  • WMS 1.3

Code Sprint on Certification

In the latter part of the year set up a code sprint to work on passing the tests:

• Initial focus on passing build jobs above

  • WCS 1.0
  • WCS 1.1
  • WFS 1.0
  • WFS 1.1
  • WMS 1.1
  • WMS 1.3

• Add new test that we are missing

  • WFS 2.0
  • WCS 2.0
  • CSW
  • WMTS 1.0
  • WPS 1.0
  • Maybe GML and KML and GeoPackage

• OGC Certification

  • Submit required paperwork, and test results, update website

• Reference Implementation

  • Stand up a reference server for OGC to use (if OSGeo or OGC provide hardware) with one test per workspace/virtual service?