Skip to content

REST API Refresh

Jump to bottom
Jody Garnett edited this page Sep 6, 2017 · 61 revisions

The REST API for GeoServer is popular, but not well maintained, collecting a large number of outstanding bugs. Many of these complain about functionality and lack of documentation. The large number of bugs (API called correctly but produced an error) may be producing more requests for documentation (developer assumed they called it incorrectly, and asks for improved documentation with an example that works).

Internally the REST API is written using an early java library called "restlet" (http://restlet.com/). There is a desire to migrate to spring rest api which is annotation driven and better supported. The risk in performing a migration to Spring MVC is introducing more bugs than are fixed. This is somewhat offset by having a larger pool of developers familiar with the codebase and the technologies used.

This activity will require quite a bit of people (all hands on deck):

Sprint Planning

Sprint Prep

Create gs-rest-ng:

Monday

  • Migrate workspaces
  • Migrate coveragestores
  • Migrate datastores
  • Migrate wmsstores / wmslayers
  • Migrate layergroup
  • Setup documentation build

Tuesday

  • Migrate workspaces
  • Migrate coveragestores
  • Migrate datastores
  • Migrate wmsstores / wmslayers
  • Migrate layergroup

Wednesday

  • TBD

Thursday

  • TBD

Friday

  • TBD
  • Documentation Wrapup
  • Code Wrapup:
    • Re-enable disabled tests (Core UI module, ...)
    • Remove gs-rest and gs-restconfig
    • Rename "restng" endpoint to "rest" (Using the RestBaseController.ROOT_PATH)
    • Ensure constructor for each class extending CatalogController is annotated with @Qualifier("catalog")

Post-sprint

  • Code Wrapup:
    • Change restconfig catalog package to rest.catalog for consistency
    • Combine multiple paths in all controllers (See StyleController for an example)
    • Make PathVariable names consistent (e.g {workspace} or {workspaceName})
    • Remove gs-rest and gs-restconfig
    • Run gsconfig and geoserver-manager integration tests against the new REST API
    • Add license headers to all files
    • Use consistent method names, e.g. "workspaceGet"

Set up Tests

Test cases should remain unchanged to verify no regressions

  • Initial review shows test-coverage for XML is pretty good, json and html are poor

Serialization / Deserialization test

  • IMPORTANT: For each endpoint - ensure that one deserialize/serialize test case is in place prior to migrating
  • This captures 90% of any XStream issues - GET an XML document, PUT it back, verify it is unchanged
@Test
public void testRoundTripCoverageStoreXML() throws Exception {
    CoverageInfo before = getCatalog().getCoverageByName(getLayerId(MockData.TASMANIA_BM));

    // get and re-write, does not go boom
    String xml = getAsString( "/rest/workspaces/wcs/coveragestores/BlueMarble.xml");
    MockHttpServletResponse response = putAsServletResponse("/rest/workspaces/wcs/coveragestores/BlueMarble.xml", xml, "text/xml");
    assertEquals(200, response.getStatus());

    // check nothing actually changed
    CoverageInfo after = getCatalog().getCoverageByName(getLayerId(MockData.TASMANIA_BM));
    assertEquals(before, after);
}

Migration Approach and Notes

These are loose notes on converting the GeoServer REST API to Spring MVC. They are derived from the experience of converting the existing Styles end point. This document is meant to be a companion to the actual code.

Approach

The general approach that I found most useful was:

  • Copy over the Unit Tests for the end point you're working on.
  • Update the URLs
  • Run, watch all the failures.
  • Create your new end point and fill it in, running the test cases as needed.
  • In general, start with the POST/PUT end points. Those tend to be the trickiest. GET and DELETE are simpler.

Sample Branch

The sample Spring MVC conversion branch is here:

https://github.com/geoserver/geoserver/tree/rest-api-refresh

Spring MVC Documentation

The Spring MVC reference docs are decent and worth at least a skimming.

https://docs.spring.io/spring/docs/current/spring-framework-reference/html/mvc.html

Architecture

The basic parts of the architecture are as follows. More information can be found in the JavaDocs for each object.

Controllers

These are the main REST request handlers. They map roughly to the existing *Resource classes in the existing rest module. For example

@GetMapping(
    path = "/styles/{styleName}",
    produces = {
        MediaType.APPLICATION_JSON_VALUE, 
        MediaType.APPLICATION_XML_VALUE, 
        MediaType.TEXT_HTML_VALUE})
protected RestWrapper<StyleInfo> getStyle(
        @PathVariable String styleName) {
    return wrapObject(getStyleInternal(styleName, null), StyleInfo.class);
}

HttpMessageConverters

These are responsible for serialization and deserialization of response objects. These correlate to the *Format objects in the existing REST API. In most cases these just need to tie into our existing serialization (XStreamPersister).

/**
 * Message converter implementation for JSON serialization via XStream
 */
public class JSONMessageConverter extends BaseMessageConverter {

    public JSONMessageConverter(ApplicationContext applicationContext) {
        super(applicationContext);
    }
    @Override
    public boolean canRead(Class clazz, MediaType mediaType) {
        return !XStreamListWrapper.class.isAssignableFrom(clazz) &&
            MediaType.APPLICATION_JSON.equals(mediaType);
    }
    @Override
    public boolean canWrite(Class clazz, MediaType mediaType) {
        return !XStreamListWrapper.class.isAssignableFrom(clazz) &&
            MediaType.APPLICATION_JSON.equals(mediaType);
    }
    @Override
    public List<MediaType> getSupportedMediaTypes() {
        return Arrays.asList(MediaType.APPLICATION_JSON);
    }
    @Override
    public Object read(Class clazz, HttpInputMessage inputMessage)
        throws IOException, HttpMessageNotReadableException
    {
        XStreamPersister p = xpf.createJSONPersister();
        p.setCatalog(catalog);
        return p.load(inputMessage.getBody(), clazz);
    }
    @Override
    public void write(Object o, MediaType contentType, HttpOutputMessage outputMessage)
        throws IOException, HttpMessageNotWritableException {
        XStreamPersister xmlPersister = xpf.createJSONPersister();
        xmlPersister.setCatalog(catalog);
        xmlPersister.setReferenceByName(true);
        xmlPersister.setExcludeIds();
        xmlPersister.save(o, outputMessage.getBody());
    }

}

RestWrapper

RestWrappers are used to provide additional configuration used by the converters alongside the objects returned by the controllers. The base controller class RestController provides utility methods wrapObject and wrapList for constructing these wrappers. All objects that get serialized by the XStreamXmlConverter, XStreamJSONConverter, or FreemarkerHtmlConverter should be wrapped (this generally only applies to GETs).

MVC configuration

MVCConfiguration is the class responsible for doing Spring MVC configuration. In particular adding converters, configuring content type negotiation, and adding intercepters. See the documentation here:

http://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/web/servlet/config/annotation/WebMvcConfigurationSupport.html

Controller Advice

RestControllerAdvice.java is primarily used to configure error message handling, but it can also be used for a number of things. See here for more controller advice functionality.

@ExceptionHandler(RestException.class)
public void handleRestException(RestException e, HttpServletResponse response, WebRequest request, OutputStream os)
    throws IOException {
    response.setStatus(e.getStatus().value());
    StreamUtils.copy(e.getMessage(), Charset.forName("UTF-8"), os);
}

HTML Output

The REST HTML format works slightly differently from the XML and JSON formats.

First of all, the HTML format only supports GET requests.

HTML output is generated using a Freemarker template (*.ftl) file. These files will generally exist alongside the controller code. Because of this, HTML gets require a bit of additional context. This is achieved by wrapping the response object in a FreemarkerConfigurationWrapper when returning from the controller. The RestController base class provides some utility methods for constructing this wrapper. This does mean that a separate get method is required for all controller endpoints that support HTML output.

For example:

@RequestMapping(
    path = "/styles/{styleName}", 
    method = RequestMethod.GET, 
    produces = {MediaType.TEXT_HTML_VALUE})
protected FreemarkerConfigurationWrapper getStyleFreemarker(
    @PathVariable String styleName) 
{
    return toFreemarkerMap(getStyleInternal(styleName, null)); //return FreemarkerConfigurationWrapper containing style object
}

Debugging Tips

Controller not hit/Response Code 415

The most common issue I've run into during the conversion was the handler method not being hit at all. This usually results in a response code 415 from Spring (media type not accepted). Debugging this ranges from simple to aggravating. Here are a few tips, from most obvious to least:

  • Is the request path correct?
  • Does your request Content-Type match the "consumes" parameter of the handler
  • Are all your path elements matched correctly?
  • Is the HttpMessageConverter you expect to be hit -- based on the requested content type -- actually being invoked? Be sure to check the canWrite/canRead method to see that it's returning true as expected.
  • Are you requesting something via extension (say .zip) that can't actually be produced (ie. POSTING to .zip when the controller only produces XML)
  • org.springframework.web.servlet.handler.AbstractHandlerMethodMapping#addMatchingMappings: This method goes through all the handlers to find the one that matches. Useful for debugging why a controller isn't being hit (415 response code). Digging around here is your last resort to find out WHY a specific handler is being rejected.

Documentation

And there is an outstanding bug to "improve" the documentation GEOS-7931.

  1. Create API documentation using Swagger.

    • See example showing styles endpoint: styles.yaml

    • Cut and paste example into online editor for writing, and then save into codebase.

  2. Restructure examples in the User Manual.

Reference API Documentation (Swagger)

A swagger "document" describes a REST interface:

  • Can be used to generate Spring MVC (if you are making a new API from scratch)
  • Can be used to generate human readable documentation
  • There are editors http://editor.swagger.io/
  • We considered "spring-fox annotions" - it did not work - it relies on static content, but Xstream is doing its serialization programmatically.

Examples of Swagger

  • See the /styles endpoint, written in Swagger, committed to the rest-api-refresh branch: styles.yaml

  • The aboveg example can be cut and pasted into the http://editor.swagger.io/ allowing you to play test the api and the doc generation.

Here is what that looks like:

swagger editor example for workspace

And a sample form the file:

---
swagger: '2.0'
info:
  version: 1.0.0
  title: GeoServer Styles
  description: A style describes how a resource is symbolized or rendered by the Web Map Service.

paths:
  /styles:
    get:
      operationId: getStyles
      summary: Get a list of styles
      description: Displays a list of all styles on the server. Use the "Accept:" header to specify format or append an extension to the endpoint (example "/styles.xml" for XML)
      produces:
        - text/html
        - application/json
        - application/xml
      responses:
        200:
          description: OK
          schema:
            $ref: "#/definitions/StyleList"
          examples:
            application/json: '
            {
                "styles": {
                    "style": [
                        {
                            "href": "http://localhost:8080/geoserver/rest/styles/burg.json",
                            "name": "burg"
                        },
                        {
                            "href": "http://localhost:8080/geoserver/rest/styles/capitals.json",
                            "name": "capitals"
                        }
                    ]
                }
            }
            '
            application/xml: '
              <styles>
                  <style>
                      <name>burg</name>
                      <atom:link xmlns:atom="http://www.w3.org/2005/Atom" rel="alternate" href="http://localhost:8080/geoserver/rest/styles/burg.xml" type="application/xml"/>
                  </style>
                  <style>
                      <name>capitals</name>
                      <atom:link xmlns:atom="http://www.w3.org/2005/Atom" rel="alternate" href="http://localhost:8080/geoserver/rest/styles/capitals.xml" type="application/xml"/>
                  </style>    
              </styles>'
        401:
          description: Unauthorized

Swagger Implementation

Location: Swagger API documentation lives in /doc/en/src/main/resources/api.

Reference

Swagger Specification

LOOK OUT When defining a parameter, the "in" property MUST BE LOWER CASE due to a bug in swagger-codegen:

      # ...
      Parameters:
        - name: styleName
          in: path # DEFINITELY NOT: Path
          required: true
          description: The name of the style to retrieve.
          type: string

REST Examples (Sphinx)

The existing documentation is written in RST, reviewing the 70+ REST API tickets documentation fixes may close a large portion of them (as many of are due to documentation confusion).

References:

REST API Example Update

Here is an example of updating a REST API example so we have multiple examples for each endpoint: styles.rst.

Listing all styles
------------------

**List all styles on the server, in JSON format:**

*Request*

.. admonition:: curl

   ::

     curl -u admin:geoserver -XGET http://localhost:8080/geoserver/rest/styles.json

.. admonition:: python

   TBD

.. admonition:: java

   TBD

*Response*

.. code-block:: json

   {"styles":{"style":[{"name":"burg","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/burg.json"},{"name":"capitals","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/capitals.json"},{"name":"dem","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/dem.json"},{"name":"generic","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/generic.json"},{"name":"giant_polygon","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/giant_polygon.json"},{"name":"grass","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/grass.json"},{"name":"green","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/green.json"},{"name":"line","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/line.json"},{"name":"poi","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/poi.json"},{"name":"point","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/point.json"},{"name":"polygon","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/polygon.json"},{"name":"poly_landmarks","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/poly_landmarks.json"},{"name":"pophatch","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/pophatch.json"},{"name":"population","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/population.json"},{"name":"rain","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/rain.json"},{"name":"raster","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/raster.json"},{"name":"restricted","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/restricted.json"},{"name":"simple_roads","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/simple_roads.json"},{"name":"simple_streams","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/simple_streams.json"},{"name":"tiger_roads","href":"http:\/\/localhost:8080\/geoserver\/rest\/styles\/tiger_roads.json"}]}}


**List all styles in a workspace, in XML format:**

*Request*

.. admonition:: curl

   ::

     curl -u admin:geoserver -XGET http://localhost:8080/geoserver/rest/cite/styles.xml

.. admonition:: python

   TBD

.. admonition:: java

   TBD

*Response*

.. code-block:: xml

   <styles>
     <style>
       <name>citerain</name>
       <atom:link xmlns:atom="http://www.w3.org/2005/Atom" rel="alternate" href="http://localhost:8080/geoserver/rest/workspaces/cite/styles/citerain.xml" type="application/xml"/>
     </style>
   </styles>

Out of Scope

The following activities are out of scope:

  • Audit functionality against GUI: As time permits make a note of any functionality missing from the REST API:

    • Shortlist missing functionality for proposal and implementation (examples recalculate feature type columns, rest layer bounds from SRS bounds, ...)
    • If there is already a JIRA issue, add the link to spreadsheet above

  • Shortlist documentation issues

  • Review outstanding REST API JIRA issues

©2020 Open Source Geospatial Foundation

Home
Proposals
Release Schedule

Clone this wiki locally