Skip to content

Usage

Jump to bottom Edit New page
Adam Franco edited this page May 27, 2016 · 2 revisions

Usage

Curvature works with Open Street Map (OSM) XML data files. While you can export these from a small area directly from openstreetmap.org , you are limited to a small area with a limited number of points so that you don't overwhelm the OSM system. A better alternative is to download daily exports of OSM data for your region from Planet.osm.

This script was developed using downloads of the US state OSM data provided at: download.geofabrik.de/openstreetmap

Once you have downloaded a .osm or .osm.pbf file that you wish to work with, you can run one of the the predefined processing changes with their default options:

./processing_chains/adams_defaults.sh -v vermont.osm.pbf

This will generate a set of compressed KMZ files that includes lines for all of the matched segments.

Use

./processing_chains/adams_defaults.sh -h

for more options.

Basic KML Output

Open the KML files in Google Earth. GoogleEarth can become overloaded with giant KML files, so the default mode is to generate a single placemark for each matching way that has a single-color line-string. On a reasonably modern computer even large files such as roads with curvature greater than 300 in California can be rendered when formatted in this way.

Colored-curves KML Output

Additionally, this script allows rendering of ways as a sequence of segments color-coded based on the curvature of each segment. These 'colorized' KML files provide a neat look at the radius of turns and are especially useful when tuning the radii and weights for each level to adjust which ways are matched. The color-coding is as follows: straight segments are green and the four levels of curves are color-coded from yellow to red with decreasing turn radii.

Unfortunately, these multi-colored ways are significantly more difficult for GoogleEarth to render than the longer single-color segments, so try to avoid files larger than about 40MB. Google Earth can easily handle smaller areas like Vermont with a curvature of 300, but larger regions (like California) may need to be trimmed down with a bounding box if you wish to make usable multicolor KML files.

Building your own processing scripts

The processing-chains in curvature/processing_chains/ are just a few examples of ways to use this program. You can easily pipe output from one filter to the next to modify the data stream to your liking.

The easiest way to create your own processing chain is to copy adams_default.sh and edit the sequence of steps to filter more or fewer road segments.

Collecting

Usually, any processing-chain will start with a call to the bin/curvature-collect script to take the input .osm.pbf data and convert it to a stream of collections. Each collection is an ordered sequence of one or more Open Street Map ways. Where possible, ways are joined end to end into long sequences that all share the same name tag or ref tag (used for routes, like US 7). This collecting process gives us the most complete linear "roads" it is possible to extract from the data and allows us to look at a long shape of a road as a unit even if it is broken up into many different ways to account for changing speed-limit tags, bridge tags, etc.

The input of the collecting process is an osm-xml file or an .osm.pbf file. The output is a MessagePack stream written to STDOUT, which can then either be piped to another script or written to a file.

Example, writing to a file:

bin/curvature-collect -v ~/Downloads/vermont.osm.pbf > vermont.msgpack

Example, piping to other scripts:

bin/curvature-collect -v ~/Downloads/vermont.osm.pbf | bin/msgpack-reader

Note the binary MessagePack format is very fast to read/write, but it is not particularly space-efficient. When writing to disk (instead of piping between scripts) it may be faster (and is definitely more space-efficient) to pipe the MessagePack data through GZIP:

bin/curvature-collect -v ~/Downloads/vermont.osm.pbf | gzip > vermont.msgpack.gz

cat vermont.msgpack.gz | gunzip | bin/msgpack-reader | head -n 50

Reading the MessagePack stream

When developing processing-chains, it is sometimes helpful to get a human-readable view of the output at each stage. The bin/msgpack-reader program takes a MessagePack stream on STDIN and writes a human-readable version of the data on STDOUT. Example:

bin/curvature-collect -v ~/Downloads/vermont.osm.pbf | bin/msgpack-reader | head -n 50

{   'join_type': 'none',
'ways': [   {   'coords': [   [43.6371221, -72.33376860000001],
                              [43.6372364, -72.33391470000001],
                              [43.6375166, -72.33431750000001],
                              [43.6376378, -72.33447530000001],
                              [43.6378658, -72.3347415],
                              [43.6379279, -72.33480050000001],
                              [43.6380124, -72.33487350000001],
                              [43.6381617, -72.33499220000002],
                              [43.6382817, -72.33507130000001],
                              [43.6384463, -72.33515430000001],
                              [43.638724499999995, -72.33528830000002],
                              [43.639139199999995, -72.33549630000002],
                              [43.63962419999999, -72.33573870000002],
                              [43.640433299999984, -72.33617290000034],
                              [43.64048309999998, -72.33620290000033],
                              [43.640566299999996, -72.33625740000002]],
                'id': 4217987,
                'refs': [   25060849,
                            25060861,
                            25060855,
                            25060841,
                            25060845,
                            25060835,
                            25062311,
                            25062317,
                            25062321,
                            25062326,
                            25062331,
                            25062337,
                            25062343,
                            652095695,
                            652095698,
                            25062350],
                'tags': {   'destination:ref': 'I 91 North',
                            'highway': 'motorway_link',
                            'oneway': 'yes',
                            'surface': 'asphalt',
                            'tiger:cfcc': 'A63',
                            'tiger:county': 'Windsor, VT'}}]}

The head post processor is also helpful for viewing output as it will truncate the output to just the first n collections (whereas the POSIX head command works on number of lines).

Example:

bin/curvature-collect -v ~/Downloads/vermont.osm.pbf | bin/curvature-pp head -n 2 | bin/msgpack-reader

Calculating

The filter, splitting, sorting, and modifying operations are handled by a set of post processors found at curvature/cuvature/post_processors/. Post processors are called by piping the MessagePack stream to the bin/curvature-pp program with the post-processor desired as the first argument.

The add_segments, add_segment_length_and_radius, add_segment_curvature, and filter_segment_deflections post processors are generally used together to analyze the geometry of each way.

  1. add_segments converts the sequence of coordinates that make up a way into pairs of coordinates that make up a line-segment. This new segment object in the data stream can then have additional calculations performed on it.

  2. add_segment_length_and_radius calculates the length of the segment as well as the curve-radius for that segment.

  3. add_segment_curvature multiplies a radius-classified weight by the segment-length to give us the cuvature value for the segment.

  4. filter_segment_deflections looks for jiggles in otherwise straight ways and filters out the curvature value for these segments to prevent noisy data from being interpreted the same as hairpin curves.

Other calculating post-processors are roll_up_length, roll_up_curvature, and remove_way_properties.

Filtering and splitting

The filter_xxxx_ways_xxxx, filter_collections_by_xxxx and split_collections_on_xxxx post processors allow you to remove collections (or portions of collections) that don't meet your criteria.

For example, you may only want to include roads that are paved, so you might use the filter_out_ways_with_tag post processor to remove ways that have surface tags that indicate an unpaved road-surface.

cat vermont.msgpack | bin/curvature-pp filter_out_ways_with_tag --tag surface --values 'unpaved,dirt,gravel,fine_gravel,sand,grass,ground,pebblestone,mud,clay,dirt/sand,soil' | bin/msgpack-reader

You can also filter on complex boolean expressions using the filter_out_ways post processor, such as all of the driveways in the US that were incorrectly imported as unnamed 'residential' ways:

cat vermont.msgpack | bin/curvature-pp filter_out_ways --match 'And(TagEmpty("name"), TagEmpty("ref"), TagEquals("highway", "residential"), TagEquals("tiger:reviewed", "no"))' | bin/msgpack-reader

The filter_xxxx_ways_xxxx and split_collections_on_xxxx post processors will break apart collections into multiple resulting collections, so if you ran the above example on an input file that had a road that started paved, became gravel, then became paved again, the output would have 2 collections (one for each paved portion) and the gravel portion would be dropped.

The filter_collections_by_xxxx post processors simply drop entire collections without modifying them. For example, you could exclude road-sections shorter than 1 mile (1609 meters) with:

cat vermont.msgpack | bin/curvature-pp filter_collections_by_length --min 1609 | bin/msgpack-reader

Output

After you have massaged the data to your liking, you can then pipe the MessagePack stream to one of the output programs like bin/curvature-output-kml or bin/curvature-output-tab to write a data-file. Example:

cat vermont.msgpack | curvature-output-kml > doc.kml
Add a custom footer
Add a custom sidebar
Clone this wiki locally