This is the backend of the new OSM Inspector Address view. OSM Inspector is a quality assurance service for OpenStreetMap data by the german consulting company Geofabrik GmbH and can be found at http://tools.geofabrik.de/osmi/?view=addresses. This software was written by Lukas Toggenburger as part of a project thesis for his master studies. It makes heavy use of libosmium (https://github.com/osmcode/libosmium) written by Jochen Topf.
I'd like to hear your feedback about this software. You can reach me per e-mail: lukas.toggenburgerXXhtwchur.ch (replace XX with @)
This software is available under the Boost Software License 1.0, see http://www.boost.org/LICENSE_1_0.txt.
Casual users interested in seeing address data may find the hosted version at http://tools.geofabrik.de/osmi/?view=addresses much more useful than running this software. A WxS is also available there, see http://wiki.openstreetmap.org/wiki/OSM_Inspector/WxS on how to use it.
If you are indeed interested in the backend, e.g. to see usage of libosmium in a bigger project, you have come to the right place. This software will take OSM data (XML or PBF; planet file or parts of it) data as input and produce SpatiaLite files (one for each layer) as output. The output format is easily changeable due to GDAL/OGR being utilized to hold and write data.
A file can be processed like this:
./osmi-addresses planet-latest.osm.pbf
By default an output directory called osmi-addresses_sqlite_out is created in the current directory (that means: not necessarily the directory the binary resides in). If a second parameter is given, the name of the output directory can be changed:
./osmi-addresses planet-latest.osm.pbf my-output-dir
An existing directory will not be overwritten, the software will abort instead.
In order to improve overall calculation speed, spatial indices are not calculated while writing the SpatiaLite files. Instead you are supposed to call
./create_spatial_indices.sh osmi-addresses_sqlite_out
to add them afterwards.
The software was tested on Ubuntu but probably runs under other Unix variants (incl. Mac OS X) as well.
You should find at least two branches in the GitHub repository:
master: The development version containing the newest features and fixescurrently_running_on_geofabrik_server: The software as it is running at http://tools.geofabrik.de/osmi/?view=addresses
You will need a 64-bit system, a C++11 compiler and libosmium to compile the software. You can find libosmium at https://github.com/osmcode/libosmium .
Jochen updates libosmium quite often, so I cloned libosmium and made a soft-link to it (after making sure that /usr/local/include did not exist before):
sudo ln -s /path/to/libosmium/include /usr/local/include
By this I can easily pull in new versions of libosmium.
libosmium itself comes with a list of prerequisites, which you can find at https://github.com/osmcode/libosmium#prerequisites.
On Ubuntu/Debian, you should be able to get all of libosmium's and osmi-addresses's dependencies with:
sudo apt-get install clang libboost-dev libboost-filesystem-dev libgdal-dev libsparsehash-dev libbz2-dev libexpat1-dev sqlite3 spatialite-bin colordiff parallel realpath
Switch to the source directory:
cd osmi
Compile using GCC:
make
Or compile using clang:
CXX=clang++ make
The compiled executable osmi is a standalone application and needs no installation.
If you have never used clang before, you should give it a try. It compiles (at least this software) slightly faster and gives better understandable error messages if you mess things up.
switch to test dir
cd test
remove the default output directory if it already exists
rm -rf osmi-addresses_sqlite_out
make all the IDs in the testzone file positive, so that this tool can handle them
./makeidpositive.sh osmi-testzone.osm
process the testzone file
../osmi/osmi-addresses pos-osmi-testzone.osm
create spatial indices
../create_spatial_indices.sh osmi-addresses_sqlite_out/
check results
./run_tests.sh osmi-addresses_sqlite_out
To test local changes, you can run a MapServer instance that serves the generated results as map overlays. The base layer tiles are loaded directly from the openstreetmap.org servers.
Install MapServer (Ubuntu):
sudo apt-get install mapserver-bin cgi-mapserver apache2 proj-data unifont
Activate CGI:
sudo a2enmod cgi; service apache2 restart
Create a logfile with suitable permissions:
sudo mkdir /var/log/mapserver/ ; sudo touch /var/log/mapserver/addresses.log ; chmod a+w /var/log/mapserver/addresses.log
Open /usr/share/proj/epsg, duplicate the line starting with <3857> and change the beginning to <900913> in one of the lines.
The addresses.map file is the configuration file as used on the production server running the full OSM inspector. The file addresses.local.map is a derived version that configures MapServer to show the local results. It is generated from adresses.map as follows:
- For each layer disable the lines starting with
TILEINDEXorTILEITEM - For each layer add a line
CONNECTION "X"where X is the path to the .sqlite file - For each layer add a line
DATA "X"where X is the name of the table in the sqlite file, e.g.osmi_addresses_nearest_roads(the tables carry always the same name as the file)
Set the default location of the .map file: Add the line
SetEnvIf Request_URI "/cgi-bin/mapserv" MS_MAPFILE=/absolute/path/to/addresses.local.map
into your Apache site config, e.g. into /etc/apache2/sites-available/000-default.conf. Restart apache with service apache2 restart.
MapServer will then access addresses.local.map, which refers to the data in test/osmi-addresses_sqlite_out/.
Copy the "viewer" directory to a place where it is served by the web server, e.g. /var/www/html. Then, open the contained index.html (e.g. http://localhost/viewer/index.html) in a browser to comfortably look at the (locally) detected address problems.
msLoadMap(): Regular expression error. MS_DEFAULT_MAPFILE_PATTERN validation failed. msEvalRegex(): Regular expression error. String failed expression test.
Make sure that your .map file actually ends with .map. See also http://gis.stackexchange.com/a/11444
The logfile's location is configured using CONFIG MS_ERRORFILE. Try /var/log/mapserver/addresses.log
Have you included the desired layer in your HTTP request?
A given input file will be processed in two passes. This happens by the classes FirstHandler and SecondHandler. The first pass is necessary to build a lookup structure to get street geometries based on the street's name. The second pass accesses this structure and writes the .sqlite files.
Here is a hierarchical overview of calls/accesses:
FirstHandlerway()addr_interpolation_node_sethighway_lookup_type->name2highway_areahighway_lookup_type->name2highway_nonarea
SecondHandlernode()ConnectionLinePreprocessor.process_node()-
EntrancesWriter.feed_node()NodesWithAddressesWriter.process_node()
way()BuildingsWriter.feed_way()ConnectionLinePreprocessor.process_way()(reads buildings, etc. withaddr:streettag)-
InterpolationWriter.feed_way()(reads interpolation lines withaddr:interpolationtag)ConnectionLinePreprocessor.process_interpolated_node()-
NodesWithAddressesWriter.process_interpolated_node()
NodesWithAddressesWriter.process_way()WaysWithAddressesWriter.feed_way()WaysWithPostalCodeWriter.feed_way()
-
- =
ConnectionLinePreprocessor.handle_connection_line()
ConnectionLineWriter.write_line()NearestRoadsWriter.write_road()XORNearestAreasWriter.write_area()NearestPointsWriter.write_point()
- =
Not having enough memory results in a segmentation fault. gdb will show a message like this:
Program received signal SIGSEGV, Segmentation fault.
google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> >::sparsegroup (this=0x1ba5d560, x=...)
at /usr/include/sparsehash/sparsetable:992
992 sparsegroup(const sparsegroup& x) : group(0), settings(x.settings) {
This is also discussed in osmcode/libosmium#23 .