ubodt_gen command line program fails to create output file.

[sardounis]

I am trying to run ubodt_gen command, providing an xml configuration file, using an edges.shp file I created via osmnx.

In this process I get the error: [critical][network.cpp:112] Id, source or target column not found
libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Id, source or target column not found
Abort trap: 6

The edges.shp file I am trying to process is:
edges.shp.zip

Below you may find all the logs of the ubodt_gen run:

ubodt_gen src/helpers/fmm/ubodt_config.xml
[info][ubodt_gen_app_config.cpp:28 ] Read configuration from xml file: src/helpers/fmm/ubodt_config.xml
[info][ubodt_gen_app_config.cpp:38 ] Read configuration from xml file done
[info][ubodt_gen_app_config.cpp:74 ] ---- Print configuration ----
[info][network_config.cpp:6 ] NetworkConfig
[info][network_config.cpp:7 ] File name: greece-attica/edges.shp
[info][network_config.cpp:8 ] ID name: id
[info][network_config.cpp:9 ] Source name: source
[info][network_config.cpp:10 ] Target name: target
[info][network_config.cpp:11 ] Mode name: drive
[info][ubodt_gen_app_config.cpp:76 ] Delta 3
[info][ubodt_gen_app_config.cpp:77 ] Output file ubodt.txt
[info][ubodt_gen_app_config.cpp:78 ] Log level 2-info
[info][ubodt_gen_app_config.cpp:79 ] Use omp false
[info][ubodt_gen_app_config.cpp:80 ] ---- Print configuration done ----
[info][ubodt_gen_app_config.cpp:97 ] Validating configuration for UBODT construction
[info][ubodt_gen_app_config.cpp:118] Validating done.
[info][network.cpp:91 ] Read network from file greece-attica/edges.shp
[critical][network.cpp:112] Id, source or target column not found
libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Id, source or target column not found
Abort trap: 6

You may find the ubodt_config.xml below:

<?xml version="1.0" encoding="utf-8"?> <config> <input> <network> <file>greece-attica/edges.shp</file> <id>id</id> <source>source</source> <target>target</target> </network> </input> <parameters> <delta>3</delta> </parameters> <output> <file>ubodt.txt</file> </output> </config>

[cyang-kth]

Please follow the tutorial https://github.com/cyang-kth/osm_mapmatching to download the edges.shp file rather than directly from osmnx because the default edges.shp saved by osmnx is still inconsistent with several issues, e.g., id field.

The default edges.shp from OSMNX does not provide source, target, you need to rename those fields yourself.

[cyang-kth]

The two differences are

• OSMNX shapefile has source field of u and target field of v
• The default osm_id of the shapefile could contain a list of original OSM id so finally a feature index is created as fid

On this page https://github.com/cyang-kth/osm_mapmatching

# We need an unique ID for each edge
    gdf_edges["fid"] = gdf_edges.index

The configuration of reading is also set as

### Read network data

network = Network("network/edges.shp","fid","u","v")

[sardounis]

Great! So if I understand correctly in order to produce a correct edges.shp, I need to change df_edges["fid"] = gdf_edges.index to df_edges["osm_id"] = gdf_edges.index. Is this correct? If not how shouls I change this assignment to create a valid edges.shp to feed ubodt_gen?

[cyang-kth]

You can just keep the fid and specify it as the id name of shapefile

You can check this link
https://github.com/cyang-kth/fmm/tree/master/example/osmnx_example

[sardounis]

Great. I changed the ubodt_config.xml network tag, to :

<network> <file>greece-attica/edges.shp</file> <id>fid</id> <source>u</source> <target>v</target> </network>

And it worked. Though now the script has stacked here (maybe due to the size of the data?):

[info][ubodt_gen_app_config.cpp:28 ] Read configuration from xml file: src/helpers/fmm/ubodt_config.xml
[info][ubodt_gen_app_config.cpp:38 ] Read configuration from xml file done
[info][ubodt_gen_app_config.cpp:74 ] ---- Print configuration ----
[info][network_config.cpp:6 ] NetworkConfig
[info][network_config.cpp:7 ] File name: greece-attica/edges.shp
[info][network_config.cpp:8 ] ID name: fid
[info][network_config.cpp:9 ] Source name: u
[info][network_config.cpp:10 ] Target name: v
[info][network_config.cpp:11 ] Mode name: drive
[info][ubodt_gen_app_config.cpp:76 ] Delta 3
[info][ubodt_gen_app_config.cpp:77 ] Output file ubodt.txt
[info][ubodt_gen_app_config.cpp:78 ] Log level 2-info
[info][ubodt_gen_app_config.cpp:79 ] Use omp false
[info][ubodt_gen_app_config.cpp:80 ] ---- Print configuration done ----
[info][ubodt_gen_app_config.cpp:97 ] Validating configuration for UBODT construction
[info][ubodt_gen_app_config.cpp:118] Validating done.
[info][network.cpp:91 ] Read network from file greece-attica/edges.shp
[info][network.cpp:185] Number of edges 296895 nodes 123531
[info][network.cpp:187] Field index: id 18 source 15 target 16
[info][network.cpp:189] Read network done.
[info][network_graph.cpp:17 ] Construct graph from network edges start
[info][network_graph.cpp:30 ] Graph nodes 123531 edges 296895
[info][network_graph.cpp:31 ] Construct graph from network edges end
[info][ubodt_gen_app.cpp:21 ] Write UBODT to file ubodt.txt
[info][ubodt_gen_algorithm.cpp:42 ] Start to generate UBODT with delta 3
[info][ubodt_gen_algorithm.cpp:43 ] Output format csv
[info][ubodt_gen_algorithm.cpp:58 ] Progress 0 / 123531

[cyang-kth]

Your delta equals 300km, which is too big, you can try 0.03 instead of 3.

[sardounis]

Great thanks, I will!

[sardounis]

I was able to produce the ubodt.txt file which got around 10 GB of storage in my disc. Then I tried to run the fmm command, though it took more than 5 minutes to finish, by only providing two coordinates to the algorithm. Is this normal? Is it possible to decrease the time drastically? Will changing k,r and e values help?

The fmm command I ran was: fmm --ubodt ubodt.txt --network greece-attica/edges.shp --network_id fid --source u --target v --gps src/helpers/fmm/traj.csv -k 8 -r 0.003 -e 0.0005 --output src/helpers/fmm/fmm_output.txt --use_omp

which produced the output below:

[info][fmm_app_config.cpp:49 ] Start reading FMM configuration from arguments
[info][fmm_app_config.cpp:78 ] Finish with reading FMM arg configuration
[info][fmm_app_config.cpp:98 ] ---- Print configuration ----
[info][network_config.cpp:6 ] NetworkConfig
[info][network_config.cpp:7 ] File name: greece-attica/edges.shp
[info][network_config.cpp:8 ] ID name: fid
[info][network_config.cpp:9 ] Source name: u
[info][network_config.cpp:10 ] Target name: v
[info][network_config.cpp:11 ] Mode name: drive
[info][gps_config.cpp:13 ] GPS format: CSV trajectory
[info][gps_config.cpp:14 ] File name: src/helpers/fmm/traj.csv
[info][gps_config.cpp:15 ] ID name: id
[info][gps_config.cpp:16 ] Geom name: geom
[info][gps_config.cpp:17 ] Timestamp name: timestamp
[info][result_config.cpp:34 ] ResultConfig
[info][result_config.cpp:35 ] File: src/helpers/fmm/fmm_output.txt
[info][result_config.cpp:36 ] Fields: cpath mgeom
[info][fmm_algorithm.cpp:27 ] FMMAlgorithmConfig
[info][fmm_algorithm.cpp:29 ] k 8 radius 0.003 gps_error 0.0005 reverse_tolerance 0
[info][fmm_app_config.cpp:103] Log level 2-info
[info][fmm_app_config.cpp:104] Step 100
[info][fmm_app_config.cpp:105] Use omp true
[info][fmm_app_config.cpp:106] ---- Print configuration done ----
[info][network.cpp:91 ] Read network from file greece-attica/edges.shp
[info][network.cpp:185] Number of edges 296895 nodes 123531
[info][network.cpp:187] Field index: id 18 source 15 target 16
[info][network.cpp:189] Read network done.
[info][network_graph.cpp:17 ] Construct graph from network edges start
[info][network_graph.cpp:30 ] Graph nodes 123531 edges 296895
[info][network_graph.cpp:31 ] Construct graph from network edges end
[info][ubodt.cpp:205] Reading UBODT file (CSV format) from ubodt.txt
[info][ubodt.cpp:233] Read rows 1000000
[info][ubodt.cpp:233] Read rows 2000000
[info][ubodt.cpp:233] Read rows 3000000
[info][ubodt.cpp:233] Read rows 4000000
[info][ubodt.cpp:233] Read rows 5000000
[info][ubodt.cpp:233] Read rows 6000000
[info][ubodt.cpp:233] Read rows 7000000
[info][ubodt.cpp:233] Read rows 8000000
[info][ubodt.cpp:233] Read rows 9000000
[info][ubodt.cpp:233] Read rows 10000000
[info][ubodt.cpp:233] Read rows 11000000
[info][ubodt.cpp:233] Read rows 12000000
[info][ubodt.cpp:233] Read rows 13000000
[info][ubodt.cpp:233] Read rows 14000000
[info][ubodt.cpp:233] Read rows 15000000
[info][ubodt.cpp:233] Read rows 16000000
[info][ubodt.cpp:233] Read rows 17000000
[info][ubodt.cpp:233] Read rows 18000000
[info][ubodt.cpp:233] Read rows 19000000
[info][ubodt.cpp:233] Read rows 20000000
[info][ubodt.cpp:233] Read rows 21000000
[info][ubodt.cpp:233] Read rows 22000000
[info][ubodt.cpp:233] Read rows 23000000
[info][ubodt.cpp:233] Read rows 24000000
[info][ubodt.cpp:233] Read rows 25000000
[info][ubodt.cpp:233] Read rows 26000000
[info][ubodt.cpp:233] Read rows 27000000
[info][ubodt.cpp:233] Read rows 28000000
[info][ubodt.cpp:233] Read rows 29000000
[info][ubodt.cpp:233] Read rows 30000000
[info][ubodt.cpp:233] Read rows 31000000
[info][ubodt.cpp:233] Read rows 32000000
[info][ubodt.cpp:233] Read rows 33000000
[info][ubodt.cpp:233] Read rows 34000000
[info][ubodt.cpp:233] Read rows 35000000
[info][ubodt.cpp:233] Read rows 36000000
[info][ubodt.cpp:233] Read rows 37000000
[info][ubodt.cpp:233] Read rows 38000000
[info][ubodt.cpp:233] Read rows 39000000
[info][ubodt.cpp:233] Read rows 40000000
[info][ubodt.cpp:233] Read rows 41000000
[info][ubodt.cpp:233] Read rows 42000000
[info][ubodt.cpp:233] Read rows 43000000
[info][ubodt.cpp:233] Read rows 44000000
[info][ubodt.cpp:233] Read rows 45000000
[info][ubodt.cpp:233] Read rows 46000000
[info][ubodt.cpp:233] Read rows 47000000
[info][ubodt.cpp:233] Read rows 48000000
[info][ubodt.cpp:233] Read rows 49000000
[info][ubodt.cpp:233] Read rows 50000000
[info][ubodt.cpp:233] Read rows 51000000
[info][ubodt.cpp:233] Read rows 52000000
[info][ubodt.cpp:233] Read rows 53000000
[info][ubodt.cpp:233] Read rows 54000000
[info][ubodt.cpp:233] Read rows 55000000
[info][ubodt.cpp:233] Read rows 56000000
[info][ubodt.cpp:233] Read rows 57000000
[info][ubodt.cpp:233] Read rows 58000000
[info][ubodt.cpp:233] Read rows 59000000
[info][ubodt.cpp:233] Read rows 60000000
[info][ubodt.cpp:233] Read rows 61000000
[info][ubodt.cpp:233] Read rows 62000000
[info][ubodt.cpp:233] Read rows 63000000
[info][ubodt.cpp:233] Read rows 64000000
[info][ubodt.cpp:233] Read rows 65000000
[info][ubodt.cpp:233] Read rows 66000000
[info][ubodt.cpp:233] Read rows 67000000
[info][ubodt.cpp:233] Read rows 68000000
[info][ubodt.cpp:233] Read rows 69000000
[info][ubodt.cpp:233] Read rows 70000000
[info][ubodt.cpp:233] Read rows 71000000
[info][ubodt.cpp:233] Read rows 72000000
[info][ubodt.cpp:233] Read rows 73000000
[info][ubodt.cpp:233] Read rows 74000000
[info][ubodt.cpp:233] Read rows 75000000
[info][ubodt.cpp:233] Read rows 76000000
[info][ubodt.cpp:233] Read rows 77000000
[info][ubodt.cpp:233] Read rows 78000000
[info][ubodt.cpp:233] Read rows 79000000
[info][ubodt.cpp:233] Read rows 80000000
[info][ubodt.cpp:233] Read rows 81000000
[info][ubodt.cpp:233] Read rows 82000000
[info][ubodt.cpp:233] Read rows 83000000
[info][ubodt.cpp:233] Read rows 84000000
[info][ubodt.cpp:233] Read rows 85000000
[info][ubodt.cpp:233] Read rows 86000000
[info][ubodt.cpp:233] Read rows 87000000
[info][ubodt.cpp:233] Read rows 88000000
[info][ubodt.cpp:233] Read rows 89000000
[info][ubodt.cpp:233] Read rows 90000000
[info][ubodt.cpp:233] Read rows 91000000
[info][ubodt.cpp:233] Read rows 92000000
[info][ubodt.cpp:233] Read rows 93000000
[info][ubodt.cpp:233] Read rows 94000000
[info][ubodt.cpp:233] Read rows 95000000
[info][ubodt.cpp:233] Read rows 96000000
[info][ubodt.cpp:233] Read rows 97000000
[info][ubodt.cpp:233] Read rows 98000000
[info][ubodt.cpp:233] Read rows 99000000
[info][ubodt.cpp:233] Read rows 100000000
[info][ubodt.cpp:233] Read rows 101000000
[info][ubodt.cpp:233] Read rows 102000000
[info][ubodt.cpp:233] Read rows 103000000
[info][ubodt.cpp:233] Read rows 104000000
[info][ubodt.cpp:233] Read rows 105000000
[info][ubodt.cpp:233] Read rows 106000000
[info][ubodt.cpp:233] Read rows 107000000
[info][ubodt.cpp:233] Read rows 108000000
[info][ubodt.cpp:233] Read rows 109000000
[info][ubodt.cpp:233] Read rows 110000000
[info][ubodt.cpp:233] Read rows 111000000
[info][ubodt.cpp:233] Read rows 112000000
[info][ubodt.cpp:233] Read rows 113000000
[info][ubodt.cpp:233] Read rows 114000000
[info][ubodt.cpp:233] Read rows 115000000
[info][ubodt.cpp:233] Read rows 116000000
[info][ubodt.cpp:233] Read rows 117000000
[info][ubodt.cpp:233] Read rows 118000000
[info][ubodt.cpp:233] Read rows 119000000
[info][ubodt.cpp:233] Read rows 120000000
[info][ubodt.cpp:233] Read rows 121000000
[info][ubodt.cpp:233] Read rows 122000000
[info][ubodt.cpp:233] Read rows 123000000
[info][ubodt.cpp:233] Read rows 124000000
[info][ubodt.cpp:233] Read rows 125000000
[info][ubodt.cpp:233] Read rows 126000000
[info][ubodt.cpp:233] Read rows 127000000
[info][ubodt.cpp:233] Read rows 128000000
[info][ubodt.cpp:233] Read rows 129000000
[info][ubodt.cpp:233] Read rows 130000000
[info][ubodt.cpp:233] Read rows 131000000
[info][ubodt.cpp:233] Read rows 132000000
[info][ubodt.cpp:233] Read rows 133000000
[info][ubodt.cpp:233] Read rows 134000000
[info][ubodt.cpp:233] Read rows 135000000
[info][ubodt.cpp:233] Read rows 136000000
[info][ubodt.cpp:233] Read rows 137000000
[info][ubodt.cpp:233] Read rows 138000000
[info][ubodt.cpp:233] Read rows 139000000
[info][ubodt.cpp:233] Read rows 140000000
[info][ubodt.cpp:233] Read rows 141000000
[info][ubodt.cpp:233] Read rows 142000000
[info][ubodt.cpp:233] Read rows 143000000
[info][ubodt.cpp:233] Read rows 144000000
[info][ubodt.cpp:233] Read rows 145000000
[info][ubodt.cpp:233] Read rows 146000000
[info][ubodt.cpp:233] Read rows 147000000
[info][ubodt.cpp:233] Read rows 148000000
[info][ubodt.cpp:233] Read rows 149000000
[info][ubodt.cpp:233] Read rows 150000000
[info][ubodt.cpp:233] Read rows 151000000
[info][ubodt.cpp:233] Read rows 152000000
[info][ubodt.cpp:233] Read rows 153000000
[info][ubodt.cpp:233] Read rows 154000000
[info][ubodt.cpp:233] Read rows 155000000
[info][ubodt.cpp:233] Read rows 156000000
[info][ubodt.cpp:233] Read rows 157000000
[info][ubodt.cpp:233] Read rows 158000000
[info][ubodt.cpp:233] Read rows 159000000
[info][ubodt.cpp:233] Read rows 160000000
[info][ubodt.cpp:233] Read rows 161000000
[info][ubodt.cpp:233] Read rows 162000000
[info][ubodt.cpp:233] Read rows 163000000
[info][ubodt.cpp:233] Read rows 164000000
[info][ubodt.cpp:233] Read rows 165000000
[info][ubodt.cpp:233] Read rows 166000000
[info][ubodt.cpp:233] Read rows 167000000
[info][ubodt.cpp:233] Read rows 168000000
[info][ubodt.cpp:233] Read rows 169000000
[info][ubodt.cpp:233] Read rows 170000000
[info][ubodt.cpp:233] Read rows 171000000
[info][ubodt.cpp:233] Read rows 172000000
[info][ubodt.cpp:233] Read rows 173000000
[info][ubodt.cpp:233] Read rows 174000000
[info][ubodt.cpp:233] Read rows 175000000
[info][ubodt.cpp:233] Read rows 176000000
[info][ubodt.cpp:233] Read rows 177000000
[info][ubodt.cpp:233] Read rows 178000000
[info][ubodt.cpp:233] Read rows 179000000
[info][ubodt.cpp:233] Read rows 180000000
[info][ubodt.cpp:233] Read rows 181000000
[info][ubodt.cpp:233] Read rows 182000000
[info][ubodt.cpp:233] Read rows 183000000
[info][ubodt.cpp:233] Read rows 184000000
[info][ubodt.cpp:233] Read rows 185000000
[info][ubodt.cpp:233] Read rows 186000000
[info][ubodt.cpp:233] Read rows 187000000
[info][ubodt.cpp:233] Read rows 188000000
[info][ubodt.cpp:233] Read rows 189000000
[info][ubodt.cpp:233] Read rows 190000000
[info][ubodt.cpp:233] Read rows 191000000
[info][ubodt.cpp:233] Read rows 192000000
[info][ubodt.cpp:233] Read rows 193000000
[info][ubodt.cpp:233] Read rows 194000000
[info][ubodt.cpp:233] Read rows 195000000
[info][ubodt.cpp:233] Read rows 196000000
[info][ubodt.cpp:233] Read rows 197000000
[info][ubodt.cpp:233] Read rows 198000000
[info][ubodt.cpp:233] Read rows 199000000
[info][ubodt.cpp:233] Read rows 200000000
[info][ubodt.cpp:233] Read rows 201000000
[info][ubodt.cpp:233] Read rows 202000000
[info][ubodt.cpp:233] Read rows 203000000
[info][ubodt.cpp:233] Read rows 204000000
[info][ubodt.cpp:233] Read rows 205000000
[info][ubodt.cpp:233] Read rows 206000000
[info][ubodt.cpp:233] Read rows 207000000
[info][ubodt.cpp:233] Read rows 208000000
[info][ubodt.cpp:233] Read rows 209000000
[info][ubodt.cpp:233] Read rows 210000000
[info][ubodt.cpp:233] Read rows 211000000
[info][ubodt.cpp:233] Read rows 212000000
[info][ubodt.cpp:233] Read rows 213000000
[info][ubodt.cpp:233] Read rows 214000000
[info][ubodt.cpp:233] Read rows 215000000
[info][ubodt.cpp:233] Read rows 216000000
[info][ubodt.cpp:233] Read rows 217000000
[info][ubodt.cpp:233] Read rows 218000000
[info][ubodt.cpp:233] Read rows 219000000
[info][ubodt.cpp:233] Read rows 220000000
[info][ubodt.cpp:233] Read rows 221000000
[info][ubodt.cpp:233] Read rows 222000000
[info][ubodt.cpp:233] Read rows 223000000
[info][ubodt.cpp:233] Read rows 224000000
[info][ubodt.cpp:233] Read rows 225000000
[info][ubodt.cpp:233] Read rows 226000000
[info][ubodt.cpp:233] Read rows 227000000
[info][ubodt.cpp:233] Read rows 228000000
[info][ubodt.cpp:233] Read rows 229000000
[info][ubodt.cpp:233] Read rows 230000000
[info][ubodt.cpp:233] Read rows 231000000
[info][ubodt.cpp:233] Read rows 232000000
[info][ubodt.cpp:233] Read rows 233000000
[info][ubodt.cpp:233] Read rows 234000000
[info][ubodt.cpp:233] Read rows 235000000
[info][ubodt.cpp:233] Read rows 236000000
[info][ubodt.cpp:233] Read rows 237000000
[info][ubodt.cpp:233] Read rows 238000000
[info][ubodt.cpp:233] Read rows 239000000
[info][ubodt.cpp:233] Read rows 240000000
[info][ubodt.cpp:233] Read rows 241000000
[info][ubodt.cpp:233] Read rows 242000000
[info][ubodt.cpp:233] Read rows 243000000
[info][ubodt.cpp:233] Read rows 244000000
[info][ubodt.cpp:233] Read rows 245000000
[info][ubodt.cpp:233] Read rows 246000000
[info][ubodt.cpp:233] Read rows 247000000
[info][ubodt.cpp:233] Read rows 248000000
[info][ubodt.cpp:233] Read rows 249000000
[info][ubodt.cpp:233] Read rows 250000000
[info][ubodt.cpp:233] Read rows 251000000
[info][ubodt.cpp:233] Read rows 252000000
[warning][ubodt.cpp:239] Load factor is too large.
[info][ubodt.cpp:240] Finish reading UBODT with rows 252452685
[info][ubodt.cpp:199] Read UBODT file in 211.104 seconds
[info][gps_reader.cpp:341] GPS data in trajectory CSV format
[warning][gps_reader.cpp:141] Timestamp column timestamp not found
[info][gps_reader.cpp:144] Id index 1 Geometry index 2 Timstamp index -1
[info][fmm_app.cpp:27 ] Progress report step 100
[info][fmm_app.cpp:29 ] Start to match trajectories
[info][fmm_app.cpp:31 ] Run map matching parallelly
[info][fmm_app.cpp:75 ] MM process finished
[info][fmm_app.cpp:79 ] Time takes 0.726
[info][fmm_app.cpp:80 ] Time takes excluding input 0.678
[info][fmm_app.cpp:82 ] Finish map match total points 2 matched 2
[info][fmm_app.cpp:83 ] Matched percentage: 1
[info][fmm_app.cpp:84 ] Point match speed: 2.75482
[info][fmm_app.cpp:86 ] Point match speed (excluding input): 2.94985
[info][fmm_app.cpp:87 ] Time takes 0.726

The traj.csv file was:

index;id;geom
1;1;LineString(23.79895 38.05781062,23.79896438 38.05837859)

The fmm_output.txt was:

id;cpath;mgeom
1;174760,174522;LINESTRING(23.7987688 38.05816,23.7988982633 38.058126099)

[cyang-kth]

That is normal, you can refer to the original paper for details. If you do not have a lot of GPS data or you have a quite large network, then you can use stmatch instead of fmm, which does not need the precomputation. #138

[sardounis]

So If I understand correctly, using fmm algorithm is inefficient in time as well as storage terms in large networks. Is this correct?

[cyang-kth]

Yes, it needs to do precomputation and for large network, the UBODT size considerably increases, where fmm could be inefficient as a lot of the time is spent in reading the UBODT file.

[sardounis]

Thanks a lot!