Need a method for "massive" new links creation

[Art-Ev]

Hi,

Currently, creating a project from existing files or GIS layer is done via adding links "one by one".
This result in pretty long running time because .spatialite file is opened, edited, consolidated and closed once per link we want to add.

Maybe we can create a function taking a pandas dataframe as input and putting the new links all at once in the spatialite file ? (or by group of several thousand of links ?)
Or maybe set the opening and closing of the spatialite file "manual" instead ?

When building project from OSM, how is the spatialite file updated @pedrocamargo ?

[pedrocamargo]

@Art-Ev , this is indeed a time-consuming process. Currently, the OSM importer does this with an executemany call to sqlite3

aequilibrae/aequilibrae/project/network/osm/osm_builder.py

Line 145 in a7339be

conn.executemany(insert_qry, self.node_df[cols].to_records(index=False))

Creating all nodes before doing the links makes it a LOT faster, which is what we do with OSM (see line above).

Whenever I do things on a "large scale", I remove all triggers, add links and nodes and then add the triggers back. That is a hack that I can't recommend, but it could've made OK by running a set of Spatialite queries AFTER all links and nodes are added to enforce consistency.