Skip to content

Commit

Permalink
last touches
Browse files Browse the repository at this point in the history
  • Loading branch information
g.trantham committed Dec 28, 2022
1 parent 3a5a1c1 commit a2ccb3b
Showing 1 changed file with 15 additions and 16 deletions.
31 changes: 15 additions & 16 deletions docs/workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,36 +13,35 @@ The crawler CLI will bulk-download feature data from pre-defined sources. The se
}
}%%
sequenceDiagram
actor CLI
CLI->>Crawler: launch
Crawler->>+NLDI-DB: Get Source Information
Note left of NLDI-DB: SELECT * FROM nldi_data.crawler_source
NLDI-DB-->>-Crawler: Sources table
Crawler->>+Feature_Source: Request Features
Note left of Feature_Source: HTTP GET ${crawler_source.source_uri}
Feature_Source-->>-Crawler: GeoJSON FeatureCollection
Crawler->>+FeatureSource: Request Features
Note left of FeatureSource: HTTP GET ${crawler_source.source_uri}
FeatureSource-->>-Crawler: GeoJSON FeatureCollection
loop foreach feature in Collection
Crawler-->>+Crawler: ORM
Note right of Crawler: Parses and maps feature to SQL
Crawler->>-NLDI-DB: Add to feature table
Note left of NLDI-DB: INSERT INTO nldi_data.features
end
Crawler->>+NLDI-DB: Relate Features
Crawler->>NLDI-DB: Relate Features
%NLDI-DB-->>-Crawler: Success
```

## Annotations

1) Connect to NLDI master database, requesting the list of configured feature sources.
2) Returns a list of feature sources. The crawler can either:
1) Launch CLI tool
2) Connect to NLDI master database, requesting the list of configured feature sources.
3) Returns a list of feature sources. The crawler can either:
* list all sources and exit
* Proceed to 'crawl' one of the sources in the table
3) For the identified feature source, make a GET request via HTTP. The URI is taken from the `crawler_sources` table.
4) The feature source returns GeoJSON. Among the returned data is a list of 'features'.

For each feature:

5) Use the ORM to map the feature data to the schema reflected from the `features` table
6) Insert the new feature to the master NLDI database


7) "Relate" features -- build the relationships matching features to their adjacent features in the NLDI topology.
4) For the identified feature source, make a GET request via HTTP. The URI is taken from the `crawler_sources` table.
5) The feature source returns GeoJSON. Among the returned data is a list of 'features'.
6) **[Per-Feature]** Use the ORM to map the feature data to the schema reflected from the `features` table
7) **[Per-Feature]** Insert the new feature to the master NLDI database
8) "Relate" features -- build the relationships matching features to their adjacent features in the NLDI topology.

0 comments on commit a2ccb3b

Please sign in to comment.