Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release #212

Merged
merged 116 commits into from
Jul 17, 2019
Merged
Changes from 1 commit
Commits
Show all changes
116 commits
Select commit Hold shift + click to select a range
ccf435e
feat(gis-export): add gis export for stops/routes
landonreed Jun 8, 2017
fa9ec49
initial work on merge feeds for MTC
landonreed Oct 11, 2018
8f8f6c4
Merge branch 'dev' into merge-feed-versions-mtc
landonreed Jan 28, 2019
9e76476
refactor(feed-merge): handle merging projects + two versions for feed…
landonreed Feb 1, 2019
cf7208c
Merge remote-tracking branch 'origin/fix-gtfs-plus-upload-job-fixes' …
landonreed Feb 5, 2019
17f651e
docs(github): add item about referencing PRs in PRs to master
evansiroky Feb 12, 2019
7bcbda7
ci: cache yarn stuff and preemptively fix semantic-release to v15
evansiroky Feb 14, 2019
88a57cd
refactor(auth0): log response body when an error from auth0 is received
evansiroky Feb 15, 2019
b6d1058
feat(merge-feeds): add MTC merge feeds strategy
landonreed Feb 15, 2019
2fc62d1
refactor(job): remove unused job type
landonreed Feb 15, 2019
1e7014b
refactor(merge-feeds): improve comments, add job#validationResult
landonreed Feb 15, 2019
ecf8388
Merge pull request #182 from conveyal/ci-improvements
Feb 15, 2019
4c93b80
Merge pull request #175 from conveyal/pr-template-addition
Feb 15, 2019
065c1ec
Merge pull request #184 from conveyal/auth0-error-response-logging
Feb 15, 2019
3b14e94
fix(users): pass through relevant auth0 failures to user API responses
evansiroky Feb 16, 2019
551a684
test(user): add CRUD tests for UserController
evansiroky Feb 16, 2019
e174aa3
refactor(test): change argument order for junit v4 compatibility
evansiroky Feb 16, 2019
c4109c4
refactor: address PR comments
landonreed Feb 18, 2019
f84dd4c
build(pom): update gtfs-lib to snapshot version for testing
landonreed Feb 19, 2019
114fc65
refactor: isolate Auth0 error handling changes
evansiroky Feb 21, 2019
5943dfb
test(user): add CRUD tests for auth0 endpoints
evansiroky Feb 21, 2019
adb9b0d
Merge branch 'dev' into merge-feed-versions-mtc
Mar 1, 2019
c5950c1
refactor(user): create methods for enabling test behavior and address…
evansiroky Mar 6, 2019
6919e14
refactor(auth0): move testing string literals into constants
landonreed Mar 7, 2019
e4bfe79
refactor(test): use PORT constant
landonreed Mar 7, 2019
c6eac0c
Merge pull request #190 from conveyal/auth0-test-tweaks
Mar 7, 2019
1e75f61
Merge pull request #187 from conveyal/auth0-error-response-improvements
Mar 11, 2019
c60b5a0
feat(project): add pinned deployments to projects
evansiroky Mar 22, 2019
f5ae65a
refactor(project): add comments about pinned deployment
evansiroky Mar 22, 2019
85661e4
feat(editor): add normalize stop times API endpoint
landonreed Mar 28, 2019
bc0b450
test(merge-feeds): add tests for MTC and REGIONAL merge types
landonreed Mar 28, 2019
9969a56
refactor(project): improve comment about pinned deployments
evansiroky Mar 28, 2019
d5e4e49
chore(pom): update gtfs-lib to 4.3.0
Apr 1, 2019
e4f089e
refactor(editor): remove unused editor controller hooks
landonreed Apr 1, 2019
a90940e
chore(pom): update gtfs-lib to 4.3.0
Apr 1, 2019
6802380
Merge pull request #193 from conveyal/gtfs-lib_4.3.0
Apr 1, 2019
566e4da
Merge pull request #191 from conveyal/pinned-deployments
Apr 1, 2019
10c253f
Merge branch 'dev' into merge-feed-versions-mtc
Apr 1, 2019
fb3a2d8
refactor(junit): refactor tests to use junit4
landonreed Apr 1, 2019
72c57f6
build(pom): add junit 4
landonreed Apr 1, 2019
6b06107
test: create database if not exists
landonreed Apr 1, 2019
c3b9c9e
test: add log
landonreed Apr 1, 2019
b0f0260
test(ci): create database using travis
landonreed Apr 1, 2019
54b0c0e
Merge pull request #192 from conveyal/normalize-stop_times
Apr 2, 2019
97e742f
build(travis): always create postgres db for tests
Apr 3, 2019
026f577
build(pom): bump gtfs-lib to 4.3.1
Apr 3, 2019
97dffea
refactor(feedValidationResultSummary): add FeedSource#publishedValida…
landonreed Apr 4, 2019
4d2d0e4
refactor(feedsource): publishedValidation -> publishedValidationSummary
landonreed Apr 4, 2019
853ddb8
Merge pull request #194 from conveyal/gtfs-lib-4.3.1
Apr 4, 2019
f5c6778
refactor(FeedSource): retrievePublished --> retrievePublishedVersion
landonreed Apr 8, 2019
f7eb459
refactor(FeedSource): add clarifying comments
landonreed Apr 8, 2019
ae95442
Merge pull request #195 from conveyal/add-published-validation
Apr 8, 2019
99e7f77
build(pom): bump gtfs-lib to 4.3.2
Apr 8, 2019
d3e8267
Merge pull request #196 from conveyal/gtfs-lib-4.3.2
Apr 8, 2019
9860110
Merge branch 'dev' into merge-feed-versions-mtc
landonreed Apr 11, 2019
c639c36
fix(users): update auth0 search engine to v3
robbi5 Apr 13, 2019
d94f40e
refactor(merge-feeds): address PR comments
landonreed Apr 23, 2019
126f61d
fix(merge-feeds): address remaining PR comments
landonreed Apr 24, 2019
9eeaac0
Merge branch 'dev' into gis-export
landonreed Apr 24, 2019
d2276a2
build(pom): bump gtfs-lib to 4.3.3
Apr 25, 2019
eccd433
feat(shapefile-export): add .shp export for routes or stops
landonreed Apr 26, 2019
98f2531
test(shp-export): re-add non-null geom assertion
landonreed Apr 26, 2019
08e5d95
Merge branch 'dev' into auth0-testing
evansiroky Apr 26, 2019
5361439
refactor(auth0): update so auth0 API calls work during testing
evansiroky Apr 26, 2019
7c03086
refactor(auth0): consolidate URI builder code
evansiroky Apr 27, 2019
11df612
refactor(test): update to latest java-snapshot-matcher
evansiroky Apr 29, 2019
6d9db08
fix(merge-feeds): add test for errors; fix missing stop_code bug
landonreed May 2, 2019
bcc5cbb
build(travis): remove oraclejdk8 to fix build
May 2, 2019
04c9c47
build(travis): use jdk 8 and linux trusty dist
landonreed May 2, 2019
5c42853
Merge pull request #198 from conveyal/snapshot-update
landonreed May 2, 2019
7f69f67
refactor(merge-feeds): improve error message for agency_id mismatch
landonreed May 2, 2019
a590fc3
build(pom): update gtfs-lib to 4.3.4
landonreed May 3, 2019
f81ba52
Merge branch 'dev' into merge-feed-versions-mtc
landonreed May 3, 2019
c43f697
refactor(export-gis): fix geotools version; add controller
landonreed May 6, 2019
6239db4
fix(permissions): use enum to check request action types
landonreed May 6, 2019
976d672
refactor(gis-export): generate shape from stops if shape missing
landonreed May 6, 2019
462d3dc
fix(export-gis): fix shapefile from pattern stops; update test
landonreed May 6, 2019
f7f6de6
build(travis): use jdk 8 and linux trusty dist
landonreed May 2, 2019
8645ab8
ci(travis): create postgres db before install
landonreed May 6, 2019
b9e95a0
refactor(merge-feeds): add description for exception
landonreed May 6, 2019
8be5e4a
Merge pull request #186 from conveyal/merge-feed-versions-mtc
landonreed May 6, 2019
d2c4703
Merge pull request #197 from robbi5/auth0-search-engine-v3
landonreed May 6, 2019
3d87e05
refactor(user): improve test readability and add comments
evansiroky May 7, 2019
8c7c051
Merge branch 'dev' into auth0-testing
evansiroky May 7, 2019
c88517c
refactor(test): update snapshots with latest snapshotter
evansiroky May 7, 2019
56589dc
Merge pull request #188 from conveyal/auth0-testing
landonreed May 7, 2019
defa58a
test(export-gis): address PR comments
landonreed May 9, 2019
e272c56
Merge branch 'dev' into gis-export
landonreed May 9, 2019
aaf1575
Merge pull request #199 from conveyal/gis-export
landonreed May 9, 2019
1bee80e
build(pom): update gtfs-lib to 4.3.5
landonreed May 21, 2019
6e304e4
Merge pull request #200 from conveyal/gtfs-lib-4.3.5
landonreed May 22, 2019
2aca2a9
fix(pom): fix jackson-databind security vuln.
landonreed May 23, 2019
e66a97e
fix(pom): fix jackson-databind to 2.9.9
landonreed May 23, 2019
6b36529
Merge pull request #201 from ibi-group/jackson-patch
landonreed May 28, 2019
8ae8498
fix(GisExportJob): fix projection for stops and routes export
landonreed May 31, 2019
9fd302d
refactor(GisExportJob): fix a few things
landonreed May 31, 2019
bdda525
test(GisExportJob): add test to check for CRS
landonreed May 31, 2019
768b7f7
fix(MergeFeedsJob): various fixes for MTC feed merge
landonreed Jun 3, 2019
b8617fd
test(GisExportJobTest): refactor CRS assertion
landonreed Jun 3, 2019
1169a5a
chore(pom): bump gtfs-lib to 4.3.6
landonreed Jun 4, 2019
47edfe4
ci(travis): change repo name for deploy
landonreed Jun 4, 2019
976aa66
Merge pull request #205 from ibi-group/mtc-merge-fixes
landonreed Jun 4, 2019
f367375
Merge pull request #204 from ibi-group/fix-shp-export
landonreed Jun 4, 2019
4fcb583
fix(GTFS+): add check for missing, but required columns
landonreed Jun 7, 2019
da07433
style: white space formatting
landonreed Jun 7, 2019
87a4769
refactor(GTFS+): move GTFS+ files around
landonreed Jun 7, 2019
4f9ef60
test(GTFS+): add basic test for GTFS+ validation
landonreed Jun 7, 2019
57e87fd
fix: remove unused/bad import
landonreed Jun 7, 2019
3bb1858
ci(travis): create GTFS+ dir before script
landonreed Jun 7, 2019
a99a4f6
Merge pull request #207 from ibi-group/gtfs+-check-missing-column
landonreed Jun 7, 2019
9f5b3f3
fix(MTC-sync): fix feed sync with MTC RTD database
landonreed Jun 24, 2019
0054341
style(MTC-sync): enhance comment
landonreed Jun 24, 2019
289ed0e
refactor(MTC-sync): address PR comments
landonreed Jun 24, 2019
eedf4e7
Merge pull request #209 from ibi-group/fix-mtc-sync
landonreed Jun 24, 2019
f8d393d
fix(MtcFeedResource): properly initialize external feed props on create
landonreed Jun 28, 2019
a68aa5f
Merge pull request #211 from ibi-group/fix-MTC-feed-source-created
landonreed Jun 28, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
refactor(export-gis): fix geotools version; add controller
  • Loading branch information
landonreed committed May 6, 2019
commit c43f69762393caa5e30e79ef0816a30170cdf5f1
4 changes: 3 additions & 1 deletion pom.xml
Original file line number Diff line number Diff line change
@@ -52,7 +52,9 @@
</scm>
<properties>
<jackson.version>2.9.8</jackson.version>
<geotools.version>20.3</geotools.version>
<!-- Using the latest version of geotools (e.g, 20) seems to cause issues with the shapefile
plugin where the_geom for each feature is null. -->
<geotools.version>17.5</geotools.version>
</properties>
<build>
<resources>
Original file line number Diff line number Diff line change
@@ -6,6 +6,7 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.File;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
@@ -19,10 +20,11 @@
*/
public abstract class MonitorableJob implements Runnable {
private static final Logger LOG = LoggerFactory.getLogger(MonitorableJob.class);
protected final String owner;
public final String owner;

// Public fields will be serialized over HTTP API and visible to the web client
public final JobType type;
public File file;
public String parentJobId;
public JobType parentJobType;
// Status is not final to allow some jobs to have extra status fields.
@@ -48,6 +50,7 @@ public enum JobType {
LOAD_FEED,
VALIDATE_FEED,
DEPLOY_TO_OTP,
EXPORT_GIS,
FETCH_PROJECT_FEEDS,
FETCH_SINGLE_FEED,
MAKE_PROJECT_PUBLIC,
@@ -90,6 +93,10 @@ private void registerJob() {
DataManager.userJobsMap.put(this.owner, userJobs);
}

public File retrieveFile () {
return file;
}

/**
* This method should never be called directly or overridden. It is a standard clean up stage for all
* monitorable jobs.
Original file line number Diff line number Diff line change
@@ -62,8 +62,7 @@ public void jobLogic() {
and(
eq("version", versionNumber),
eq(Snapshot.FEED_SOURCE_REF, feedId)
),
null
)
);
boolean snapshotExists = true;
if (matchingSnapshot == null) {
@@ -351,4 +350,4 @@ private int handleBatchExecution(int batchSize, PreparedStatement ... preparedSt
return batchSize;
}
}
}
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package com.conveyal.datatools.manager.controllers.api;

import com.conveyal.datatools.common.utils.SparkUtils;
import com.conveyal.datatools.manager.DataManager;
import com.conveyal.datatools.manager.auth.Auth0UserProfile;
import com.conveyal.datatools.manager.jobs.CreateFeedVersionFromSnapshotJob;
@@ -22,7 +23,6 @@

import javax.servlet.http.HttpServletResponse;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.Arrays;
import java.util.Collection;
@@ -35,6 +35,7 @@
import static com.conveyal.datatools.common.utils.SparkUtils.formatJobMessage;
import static com.conveyal.datatools.common.utils.SparkUtils.logMessageAndHalt;
import static com.conveyal.datatools.manager.controllers.api.FeedSourceController.checkFeedSourcePermissions;
import static com.mongodb.client.model.Filters.eq;
import static spark.Spark.delete;
import static spark.Spark.get;
import static spark.Spark.post;
@@ -212,8 +213,8 @@ private static HttpServletResponse downloadFeedVersionDirectly(Request req, Resp
* Returns credentials that a client may use to then download a feed version. Functionality
* changes depending on whether application.data.use_s3_storage config property is true.
*/
private static Object getFeedDownloadCredentials(Request req, Response res) {
FeedVersion version = requestFeedVersion(req, "view");
private static Object getDownloadCredentials(Request req, Response res) {

if (DataManager.useS3) {
// Return pre-signed download link if using S3.
@@ -267,24 +268,52 @@ private static FeedVersion publishToExternalResource (Request req, Response res)
}
}

public static FileInputStream exportGis (Request req, Response res) throws IOException {
/**
* HTTP endpoint to initiate an export of a shapefile containing the stops or routes of one or
* more feed versions. NOTE: the job ID returned must be used by the requester to download the
* zipped shapefile once the job has completed.
*/
private static String exportGis (Request req, Response res) throws IOException {
String type = req.queryParams("type");
Auth0UserProfile userProfile = req.attribute("user");
List<String> feedIds = Arrays.asList(req.queryParams("feedId").split(","));
File temp = File.createTempFile("gis_" + type, ".zip");
// Create and run shapefile export.
GisExportJob gisExportJob = new GisExportJob(
GisExportJob.ExportType.valueOf(type),
temp,
feedIds,
userProfile.getUser_id()
);
DataManager.heavyExecutor.execute(gisExportJob);
// Do not use S3 to store the file, which should only be stored ephemerally (until requesting
// user has downloaded file).
FeedDownloadToken token = new FeedDownloadToken(gisExportJob);
Persistence.tokens.create(token);
return SparkUtils.formatJobMessage(gisExportJob.jobId, "Generating shapefile.");
}

GisExportJob gisExportJob = new GisExportJob(GisExportJob.Type.valueOf(type), temp, feedIds);
gisExportJob.run();

FileInputStream fis = new FileInputStream(temp);

res.type("application/zip");
res.header("Content-Disposition", "attachment;filename=" + temp.getName().replaceAll("[^a-zA-Z0-9]", "") + ".zip");

// will not actually be deleted until download has completed
// http://stackoverflow.com/questions/24372279
temp.delete();

return fis;
/**
* Public HTTP endpoint to download a zipped shapefile of routes or stops for a set of feed
* versions using the job ID that was used for initially creating the exported shapes.
*/
private static HttpServletResponse downloadFeedVersionGis (Request req, Response res) {
FeedDownloadToken token = Persistence.tokens.getOneFiltered(eq("jobId", req.params("jobId")));
File file = new File(token.filePath);
try {
return downloadFile(file, file.getName(), req, res);
} catch (Exception e) {
logMessageAndHalt(req, 500, "Unknown error occurred while downloading feed version shapefile", e);
} finally {
if (!file.delete()) {
LOG.error("Could not delete shapefile {}. Storage issues may occur.", token.filePath);
} else {
LOG.info("Deleted shapefile {} following download.", token.filePath);
}
// Delete token.
Persistence.tokens.removeById(token.id);
}
return null;
}

/**
@@ -319,19 +348,21 @@ public static void register (String apiPrefix) {
// previous version of data tools.
get(apiPrefix + "secure/feedversion/:id", FeedVersionController::getFeedVersion, json::write);
get(apiPrefix + "secure/feedversion/:id/download", FeedVersionController::downloadFeedVersionDirectly);
get(apiPrefix + "secure/feedversion/:id/downloadtoken", FeedVersionController::getFeedDownloadCredentials, json::write);
get(apiPrefix + "secure/feedversion/:id/downloadtoken", FeedVersionController::getDownloadCredentials, json::write);
post(apiPrefix + "secure/feedversion/:id/validate", FeedVersionController::validate, json::write);
get(apiPrefix + "secure/feedversion", FeedVersionController::getAllFeedVersionsForFeedSource, json::write);
post(apiPrefix + "secure/feedversion", FeedVersionController::createFeedVersionViaUpload, json::write);
post(apiPrefix + "secure/feedversion/shapes", FeedVersionController::exportGis, json::write);
post(apiPrefix + "secure/feedversion/fromsnapshot", FeedVersionController::createFeedVersionFromSnapshot, json::write);
put(apiPrefix + "secure/feedversion/:id/rename", FeedVersionController::renameFeedVersion, json::write);
post(apiPrefix + "secure/feedversion/:id/publish", FeedVersionController::publishToExternalResource, json::write);
delete(apiPrefix + "secure/feedversion/:id", FeedVersionController::deleteFeedVersion, json::write);

get(apiPrefix + "public/feedversion", FeedVersionController::getAllFeedVersionsForFeedSource, json::write);
get(apiPrefix + "public/feedversion/:id/downloadtoken", FeedVersionController::getFeedDownloadCredentials, json::write);
get(apiPrefix + "public/feedversion/:id/downloadtoken", FeedVersionController::getDownloadCredentials, json::write);

get(apiPrefix + "downloadfeed/:token", FeedVersionController::downloadFeedVersionWithToken);
get(apiPrefix + "downloadshapes/:jobId", FeedVersionController::downloadFeedVersionGis, json::write);

}
}
Loading