Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tweaks for MTC enhancements #200

Merged
merged 5 commits into from
Feb 19, 2019
Merged

Tweaks for MTC enhancements #200

merged 5 commits into from
Feb 19, 2019

Conversation

landonreed
Copy link
Contributor

This PR modifies and makes static/public a few internal methods for Table and JdbcTableLoader that are used in ibi-group/datatools-server#186.

@codecov-io
Copy link

codecov-io commented Feb 15, 2019

Codecov Report

Merging #200 into dev will increase coverage by 0.06%.
The diff coverage is 79.41%.

Impacted file tree graph

@@             Coverage Diff             @@
##               dev     #200      +/-   ##
===========================================
+ Coverage     57.5%   57.57%   +0.06%     
- Complexity     800      808       +8     
===========================================
  Files          144      145       +1     
  Lines         7314     7314              
  Branches       850      851       +1     
===========================================
+ Hits          4206     4211       +5     
+ Misses        2769     2763       -6     
- Partials       339      340       +1
Impacted Files Coverage Δ Complexity Δ
...ava/com/conveyal/gtfs/loader/ReferenceTracker.java 100% <100%> (ø) 1 <1> (?)
src/main/java/com/conveyal/gtfs/loader/Field.java 76.66% <100%> (+1.66%) 19 <4> (+3) ⬆️
src/main/java/com/conveyal/gtfs/loader/Table.java 84.56% <75.92%> (-0.66%) 84 <21> (+9)
.../java/com/conveyal/gtfs/loader/JdbcGtfsLoader.java 68.68% <87.5%> (-1.66%) 20 <6> (-7)
.../java/com/conveyal/gtfs/error/SQLErrorStorage.java 76.66% <0%> (+4.44%) 14% <0%> (+2%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eca69fb...5d58f2a. Read the comment docs.

@@ -562,24 +600,12 @@ private void setFieldToNull(boolean postgresText, String[] transformedStrings, i
*
* TODO add a test including SQL injection text (quote and semicolon)
*/
public String sanitize (String string) throws SQLException {
public static String sanitize (String string, SQLErrorStorage errorStorage) {
String clean = string.replaceAll("[^\\p{Alnum}_]", "");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't quite understand this regex. Can you make a comment for it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@abyrd, can you provide a description of this? (I can add your comments to code).

Copy link
Member

@abyrd abyrd Feb 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the undocumented regex, obviously these things should always have a comment. This removes all characters that are not alphanumeric or an underscore. The reasoning being that all GTFS field names are composed of only alphanumeric characters and underscores, and limiting all supplied strings to those characters should completely preclude SQL injection.

Since this is already merged, I think it would be OK to make a commit directly to dev to add this information in a comment.

Copy link
Contributor

@evansiroky evansiroky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs a few more comments, and I think it'd be good to move the mentioned methods out of the JdbcGtfsLoader table to either the Table or Field class.

@evansiroky evansiroky assigned landonreed and unassigned evansiroky Feb 18, 2019
@landonreed landonreed assigned evansiroky and unassigned landonreed Feb 18, 2019
@landonreed
Copy link
Contributor Author

@evansiroky, thanks for the suggestions. I've addressed the comment gaps and moved methods to Table/Field. I don't know exactly what the regex does, but I've tagged @abyrd who wrote that method originally. If you'd like we can wait for his comments before merge.

@evansiroky evansiroky assigned landonreed and unassigned evansiroky Feb 19, 2019
@landonreed landonreed merged commit 5ec5285 into dev Feb 19, 2019
@landonreed landonreed deleted the mtc-enhancements-20190214 branch February 19, 2019 19:01
@landonreed landonreed mentioned this pull request Apr 1, 2019
7 tasks
@landonreed
Copy link
Contributor Author

🎉 This PR is included in version 4.3.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants