deriva-upload fails on csv files unless every column is accounted for

[ljpearlman]

I don't seem to be able to upload records to tables from CSV files using the uploader tool without accounting for every column in the table, (i.e., by making sure that every column appears in either the CSV file itself or the default_columns list in the config). This limitation does not appear to affect JSON files. This is becoming problematic for RBK, because we're beginning to support the "create a spreadsheet, save to CSV, upload" workflow for a few users.

To demonstrate the problem, I've created a very simple test table (https://dev.rebuildingakidney.org/ermrest/catalog/2/schema/Common/table/upload_test_table):

        Table.define("upload_test_table",
                     [
                         Column.define("col1", builtin_types.text, nullok=True),
                         Column.define("col2", builtin_types.text, nullok=True)
                     ]
        )

and I'm using this as my upload configuration (in a local config file):

{
  "asset_mappings": [
    {
      "asset_type": "table",
      "default_columns": [
        "RID",
        "RCT",
        "RMT",
        "RCB",
        "RMB"
      ],
      "ext_pattern": "^.*[.](?P<file_ext>json|csv)$",
      "file_pattern": "^((?!/assets/).)*/records/(?P<schema>.+?)/(?P<table>.+?)[.]"
    }
  ]
}

I can upload this json file successfully:

[
    { "col1" : "uploaded from json file"}
]

But when I attempt to upload this csv file:

col1
"uploaded from csv file"

I get this error:

/home/laura/upload-test/deriva/records/Common/upload_test_table.csv -- [DerivaUploadCatalogCreateError] [HTTPError] 400 Client Error: Bad Request for url: [https://dev.rebuildingakidney.org/ermrest/catalog/2/entity/Common:upload_test_table?defaults=RID,RCT,RMT,RCB,RMB] Details: b'400 Bad Request\nThe request is malformed. Missing expected CSV columns: "col2".\n' - Server responded: 400 Bad Request: The request is malformed. Missing expected CSV columns: "col2".

I'm using deriva-upload 0.8.4 from the client-tools distribution.