Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Continue file upload if recordedit hasn't reloaded #2457

Merged
merged 11 commits into from
May 29, 2024
Merged

Conversation

jrchudy
Copy link
Member

@jrchudy jrchudy commented May 7, 2024

The changes in this PR for continuing file upload include tracking details about current upload jobs in case something occurs during upload and the upload is resubmitted and resumed.

The changes include:

  • Adding rowIdx, partialUpload, and uploadKey to the UploadFileObject
    • uploadKey is the key used in lastContiguousChunk for this UploadFileObject
      • intended to be a unique string made up of the file's md5_base64, the column name for the file, and the rowIdx of the record in recordedit app
      • more details about lastContiguousChunk values below
    • rowIdx is used for generating the uploadKey
    • partialUpload set to true if the generated uploadKey has a match
      • if there is a match for uploadKey in the lastContiguousChunk map that means a partial upload might exist for the current file being uploaded
      • check the stored jobUrl in lastContiguousChunk map with the generatedUrl after calculating the checksum
      • also check the stored file size with the new file size and that no uploadVersion exists yet
      • if the jobUrl contains the generatedUrl, then we can mark partialUpload to true and resume upload for the stored jobUrl
        • if the files match, the jobUrl should be the same as the generatedUrl + ';upload/job_hash
  • Adding lastContiguousChunkRef to recordedit provider for tracking if partial upload jobs exist after an error occurs
    • keys in this object are the uploadKey mentioned above in the form of ${file.checksum}_${column.name}_${form_index}
    • there are 4 values stored for each key
      • lastChunkIdx: the numeric index of the last successfully uploaded chunk
      • jobUrl: the previous job url for a partial upload job
      • fileSize: the file size for the file in the previous upload job
      • uploadVersion: the version of the file signaling that the job was already completed
  • For fileExists, if there's a partial upload, pass the jobUrl from the matching entry in lastContiguousChunk for use if a 409 is returned when checking for the file
  • For start, if there's a partial upload, use the lastChunkIdx from the matching entry in lastContiguousChunk to start the upload process from

Other details about the implementation:

  • When calculating if a partialUpload exists for the current file being uploaded, we have to take the generatedUrl and split it into parts, url encode those parts, then recombine the parts so we can compare with the stored jobUrl (which is url encoded by the server during the previous upload job that was not completed)
  • When updating lastContiguousChunk, we can stop after the first null or undefined entry is found since we are looking for true values in the chunkTracker array before the first null/undefined.

These changes resolve step 1 of issue #2379

…e checksum in reupload with stored key and then verify using the new url and the stored url if the paths are the same for continuing the upload job
@jrchudy jrchudy added recordedit hatrac Anything related to hatrac/uploader labels May 7, 2024
@jrchudy jrchudy self-assigned this May 7, 2024
@jrchudy jrchudy requested a review from RFSH May 9, 2024 01:09
@jrchudy jrchudy merged commit 818d2f4 into master May 29, 2024
1 check passed
@jrchudy jrchudy deleted the continue-upload branch May 29, 2024 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
hatrac Anything related to hatrac/uploader recordedit
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant