Upload
Once a Sequence passes client dies validation, upload begine.

Large file size/volume upload

Use case

It is likely our users will upload videos regularly of around 6gb, and images of 20mb (but potentially 2000 at a time). This poses a number of challenges.

Sequence upload flow

The upload flow happens as follow:
  1. 1.
    Create sequence: this is user inputting sequence info and files that pass validation. If validation passes
  2. 2.
    Web app makes call to Lamba function to generate a time limited (3 hours) presigned S3 bucket directory for the files to be uploaded to
  3. 3.
    Lambda sends presigned bucket URL to web app
  4. 4.
    Web app sends presigned bucket URL to Client
  5. 5.
    Client starts upload of files from their computer to S3. Note, in the case of videos, videos will have been chunked.
  6. 6.
    When upload complete (number of complete uploads = number of submitted uploads), the client closes the upload session on the web app
  7. 7.
    Web app triggers Lambda function to close S3 upload session (and destroy presigned URL)

Technical challenges for large filesizes

We use the AWS SDK functions to ensure large files have support for network failures (resumable uploads: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/welcome.html)
Upload in the browser poses a number of challenges.
In the case of video, it is due to file size (<6GB).
In the case of image, it is due to the amount (<=3000 images).
Files should be directly uploaded to S3, as they will require post processing after upload (using Lamda).
Stability and resume-ability of uploads are therefore important.
If upload abandoned or enters failed state, any uploaded content is deleted.

Image files

If any upload errors, we should attempt to retry 3 times, 1 minute after each failure.
If fails 3 times, upload considered failed. User shown error: "There was an error during upload. Please retry to create this Sequence. If this problem persists, please reduce the number of images in your Sequence."

Video files

Due to large sizes, videos should be chunked and uploaded in parallel.
If any upload chunk errors, we should attempt to retry 3 times, 1 minute after each failure.
If fails 3 times, upload considered failed. User shown error: "There was an error during upload. Please retry to create this Sequence. If this problem persists, please try a faster connection to upload this video."

Technical challenges

To save bandwidth costs, files are uploaded directly to S3 (and not touch webserver).

User experience challenges

No user wants to wait for a long time for upload, so upload time should is reduced as much as possible.
However, when dealing in such large files and varying connection speeds on users machine this is somewhat an unavoidable challenge.
Therefore, the app provides a clear interface about the progress of uploading steps.
This is achieved using a % complete bar. User must keep tab open until upload complete (and are warned if they attempt to close tab). In the case of image files, count of images selected for upload vs completed upload is used to show % complete.
In the case of video files, count of chunks for complete video vs uploaded chunks is used to show % complete.

Last modified 1mo ago