AWS: Upload/Download Large Files to S3 using PreSigned URL

You could upload to S3 using API Gateway directly, or through a Lambda Proxy but there are couple benefit of uploading and downloading using a pre-signed URL!

1. You don’t have to worry about setting the correct API Gateway binary Media Types (especially if you need to handle multiple different kinds)
2. Size limit: AWS API Gateway has a payload size limit of 10MB whereas uploading directly to S3 can handle larger file sizes up to 5TB.
3. Different from make the bucket entirely public, you could control the expiration time and restrict the url capabilities. (That is if don’t mind making S3 Bucket public, I guess you could skip the rest of this article…)

I will be using Rust aws_sdk_s3 here, but same idea regardless of the language you are using!

Let’s check it out!

Upload

Step 1: Generate URL

Instead of specifying a ByteStream body like we normally do with the put_object action, we use the presigned method to ask for an upload URL.

I am setting the expiration time to be 1 hour here.

pub async fn put_object_presigned(
        client: &aws_sdk_s3::Client,
        bucket_name: &str,
        key: &str,
        content_type: &str,
    ) -> Result<String> {
        let expires_in = Duration::from_secs(3600);
let presigned_request = client
            .put_object()
            .content_type(content_type)
            .bucket(bucket_name)
            .key(key.to_owned())
            .presigned(PresigningConfig::expires_in(expires_in)?)
            .await?;
println!("Object URI: {}", presigned_request.uri());
        Ok(presigned_request.uri().to_owned())
    }

Step 2: Put Data to the URL

YES, USE PUT METHOD! Not POST!

Add the data as binary body and simply send your request. You don’t need further authorization! Also, I will strongly recommend to include a Content-Type header.

That’s it!

No need to worry about size limit, no need to try to break up large files into multiple parts! Life is great!

Download

Even easier!

No need to try to figure out the content type, collecting byte streams, converting to vectors, encoding and decoding, and blah!

pub async fn get_object_presigned(
    client: &aws_sdk_s3::Client,
    bucket_name: &str,
    key: &str
) -> Result<String> {
    let expires_in = Duration::from_secs(PRESIGNED_VALID_DURATION);
let presigned_request = client
        .get_object()
        .bucket(bucket_name)
        .key(key.to_owned())
        .presigned(PresigningConfig::expires_in(expires_in)?)
        .await?;
println!("Object URI: {}", presigned_request.uri());
Ok(presigned_request.uri().to_owned())
}

You can then use the URL directly to view the content or to download it as you wish!

Life is easy thanks to the pre-signed URLs!

Of course, there are drawbacks! Obviously, those urls are less secure and you get less control on who can upload/download your contents!

Anyway!

Thank you for reading!

Happy pre-signing!