AWS S3 Upload Presigned Post URLs

There might be scenarios where your application users might need to upload or download files from s3 buckets without providing them explicit access to those buckets. In such cases presigned URLs will be useful.

There are two types of presigned URLs

1. Presigned Post URL
2. Presigned Get URL

In this post I will be explaining the upload part and in the next post I was planing to explain about presigned Get Url.

Presigned Post URL

S3 allows us to generate a URL with which we can upload files to s3 for a limited period of time without providing explicit write access to the user.

Architecture:

Nodejs code for generating presigned post URL:

// backend
const AWS = require('aws-sdk');
const s3 = new AWS.S3();

const generateS3PreSignedPostUrl = (
  key, // path at which our file needs to be saved
  maxSize,
  expiry = constants.POST_SIGNED_URL_EXPIRY,
  fileId, // 12345
  isUnverified = false,
) => {
  const params = {};

  params.Bucket = process.env.bucketName;
  params.Expires = expiry;
  params.Fields = {
    key,
  };
  params.Conditions = [
    ['content-length-range', 1, maxSize],
    ['eq', '$Cache-Control', 'max-age=86400'],
    ['eq', '$x-amz-meta-file_id', fileId],
    ['eq', '$Tagging', `<Tagging><TagSet><Tag><Key>isUnverified</Key><Value>${isUnverified}</Value></Tag></TagSet></Tagging>`],
    ['starts-with', '$Content-Type', 'image/'],
  ];
  return new Promise((resolve, reject) => {
    s3.createPresignedPost(params, (err, presignedUrl) => {
      if (err) {
        return reject(err);
      }
      const s3PresignedUrl = presignedUrl;
      s3PresignedUrl.fields['x-amz-meta-file_id'] = fileId;
      let s3Key = s3PresignedUrl.fields.key;
      try {
        s3Key = encodeURI(s3Key);
      } catch (error) {
        console.log('create post signed url failed', error);
        return reject(error);
      }
      s3PresignedUrl.s3_key = s3Key;
      return resolve(s3PresignedUrl);
    });
  });
};

The above function would generate a presigned post URL with specified expiry time. While uploading a file with this url the content-type should start with image/ and max size should not exceed the maxSize(in bytes). We can also add additional metadata under x-amz-meta-#your meta data name# in params.Conditions. This would make the user who is using this singed url to set those metadata while uploading or else the upload would fail.

In here we can generate the key as:

fileId = uuidv4(); // to prevent file name collision
key = `${s3_folder_path}/${originalFileName}_${Date.now()}_${file-id}.${extension}`

This key can be stored in our local db along with our file-id and original file name for further file retrieval.

The response of the function would look like:

{ 
          "url": "https://s3.ap-south-1.amazonaws.com/<your bucket name>",
                "fields": {
                    "key": "<your folder path in aws s3>/sample_1624816164229_3e01f5e9-89b0-4a85-a648-fa278ce11a08.jpg",
                    "X-Amz-Algorithm": "AWS4-HMAC-SHA256",
                    "X-Amz-Credential": "xxxxxxxxxx/ap-south-1/s3/aws4_request",
                    "X-Amz-Date": "20210926T113956Z",
                    "X-Amz-Security-Token": "xxxxxxx",
                    "Policy": "jwt token",
                    "X-Amz-Signature": "signature",
                    "x-amz-meta-file_id": "12345"
                }
       "s3_key": "<your folder path in aws s3>/sample_1624816164229_3e01f5e9-89b0-4a85-a648-fa278ce11a08.jpg",
}

We can now Upload the file (sample.jpg) using this presigned post url as form-data to https://s3.ap-south-1.amazonaws.com/#your bucket name#

Our file would get uploaded to s3 and metadata, tags and system defined data (content-type, Cache-Control) will also be set.

Front end code to upload file using this post signed URL:

// front end code
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');
const data = new FormData();
data.append('key', '<your folder path in aws s3>/sample_1624816164229_3e01f5e9-89b0-4a85-a648-fa278ce11a08.jpg');
data.append('X-Amz-Algorithm', 'AWS4-HMAC-SHA256');
data.append('X-Amz-Credential', 'xxxxxxxxxxxx/20210608/ap-south-1/s3/aws4_request');
data.append('X-Amz-Date', '20210608T093333Z');
data.append('Policy', 'jwt-token');
data.append('X-Amz-Signature', 'xxxxxx');
data.append('x-amz-meta-file_id', '12345');
data.append('Tagging', '<Tagging><TagSet><Tag><Key>isUnverified</Key><Value>true</Value></Tag></TagSet></Tagging>');
data.append('Content-Type', 'image/jpeg');
data.append('Cache-Control', 'max-age=86400');
data.append('file', fs.createReadStream('/home/ramprakash/Downloads/sample.jpeg'));

const config = {
  method: 'post',
  url: 'https://s3.ap-south-1.amazonaws.com/<your bucket name>',
  headers: { 
    ...data.getHeaders()
  },
  data : data
};

axios(config)
.then(function (response) {
  console.log(JSON.stringify(response.data));
})
.catch(function (error) {
  console.log(error);
});

Without this approach if we need to upload files to our s3 bucket we need to send files from front end to back-end and then to s3. This would be an additional network and resource burden to our back-end.

Tagging

['eq', '$Tagging', `<Tagging><TagSet><Tag><Key>isUnverified</Key><Value>${isUnverified}</Value></Tag></TagSet></Tagging>`],

If we look at our back-end code params.Conditions there is a field called tagging. AWS supports tags to be added to any resources. Here we have added isUnverified tag with value true in condition so while uploading we are enforcing front-end to add this tag value pair. We can now add a bucket policy which will delete all files in s3 with isUnverified = 'true'.

This will be helpful in the following scenario: Let's say we have front end application where user can apply for job in your company. The front end would have a form which would collect basic details of the candidates along with few supporting documents like resume, photo etc. If the user has uploaded his resume and photo via presinged post url and closes the browser without submitting the form. The uploaded files remains in s3 even though the user failed to submit the form. In such scenario setting bucket policy for file expiry with tags would be useful. So on initial upload we would be upload files with isUnverified as 'true'. Now when the user submits the form which would would contain basic details plus the file metadata(filename, file-id) to back-end we can either remove the tag or change its value by making an request to s3 and thereby verifying it.

Here with the file-id we can query our db to get our Key(path to the file) which we have stored while generating presinged post url.

Nodejs code for changing tag value:

const putObjectTags = (key, TagSet) => new Promise((resolve, reject) => {
  let s3Key = key;
  try {
    s3Key = decodeURI(s3Key);
  } catch (error) {
    console.log('error in decode', error);
    reject(error);
  }
  const params = {
    Bucket: process.env.bucketName,
    Key: s3Key,
    Tagging: { TagSet },
  };
  s3.putObjectTagging(params, (err, data) => {
    if (err) {
      return reject(err);
    }
    return resolve(data);
  });

putObjectTags(key, [{ Key: 'isUnverified', Value: 'false' }]);

Without this tag approach we might need to run some batch process in our back-end to check what are the files which are unverified and then deleting them in s3. Again this would be a resource intensive and time consuming.

Advantages:

1. Able to upload files to s3 bucket directly and securely without sending it to our back-end server and then uploading it to s3.
2. Able to delete unverified files without any batch process.

Resources

1. S3 signed url post policy: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTConstructPolicy.html

PS: if you have any feedback or doubts please add a comment : )