Sometimes, we want to iterate through all files in a bucket with Node.js and Amazon S3.

In this article, we’ll look at how to iterate through all files in a bucket with Node.js and Amazon S3.

How to iterate through all files in a bucket with Node.js and Amazon S3?

To iterate through all files in a bucket with Node.js and Amazon S3, we can use the listObjectsV2 method.

For instance, we write

const {
  S3
} = require("aws-sdk");
const s3 = new S3();

async function* listAllKeys(opts) {
  const newOpts = {
    ...opts
  }
  do {
    const data = await s3.listObjectsV2(newOpts).promise();
    newOpts.ContinuationToken = data.NextContinuationToken;
    yield data;
  } while (newOpts.ContinuationToken);
}

to add a do-while loop that calls s3.listObjectsV2 with newOpts and then call promise to return a promise with the data from the bucket.

Then we set newOpts.ContinuationToken to data.NextContinuationToken to continue to the next iteration if newOpts.ContinuationToken isn’t null.

We use yield to return the data returned from the promise.

Conclusion

To iterate through all files in a bucket with Node.js and Amazon S3, we can use the listObjectsV2 method.