How to loop through all the files in a Digital Ocean Spaces 'folder' with Node.js

Let's start by saying that 'folders' or 'directories' actually don't exist in Digital Ocean Spaces (just like they don't exist in AWS S3). However, when you visit the Digital Ocean Spaces web interface, you can still create folders. Behind the scenes, this will just add a prefix to the filename.

With that out of the way, let's find out how we can loop through all the items in a Digital Ocean Spaces 'folder'.

Let's say that we have a Digital Ocean Spaces that includes pictures of Marvel characters.

Our daily challenge: get all the pictures inside of the iron-man "folder"

Get an API key

Before diving in the code, let's get an API key. You can generate one through on the 'Applications & API' settings in the Digital Ocean dashboard.

Install the aws-sdk

To communicate with our Digital Ocean Spaces object storage, we can use the AWS SDK

npm install aws-sdk

At the top of our code, let's require the SDK:

const AWS = require('aws-sdk');

Define the variables

Let's create a constant that contains the configuration of our Digital Ocean Space.

const s3 = new AWS.S3({
        endpoint: "https://ams3.digitaloceanspaces.com", 
        region: "us-east-1", 
        credentials: {
            accessKeyId: "<insert access key here>", 
            secretAccessKey: "<insert secret key here>" 
        }
    });

Let's note some things:

• You can find the endpoint in the Digital Ocean Spaces dashboard. This does not include your Spaces name.
• The region is required for the AWS SDK, but isn't actually used. Even though the region of my Digital Ocean Space is ams3, I can still use "us-east-1" here without any consequences.
• Ideally you'd use environment variables for the credentials.

With that out of the way, let's write a (very simplified) function that loops through all the images in the 'iron-man' folder and logs the filename to the console.

async function getAllImages(bucket,prefix) {
        let params = {Bucket: bucket,Prefix:prefix};
        const response = await s3.listObjects(params).promise();
        response.Contents.forEach(item => {
            // Display the filename on the console
            console.log(item.Key);
        });
}

getAllImages('marvelgallery','iron-man');

So, what's happening here:

• In the function 'getAllImages' we take 2 parameters: the bucket name and a prefix. The prefix is where the magic happens.

If we don't enter any prefix, this is the result we get:

So we can add a prefix like 'iron-man/' to only return the objects in the "folder" of iron-man.

• We assign these parameters to an object 'params', and we use that object as the parameter for the s3.listObjects method.
• We loop through all the results in the response, and we log the item.Key (which corresponds with the filename).

This code works, but let me emphasize that it's very simplified for the purpose of this demo. The 'listObjects' method may not return all objects in one call since it's limited to 1000 objects. If you have a lot of files, you can use 'isTruncated' and 'marker' to set up some form of pagination.

Full code example

const AWS = require('aws-sdk');

const s3 = new AWS.S3({
    endpoint: "https://ams3.digitaloceanspaces.com",
    region: "us-east-1",
    credentials: {
        accessKeyId: "<your-access-key>",
        secretAccessKey: "<your-secret-key>"
    }
});

async function getAllImages(bucket,prefix) {
        let params = {Bucket: bucket,Prefix:prefix};
        const response = await s3.listObjects(params).promise();
        console.log(response);
        response.Contents.forEach(item => {
            // Display the filename on the console
            console.log(item.Key);
        });
}

getAllImages('marvelgallery','iron-man');