Syncing an Entire S3 bucket to local

Syncing an entire S3 bucket to your local machine

Recently I knew for a fact that I would be loosing one of my AWS accounts and I had a specific bucket which held some files that were near and dear to my heart. I offloaded the files into S3 knowing that they would be safe and sound and would not cluster up any local storage on my PC. All said these particular files were large video files which came to a total of about 150 GB.

Listing all available buckets to current account configured in the AWS CLI.

As a refresher, you can view all the S3 buckets you have by running the following command :

aws s3api list-buckets

Viewing the total size and number of objects in a bucket

To view the total size and number of objects in a bucket, navigate to the “Metrics” tab of your specific bucket and you will be able to view this data :

Model Results

So I wanted to figure out how can I get all of these files which have several different folders inside a single S3 bucket to my local machine in the fastest way possible?

The answer is with the s3 sync command, this command will allow you to take the remote files in a s3 location and write them to your local system.


For this example I created a folder on my C/: called Test which will be the location I pull my files into.

Open your favorite command line tool and change the working directory to the target you would like.

Next fire the aws s3 sync command with the target s3 bucket you are wanting to pull files from. The . in this context simply specifies to pull all remote files to the current directory, in this case C:\Test.

cd C:\Test

aws s3 sync s3://MyTargetBucket .

When you run this command you may be flooded with messages into your command line interface, but it will look similar to this:

Model Results

Giving you an indication of the process of the sync and exactly which files in the batch are being moved over.

And that’s it, when the command is done, you will have all the files in that S3 bucket on your local machine!

Ian Fogelman

Ian Fogelman

My name is Ian Fogelman. I like to develop data driven solutions with SQL Server, Python, .NET and predictive analytics.