Amazon AWS GovCloud S3 Syncing .... and you can too!

So, you're in fancy pants AWS GovCloud and you need to sync between a directory on your EC2 instance and an S3 bucket, right? ARE YOU IN LUCK MY FRIEND:

What you'll need:

1. An AWS access key and the corresponding secret key. You can find these in IAM and create one just for this; however, you are limited to 2 AWS access keys for any API fancy connections. Always save your keys somewhere super safe.
2. Some EC2 instance that you need to either backup or sync contents.

How to:

1. On your EC2 instance you first need to setup "aws configure" to let the instance know HOW to connect to your s3 bucket with your credentials, so we're going to type:

[ec2-user@ip]$ aws configure

2. This will bring up the following prompts and these instructions are for GovCloud users, but you can use whatever region you are in:

  a. Paste in your access key ID:  AWS Access Key ID [ ]: 
 b. Paste in your secret key:  AWS Secret Access Key [ ]: 
  c. Enter your default region name: Default region name [ ]: us-gov-west-1
  i. NOTE: This has to be exact otherwise the aws command line will not know how to get to your bucket. If you don't know it, login to S3, click on any file you have in there and take a look at the properties, then check out the "Link:". Everything after the "s3-" up to the first "." is your region. E.g., mine is always s3-us-gov-west-1.amazonaws.com. So my region is: us-gov-west-1
  d. Leave default output format at none: Default output format [None]:
 e. Hit enter at the last step and it will spit you back out to the command prompt. 

3. Yay! That part is done. Now we can test by listing the contents of a bucket:

aws s3 ls s3://nameofbucket

  Note: You literally just need the name of the bucket in S3. So you don't need to put s3-us-gov-west-1.amazonaws.com/bucketofstuff ... from here on out, just but s3://bucketofstuff

4. If that worked you should see a nice ls listing of your bucket on s3! We're getting closer! 

5. Now to test syncing all the data FROM an S3 bucket to a directory on your EC2 instance:

  a. cd into the directory you'd like the data in first
  b. Then type: aws s3 sync s3://nameofbucket .
  c. It should now spit out the downloaded files into the directory your in. Sweet! 
  d. I'm old school so I want to SEE the data I'm putting in a dir, but yes, you can skip the "." and put in the literal path, e.g., /home/ec2-user/stuff or whatever dir you're syncing data with. 

6. Now to go the other way:

aws s3 sync /name/of/dir/ s3://nameofbucket
  
  a. Note: this will push EVERYTHING in the dir to the s3 bucket and overwrite
  b. In the case of the step 5 example where we're IN the directory we're putting data in, you'd just type: aws s3 sync . s3://nameofbucket 

7. AWESOME. Now you can setup a crontab for backups or get super-mega-fancy and put the AWS command line stuff on your Windows or Mac and push to S3 from ANYWHERE! Just remember to always have your AWS access key's handy and you're set. 



P.S. If you get lost, here's the S3 command line page and it will show you how to do wget's, cp's, and of course sync.

P.P.S. Don't forget to check permissions on your S3 bucket if you can't upload/download. You'll need to have at least "Upload/Delete" for an authenticated AWS user.