How can I clear S3 bucket before travis deploy

Question

This is my .travis.yml:

deploy:
  provider: s3
  access_key_id: $AWS_ACCESS_KEY
  secret_access_key: $AWS_SECRET_KEY
  bucket: domain.com
  skip_cleanup: true
  acl: public_read
  region: ap-northeast-1
  endpoint: domain.com.s3-website-ap-northeast-1.amazonaws.com
  detect_encoding: true
  on:
    branch: master

But it is only upload files to bucket, not sync. How can I sync or clear S3 bucket files?

Answer 1

Thanks Guys, The scripts above helped me to build the following that worked for me

before_script: - pip install awscli - export PATH=$PATH:$HOME/.local/bin - AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY aws s3 rm s3://$BUCKET --recursive --region=$REGION

Answer 2

The update helped me find a solution. Thanks! It's cleaner than the answer proposed by hussfelt.

Using awscli

Since using the listed command required some research I'll explain how I had to change my .travis.yml for any others who find this post.

before_deploy: pip install --user awscli

First install awscli to enable syncing with your S3 bucket. To run on Travis' container-based architecture we can't use sudo, so install to the home directory with --user. On Linux, which is the default OS on Travis, binaries installed with this option is located in ~/.local/bin/-

deploy:
  provider: script

Next, use the script provider to run a custom command as the deployment method.

  script: ~/.local/bin/aws s3 sync dist s3://example.com --region=eu-central-1 --delete

This line is were your files are uploaded. The aws s3 sync is used to sync files between a local machine and buckets. The full documentation is available here.

In my example dist is the build folder we want to upload to S3. Your build system might call it build or something else. "example.com" is the name of your bucket. The region argument is required to uniquely identify your bucket.

The really interesting bit in this command is the --delete switch which is the solution to our problem. When set, aws will delete any files found in your bucket but not in your build directory.

  skip_cleanup: true
  on:
    branch: master

skip_cleanup should be set or none of your files will be uploaded. Personally I like to have Travis deploy only commits to master, but any configuration is possible here. See the docs for more info.

Environment

We need to give aws our AWS credentials to authorise any interaction. The environment variables used by aws are AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. hussfelt writes how to provide these in their answer. The process is also described in the Travis documentation: encryption and AWS specifics.

Full solution

# Deploy using awscli to enable pruning of removed files
before_deploy: pip install --user awscli
deploy:
  provider: script
  script: ~/.local/bin/aws s3 sync dist s3://example.com --region=eu-central-1 --delete      
  skip_cleanup: true
  on:
    branch: master

Answer 3

To solve this I installed the AWS cli from pip and executed a before-deploy script.

This is what you need in your .travis.yml:

before_install:
  - pip install --user awscli
  - export PATH=$PATH:$HOME/.local/bin
before_deploy: bin/deploy.sh

You also need to secure two environment variables inside your .travis.yml which is ready by aws-cli:

travis encrypt AWS_ACCESS_KEY_ID=YOUR_KEY_HERE --add
travis encrypt AWS_SECRET_ACCESS_KEY=YOUR_SECRET_HERE --add

Your bin/deploy.sh should look something like the following

#!/bin/sh

echo "Clearing bucket: s3://your-bucket/path/inside/bucket/if/you/want"
aws s3 rm s3://your-bucket/path/inside/bucket/if/you/want --recursive --region eu-central-1

Not that we specify the region, which seems mandatory for aws cli here.

Hope this helps!

Answer 4

Just want to add on everybody else's answer. Unfortunately I can't comment because I have low reputation.

installing awscli

I am getting error on the pip install awscli or pip install awscli --user script. Here's what worked for me for installing awscli at travisCI:

before_deploy:
  - pip install awscli --upgrade --user

adding credentials

and if anyone wants also know how to add aws credentials, I found this good answer here.

1. Setup your ENV variables, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY at travis-ci dashboard settings. Your aws s3 commands should work if these variables are setup.