Artifact Deployment via Google Drive

Artifact Deployment via Google Drive
July 13, 2016

The Twelve-Factor App is a set of best practices for building web applications. Every project I build adheres to the principles laid out at the above website and I highly recommend internalizing them if you’re a full stack developer or in dev ops.

One of the key points is “dev/prod parity”: minimal difference between development, staging and production environments. For staging and production, the best practice is produce an artifact (typically an archive of everything needed to run your application, including dependencies) during your build step and deploy the same artifact to both environments. Deployment of a single artifact with its dependencies guarantees all platforms are running the exact same code, they only differ in configuration.

It’s common to see Amazon S3 used for artifact hosting, particularly if the app is also deployed on Amazon EC2. While S3 is cheap, it’s not free :). In my quest to build a completely free web application stack, I’ve built a way to use Google Drive as the artifact host. This blog post will break down how to upload your artifact to Google Drive from a Continuous Integration (CI) tool.

Continuous Integration (CI)

My CI tool of choice is Shippable. Shippable is similar to CircleCI or TravisCI (in fact it supports the .travis.yml format verbatim!), however it’s free for private repos for BitBucket and GitHub. I use it primarily with BitBucket for a one-two punch of free code hosting and CI.

Google Drive

Interaction with Google Drive is done through prasmussen/gdrive, a command line Google Drive client written in Go. It supports most platforms (Windows, Linux, Mac OS X, *BSD). It is single binary, meaning it’s easily bundled with our application.

The Magic

Shippable works by reading the steps in shippable.yml. Here’s an example shippable.yml for a NodeJS project:

node_js:
  - 6.2.1
language: node_js
env:
  global:
    - secure: [OMITTED]
script:
  - npm run test
  - NODE_ENV=production npm run deploy
after_script:
  - ./bin/upload.sh

This implicitly runs npm install loading dependencies from our package.json, runs our tests and builds the final artifact. Once that is all completes successfully, the after_script step is run. This is where the magic happens. In our project we’ve defined a shell script (./bin/upload.sh) that archives the app (and node_modules) and uploads to Google Drive. Here it is:

#!/bin/sh
FIXED_BRANCH=$(echo $BRANCH | sed 's/\//-/g')
ARCHIVE=$REPO_NAME-$FIXED_BRANCH-$(date +%Y-%m-%d_%H_%M_%S)-$COMMIT.tar.bz2
echo "Creating archive $ARCHIVE"
tar cfj $ARCHIVE dist
FILESIZE=$(stat -c%s "$ARCHIVE")
echo "Finished archive (size $FILESIZE), starting Google Drive upload"
./bin/gdrive upload --refresh-token $GDRIVE_REFRESH_TOKEN --parent $GDRIVE_DIR "$ARCHIVE"
echo "Finished Google Drive upload"

This script uses the prasmussen/gdrive binary ./bin/gdrive. Download the appropriate gdrive binary for your build system and put it in your bin folder in your project.

There are a number of environment variables used in the above script. Some are exposed by Shippable automatically (see Configure Your Build - Using environment variables) and some we will need to provide ourselves. I’ll go into detail explaining each one:

$BRANCH: Git branch name.
$FIXED_BRANCH: File system safe version of the git branch name – replaces / in the branch name with a - using sed. This may not make the branch name 100% safe, but / is the most common character in git branch names that is invalid in file names.
$REPO_NAME: Repository name.
$COMMIT: Commit hash for the latest commit in the branch.
$ARCHIVE: Artifact filename. It is a concatenation of $REPO_NAME, $FIXED_BRANCH, a datetime and $COMMIT.
$FILESIZE: Filesize in bytes of the artifact archive
$GDRIVE_REFRESH_TOKEN: Google Drive OAuth2 refresh token. This is a persistent authentication key used by the Google Drive client. We need to provide this!
$GDRIVE_DIR: Google Drive parent directory ID. We need to provide this!

There are two keys we need to provide to make this work. How do we get them and how to we let our build process know about them?

Google Drive Refresh Token

Download the prasmussen/gdrive binary for your system and run the following command.

$ ./gdrive list
Authentication needed
Go to the following url in your browser:
https://accounts.google.com/o/oauth2/auth?access_type=offline&client_id=...

Enter verification code: ...
Id                             Name                                       Type   Size      Created
0B1h7GlNyl9MaSnNMY0RRWW5PREU   app-client-feature-...3689705bfe.tar.bz2   bin    8.0 MB    2016-07-11 22:25:17
0B1h7GlNyl9MaWE8ybFk1SjROMWc   app-server-master-2...34b0e75e79.tar.bz2   bin    16.3 MB   2016-07-11 22:22:59

It will ask you to authenticate at a Google URL. Go to the URL, login with your Google Account, permit the app to view/edit your Google Drive files and copy/paste the verification code you’ve received back into the command line prompt.

Notice that when you run ./gdrive list a second time it no longer prompts you for authentication. This is because the client has cached your OAuth2 access tokens and refresh tokens in the file ~/.gdrive/token_v2.json:

{
  "access_token": "...",
  "token_type": "Bearer",
  "refresh_token": "...",
  "expiry": "2016-07-13T11:47:00.424973987-04:00"
}

Make note of your refresh token for later.

Google Drive Folder ID

Go to Google Drive and navigate to the folder where you wish artifacts to be uploaded. Copy the ID from the URL bar.

Google Drive folder URL bar

Encrypted Environment Variables

Shippable has this nifty feature that lets you provide encrypted environment variables to your build environment. On Shippable navigate to Your Project -> Settings -> Encrypt.

Encrypt

Here we can encrypt the two environment variables we need, $GDRIVE_REFRESH_TOKEN and $GDRIVE_DIR:

Environment Variables

Copy the secure value to your shippable.yml. Shippable holds the secret key to the encrypted phrase so only they can decrypt it on build. That should be all you need to get your project artifact uploaded to Google Drive on every successful build!

Deployment with Ansible

Ansible uses a similar deployment script, ./bin/download.sh. This will connect to your Google Drive, read the contents of your $GDRIVE_DIR and find the most recent artifact that matches. It will then download this artifact to a /tmp folder locally.

#!/bin/sh
ARGS=("$@")
TMP_DIR=/tmp
PROJECTS="app-server"
BRANCHES="master"
GDRIVE_DIR="..."
GDRIVE_REFRESH_TOKEN=${ARGS[0]}
GDRIVE_BIN=./bin/gdrive

GDRIVE_OUTPUT=$($GDRIVE_BIN --refresh-token "$GDRIVE_REFRESH_TOKEN" list -q "name contains '$PROJECT-$BRANCH-'" --order "name desc" -m 1 --no-header --name-width 0)
IFS='   ' read -a GDRIVE_FIELDS <<< "$GDRIVE_OUTPUT"
$($GDRIVE_BIN --refresh-token "$GDRIVE_REFRESH_TOKEN" download --path "$TMP_DIR" --no-progress -f "${GDRIVE_FIELDS[0]}" >/dev/null 2>&1)
echo "${GDRIVE_FIELDS[1]}"

Below is the ansible task that runs ./bin/download.sh, downloads the artifact locally, uploads it to your server, unarchives it and starts the service! Variables such as the gdrive_refresh_token are stored in Ansible’s encrypted vault.

- name: Download latest release from Google Drive
  register: gdrive_downloads
  local_action: shell ./bin/download.sh "{{ gdrive_refresh_token }}"
  become: false

- name: Copy app-server artifact {{ gdrive_downloads.stdout_lines[0] }}
  copy: src={{ temp_dir }}/{{ gdrive_downloads.stdout_lines[0] }} dest={{ common_deployer_home }}

- name: Unarchive app-server
  unarchive: src={{ gdrive_downloads.stdout_lines[0] }} copy=no dest={{ common_deployer_home }}

- name: start app-server
  service: name=app-server state=started

Moving Forward

This approach emphasizes a moving-forward mentality: code deployed is always the latest from the master branch. To “rollback” code, you can simply issue a git revert on an erroneous commit, push it up, wait for this commit to be build and uploaded, and then redeploy. In emergencies the logic in ./bin/download.sh can be modified locally to return older artifacts.

That’s it!

There you have it. I’ll leave the rest of the Ansible deployment script as an exercise to the user, but the core idea of uploading and downloading artifacts to Google Drive has been demonstrated.

Upshot

Once this build flow is established, I’ve found it to be a remarkably stable way of doing deployment. Deploys are simple, fast and reproducible. There is no fetching dependencies on the server. When combined with Google Drive, Shippable, Bitbucket and Amazon EC2‘s free tier, this is a complete free web application stack that follows Twelve-Factor best practices.

Happy deploying!

devops

Joey Robert