Compilation and deploy via SSH in Gitlab CI

Let's take a look to how GO compilation works and use Gitlab CI for that.

Compilation

To compile we should run go build -o binary_name.

If there are imports of 3rd party code, that means there are external dependencies in project. To build binary all source code is required (including the source code of 3rd party libraries).

Vendor or loading

Currently the latest GO version is 1.14.2. Staring from GO 1.11 there is go modules functionality in GO.

Go modules makes GO download every dependency when go run, go build is called. Also, we can explicitly download dependencies by calling go get ./....

Downloaded dependency code is cached in $GOPATH/pkg/mod or in $HOME/go/pkg/mod, if $GOPATH variable is not set.

There is also vendor approach that can be used on dependencies. If -mod=vendor_dir parameter is set, dependencies will be downloaded to vendor_dir folder. That folder can be located inside project repo and stored in Git.

If dependencies source code stored with the project code, there is no need to download dependencies.

Gitlab

Gitlab allows us to run some actions after code push.

To make Gitlab do something with your code, we need to put .gitlab-ci.yml file with instruction into the root of repository.

Compilation with Gitlab

We need to set particular stages and actions to make Gitlab perform them. For every stage and action we can specify docker image which will be downloaded.

The source code is downloaded into /builds/{project_group}/{project_name} folder, that means there is no need to do anything to get the source code.

I created build stage and the same name action:

stages:
  — build
build:
  image: rhaps1071/golang-1.14-alpine-git
  stage: build
  script:
    — go get ./...
    — GOARCH=amd64 GOOS=linux go build -ldflags "-extldflags '-static'" -o $CI_PROJECT_DIR/binary
  artifacts:
    paths:
      — binary

I used custom docker image rhaps1071/golang-1.14-alpine-git to perform build.
There is git command i added to my custom docker image which is absent in golang:1.14-alpine. That is critical to make go get ./... work, which downloads dependencies by using git clone.
golang:1.14-alpine is based on Alpine Linux, that is very small (about 3MB).
But golang:1.14-alpine size is 370MB, because of GO in it. But there is 2x image size economy compared to golang:1.14 (809MB), that is based on Ubuntu.

There is no vendor approach in the project, so i have to download the dependencies. That's why go get ./... is used.

As mentioned before, the source code is placed into /builds/{project_name}/{project_folder} folder by Gitlab. That path is outside $GOPATH.
To make GO commands work outside $GOPATH go.mod file is required to be placed in the root of project files. That file can be created with go mod init command. If there is no go.mod file go get ./... (also, run and build) will fail to run outside $GOPATH.

If there are no GO modules your project, your source code needs to be placed inside $GOPATH.
In Gitlab CI we can make that copying following way:
- cp /builds/* $GOPATH/src/

artifacts in .gitlab-ci.yml allows us to save any file from CI to manual download via Gitlab interface or to pass that file to following stages/actions.

Deploy via SSH

Let's take a look to simple binary deploy via SSH:

deploy_stage:
  image: kroniak/ssh-client
  stage: deploy
  environment:
    name: stage
    url: http://stage.project.com
  when: manual
  script:
    — mkdir -p ~/.ssh
    — echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
    — chmod -R 700 ~/.ssh
    — echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
    — chmod 644 ~/.ssh/known_hosts
    — echo "$CONFIG" > ./config.json
    — scp -P$SSH_PORT -r ./config.json $SSH_USER@$SSH_HOST:/var/www/project/config/
    — scp -P$SSH_PORT -r ./binary $SSH_USER@$SSH_HOST:~/binary_tmp
    — ssh -p$SSH_PORT $SSH_USER@$SSH_HOST 'sudo service project stop && cp ~/binary_tmp /var/www/project/binary && sudo service project restart'

Here we are using custom docker image based on Alpine Linux again — kroniak/ssh-client size of 12.1MB. But at this time there is ssh-client installed, which allows us to use ssh and scp commands.

The deploy logic is following:

There are following Gitlab variables are set: $SSH_PRIVATE_KEY - private SSH key to access server;
$SSH_USER, $SSH_HOST, $SSH_PORT — SSH credentials;
$SSH_KNOWN_HOSTS — data for .ssh/known_hosts file, which helps us to validate that we are deploying to the exact server;
$CONFIG — contents of service config file in json;
On deploy we put data from some Gitlab variables into files;
All files are copied with scp command. We could use rsync here, because it only copies changed files. But because there are only 2 files needs to be copied, it does not matter;
Finally we change our binary and restart the service;

Full contents of .gitlab-ci.yml is below:

stages:
  — build
  — deploy
build:
  image: rhaps1071/golang-1.14-alpine-git
  stage: build
  script:
    — go get ./...
    — GOARCH=amd64 GOOS=linux go build -ldflags "-extldflags '-static'" -o $CI_PROJECT_DIR/binary
  artifacts:
    paths:
      — binary

deploy_stage:
  image: kroniak/ssh-client
  stage: deploy
  environment:
    name: stage
    url: http://stage.project.com
  when: manual
  script:
    — mkdir -p ~/.ssh
    — echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
    — chmod -R 700 ~/.ssh
    — echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
    — chmod 644 ~/.ssh/known_hosts
    — echo "$CONFIG" > ./config.json
    — scp -P$SSH_PORT -r ./config.json $SSH_USER@$SSH_HOST:/var/www/project/config/
    — scp -P$SSH_PORT -r ./binary $SSH_USER@$SSH_HOST:~/binary_tmp
    — ssh -p$SSH_PORT $SSH_USER@$SSH_HOST 'sudo service project stop && cp ~/binary_tmp /var/www/project/binary && sudo service project restart'