GitLab Pipeline

Speed up Deploy via SCP in GitLab CI/CD

Deploying thousands of small files via SCP to a server takes an inordinately long time, even via a very fast network, much longer than transferring one large file. Here’s a tutorial on my Gitlab CI setup for compressing all my deployment files into one large tarball, transferring it to to the server then uncompressing it there. 

I have been experimenting with GitLab CI/CD for use with my Swimming Management System projects for Masters Swimming Queensland. It’s a legacy project that I’m now gradually transitioning to modern standards. I’ve set up a Pipeline that will allow me to test commits in PHPUnit, then have them automatically deploy to a dev/test server. 

The project now uses both Composer and with some Angular modules it also uses NPM. In the build phase on GitLab CI, composer install is run to get the dependancies into the vendor directory. 

To do it this way, you’ll need to have shell access via ssh to your server. 

I’ve set up the test server deployment details and authentication credentials as in GitLab Variables:

Configuration of GitLab CI/CD Variables
Configuration of GitLab CI/CD Variables

In the deploy stage I’ve added the following code prior to upload via scp:

- tar -czf /tmp/build.tar.gz .
- echo "TEST_SSHPATH=${TEST_SSHPATH}" > sshenv
- sshpass -e scp -P ${TEST_SSHPORT} -r -o stricthostkeychecking=no sshenv ${TEST_SSHUSER}@${TEST_SSHHOST}:~/.ssh/environment

In this case, the target ‘.’ indicates that we are tarballing the current directory.

In the second line we create a file that contains the definition of an environment variable on the target server, which is set to the GitLab CI variable TEST_SSHPATH. TEST_SSHPATH contains the path to the root of my code deployment on the server.

Finally, we then scp this file to the target server, putting it in ~/.ssh/environment. This will mean that when we ssh into the server, that environment variable will be available to us, containing the value from the GitLab CI variable. 

Now we can transfer the build.tar.gz file and un-tar it. 

- sshpass -e scp -P ${TEST_SSHPORT} -r -o stricthostkeychecking=no /tmp/build.tar.gz ${TEST_SSHUSER}@${TEST_SSHHOST}:${TEST_SSHPATH}
- sshpass -e ssh -p ${TEST_SSHPORT} -o stricthostkeychecking=no ${TEST_SSHUSER}@${TEST_SSHHOST} "cd ${TEST_SSHPATH}; tar -zxf build.tar.gz"

The first line in this block does the transfer via scp, to our the path provided in the variable TEST_SSHPATH.

The second line, connects via ssh , changes directory to the code deployment path, then extracts the build files. Tar -zxvf removes the .tar.gz file after it’s done, so there’s no need to seperately delete it.

Before making this change, my swimman project would take 16:30 to build and install. With this change it’s down to 4:35. That’s a saving of 12 minutes which makes a big difference when deploying a quick fix to the test environment.