Speed up Deploy via SCP in GitLab CI/CD

Deploying thousands of small files via SCP to a server takes an inordinately long time, even via a very fast network, much longer than transferring one large file. Here’s a tutorial on my Gitlab CI setup for compressing all my deployment files into one large tarball, transferring it to to the server then uncompressing it there. 

I have been experimenting with GitLab CI/CD for use with my Swimming Management System projects for Masters Swimming Queensland. It’s a legacy project that I’m now gradually transitioning to modern standards. I’ve set up a Pipeline that will allow me to test commits in PHPUnit, then have them automatically deploy to a dev/test server. 

The project now uses both Composer and with some Angular modules it also uses NPM. In the build phase on GitLab CI, composer install is run to get the dependancies into the vendor directory. 

To do it this way, you’ll need to have shell access via ssh to your server. 

I’ve set up the test server deployment details and authentication credentials as in GitLab Variables:

Configuration of GitLab CI/CD Variables
Configuration of GitLab CI/CD Variables

In the deploy stage I’ve added the following code prior to upload via scp:

- tar -czf /tmp/build.tar.gz .
- echo "TEST_SSHPATH=${TEST_SSHPATH}" > sshenv
- sshpass -e scp -P ${TEST_SSHPORT} -r -o stricthostkeychecking=no sshenv ${TEST_SSHUSER}@${TEST_SSHHOST}:~/.ssh/environment

In this case, the target ‘.’ indicates that we are tarballing the current directory.

In the second line we create a file that contains the definition of an environment variable on the target server, which is set to the GitLab CI variable TEST_SSHPATH. TEST_SSHPATH contains the path to the root of my code deployment on the server.

Finally, we then scp this file to the target server, putting it in ~/.ssh/environment. This will mean that when we ssh into the server, that environment variable will be available to us, containing the value from the GitLab CI variable. 

Now we can transfer the build.tar.gz file and un-tar it. 

- sshpass -e scp -P ${TEST_SSHPORT} -r -o stricthostkeychecking=no /tmp/build.tar.gz ${TEST_SSHUSER}@${TEST_SSHHOST}:${TEST_SSHPATH}
- sshpass -e ssh -p ${TEST_SSHPORT} -o stricthostkeychecking=no ${TEST_SSHUSER}@${TEST_SSHHOST} "cd ${TEST_SSHPATH}; tar -zxf build.tar.gz"

The first line in this block does the transfer via scp, to our the path provided in the variable TEST_SSHPATH.

The second line, connects via ssh , changes directory to the code deployment path, then extracts the build files. Tar -zxvf removes the .tar.gz file after it’s done, so there’s no need to seperately delete it.

Before making this change, my swimman project would take 16:30 to build and install. With this change it’s down to 4:35. That’s a saving of 12 minutes which makes a big difference when deploying a quick fix to the test environment. 

Tasmania Holiday 2018 – Part 1

In late February 2018 we took our young family(2 3/4 years and 7 month old) to Tasmania to visit my Grandmother who lives in Pontypool on the East Coast of Tasmania. This blog post has a short listing of what we did and tips for doing a similar trip with young children.

Parking

Having two young children meant getting a lift to the airport wouldn’t really work for us. Also as we needed our pram and car seats in Tasmania, the AirTrain wouldn’t work either. So we decided to use an airport parking service. As it turned out there was a special on the Brisbane Airport ParkValet service. 

This option was fantastic for us. We were able to drive straight in and had plenty of space to unload the car seats and luggage from the car. There was also a concierge option that we probably would have taken, but it was only offered when we first booked and we couldn’t add it later. We didn’t really need it in the end though.

All our luggage and car seats unloaded at Brisbane Airport ParkValet
All our luggage and car seats unloaded at Brisbane Airport ParkValet

The Flight to Hobart

Our flights were on Virgin Australia, who helpfully let you take any baby related stuff on your flight without any excess baggage charges. We needed to book a seat for Lily as she is over 2, but Jasmine rode in Jacqui’s lap.

We got to board the plane first, with passengers who had special needs. This gave us time to get the kids on board, carry-on stowed away and everyone settled. Jasmine had a special infant seatbelt that attached on to Jacqui’s. She didn’t much like being strapped in and tried to squirm out as much as possible. 

On the way down Lily sat between us and I(David) sat by the window. Lily is prone to being very upset by loud noises such as motorbikes. However she was actually excited by the take off and wasn’t upset at all. We didn’t have any ear problems on the ascent either.

David, Lily, Jacqui and Jasmine on our flight to Hobart
David, Lily, Jacqui and Jasmine on our flight to Hobart

We were able to keep Lily amused with toys, colouring-in and for a short while the iPhone. She was a bit annoyed that she couldn’t access Netflix or ABC iView and didn’t like anything on Virgin’s entertain app. 

On descent Lily did get quite upset which we believe was due to pain in her ears. We did try a few things to get her to equalise the pressure but she wasn’t able to understand. She didn’t settle until just before landing.

Arrival

On arrival we waited until everyone else was off the plane to get out, so we could pick up all the lost toys from under our seats. On the tarmac we saw a business jet from the USA that had been equipped with weather research equipment for the SOCRATES project, studying the interactions between clouds and particles naturally produced by the ocean, such as sea salt and biogenic particles.

National Center for Atmospheric Research aircraft at Hobart Airport
National Center for Atmospheric Research aircraft at Hobart Airport

When we walked into the terminal we were right in front of the Melbourne Demons AFL team arriving from Melbourne, so there was a WIN TV crew filming us. We were told that Jacqui and Jasmine appeared in the preview and sports news item about it. 

The Hobart terminal arrivals area is quite small so there was a massive crowd around the baggage carousel when I got there. I managed to find a spot near the end and was surprised that the pram and car seats which were taken in oversize luggage in Brisbane came out on the carousel. 

By the time I’d come back Lily had made a friend in the waiting area. The game had become that their daughter would give Lily a lolly, she’d give it to Mum because she didn’t like it, then Jacqui would pass it back to the little girl’s brother. This went on for some time while I organised the hire car pick up. 

Lily made a friend in the terminal
Lily made a friend in the terminal

Hire car pick up was a tag team effort as Jacqui and I swapped duties watching the luggage and filling out paperwork at the Hertz desk. Eventually we were all sorted and we left the terminal. The little girl Lily had befriended was quite upset by this. 

Hire Car

We thought we’d be smart and hire a larger vehicle for our trip. We’d had a struggle fitting our luggage into our Corolla, with one suitcase having to go in the back seat and the other blocking access to the pram in the boot. 

We hired a medium sized SUV, listed on Hertz as a Nissan Qashqai or similar. We ended up with a Mitsubishi ASX. Immediately I noted a problem. There was no way to fit a pram and suitcases in the boot. In fact all it would fit was a pram. Even if we removed the rear parcel shelf cover there’d still be not enough space to fit them and it’d be dangerous without a cargo barrier.

So it turns out a Toyota Corolla sedan actually has more cargo space than a medium SUV Mitsubishi ASX.

It took quite some time to get the car seats installed and adjusted. This was complicated by light rain at the time. One frustrating thing I found was that after I’d installed Lily’s car seat, the rear seatbelt was looped in the wrong place. So I had to try to move the seat forward without completely removing the car seat. After what seemed like forever and several escape attempts by Lily, we got in the car and headed off to go up the East Coast to Grandma’s place.

Brat Runs Amok on National TV

Today’s cutesy viral video is from the UK where a mother was being interviewed on TV with her two children. The younger toddler runs around the studio, climbing up on the desk and everyone ignores it. In fact everyone laughs and thinks it was cute!

As the father of two children under 3 I find this behaviour absolutely disgusting. Not the toddler’s behaviour, I know toddlers sometimes do run amok even with the best discipline and training. 

The problem I have with this is everyone’s reaction, especially the mother’s. It’s not okay to ignore such terrible behaviour in public. It’s not funny, it’s not cute, it’s unacceptable. 

You see this in public places every day. Parents are standing in the queue for a bank teller for instance, meanwhile their kids are terrorising the whole bank, climbing on chairs and counters, drawing all over forms and making too much noise. Usually the parents are completely oblivious to their little brats anti-social behaviour. 

Children must be taught that there are times where they must stand still and quiet with their parents. On several occasions I stood at the swimming pool holding my daughter’s hand while she pulled and screamed as we waited for her mother to get changed. She just wanted to run around and play.

However after consistently making her stand still and asking her to be quiet, now when I do this she does stand still with me and remain mostly quiet while waiting. She’ll also sit with me in a chair quietly and wait for long periods of time. Sure I often have to remind her to sit still and quiet, but she will do it most of the time. Yes she tries to test the boundaries, but with constant reinforcement it’s possible to keep her behaving. 

Courteous behaviour in public, respect for other people and their property and waiting are all major life skills that children need to learn. We do nothing to help them learn those skills by laughing at or calling it cute when they misbehave.

How to Restore an iCloud Backup to a New iPhone with 2 Factor Authentication

My wife recently purchased a new iPhone 7. We backed up her old iPhone to iCloud, then proceeded to start up her new iPhone, selecting the Restore from iCloud Backup option. We then hit a snag. 

I had enabled 2 Factor Authentication for iCloud access. This meant to be able to access her iCloud account from the new phone, she needed to provide a code that would be SMS’d to her iPhone. However has her number had been transferred to the SIM card in the new iPhone, this meant the SMS would go to the new iPhone, the one we were trying to set up. Because we were still trying to set it up, we couldn’t see the code. We also didn’t have any other phone we could put that SIM card in.

The only other option would have been to recover the iCloud account via an alternate method., which takes several days. Not what we wanted to do. 

After much frustration and a phone call to AppleCare, it turned out the solution is simple:

Select the Resend Verification option

The initial SMS was hidden and didn’t automatically advance us through the process, but tapping the option to resend the code enabled us to continue forward. We still didn’t see the message, but the phone automatically recognised it had been sent the verification code and allowed us to continue to restore the new phone from the iCloud backup.

Getting Started with Google AdSense

Here’s the companion resource links to go with my presentation on Network Live Virtually 30/05/2017 on earning passive income via Google AdSense

I also mentioned there are other better tools you can use for Keyword Research if you want to go to the next level. Here’s the tool I was referring to:

  • Long Tail Pro(affiliate link) – I have used a trial version and it seemed pretty good. I don’t use it right now as I’m not actively building new sites.

I’ll link to the presentation replay once it’s posted. 

JSON Feed – The New Way to Syndicate Posts

There’s a new way to provide syndication feeds for websites. JSONFeed does basically the same thing as the traditional RSS or Atom feed, but instead of using XML it uses JSON. The big advantage of this is that the file size of the feed is much smaller and processing of the feeds can be much more efficient. 

You can enable it now on WordPress using the JSONFeed Plugin for WordPress. You can find it by searching for JSONFeed via the Plugins->Add New option in the WordPress Administration Dashboard. 

Enabling the plugin will make your feed accessible via JSONFeed. You can see an example by viewing the davidfindlay.com.au JSONFeed. I’m not aware of any feed readers yet that use this, but you can be sure they’ll come soon. It’s a bit of a chicken and egg problem. I’m always keen on being an early adopter on things like this. 

My Site Moves to Amazon Web Services

I’ve finally moved my site onto Amazon Web Services. It’s now running on a t2.micro EC2 instance in Amazon’s Asia Pacific Sydney region. It’s using Amazon’s linux AMI, with Apache httpd. MySQL is served via an Amazon RDS MySQL instance. The domain is delegated to Route 53 for DNS. 

So far it seems to be faster than my traditional shared hosting and that’s without even looking at any particular optimisations yet. I’m going to try to get some metrics soon to prove it. I also plan to transition all my other sites across to this type of hosting. 

The only thing still running on the old hosting is email. Amazon still has a particular gap here. I could run my own email server, but I’d prefer not to. I’m going to look into some options though. 

Fitness and Swimming Goals

I’ve been pretty lazy and busy lately, so my fitness has dropped off. I got to 85kg, the heaviest I’ve ever been. So I’ve set a goal to get back to peak fitness again. 

Since I’ve been so busy with our baby, I’ve not been swimming as much. I used to swim at least twice a week and was state masters champion in some events. I’ve fallen a lot since then. In the last 12 months I have barely swum. For the last few months I’ve been swimming once a week when I can. Time to get back into it!

So as of today I’m 80kg, thanks to cutting back on sugar, particularly Coke, and walking a lot. I usually walk 8000 steps a day or 4-6km a day, according to my iPhone 6. 

Swimming Target

It’ll take a long time to get my distance swimming endurance back. However if I work hard on my strength and anaerobic fitness, I should be able to get back into sprinting quickly. I’ll need to improve my technique as well, as it’s not as good as it used to be. 

Here’s my goal:

To swim 50m in under 30 seconds by the end of 2017

I’ve never done it before. My personal best is 30.51 in the 50m Long Course at Somerville House on 17/03/2013 at the Masters Swimming Queensland State Championships. 

Bodyweight Fitness

I discussed Bodyweight Fitness in Reddit’s /r/bodyweightfitness. I quite like its philosophy. I’m planning to use it to improve my muscle strength. I’ve already started, using an iOS app by one of the people involved in the subreddit. You can find it here: The ultimate mobile app for Bodyweight Fitness.

Training partner

I did my first set using the app today, but didn’t make it all the way through. The exercises feel good though, and even my toddler daughter seemed to want to get in on it. Might have a training partner soon!

How I Made $1000 From Google AdSense on My Authority Site

AdSense mobile app
AdSense mobile app – $1000 lifetime earnings

On Thursday 5th of January 2017, I hit the $1000 mark on Lifetime earnings on Google AdSense. All but about $30 of that was earned in the immediately preceding 12 months. This is a tale of how you can have some minor success with time and effort. 

You can see a presentation on this in the presentation section at the bottom of the page.

I started my site, Digital TV Help, in February 2014 on the topic of Do It Yourself TV Antenna work and TV setup. It was something I had previously done as a self employed technician. 

Then I had basically zero traffic and zero income from the site. In the month of December 2016, according to Google Analytics, I had 9775 sessions with 8775 users. Last month I had estimated earnings of $102.42 Australian and just under a week later I reached $1000.11 lifetime estimated earnings.

How I got there

I started the site after discovering Pat Flynn’s Smart Passive Income podcast in December 2013. The site was created in January 2014.

Initially my site was very unfocused and I had the idea of educating both the DIYer and people entering the industry. I didn’t have a very good design, just a typical blog design, and I only had a few articles of poor quality. Didn’t have that many pictures either only the ones I’d taken while running my business.

In August 2014 I went to Pat Flynn and Chris Ducker’s 1 Day Business Breakthrough seminar at Problogger on the Gold Coast. The feedback I received there helped me to change to a narrower focus just on the do-it-yourselfer audience. Unfortunately it took me a long time to implement things suggested.

By December 2014 I had written 65 posts, ranging from articles about how things worked, to tip of the day articles and only a few how-to articles. I’ve always taken a publish first, improve later approach. In December 2014 I had 494 sessions and 437 users, with $2.19 of estimated earnings. In the whole year of 2014 I had estimate earnings of $11.18.

A bit demotivated and busy with full-time work, study, the birth of our first child and other commitments, I didn’t do much over the following year. After March 2015 I didn’t do anything to my site until March 2016. 

Monthly Sessions on Google Analytics
Monthly Sessions on Google Analytics

In January 2016 I discovered my site had received a huge increase of users and was earning me a lot more money suddenly. In that month I had estimated earnings of $7.56 from 4506 sessions and 4008 users. From January 2014 to December 2015 my estimated earnings were $33.70. 

Between January and March 2016 I did some changes to my AdSense advertising based on research on various SEO sites. After checking my stats and doing some experimentation I determined that none of my AdSense income came from header or sidebar ads. All of it came from the ads in my articles. I eventually discovered best income came from 2 or 3 horizontal banners in my articles, one after the second paragraph, one before the last paragraph and maybe one in the middle.

Earnings steadily increased over following months until between March and July each month I had between $70 and $85 each month. I started receiving a bank transfer from Google every second month. Regaining motivation in March I started reviewing and improving some of my content and adding some new stuff. I now have 88 posts and about 6 videos. I still need to do a lot of work. (See SPI: 200 on Content Audits)

Eventually I started getting $100 or more a month in estimated earnings. From July to October I got a regular monthly payment of just over $100 from Google. A slight drop in earnings in November meant I didn’t get a December cheque. Since then the income seems to regained its losses. 

Month by Month Google AdSense Estimated Earnings
Month by Month Google AdSense Estimated Earnings

On January 5th 2017 I hit the $1000 lifetime estimated earnings mark. Finally! 

What Made the Difference?

The vast majority of my income has come from one post. It was post aimed at do-it-yourselfers. Over time it had just become my number one landing page with it’s acquisitions coming from Google. It seems I had just done a really good job of making that article. Over the whole time since I wrote it, the article has gradually increased in Google Search hits and income. 

Over the lifetime of the site, it had these stats:

  • 40655 pageviews (24.91% of site pageviews)
  • $215.71 Australian (30.19% of site income)

It is titled “DIY Antenna Alignment Methods for Digital TV”. This title immediately explains its usefulness to people as does the blurb. It matches pretty well to a search of “How to align a Digital TV Antenna” and the alternate searches people might use for that topic. It contains 756 words and is quite comprehensive on the topic. It has pictures, multiple headings and an embedded YouTube video. 

I believe it’s just a matter of that as more Google search traffic clicked that post, the higher the pagerank rose. The quality of the article got it into the search initially, but clicks brought higher pagerank and more clicks.

One thing I did do to the article in December 2014 was to add a strong first sentence. A member of the mastermind group I meet with pointed out that the blurb that appeared for the article on Google Search was “In a previous article blah blah blah…” It didn’t say anything about this article. I changed it to add a new first paragraph(which is still there). It now says exactly what the user will get from the article. I think it helped increase clicks, which as I say bring more clicks.

Other articles on the sites do get hits and income. The 2nd best article on the site is another How To article. However it gets only half the landings and income of the best article. The rest of the landings are spread across the site. 

I don’t get many landings on my front page(only 6%). Almost all my traffic comes from organic Google Search. I don’t do much social media promotion and no paid promotion.

How to do it yourself

These are my recommendations. I’m not an expert in SEO or writing. I’ve just followed what I’ve read from experts I follow and done some experiments. I certainly haven’t tried everything I could have tried and still have a lot of work to do to increase the quality of my site and increase my income. 

This is what I think works for me: 

  • Use the official Google AdSense WordPress plugin – it automatically makes ads responsive for mobile
  • Use only 2 or 3 horizontal banner ads, set between paragraphs of your posts
  • Allow graphics or text
  • Write lots of deep, quality content, including embedded video and photos
  • Write for people not search engines
  • Write useful articles, like do it yourself or how to content
  • Strong first sentence and paragraph – tell the user what they’ll get from your article – e.g. “Here are four methods to do blah blah blah….”
  • Wait

I think I could have reached where I am more quickly had written a lot more high quality do-it-yourself how-to articles earlier. Also more videos and social media promotion may have got me here quicker. 

My aim now is to follow a process of reviewing and improving my existing content. I’m looking to significantly increase the number of high quality how to articles and embedded videos I have. Hopefully this will get me to a level of income where it can become my day job. 

Presentations

I’ve recently presented my story and Google AdSense advice on Network Live Virtually with Sabrina Watson.  You can see a replay here:

http://networklivevirtually.com/how-i-made-1k-with-google-adsense/

Resources

Sites with useful authority site information:

Here’s what I’ve used:

Skiiddii Bicycle Child Trailer Review

We purchased a trailer for our bikes to enable us to carry our toddler with us when we go for bike rides. We purchased the Skiidii Bicycle Trailer for $149 on eBay from seller Faji Plaza. The trailer has provision for two small children to ride on it side by side.

The trailer arrived in a flat square box weighing 17kg. We had ours delivered to our Australia Post postbox without any problems. Delivery occurred within a week of order.

On opening the box you find a large folded trailer body, some tubular sections, some wheels and other components. The tools required to assemble the trailer are provided with the kit, a couple of small spanners and an Allen key. I chose to use a socket set and ratchet spanner as this was easier. You’ll need an 11mm hex socket.

At first I thought no instructions were supplied and this made it difficult to figure out how to get the trailer properly assembled. However it turned out the instruction booklet had been wrapped up in the fabric cover for the trailer. Unfortunately, the instructions are pretty badly translated.

I started by unfolding the trailer body and installing the cross bar that supports the roof of the trailer and holds the uprights in the vertical position. It fits together using plastic brackets on each end and some locking pins.

Locking pin
Locking pin

After this I installed the axle(which required adjusting the holes it goes through). The wheels are locked into place with plastic clips that fit into retention grooves at the end of the axle. The diagrams in the assembly guide aren’t too helpful. Wheel protection bars are then connected which go around the outside of the wheels, retained by spring loaded locking pins.

The trailer can be used as a running pram, so either a front wheel or the tow bar can be attached. The tow bar attaches with two locking pins. You can then remove the bike attachment from the tow bar using its locking pin and install it on the bike’s rear axle. You simply remove one of the screws and put it on the axle before screwing it back on. I installed it on the side that didn’t have the gears.

The plastic brackets for the jogging handle are then installed, but I didn’t install the handle as I won’t be jogging with it. An orange safety flag is also provided. The fabric cover is attached with velcro at the front and back and provides a weathershield as well. There is pretty good visibility for your child to be able to see out.

Assembly took me about 45 minutes in total. Once done I attached it to the bike and gave it a try. At first I accidentally knocked the bike over but the trailer stayed upright. I was happy about that. Riding around with the trailer empty it bounces around quite a lot. With our toddler aboard it still bounces a lot but not too badly. The wheel protection bars do rattle a lot however. They don’t fit snuggly into their slots. So while they won’t fall off, they do make a bit of noise. I may look at putting some thin rubber in the slots so they don’t rattle.

Bike and trailer
Bike and trailer

The child harness isn’t particularly fancy, just being a strap system. This may be a bit of an issue for younger children. Our baby has no problems walking or sitting up but did tend to end up slumping down on the seat. For larger babies I don’t think this will be a problem.

Baby in trailer harness
Baby in trailer harness

Overall I’m happy with the Skiidii Bicycle Trailer so far. We haven’t taken it on long rides yet, but will in coming weeks. Build quality is pretty good for the price. The only real issue is the rattling of the wheel protection bars. I’m sure more expensive models would be much more solid. As noted the assembly manual isn’t too good either.

One other issue we did have was finding a suitable helmet. It seems they start at 48cm at the smallest. A smaller one would have been better for us.