Your CI builds are failing and you're not sure why. Troubleshooting a broken Travis CI build has a fairly long tweak-test cycle time: you add some debugging statements, commit and push, wait for the build to start, maybe get a few more clues, then rinse and repeat until you've found the problem and fixed the build. If only you could get a shell in the build environment, and troubleshoot interactively!
This feature isn't built into Travis CI, but it can be obtained with the following recipe. Basically, we'll instruct the Travis build environment to create a reverse SSH tunnel to a host that we control, exposing its local port 22. From there, we can create an interactive SSH session back to the Travis build environment.
(For us, Travis CI had released a new Trusty 14.04 image with a broken APT repository for Docker, which prevented us from adding a few PPAs required by our application. This was made immediately clear after getting a shell and looking around in /etc/apt/sources.list.d.)
Prepare Bounce Host
You need a host separate from the Travis build environment which has SSH exposed to the world. A disposable cloud instance is perfect for this task. Let's call it our bounce host.
- Create a user (
adduser travis), set a password when prompted - Ensure you can SSH to your bounce host with
travisand the specified password. (modify yoursshd_configas needed and restart the ssh service if necessary)
Next, set a couple of encrypted environment variables for your Travis CI project: "sshpassword" (storing the password you just set for the travis user on your bounce host) and "bouncehostip" (naturally, to the public IP address of your bounce host).
Populate .travis.yml
Put these lines in the script: section of your .travis.yml, perhaps at the end, after the other build commands for your project. (trivial example repo)
- echo travis:$sshpassword | sudo chpasswd
- sudo sed -i 's/ChallengeResponseAuthentication no/ChallengeResponseAuthentication yes/' /etc/ssh/sshd_config
- sudo service ssh restart
- sudo apt-get install sshpass
- sshpass -p $sshpassword ssh -R 9999:localhost:22 -o StrictHostKeyChecking=no travis@$bouncehostip
Here, we're setting the password for the travis user, enabling SSH password authentication, and creating an SSH connection to the bounce host with remote port forwarding: port 9999 on the bounce host is tunneled to port 22 on localhost (i.e. the Travis build environment).
(We are showing some bad security habits here, like disabling host key checking and using sshpass. Our use case is a public repository for an open-source project -- at worst case, a man-in-the-middle attacker could access our bounce host or corrupt our Travis CI builds.)
Commit and Push
Commit and push your project to GitHub, and watch the build output on Travis CI. If everything was set up correctly, it should make an SSH connection to your bounce host.
Get the Shell
SSH into your bounce host if you haven't already. From here, ssh -p 9999 -o StrictHostKeyChecking=no travis@localhost should initiate an SSH connection back to the Travis build environment from the bounce host. Enter the password when prompted. (Yes, the same password that Travis CI uses to connect to the bounce host.)
cmart@vm142-70:/root/.ssh$ ssh -o StrictHostKeyChecking=no travis@localhost -p 9999
Password:
Last login: Fri Jun 23 18:31:18 2017 from localhost
travis@testing-gce-121e13a6-2777-4c68-9c5c-d51c12aced8a:~$ sudo su
root@testing-gce-121e13a6-2777-4c68-9c5c-d51c12aced8a:/home/travis# echo "got shell!" | wall
Broadcast Message from travis@testing-
(/dev/pts/9) at 18:32 ...
got shell!
You should also see the broadcast message in the Travis build output. Congratulations! Now you can troubleshoot interactively, run build steps manually, and see what's broken.
Clean Up
When you're done, simply cancel the build, or kill the SSH process on your bounce host.
root@vm142-70:/var/log# ps aux | grep ssh
root 21470 0.0 0.0 55516 2972 ? Ss Jun22 0:00 /usr/sbin/sshd -D
root 24333 0.0 0.1 163168 5756 ? Ss 07:48 0:00 sshd: root@pts/1
root 24918 0.0 0.1 158984 5584 ? Ss 08:04 0:00 sshd: travisci [priv]
travisci 24971 0.0 0.0 158984 2260 ? S 08:04 0:00 sshd: travisci@pts/2
root 24992 0.5 0.1 161952 5372 ? Ss 08:06 0:00 sshd: root [priv]
sshd 24993 0.0 0.0 64832 1544 ? S 08:06 0:00 sshd: root [net]
root 24995 0.0 0.0 11744 920 pts/4 S+ 08:06 0:00 grep ssh
root@vm142-70:/var/log# kill 24971
# Meanwhile in Travis CI land:
Done. Your build exited with 1.
Plug for My Employer
CyVerse builds free and open-source platforms that empower scientific research computation. We need help all the way from the back-end to the front: Linux, Ansible, OpenStack, Ceph, Django, React. If you like working with these technologies and a whole bunch of others, get in touch! cmart at cyverse dot org.
(Photo above via Josh Blair on Flickr.)
Thanks for another fantastic post. The place else may just anybody get that kind of info in such a perfect means of writing? I have a presentation subsequent week, and I'm at the look for such info.
Visit my web page - live chat totobet