Shell into Travis CI Build Environment

on Chris Martin's Blog

yellow hard hat as turtle shell

Your CI builds are failing and you're not sure why. Troubleshooting a broken Travis CI build has a fairly long tweak-test cycle time: you add some debugging statements, commit and push, wait for the build to start, maybe get a few more clues, then rinse and repeat until you've found the problem and fixed the build. If only you could get a shell in the build environment, and troubleshoot interactively!

This feature isn't built into Travis CI, but it can be obtained with the following recipe. Basically, we'll instruct the Travis build environment to create a reverse SSH tunnel to a host that we control, exposing its local port 22. From there, we can create an interactive SSH session back to the Travis build environment.

(For us, Travis CI had released a new Trusty 14.04 image with a broken APT repository for Docker, which prevented us from adding a few PPAs required by our application. This was made immediately clear after getting a shell and looking around in /etc/apt/sources.list.d.)

Prepare Bounce Host

You need a host separate from the Travis build environment which has SSH exposed to the world. A disposable cloud instance is perfect for this task. Let's call it our bounce host.

  • Create a user (adduser travis), set a password when prompted
  • Ensure you can SSH to your bounce host with travis and the specified password. (modify your sshd_config as needed and restart the ssh service if necessary)

Next, set a couple of encrypted environment variables for your Travis CI project: "sshpassword" (storing the password you just set for the travis user on your bounce host) and "bouncehostip" (naturally, to the public IP address of your bounce host).

Populate .travis.yml

Put these lines in the script: section of your .travis.yml, perhaps at the end, after the other build commands for your project. (trivial example repo)

    - echo travis:$sshpassword | sudo chpasswd
    - sudo sed -i 's/ChallengeResponseAuthentication no/ChallengeResponseAuthentication yes/' /etc/ssh/sshd_config
    - sudo service ssh restart
    - sudo apt-get install sshpass
    - sshpass -p $sshpassword ssh -R 9999:localhost:22 -o StrictHostKeyChecking=no travis@$bouncehostip

Here, we're setting the password for the travis user, enabling SSH password authentication, and creating an SSH connection to the bounce host with remote port forwarding: port 9999 on the bounce host is tunneled to port 22 on localhost (i.e. the Travis build environment).

(We are showing some bad security habits here, like disabling host key checking and using sshpass. Our use case is a public repository for an open-source project -- at worst case, a man-in-the-middle attacker could access our bounce host or corrupt our Travis CI builds.)

Commit and Push

Commit and push your project to GitHub, and watch the build output on Travis CI. If everything was set up correctly, it should make an SSH connection to your bounce host.

Get the Shell

SSH into your bounce host if you haven't already. From here, ssh -p 9999 -o StrictHostKeyChecking=no travis@localhost should initiate an SSH connection back to the Travis build environment from the bounce host. Enter the password when prompted. (Yes, the same password that Travis CI uses to connect to the bounce host.)

cmart@vm142-70:/root/.ssh$ ssh -o StrictHostKeyChecking=no travis@localhost -p 9999
Password:
Last login: Fri Jun 23 18:31:18 2017 from localhost
travis@testing-gce-121e13a6-2777-4c68-9c5c-d51c12aced8a:~$ sudo su
root@testing-gce-121e13a6-2777-4c68-9c5c-d51c12aced8a:/home/travis# echo "got shell!" | wall

Broadcast Message from travis@testing-                                         
        (/dev/pts/9) at 18:32 ...                                              

got shell!

You should also see the broadcast message in the Travis build output. Congratulations! Now you can troubleshoot interactively, run build steps manually, and see what's broken.

Clean Up

When you're done, simply cancel the build, or kill the SSH process on your bounce host.

root@vm142-70:/var/log# ps aux | grep ssh
root     21470  0.0  0.0  55516  2972 ?        Ss   Jun22   0:00 /usr/sbin/sshd -D
root     24333  0.0  0.1 163168  5756 ?        Ss   07:48   0:00 sshd: root@pts/1    
root     24918  0.0  0.1 158984  5584 ?        Ss   08:04   0:00 sshd: travisci [priv]
travisci 24971  0.0  0.0 158984  2260 ?        S    08:04   0:00 sshd: travisci@pts/2
root     24992  0.5  0.1 161952  5372 ?        Ss   08:06   0:00 sshd: root [priv]   
sshd     24993  0.0  0.0  64832  1544 ?        S    08:06   0:00 sshd: root [net]    
root     24995  0.0  0.0  11744   920 pts/4    S+   08:06   0:00 grep ssh
root@vm142-70:/var/log# kill 24971
# Meanwhile in Travis CI land:
Done. Your build exited with 1.

Plug for My Employer

CyVerse builds free and open-source platforms that empower scientific research computation. We need help all the way from the back-end to the front: Linux, Ansible, OpenStack, Ceph, Django, React. If you like working with these technologies and a whole bunch of others, get in touch! cmart at cyverse dot org.

(Photo above via Josh Blair on Flickr.)


Comments

No comments yet, maybe you should post one!