SSH has to be one of the great unsung heros of the average sysadmins toolkit.
Much of this information was originally published at blog.ktz.me - SSH Tips and why ProxyJump is awesome.
What is SSH?¶
The SSH protocol (also referred to as Secure Shell) is a method for secure remote login from one computer to another. It provides several alternative options for strong authentication, and it protects the communications security and integrity with strong encryption. It is a secure alternative to the non-protected login protocols (such as telnet, rlogin) and insecure file transfer methods (such as FTP) 1.
Keys are an absolute fundamental for using SSH. They mean you don't need to use passwords to login to your remote systems and are much more secure. The gold standard for many years has been the rsa key type but more recently I've started using the ed25519 format instead. Here are two public keys:
1 2 3 4 5
ed25519 public key makes the
rsa key look a bit gross doesnt it? The caveat here is that older versions of SSH don't support ed25519 but you probably shouldn't be exposing those old versions to the internet anyways. There's a great stackoverflow thread2 discussing the various merits of key types, enjoy that rabbit hole.
Generating a key is simple enough:
Add a passphrase if you like. Tips on passphrases can be found here.
You can view the generate keys in
id_ed25519 for the private key and
id_ed25519.pub for the public key. Guard the private key as you would a password. Do not share it, do not copy it anywhere else as it is your key to the kingdom. The public key is safe to share however, if you're interested as to why then check this video out:
It's also worth understanding
AgentForwarding too. GitHub have some nice documentation on this topic. It allows you to use your local SSH keys instead of leaving keys (without passphrases!) sitting on your server. In their documentation pay attention to the ssh-agent section, it's important for all this to work.
SSH Key Management¶
Copying keys to the remote host can be done in a few ways.
The first requires that you already have SSH connectivity to the host (perhaps with temporary password authentication).
ssh-copy-id user@host:port will copy the keys over and subsequent SSH sessions will automatically try the key first. If you temporarily enabled password authentication for SSH, disable it now.
The second method requires console access to the host itself and hand editing the file
~/.ssh/authorized_keys. Simply copy the public key from your client to a new line in this file. Watch that you don't add extra spaces or clip characters. A successful file should look something like this (this example has two public keys in it of differening types):
1 2 3
The third way is to use
ssh-import-id gh:ironicbadger to import all SSH keys stored on GitHub under a specific users account. Unfortunately this is limited to Ubuntu / Deb systems only. It makes use of the SSH keys you store on Github.
Remember public keys are safe to expose.
Every user exposes their keys publicly at github.com/ironicbadger.keys.
Finally, you could use Ansible but again that requires SSH in the first place so it can be somewhat of a chicken and egg problem. This method is useful if you have rotated keys elsewhere in your infrastructure and your Ansible host remains able to access the remote host. Here's an example task:
1 2 3 4 5
The Bastion Host¶
Given that SSH is so powerful, many bad actors target port 22 with scripts and other nastiness. For this reason it is good practice to only expose as little to the internet as possible rather than forwarding dozens of ports in your firewall to dozens of individual hosts.
This is common in enterprise environments but makes sense in most small networks where you have multiple hosts. The bastion server is your single exposed entrypoint so focus on making it as secure as possible.
Common things to look for when securing SSH are running on a non-standard port (i.e. not 22), disallow Root login, disallow password-authentication and so on. I use the Geerlingguy security Ansible role in my ironicbadger/infra repo to change dozens of parameters.
Once on the bastion, you can probably reach most hosts in the rest of your LAN. This is sometimes known therefore as a jump host or island hopping. Whatever you call it, it means that now to get to a server that used to involve a single hop now requires two or more hops. We'll come onto
ProxyJump in a moment but first we need to understand the SSH config file.
We haven't talked much about the
~/.ssh/config file yet but it is extremely useful. Define multiple hosts by pet names, specify custom ports, hostnames, users, etc. Take the following command:
By using an SSH config file we can reduce this to:
Here's the SSH config snippet needed for that:
1 2 3 4 5 6
We can also glob hosts together so that rules and paramaters defined apply to multiple hosts at once:
1 2 3 4 5
And finally, it's time for the coolest thing here.
ProxyJump makes it super simple to jump from one host to another totally transparently. For example:
1 2 3 4 5 6 7 8 9 10 11 12 13 14
In the above example when we execute
ssh lanserver we first connect to
bastion before connecting to our final destination of
In the old days we had to use
ProxyCommand with a bunch of crap to make this happen but now it's very elegant using