The system-wide SSH configuration files reside in /etc/ssh whereas the user ones are in their respective $HOME/.ssh. The configuration files are:
-
/etc/ssh/sshd_config → SSH daemon configuration file
/etc/sysconfig/sshd → SSH daemon configuration file with more options
/etc/ssh/ssh_config → systemwide client configuration file
/etc/ssh/ssh_host_ecdsa_key → ECDSA private key file
/etc/ssh/ssh_host_ecdsa_key.pub → ECDSA public key file
/etc/ssh/ssh_host_rsa_key → RSA private key file
/etc/ssh/ssh_host_rsa_key.pub → RSA public key file
/etc/pam.d/sshd → PAM configuration file for sshd
~/.ssh/config → client configuration file that overrides the one above
~/.ssh/authorized_keys → list of public keys authorized to connect to this host
~/.ssh/id_ecdsa → ECDSA private key for the user
~/.ssh/id_rsa → RSA private key for the user
~/.ssh/id_ecdsa.pub → ECDSA public key for the user
~/.ssh/id_rsa.pub → RSA public key for the user
~/.ssh/known_hosts → list of pubkeys of known SSH servers
The following files belong to SSH version 1 and their existence points to the use of version 1.
-
/etc/ssh/ssh_host_key
/etc/ssh/ssh_host_key.pub
~/.ssh/identity
~/.ssh/identity.pub
That is a security threat and should be dealt with by stating SSHv2 as the only protocol made available by the sshd daemon (/etc/ssh/sshd_config). We shall see how to do that a bit further down.
To ensure the tightest security it is best to create public/private keys, distribute them and disable password authentication. We do that with the ssh-keygen command:
# ssh-keygen -t ecdsa -b 521
Generating public/private ecdsa key pair.
Enter file in which to save the key (/home/marc/.ssh/id_ecdsa):
/home/marc/.ssh/id_ecdsa already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/marc/.ssh/id_ecdsa.
Your public key has been saved in /home/marc/.ssh/id_ecdsa.pub.
The key fingerprint is:
6a:8d:fe:80:3a:76:3a:e7:3b:fb:33:c4:36:91:c5:74 marc@envy17.bogomips.net
The key’s randomart image is:
+–[ECDSA 521]–+
| o. E |
| o. |
| o |
| o |
| . .S |
| .=+ |
| .o=.. |
| +.+oo. |
| .oB+=o+. |
+—————-+
In the example above we created an ECDSA key pair with 521 bits in length (ECDSA keys can only be 256, 384 or 521 bits). We could have created an RSA key pair (1024, 2048 or 4096 bits) or DSA (only 1024 bits) with the “-t rsa” or “-t dsa” flags. And we could have created all key-pairs in one go with…
# ssh-keygen -A
Once we have the keys, we have to send them to all the remote hosts we want to logon to:
# ssh-copy-id marc@rhel7
# ssh-copy-id -i ~/.ssh/id_ecdsa.pub marc@rhel7
The first ssh-copy-id sends to the rhel7 remote host the latest generated public key, whereas the second one sends a specific public key (~/.ssh/id_ecdsa.pub).
With the keys already distributed to the required hosts, now we should (ideally) disable password prompting and force the use of those keys. We can do that by editing /etc/ssh/sshd_config, setting the 3 parameters below…
PasswordAuthentication no
ChallengeResponseAuthentication no
PubkeyAuthentication yes
and restarting the sshd daemon…
# systemctl restart sshd
The first time we connect to a given remote host, that host’s public key is stored in ~/.ssh/known_hosts if we accept it (we should either take it for good or double-check that the fingerprints match). From then on, we will get a warning if at connection time the keys fingerprints do not match as it might be a sign of a man-in-the-middle attack. If the keys were regenerated, we can remove the outdated entry from known_hosts either manually (i.e. with vi) or with the command:
# ssh-keygen -R server.domain.com
The command above might not be necessary to remove outdated entries from an unhashed known_hosts file such as the one below…
# cat ~/.ssh/known_hosts
192.168.0.111 sshrsa AAAAB3NzaC1yc2EAAAABIwAAAIEApZ51Bvqcco/Fwzuv […]
s10nfs sshrsa AAAAB3NzaC1yc2EAAAABIwAAAIEApZ51Bvqcco/FwzuvflPldMU […]
vsphere,192.168.0.200 sshrsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC83lW1 […]
sl7ora121,192.168.0.130 ecdsasha2nistp256 AAAAE2VjZHNhLXNoYTItb […]
192.168.122.194 ecdsasha2nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNT […]
sl7ora112 ecdsasha2nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAI […]
But if, for the sake of tighter security, the file was hashed with the “-H” flag…
# ssh-keygen -H
Original contents retained as /home/marc/.ssh/known_hosts.old
WARNING: /home/marc/.ssh/known_hosts.old contains unhashed entries
Delete this file to ensure privacy of hostnames
.
# cat known_hosts
|1|W0N1XLThj+gyPRgfvGvt2VJrW+A=|5Zhx+5r2iEPe54NQPFuZsStiDH0= sshrsa AAAAB3NzaC1yc2EAAAABIwAAAI
|1|xOyCU12QxL30k9ARbBXy3miTbZM=|/wor9UW3ku0YNyRfmp/k8cxnsg4= sshrsa AAAAB3NzaC1yc2EAAAABIwAAAI
|1|Ac1kuB1rOB58q63i+SLhukfparQ=|jt3arOEz5pKEqxFETDO9DYY0dRw= sshrsa AAAAB3NzaC1yc2EAAAADAQABAA
|1|a9n81EVo4AHpl04JEY7Tie8PHeI=|Tu8xZm1E4s8muki5kkiFlmPWeCI= sshrsa AAAAB3NzaC1yc2EAAAADAQABAA
|1|67HPy4uV9HYyS/+g7NSvQupxkfY=|O1p/UtNNicXZakX9hSrenCUugzc= ecdsasha2nistp256 AAAAE2VjZHNhLX
|1|7VtCYkl2EkuEQIPof/finniP/q4=|YpEucouL1lS7F1ZvotAl1LqSJ4s= ecdsasha2nistp256 AAAAE2VjZHNhLX
|1|COI/rW2zZE1aRZikMiR2ZLF4gm0=|QdwW8+7o2MluAstlYkwjg0vAegw= ecdsasha2nistp256 AAAAE2VjZHNhLX
|1|2sSVAxxLPz9lK5e/SqzhlS3x+fg=|r3oMCGUbdMr94vZ/P2KTuQ90xjk= ecdsasha2nistp256 AAAAE2VjZHNhLX
… then knowing what key we need to remove becomes a bit harder and using “ssh-keygen -R” is almost a must. I say almost because the “-F” shows us the entry for a particular host in a hashed file…
# ssh-keygen -F 192.168.0.111
# Host 192.168.0.111 found: line 1 type RSA
|1|W0N1XLThj+gyPRgfvGvt2VJrW+A=|5Zhx+5r2iEPe54NQPFuZsStiDH0= sshrsa
AAAAB3NzaC1yc2EAAAABIwAAAIEApZ51Bvqcco/FwzuvflPldMUefD+vBtuzSeIL0Fwq1Pv1l/IsVpOJlWqokvXcBLJRS16
QBZ+BHFUJp7P8yfqTAHSvYqgh9VnqcQ2b5pS6bm50+vUJR4gOQx/TutC2h6j57V6BBvMfOezk4O0CTo3ydySlUFVI3LF9yJ
5COkxQuI0=
… so we could find the host with the re-generated keys with “-F” and then remove the key with vi. But why bother when we can do the same in one go with the “-R” option?
Once public key password-less connection can be established, we can easily run remote commands or copy files/directories with scp. Let’s look at some examples to get familiar with the options we can use in the CLI of ssh & scp:
/* run the commands on the remote host and show output locally */
# ssh marc@rhel7 “ls -la $HOME/.ssh ; hostname”
.
/* same as above but binding the source address to a specific local IP */
# ssh -b 192.168.0.15 -l marc rhel7 “ls -la $HOME/.ssh ; hostname”
.
/* connect to rhel7 using one of the options available in ssh_config file (see further down) */
# ssh -o ConnectionAttempts 3 marc@rhel7
.
/* copy from remote to local */
# scp marc@rhel7:/tmp/test.log /tmp
.
/* copy from local to remote */
# scp /tmp/test.log marc@rhce7:/tmp
.
/* copy from remote to remote directly */
# scp marc@rhel7:/tmp/test.log marc@rhce7:/tmp
.
/* copy from remote to remote hopping through local (firewall restrictions?) */
# scp -3 marc@rhel7:/tmp/test.log marc@rhce7:/tmp
.
/* copy to remote host forcing SSHv2, IPv4, compression and port 2022 */
# scp -24C -p 2022 /tmp/test.log marc@rhce7:/tmp
.
/* copy to remote host forcing SSHv1 & IPv6 and placing session in background (same as “&”) */
# scp -16f /tmp/test.log marc@rhce7:/tmp
.
/* same as above but forcing batch mode: if public key authentication fails it quits without prompting for a password */
# scp -16fB /tmp/test.log marc@rhce7:/tmp
.
/* same but limiting bandwidth to 1000 kbit/sec */
# scp -16fB -l 1000 /tmp/test.log marc@rhce7:/tmp
.
/* copy recursively john’s HOME directory and everything underneath to remote host */
# scp -r /home/john marc@rhce7:/home/john
.
/* same but preserves mtime, atime and permissions */
# scp -rp /home/john marc@rhce7:/home/john
The examples above have shown us most of the options and flags we can use with both ssh & scp. But there are further options that we can use (with the “-o” flag) in the command line and in the SSH configuration files.
There are many parameters in /etc/ssh/sshd_config that are worth studying (man sshd_config) to customise settings to our needs. Some of them are used in very specific conditions (thus rarely changed) and with some others their name states clearly their intended use. So let’s review the rest:
• AcceptEnv is set to no by default as no environment variables are sent to the remote server for security reasons but it can be changed by setting it explicitly to yes.
• AddressFamily can be set to any (default), inet or inet6 and is used to force a certain IP version.
• Port by default is set to 22. We can change that if we need to but we should restart the sshd daemon and in environments where SELinux is used we should inform it of the change with the command:
# semanage port -a -t ssh_port_t -p tcp <new port number>
• ListenAddress by default is set to “0.0.0.0” and “::” meaning all IPv4 & IPv6 addresses. We can
change that to only listen to one or a few IPs to restrict SSH connectivity to certain networks.
• Protocol should always be set to 2 as allowing protocol 1 is a big no-no from the security POV.
• AllowAgentForwarding is set to yes by default. If we need to set it to no we should also deny shell
access to users as they can always install their own forwarders. Using ssh-agent can save us time by
remembering the passwords we typed and not prompting for them again.
• AllowUsers & AllowGroups can be used to restrict SSH use to certain users or groups. The groups can be primary or secondary (it makes no difference) and the directives DenyUsers & DenyGroups can also be used together with the ones above. If that’s the case though, the order of processing is DenyUsers, AllowUsers, DenyGroups & AllowGroups. We can allow or deny access to SSH with the pattern user@host for maximum flexibility.
• AllowTcpForwarding can be set to yes (unrestricted), no (totally banned), local (listening socket on local host) or remote (listening socket on remote host). If we need to set it to no we should also deny shell access to users as they can always install their own forwarders.
• AuthorizedKeysCommand specifies the script to be used to retrieve the user’s public keys in case
they are not available in the file system (i.e. when using LDAP). By default this parameter is left
blank and not used.
• AuthorizedKeysCommandUser specifies the user that would run the command above.
• Banner specifies the name of the file whose contents will be shown to any user before
authentication starts. By default it is set to none.
• PasswordAuthentication & ChallengeResponseAuthentication are set to yes by default to enable password prompted logins. If we want to ban those and force public key authentication only, then we have to set both to no.
• ClientAliveInterval sets a timeout interval in seconds after which if no data has been received from
the client, sshd will send a message through the encrypted channel to request a response from the
client. The default is 0, indicating that these messages will not be sent to the client. This option
applies to protocol version 2 only.
• ForceCommand forces the execution of the specified command ignoring whatever command the
user intended to run. That command is executed using the user’s login with the -c option. This option
is always used inside a Match block (explained later) and is extremely useful when we need a
remote user restricted to executing just one command. In the following example we limit the
“backup” user so that using SSH to connect to our server it can only execute the given script and
nothing else.
Match User backup
. PasswordAuthentication no
. PubkeyAuthentication yes
. AllowTCPForwarding no
. X11Forwarding no
. ForceCommand /home/backup/dailybackup.sh
This option is useful if we just need one command executed. However, there is an alternative method to achieve the same goal (limit a user to executing something pre-ordained) without that one command limitation. We can leave the configuration files alone and directly edit the authorized_keys file in the remote server as in the example below:
# cat ~/.ssh/authorized_keys ← before edit
sshrsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCt6p9 john@server1
.
# cat ~/.ssh/authorized_keys ← after edit
command=”date; hostname”,nopty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCt6p9 john@server1
• GatewayPorts port forwards by default are only available to the localhost as they are bound to the
loopback address (127.0.0.1). If we need remote client connectivity to the forwarded ports (i.e. 3-way
connections) we must set this parameter explicitly to yes as that would bind the forwarded ports to
the wildcard “*” address meaning all IPs in use.
• LoginGraceTime is set by default to 2 minutes but it can be changed to any value we deem best.
• LogLevel specifies the verbosity of logging and can be set to: QUIET, ERROR, FATAL, INFO,
VERBOSE, DEBUG1, DEBUG2, DEBUG3. By default it is set to INFO.
• MaxAuthTries specifies the maximum number of authentication attempts permitted by connection.
When the number of failures reaches half this number, subsequent attempts are logged.
• PermitOpen is by default set to any and it allows TCP port forwarding from any local IP & port. We
can restrict that by specifying the list of ports available for forwarding as follows:
PermitOpen 192.168.122.15:10050 192.168.15.15:10050
• PermitRootLogin is set to yes by default but it can be changed to no and 2 other values:
– without-password forces the use of public keys and disables password authentication but it won’t work unless PasswordAuthentication is set to no for all users or for just root (with the Match
option below):
Match User root
. PasswordAuthentication no
– force-commands-only does exactly as its name implies: restricts the SSH connection to running
one or a few preset commands. Before we set PermitRootLogin to force-commands-only we have to
specify what those commands are. To do that, we should prefix those commands at the beginning of
the public key in the authorized_keys file.
Before: ssh-rsa AAAAB3NzaC1yc2EAAAADA […] root@server100
After: command=”/bin/date” ssh-rsa AAAAB3NzaC1yc2EAAAADA […] root@server100
• PermitTunnel is by default set to no. If we want to enable SSH tunneling we can set it to yes,
point-to-point or ethernet.
• Match if we use this option it should always be found at the end of the sshd_config file. It allows us
to change some of the previous parameters for specific users or groups as shown above with
PermitRootLogin example before. Every match keyword is followed by one or more users/groups
and with the options that apply to that/those users. We can have multiple match statements in the
sshd_config file as shown below:
Match User root
. PasswordAuthentication no
. AllowTcpForwarding yes
. GatewayPorts yes
. ForwardX11 yes
.
Match Group wheel,dba
. PasswordAuthentication no
. AllowTcpForwarding yes
. ConnectionAttempts 3
. ConnectTimeout 10
. ForwardX11 yes
.
Match Address 192.168.100.99
. LogLevel debug1
.
Match LocalUser backup
. Compression yes
. ConnectionAttempts 10
. ConnectTimeout 15
The sshd_config file explained above applies to the SSH server only (the sshd daemon). For clients, we have the files /etc/ssh/ssh_config & ~/.ssh/config. The options available for both files are exactly the same but the former applies system-wide (sets the defaults) whereas the latter only applies to a specific user.
Some of the options in ssh_config are exactly the same as those in sshd_config so we won’t mention them again. Some of the other interesting options are as follows:
• Batchmode when this option is set to yes (default is no), an ssh/scp connection to a remote host will fail immediately unless public key password-less login is enabled. This is to avoid hangouts in
ssh/scp connections launched from non-interactive scripts.
• BindAddress specifies the IP source address for the SSH connection. It might be necessary to deal
with firewalls for instance.
• Compression by default compression is not used in SSH connections. If we want to change it (i.e.
because of limited bandwidth or large file transfers) we can set it to yes.
• ConnectionAttempts is set to 1 by default but it can be changed to a larger integer if SSH
connections fail often (i.e. network blackouts, excessive server workload, etc).
• ConnectTimeout this parameter is not used by default as the timeout is the default system TCP
timeout. If we need to specify a much shorter timeout in seconds, then we can use this option for
that.
• ForwardX11 if set to yes (default is no) it redirects X11 connections over SSH channel to
$DISPLAY.
• LocalForward specifes the local port, remote host & remote port for a forward. See further down
for an example.
• RemoteForward this is similar to LocalForward but the connection is the other way around: from
the remote host to the local one. We shall see an example a bit further down.
In the ssh_config file we find first the “Host *” pattern that matches all hosts and sets the default options. After that we can set specific directives for specific hosts. Let’s see some examples below:
Host oradb1
. User oracle
. HostName oradb1.bogomips.net
. Port 22
. Protocol 2
. IdentityFile ~/.ssh/id_ecdsa
The record above would allow us to type “ssh oradba” instead of the equivalent:
ssh -l oracle -p 22 -2 -i ~/.ssh/id_ecdsa oradb1.bogomips.net
Let’s look at something a bit more complex:
Host http-webserver5
. User http
. HostName webserver5.bogomips.net
. LocalForward 8080 localhost:80
The entry above would be the equivalent of the command:
ssh -L 8080:localhost:80 http@webserver5.bogomips.net
What we are doing here is create a port forward from our localhost on port 8080 to the webserver5 on port 80. So everytime we connect to http://localhost:8080 we will be connecting to the webserver5 on port 80 but using an encrypted channel through port 22 of that webserver5.
Many MySQL databases accept only connections from localhost for security reasons. That can be overcome by setting up a local port forward as in the example above: connections to our localhost on port 9306 will be forwarded to the server mysql9.bogomips.net through port 22, bound to its loopback interface and then sent to port 3306. Thus, the connections to the MySQL database will appear to come from the localhost instead of the network.
Host sql-mysql9
. User mysql
. HostName mysql9.bogomips.net
. LocalForward 9306 localhost:3306
The entry above would be the equivalent to:
ssh -L 9306:localhost:3306 mysql@mysql9.bogomips.net
Let’s look now at a slight variation on the example above:
Host sql-mysql6
. User mysql
. HostName mysql6.bogomips.net
. LocalForward 9306 192.168.118.105:3306
… or its equivalent…
ssh -L 9306:192.168.118.105:3306 mysql@mysql6.bogomips.net
The server mysql6 has a few NICs connected to different networks but the MySQL database is listening only on 192.168.118.105 (let’s assume 127.0.0.1 can’t be used). What do we do? As long as we have connectivity to mysql6 on port 22, we can set up a local port forward as shown above and bind it to that IP rather than localhost/127.0.0.1. We have to bear in mind though that to remotely bind a socket to a non-loopback address in mysql6 will require the GatewayPorts parameter in that server (mysql6) to be set to either “yes” (bound to all available addresses) or “clientspecified”.
What happens in we have multiple IP addresses in our client but only one can go through the firewall onto mysql4:22 ? Then we bind the outgoing connection to the local IP we want as shown below…
Host sql-mysql4
. User mysql
. HostName mysql4.bogomips.net
. BindAddress 192.168.15.16
. LocalForward 9306 192.168.119.105:3306
… equivalent to…
ssh -b 192.168.15.16 -L 9306:192.169.118.105:3306 mysql@mysql6.bogomips.net
To summarise, local port forwards always open a listening socket on the local machine and forward the connections to a remote IP and port.
Remote port forwards work in reverse: a listening socket is open in a remote IP:port and connections
established there are forwarded to a local IP:port. Let’s look at an example…
We have 3 servers: externalserv, middleserv & internalserv. There is no direct communication possible between externalserv and internalserv because of the firewall. But both servers can access port 22 in middleserv. We need externalserv to be able to connect to port 80 on internalserv on an interim basis and remote port forward is the easiest way to achieve that.
First we must configure middleserv so that internalserv can open a remote listening socket there in a non-loopback address. We do that by editing /etc/ssh/sshd_config in middleserv and setting:
AllowTcpForwarding yes
GatewayPorts yes
PermitOpen 192.168.100.15:7890
AllowTcpForwarding does what its name implies.
GatewayPorts allows remote clients to connect to forwarded ports. By default all forwarded ports bind to localhost/127.0.0.1 and that prevents remote clients from connecting to them (security first!). By setting “GatewayPorts yes” we allow a forwarded port to bind to a non-loopback address and that opens it up to a connection from the network, which is exactly what we want.
The PermitOpen parameter is optional and it restricts the IPs and ports we can use for remote forwards (i.e. so that the firewall rules can be worked around this scenario).
Once middleserv is ready, we can establish the remote forward from internalserv in 2 ways:
ssh -R 7890:localhost:80 john@middleserv
# record in ssh_config (192.168.100.15 is middleserv)
Host ext-to-int
. User john
. HostName middleserv.bogomips.net
. RemoteForward 192.168.100.15:7890 localhost:80
Everything should be ready now! If we access http://middleserv:7890 from externalserv we should be
transparently redirected to http://internalserv:80.