Detecting SSH tunnels

SSH is an incredibly powerful protocol whose footprint needs to be monitored closely in enterprises. The most common use of SSH is for totally legitimate purposes like terminal (ssh) or for file transfer (scp,sftp). Many users also use the less well known port forwarding feature of SSH to create ‘tunnels’. SSH tunnels bore through firewalls, NATs, and are almost totally opaque to Network Security Monitoring tools like Trisul, Bro, Suricata, Snort, and others. SSHv2 even has SOCKS5 support – this allows anyone to setup a full SOCKS5 proxy outside your network and hide all HTTP activity from the prying eyes of NSM tools. With HTTPS/SSL, security tools can get atleast a look at the unencrypted certificates and perform checks, with SSH everything goes dark right after the initial capabilities exchange.

There are two types of SSH tunnels. The forward tunnel allows an insider to get on the outside bypassing the NSM and Firewall/NAT sentries. The reverse tunnel allows an outsider to get on the inside. The reverse tunnel is also called an autossh tunnel after the popular tool used to setup and maintain this connection.

Here is how a Forward SSH tunnel looks like

and a Reverse SSH , or autossh Tunnel

3 ways in which SSH tunnels create blind spots for monitoring

Here are the top three ways in which SSH tunnels create blind spots for NSM tools

1. single flow : when you are monitoring on the perimeter SSH tunnels show up as a single flow while they multiplex several SSH or SFTP channels. So you miss that flow level insight.
2. dynamic forwarding used as proxy using dynamic port forwarding users can proxy web traffic through the encrypted tunnel. This pretty much defeats every policy control you have at the boundary.
3. 24×7 access into your network : reverse ssh tunnels can give outsiders persistent presence into deep parts of your inside network. Autossh is a very advanced tool that maintains the tunnel and keeps it from timing out.

It is quite obvious that any monitoring platform needs to help organizations stay right on top of SSH. Lets see what can be done.

Tracking and detection of SSH tunnels

In any large organization SSH is going to be pervasive. Therefore any naive SSH flow monitoring is going to be dominated by legitimate business applications. Here are some techniques we use in Trisul, you should be able to adapt these to your own toolchain.

We create a SSH Flow Tracker to first get baseline visibility and then to incrementally refine the tracking until we get a workable list of suspect flows. Flow Tracking is a streaming algorithm which plugs into the flows stream and continously snapshots Top-K every minute. There are several built-in Top-K algorithms (volume, upload, download, duration, based on ports, IPs, subnets, netflow). You can even write a fully customized tracker using the flowtracker LUA script ).

When you first enter a unmapped network, you need to build a baseline statistical and flow profile first. Statistical visibility includes actual bandwidth of SSH traffic, servers, clients, gateways, number of SSH flows, number of concurrent SSH flows , scanning activity, etc. Then you need to step down one level into flow visibility. Identify the normal SSH flows, the elephant flows, the long running but low traffic flows, flows that involved scanners, malicious hosts,etc.

Here we explain how we can use a ‘flow tracker’ to build a baseline model and then refine it in three steps and then end with a discussion of detection of SSH tunnels.

• Visibility stage 1 : Use a “SSH Flow Tracker” – track Top-K SSH flows by volume – this gives you a baseline idea of very large elephant SSH flows. Most of them will be normal business activity that is okay. If you dont even have this insight, you are basically running blind.
• Visibility stage 2 : Use a “SSH Duration Tracker” – track Top-K SSH flows by duration – this gives you a really good idea of very long lived streams. Once again many of them will be legitimate business activity. That is okay – we will refine them and create a whitelist. Malicious Reverse SSH tunnels will typically not transmit much data as they “lie in wait” for the attacker to log in. They will be kept alive only because of ‘autossh’ keepalives. So right off the bat if you see long lived flows with very low traffic you can mark them for investigation. You can also create trackers for SSH flows by upload if you want to dig deeper.

Once you have the baseline normal visibility we need to feedback knowledge gained from (1) and (2) into the realtime part of Trisul. To do this

• Visibility stage 3 : Use “SSH Trackers but use a whitelist” – in addition to the two trackers above you create yet another trackers using the flowtracker LUA API that would track “All SSH flows by duration that arent whitelisted”. It is not that difficult because the whitelists can be automatically generated from (1) and (2). For example you can identify host pairs or subnet pairs that constitute normal traffic.
• Detection 1 : “Add detection” – In addition to the deep visibility above, if you can automatically detect SSH tunnels that would give you immediate rewards.

In the next section, we show how you can plug in the detection part using a simple example.

Detecting SSH tunnels using traffic analysis

Recently I came across a great presentation by John B Althouse III [ link ] where he used Bro (bro.org) to detect SSH tunnels carrying TTY (terminal) using traffic analysis.

The key insight found by the author is : if you observe the packet lengths of keystrokes attached to SSH you notice the following.

1. when SSH directly transports TTY keystrokes,
   1. packet length = SSH header + 1 byte char code + padding + HMAC. This could be 36,40,48 bytes or so
2. when SSH tunnels another SSH channel transporting TTY traffic
   1. packet length = SSH header + [previous SSH pkt] + HMAC. This could be 76, 84, 98 bytes and so on
   2. the exact lengths depend on the encryption block size and the HMAC algorithm + implementation of clients and servers.
   3. each keystroke is echoed back, so we can use that fact to tighten it up a bit

If you can rig a NSM tool to detect consecutive tunnel packet sizes of Server:76, Client:76,S:76,C:76,S:76,C:76 this is almost certainly a person typing in an interactive terminal on an SSH tunnel and getting echoed back. There you have it.

This is a traffic analysis attack essentially. OpenSSH does not implement the “random padding” feature of RFC4253 The secure shell protocol which says ” the insertion of variable amounts of ‘random padding’ may help thwart traffic analysis”. Until they fix that we can use this technique to detect tunnels.

Lets see how you can implement this analysis using the Trisul LUA API.

rev-ssh.lua

You can think of Trisul roughly as Bro but with LUA and an emphasis on traffic metering and streaming analytics. The LUA API lets you hook into TCP reassembly , HTTP file extraction, and a number of other points.

For more details about the script please visit Githiub rev-ssh.lua

4. Alerting and tagging

We dont want a flow to keep firing alerts when the user continues to type. So we use a dampening interval of 5 minutes.

-- tag flow 
    engine:tag_flow(flowkey:id(),"REVSSH");

    -- alert 
    engine:add_alert("{B5F1DECB-51D5-4395-B71B-6FA730B772D9}", flowkey:id(),"REVSSH",1,"rev ssh detected by keypress detect method");

    sshF.lastalertts=timestamp
  end

Flow tagging allows you to generate a ‘tag stream’ that are merged with the main “flows stream” and persisted at the end of a time window. Flow tags are nothing but text labels that are attached to flows. You can then search for flow by the tag value.

Conclusion

We think traffic and flow visibility is a very important part of Network Security Monitoring. As encryption spreads throughout the network we are going to need more and more of statistical and flow based approaches to gain insights into the network. You also need a real time API so you can plug in knowledge learnt from earlier “hunting expeditions”.

Getting started

If you want to get started on the Trisul platform and play with the scripting API, all you have to do is

1. install Trisul using apt-get or yum (no signups or emails asked just apt-get/ yum)
2. it runs free for the most recent 3 days window
3. the LUA API is fully enabled
4. install the LUA script into the directory mentioned in Installing LUA scripts
5. now you can capture live traffic or process PCAP dumps

Free Download Trisul 6.0 ! Ready to go packages for Ubuntu and CentOS.