Who Has a Copy of Your SSH Keys? The Unlocked Back Door of Encrypted Network Security

- Jason Thompson, global director of marketing for SSH Communications Security, says:

The secure shell (SSH) data-in-transit protocol was invented in 1995, but since then has rapidly become an indispensable tool in the arsenal of network administrators the world over.  SSH has been used within networks of all varieties and sizes to securely transfer data from device to device and allow administrators to access networks remotely. A version of SSH is accompanied with every type of Linux, Unix and Mac OS, and is becoming rapidly adopted in the Windows world as well. Virtually half of all of the websites in the world use some version of SSH. While reaching an exact estimation of the number of SSH implementations worldwide is impossible, the number is estimated in the millions, making SSH a trusted workhorse in the network security ecosystem.

Since its launch 20 nearly two decades ago, SSH has protected billions of corporate transactions without any major security breaches. Even though the protocol itself is highly secure, today’s quickly-changing threat environment means that companies must take a serious look at how they manage SSH key creation, rotation and removal.  

Making Copies of Keys

Ordinarily, SSH is used to transfer sensitive data from one point to another within the network, such as between a user’s computer and the server.  This data can include personal identity information, credit card numbers, classified intelligence and healthcare records. From the perspective of a malicious insider or a hacker, SSH protects a smorgasbord of vital organizational information.

Nonetheless, since SSH itself is iron-clad, how would a hacker obtain access to the sensitive data it protects?  That’s where SSH key mismanagement comes in.

When a user connect to the server via SSH, a trust relationship between the user’s computer and the server is created using a cryptographic key pair.  These trust relationships are generated and managed by the organization’s IT department, often via systems dating back decades.  None of these older systems have the ability to search for or find these SSH key trust relationships on the network.  Therefore, tracking trust relationships must be done manually.  When a network ecosystem potentially contacts hundreds of thousands of keys, these trust relationships are unavoidably lost or misplaced.  An attacker with access to one of these trust relationships can imitate an authorized user with freedom to access valuable company information.

Therefore, sloppy management of SSH keys presents an avenue to exploitation by attackers searching for access to sensitive data. After conducting a study on the administrative operations of some major worldwide organizations, an alarming trend emerged: 

• Nearly 10 percent of all SSH user keys deliver root access, a major security and compliance violation
• Companies often assign the same SSH host key to thousands of devices, leaving the network defenseless to man-in-the-middle attacks
• Businesses rarely understand what each key is used for, displaying not only a security risk, but also a business stability risk
• Many SSH keys that allow access to critical servers are abandoned and no longer used
• A number of organizations authorize administrators to create or delete SSH user keys at will – without approvals or control – essentially permitting unfettered access to systems and people
• Very few organizations ever remove SSH user keys, or even rotate them when a user departs or an application is withdrawn
• Key-based access grants are fundamentally permanent, in direct offence of SOX, PCI and FISMA requirements for improperly terminating access, leaving the network vulnerable to treats

With increasingly sophisticated threats becoming more frequent, organizations without suitable SSH key management protocols in place are taking on water. The further an organization strays from a best practices approach to SSH key management, the greater the danger becomes.

In addition to the security implications of SSH key mismanagement, organizations need to be conscious of what federal standards – such as PCI, SOX, NIST and HIPAA – demand from them as well.  Specifically, these federal regulations require that organizations retain a high degree of control over access to sensitive network data, or face expensive fines. These factors even leave out the economic argument, which is compelling in and of itself. Major organizations today often have over 20,000 servers. The cost of manual SSH key management for a server environment of this size is projected to be close to $40 million over the next decade.  Add in the excruciatingly visible reputation damage caused by a security breach that plays out in public view, and organizations are soberly considering a host of motivations to fix their SSH key management practices.

Key Management Practices Need to Change

Luckily, security concerns in the secure shell environment are not due to any flaws or vulnerabilities in the SSH protocol itself. Rather, the security and compliance risks discussed above are caused by:

1. Lack of clear guidelines or procedures for years relating to SSH key management
2. Lack of understanding of the extent and consequences of the issues
3. An inadequate amount of time and resources needed to gain understanding and develop solutions
4. A lack of worthy tools and procedures early on for resolving key management issues
5. A hesitation on the part of auditors to identify issues for which they do not have efficient solutions
6. The attention of the access management field on shared users without addressing computerized access

It’s understandable to wonder why this issue has stayed hidden for so long, given the consequences of exploitation. Unfortunately, the simple answer is that because SSH key management is so deeply technical, it has stayed hidden and obscured within the field of system administrators. Each system manager usually only sees a small portion of the IT environment, and does not have the full picture. Administrators today are extremely busy – particularly with staff cutbacks in the recent years – and they might not acknowledge that there is an issue. Since management is several steps removed from the problem – and its potentially destructive consequences – too often, nothing is done about the issue.

But the risk remains.

SSH Key Management Prevention – Best Practices

Because exposure is commonly found in all Unix/Linux servers and many Windows servers, the requirements needed to fix the problem typically involves several teams within IT operations.  The potential liability and compliance issues require the understanding and buy-in from executive management as well.

Best practices to prevent these problems include:

Learning about all existing users, shared and private keys, and recording trust between servers and users

Observing the environment to establish which keys are actually used, and eliminating keys no longer in use

Enforcing appropriate approvals for all key systems

Automating key formats and key removals; reducing manual work and human faults. This step reduces the number of administrators necessary for key setups from potentially several hundred to only a couple extremely reliable administrators

Alternating keys frequently, so that copied keys stop working and proper termination of access can be guaranteed

Restricting where each key has access and what instructions can be implemented using the key

To reduce risk further, SSH key management should involve the establishment of internal limitations within the organization.  An organization should rigorously control what key-based trust relationships can be accepted by which boundaries, while implementing an agreed IP address and “forced command” restrictions for all authorized keys concerning trust relationships crossing such boundaries.

Although SSH is generally recognized as the standard for data-in-transit security, the existing threat landscape demands that organizations reconsider how they manage encrypted networks access. Using these best security practices will help position a company to better prepare for security breaches and new compliance authorizations before they occur. 

About the Author:

Jason Thompson is director of global marketing for SSH Communications Security. Mr. Thompson brings more than 12 years of experience launching new, innovative solutions across a number of industry verticals. Prior to joining SSH, Mr. Thompson worked at Q1 Labs where he helped build awareness around security intelligence and holistic approaches dealing with advanced threat vectors. Mr. Thompson holds a BA from Colorado State University and an MA for the University of North Carolina at Wilmington.