Skip to content

Using Nagios exclusively to monitor the YubiCloud

2016-09-08

Nagios is a very important part of our infrastructure. We rely on it to provide us with extensive and detailed checks on all our machines and services. We don’t just monitor if a service is up or down, but also a lot of other metrics which allow us to quickly understand the origin of a problem and hopefully fix it before the external service is effected.

Up until recently we also used Pingdom in conjunction with our Nagios infrastructure. Pingdom made monitoring an external service easy enough, while providing us with graphs we could display here on this blog.

Unfortunately, for various reasons, we have come to the decision to stop using Pingdom and rely solely on Nagios instead.

For the time being this means that we cannot, as transparently as before, display uptime statistics for our api*.yubico.com endpoints. We are investigating alternative ways in how we can render this data from the Nagios monitoring system.

Below are uptime statistics for each of our YubiCloud endpoint for the past year.

api.yubico.com
api2.yubico.com
api3.yubico.com
api4.yubico.com
api5.yubico.com

You may have noticed that api2 and api5 have considerably higher downtime than other endpoints. These two machines are currently hosted with Linode, which went through several DDOS attacks between December 25th 2015 and January 5th 2016. We are considering to move one of these machines to another hosting provider.

The rest of the outages were caused by restarts to the machines, due to updates and security patches. Please note that as a whole, the YubiCloud still has had 100% uptime since its inception.

YubiCloud no longer supports DH cipher suites

2016-09-01

Following up on our previous post, the YubiCloud, as of today no longer supports DH cipher suites. The change was made at exactly 2016-09-01 08:00:00 UTC.

The TLS configuration across all five api machines now is:

ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers    "EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES";
ssl_prefer_server_ciphers  on;

api 1, 2, 3 & 5 have already been using such configuration for a couple of weeks. Today, the change was made to api 4, bringing all api machines to a standardized TLS configuration.

testssl.sh results for api4 before and after the change was made are available. Note that the files are plain-text, but with a .doc extension to workaround a WordPress limitation.

[2016-09-01 08:00:00 UTC] YubiCloud cipher changes and statistics

2016-08-30

In the past, the TLS configuration for YubiCloud has differed slightly between each api*.yubico.com machine, due to different web server, SSL/TLS libraries and operating system versions.

When deciding which TLS cipher suites to support, and their preferred order, we typically follow Mozilla TLS guidelines. However, these are only a rough guide, and sometimes we have to make slight changes to support different client configurations.

YubiCloud has a certain requirement where a wide array of clients has to be supported, some of which are using very old software.

A little bit over a year ago, while upgrading from Ubuntu 12.04 to newer LTS releases, we ran into a problem with some of our Java-based clients. The issue was that we had DH key-exchange enabled with a minimum prime bit length of 1024, but the newer Apache HTTP Server bundled with Ubuntu, only support >= 2048-bit primes.

These clients abort the TLS negotiation and hence fail to establish a connection with YubiCloud.

In the rush to solve this issue, we disabled DH ciphers on api.yubico.com and api3.yubico.com, so that these clients could communicate with some of the YubiCloud machines using RSA+AES. This allowed such clients to stop trying to negotiate DH (where they would fail due to large primes) and fallback to the next preferred cipher suite of RSA+AES (which, unlike DH, does not support Perfect Forward Secrecy).

This Summer, we have been working on standardizing the TLS configuration across the YubiCloud. To start with, we looked at what cipher suites were actually being used by clients.

We realized that roughly 95% of our clients negotiate TLS sessions using ECDH. The rest use a mixture of DH and RSA. The ones using RSA for key exchange typically fall into using AES or 3DES as the encryption algorithm.

To have the same TLS cipher suite (and preference order) across all api*.yubico.com machines, we needed to make a decision on whether to support DH key-exchange with a minimum of 2048-bit primes, or disable it completely, hence leaving new clients to use ECDH exclusively and old clients to use RSA+AES.

Considering that less than 1% of our clients use DH, we decided to drop it. Such clients that want PFS should consider upgrading their TLS stack to support ECDH.

Up until now, DH has been disabled on all api* machines except for api4.yubico.com. On September 1st, 2016, we will disable DH completely across YubiCloud. The TLS configuration will hence be as follows:

~$ openssl ciphers "EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES" | tr ':' '\n'
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-SHA256
ECDHE-ECDSA-AES128-SHA256
ECDHE-RSA-AES128-SHA
ECDHE-ECDSA-AES128-SHA
AES128-GCM-SHA256
AES128-SHA256
AES128-SHA
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-SHA384
ECDHE-ECDSA-AES256-SHA384
ECDHE-RSA-AES256-SHA
ECDHE-ECDSA-AES256-SHA
AES256-GCM-SHA384
AES256-SHA256
AES256-SHA
ECDHE-RSA-DES-CBC3-SHA
ECDHE-ECDSA-DES-CBC3-SHA
DES-CBC3-SHA

Clients using DH should automatically have their TLS negotiation to fallback to RSA+AES (one of the AES128* or AES256* suites). We have observed this behavior while switching off DH on api 1, 2, 3 and 5.

We highly recommend to upgrade your software to support ECDH.

Some statistics showing cipher suite usage across the YubiCloud are displayed below. This data represents OTP verification requests using TLS between 2016-08-15 and 2016-08-28 (inclusive).

Cipher suite usage for each YubiCloud api*.yubico.com machine:

API 1:

87.01% ECDHE-RSA-AES128-GCM-SHA256
 7.09% ECDHE-RSA-AES128-SHA
 3.30% ECDHE-RSA-AES128-SHA256
 1.72% AES128-SHA
 0.47% DES-CBC3-SHA
 0.38% AES128-SHA256
 0.04% AES128-GCM-SHA256
 0.00% ECDHE-RSA-AES256-SHA

API 2:

65.79% ECDHE-RSA-AES128-GCM-SHA256
26.44% ECDHE-RSA-AES128-SHA
 4.72% AES128-SHA
 1.98% DES-CBC3-SHA
 0.57% ECDHE-RSA-AES128-SHA256
 0.45% AES128-SHA256
 0.04% AES128-GCM-SHA256

API 3:

78.73% ECDHE-RSA-AES128-GCM-SHA256
16.22% ECDHE-RSA-AES128-SHA
 3.20% AES128-SHA
 1.18% DES-CBC3-SHA
 0.39% AES128-SHA256
 0.24% ECDHE-RSA-AES128-SHA256
 0.04% AES128-GCM-SHA256

API 4:

62.29% ECDHE-RSA-AES128-GCM-SHA256
31.33% ECDHE-RSA-AES128-SHA
 3.02% DHE-RSA-AES128-SHA
 2.06% DES-CBC3-SHA
 0.79% DHE-RSA-AES128-SHA256
 0.49% DHE-RSA-AES128-GCM-SHA256
 0.02% ECDHE-RSA-AES128-SHA256
 0.00% AES128-SHA

API 5:

63.00% ECDHE-RSA-AES128-GCM-SHA256
27.04% ECDHE-RSA-AES128-SHA
 5.94% AES128-SHA
 2.22% ECDHE-RSA-AES128-SHA256
 1.09% DES-CBC3-SHA
 0.68% AES128-SHA256
 0.04% AES128-GCM-SHA256

Global cipher suite usage across the YubiCloud:

78.35% ECDHE-RSA-AES128-GCM-SHA256
15.21% ECDHE-RSA-AES128-SHA
 2.64% AES128-SHA
 2.03% ECDHE-RSA-AES128-SHA256
 0.98% DES-CBC3-SHA
 0.39% AES128-SHA256
 0.26% DHE-RSA-AES128-SHA
 0.07% DHE-RSA-AES128-SHA256
 0.04% DHE-RSA-AES128-GCM-SHA256
 0.03% AES128-GCM-SHA256
 0.00% ECDHE-RSA-AES256-SHA

Cipher suite usage grouped by key exchange algorithm (and encryption where applicable), for each YubiCloud api*.yubico.com machine:

API 1:

97.40% ECDH
 2.13% RSA-AES
 0.47% RSA-3DES

API 2:

92.79% ECDH
 5.22% RSA-AES
 1.98% RSA-3DES

API 3:

95.18% ECDH
 3.63% RSA-AES
 1.18% RSA-3DES

API 4:

93.64% ECDH
 4.30% DH
 0.00% RSA-AES
 2.06% RSA-3DES

API 5:

92.26% ECDH
 6.65% RSA-AES
 1.09% RSA-3DES

Cipher suite usage grouped by key exchange algorithm (and encryption where applicable), globally across the YubiCloud:

95.59% ECDH
 3.06% RSA-AES
 0.98% RSA-3DES
 0.37% DH

It is worth noting that some connections use 3DES. These are most likely Windows XP machines which did not have any Service Pack installed, otherwise AES support would have been added.

Interestingly enough, Windows XP machines are not something that you want anywhere near close to the Internet, considering that Microsoft is no longer providing security patches and it was originally released 15+ years ago.

For astute readers wondering if 3DES should be disabled, our server preference declares that 3DES cipher suites are the least preferred ones, meaning that newer clients will always use a more modern cipher suite.

api3.yubico.com has been migrated

2016-07-19

Following up on our previous post, api3.yubico.com has been migrated to the new host. The DNS update took place at 08:00 UTC.

The new machine immediately started serving traffic and no interruptions should have been experienced.

The old IP addresses, 104.236.200.122 and 2604:a880:800:10::166:8001, are no longer accepting any connections.

[2016-07-19 08:00:00 UTC] Upcoming changes to api3.yubico.com

2016-07-14

On 19th July 2016 at around 08:00 (am) UTC the DNS record for api3.yubico.com will be updated to point to a new host.

The machine will be moved from Digital Ocean (New York City) to Vultr (New Jersey).

Software updates will also be applied, in particular, upgrading to latest stable releases in terms of both the application and the underlying operating system.

While you should not hardcode any of our IPs in your application or firewall, please do note the new addresses below.

Current IPv4: 104.236.200.122

Current IPv6: 2604:a880:800:10::166:8001

New IPv4: 45.63.8.184

New IPv6: 2001:19f0:5:107:5400:ff:fe2c:a50b

If you must firewall your outgoing connections, you might be able to use DNS hostnames in your ruleset.

iptables and pf support this, but you must make sure to periodically reload your ruleset, otherwise changes to the DNS zones are not picked up.

api4.yubico.com has been migrated

2016-06-09

Following up on our previous post, api4.yubico.com has been migrated to the new host. The DNS update took place at 07:05 UTC.

The new machine immediately started serving traffic and no interruptions should have been experienced.

The old IP addresses,178.62.237.116 and 2a03:b0c0:2:d0::3f:1001, are no longer accepting any connections.

[2016-06-09 07:00:00 UTC] Upcoming changes to api4.yubico.com

2016-06-07

On 9th June 2016 at around 07:00 (am) UTC the DNS record for api4.yubico.com will be updated to point to a new host.

The machine will be moved from Digital Ocean (Amsterdam) to Hetzner (Germany).

Software updates will also be applied, in particular, upgrading to latest stable releases in terms of both the application and the underlying operating system.

While you should not hardcode any of our IPs in your application or firewall, please do note the new addresses below.

Current IPv4: 178.62.237.116

Current IPv6: 2a03:b0c0:2:d0::3f:1001

New IPv4: 78.47.118.220

New IPv6: 2a01:4f8:c17:2bfe::2

If you must firewall your outgoing connections, you might be able to use DNS hostnames in your ruleset.

iptables and pf support this, but you must make sure to periodically reload your ruleset, otherwise changes to the DNS zones are not picked up.