Skip to content unscheduled downtime


One of the YubiCloud endpoints,, was down between 2017-03-30 22:48:00 and 2017-03-30 23:20:00 UTC.

The problem was caused by the physical host at Rackspace, where the virtual machine for this endpoints resides.

All other YubiCloud endpoints (i.e. api2, api3, api4 and api5) were operational during this time window. unscheduled downtime


The physical host where our VPS was hosted developed an issue, and as a result, Rackspace migrated us to a different physical host.

Service was disrupted between 2017-01-11 07:00:00 and 2017-01-11 09:00:00 UTC.

Issues with api last weekend


One of the YubiCloud endpoints,, experienced some unscheduled downtime due to issues with our hosting provider.

Our VPS had to be migrated to a different physical host, due to hardware issues.

The downtime started at 2016-12-03 00:47:00 and ended at 2016-12-03 01:17:00 UTC.

All other api machines (i.e. api2, api3, api4 and api5) were still available at all times. For properly configured clients, no noticeable downtime should have been experienced.

Issues with api2 last Saturday


Last Saturday, Nov 26th 2016, api2 experienced some unscheduled downtime due to issues with our hosting provider.

There were three instances of downtime as shown below (all times are UTC):

2016-11-26 17:50:00 down
2016-11-26 18:22:00 up

2016-11-26 18:36:00 down
2016-11-26 19:28:00 up

2016-11-26 19:33:00 down
2016-11-26 21:18:00 up

All other api machines (i.e. api, api3, api4 and api5) were still available at all times. For properly configured clients, no noticeable downtime should have been experienced.

Using Nagios exclusively to monitor the YubiCloud


Nagios is a very important part of our infrastructure. We rely on it to provide us with extensive and detailed checks on all our machines and services. We don’t just monitor if a service is up or down, but also a lot of other metrics which allow us to quickly understand the origin of a problem and hopefully fix it before the external service is effected.

Up until recently we also used Pingdom in conjunction with our Nagios infrastructure. Pingdom made monitoring an external service easy enough, while providing us with graphs we could display here on this blog.

Unfortunately, for various reasons, we have come to the decision to stop using Pingdom and rely solely on Nagios instead.

For the time being this means that we cannot, as transparently as before, display uptime statistics for our api* endpoints. We are investigating alternative ways in how we can render this data from the Nagios monitoring system.

Below are uptime statistics for each of our YubiCloud endpoint for the past year.

You may have noticed that api2 and api5 have considerably higher downtime than other endpoints. These two machines are currently hosted with Linode, which went through several DDOS attacks between December 25th 2015 and January 5th 2016. We are considering to move one of these machines to another hosting provider.

The rest of the outages were caused by restarts to the machines, due to updates and security patches. Please note that as a whole, the YubiCloud still has had 100% uptime since its inception.

YubiCloud no longer supports DH cipher suites


Following up on our previous post, the YubiCloud, as of today no longer supports DH cipher suites. The change was made at exactly 2016-09-01 08:00:00 UTC.

The TLS configuration across all five api machines now is:

ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers    "EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES";
ssl_prefer_server_ciphers  on;

api 1, 2, 3 & 5 have already been using such configuration for a couple of weeks. Today, the change was made to api 4, bringing all api machines to a standardized TLS configuration. results for api4 before and after the change was made are available. Note that the files are plain-text, but with a .doc extension to workaround a WordPress limitation.

[2016-09-01 08:00:00 UTC] YubiCloud cipher changes and statistics


In the past, the TLS configuration for YubiCloud has differed slightly between each api* machine, due to different web server, SSL/TLS libraries and operating system versions.

When deciding which TLS cipher suites to support, and their preferred order, we typically follow Mozilla TLS guidelines. However, these are only a rough guide, and sometimes we have to make slight changes to support different client configurations.

YubiCloud has a certain requirement where a wide array of clients has to be supported, some of which are using very old software.

A little bit over a year ago, while upgrading from Ubuntu 12.04 to newer LTS releases, we ran into a problem with some of our Java-based clients. The issue was that we had DH key-exchange enabled with a minimum prime bit length of 1024, but the newer Apache HTTP Server bundled with Ubuntu, only support >= 2048-bit primes.

These clients abort the TLS negotiation and hence fail to establish a connection with YubiCloud.

In the rush to solve this issue, we disabled DH ciphers on and, so that these clients could communicate with some of the YubiCloud machines using RSA+AES. This allowed such clients to stop trying to negotiate DH (where they would fail due to large primes) and fallback to the next preferred cipher suite of RSA+AES (which, unlike DH, does not support Perfect Forward Secrecy).

This Summer, we have been working on standardizing the TLS configuration across the YubiCloud. To start with, we looked at what cipher suites were actually being used by clients.

We realized that roughly 95% of our clients negotiate TLS sessions using ECDH. The rest use a mixture of DH and RSA. The ones using RSA for key exchange typically fall into using AES or 3DES as the encryption algorithm.

To have the same TLS cipher suite (and preference order) across all api* machines, we needed to make a decision on whether to support DH key-exchange with a minimum of 2048-bit primes, or disable it completely, hence leaving new clients to use ECDH exclusively and old clients to use RSA+AES.

Considering that less than 1% of our clients use DH, we decided to drop it. Such clients that want PFS should consider upgrading their TLS stack to support ECDH.

Up until now, DH has been disabled on all api* machines except for On September 1st, 2016, we will disable DH completely across YubiCloud. The TLS configuration will hence be as follows:

~$ openssl ciphers "EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES" | tr ':' '\n'

Clients using DH should automatically have their TLS negotiation to fallback to RSA+AES (one of the AES128* or AES256* suites). We have observed this behavior while switching off DH on api 1, 2, 3 and 5.

We highly recommend to upgrade your software to support ECDH.

Some statistics showing cipher suite usage across the YubiCloud are displayed below. This data represents OTP verification requests using TLS between 2016-08-15 and 2016-08-28 (inclusive).

Cipher suite usage for each YubiCloud api* machine:

API 1:

87.01% ECDHE-RSA-AES128-GCM-SHA256
 3.30% ECDHE-RSA-AES128-SHA256
 1.72% AES128-SHA
 0.47% DES-CBC3-SHA
 0.38% AES128-SHA256
 0.04% AES128-GCM-SHA256

API 2:

65.79% ECDHE-RSA-AES128-GCM-SHA256
 4.72% AES128-SHA
 1.98% DES-CBC3-SHA
 0.57% ECDHE-RSA-AES128-SHA256
 0.45% AES128-SHA256
 0.04% AES128-GCM-SHA256

API 3:

78.73% ECDHE-RSA-AES128-GCM-SHA256
 3.20% AES128-SHA
 1.18% DES-CBC3-SHA
 0.39% AES128-SHA256
 0.24% ECDHE-RSA-AES128-SHA256
 0.04% AES128-GCM-SHA256

API 4:

62.29% ECDHE-RSA-AES128-GCM-SHA256
 3.02% DHE-RSA-AES128-SHA
 2.06% DES-CBC3-SHA
 0.79% DHE-RSA-AES128-SHA256
 0.49% DHE-RSA-AES128-GCM-SHA256
 0.02% ECDHE-RSA-AES128-SHA256
 0.00% AES128-SHA

API 5:

63.00% ECDHE-RSA-AES128-GCM-SHA256
 5.94% AES128-SHA
 2.22% ECDHE-RSA-AES128-SHA256
 1.09% DES-CBC3-SHA
 0.68% AES128-SHA256
 0.04% AES128-GCM-SHA256

Global cipher suite usage across the YubiCloud:

78.35% ECDHE-RSA-AES128-GCM-SHA256
 2.64% AES128-SHA
 2.03% ECDHE-RSA-AES128-SHA256
 0.98% DES-CBC3-SHA
 0.39% AES128-SHA256
 0.26% DHE-RSA-AES128-SHA
 0.07% DHE-RSA-AES128-SHA256
 0.04% DHE-RSA-AES128-GCM-SHA256
 0.03% AES128-GCM-SHA256

Cipher suite usage grouped by key exchange algorithm (and encryption where applicable), for each YubiCloud api* machine:

API 1:

97.40% ECDH
 2.13% RSA-AES
 0.47% RSA-3DES

API 2:

92.79% ECDH
 5.22% RSA-AES
 1.98% RSA-3DES

API 3:

95.18% ECDH
 3.63% RSA-AES
 1.18% RSA-3DES

API 4:

93.64% ECDH
 4.30% DH
 0.00% RSA-AES
 2.06% RSA-3DES

API 5:

92.26% ECDH
 6.65% RSA-AES
 1.09% RSA-3DES

Cipher suite usage grouped by key exchange algorithm (and encryption where applicable), globally across the YubiCloud:

95.59% ECDH
 3.06% RSA-AES
 0.98% RSA-3DES
 0.37% DH

It is worth noting that some connections use 3DES. These are most likely Windows XP machines which did not have any Service Pack installed, otherwise AES support would have been added.

Interestingly enough, Windows XP machines are not something that you want anywhere near close to the Internet, considering that Microsoft is no longer providing security patches and it was originally released 15+ years ago.

For astute readers wondering if 3DES should be disabled, our server preference declares that 3DES cipher suites are the least preferred ones, meaning that newer clients will always use a more modern cipher suite.