Close
Glad You're Ready. Let's Get Started!

Let us know how we can contact you.

Thank you!

We'll respond shortly.

  • Blog Navigation
Challenges With Randomness In Multi-tenant Linux Container Platforms

featured-dice-containersUsing Linux Containers for tenant isolation rather than OS isolation from VMs or non-virtualized hosts exposes unique challenges. The Cloud Foundry team has worked on problems like that this quite a few times over the last few years. Recently we have been working on what we believe to be an unsolved isolation issue in Linux Containers for the /dev/random device. This new issue came to light after investigating reports of poor user experience on the mailing lists and attendees of Pivotal’s Cloud Platform road show. Sharing /dev/random is a challenge for any Linux container technology including Cloud Foundry Garden, Docker, LXC, etc. We are hopeful that we can work with the Linux community to help us weigh the various trade-offs and improve the experience and isolation for running multi-tenant Linux Containers.

Finding The Root Cause of Poor User Experience

During the public road shows, attendees learn about Cloud Foundry and they walk through an exercise to deploy a demo Java application running Apache Tomcat. Typically the application starts up in several seconds, but we started getting reports that users would consistently not be able to start the application within the 60 second default timeout and sometimes not even after multiple attempts. Why would a simple application that normally starts in several seconds suddenly start taking over 60 seconds?

A standard java debugging technique is to take a JVM thread dump, which will show you what the JVM threads are working on at any given point in time. This technique is very useful in identifying where something may be stuck, especially when comparing several thread dumps taken sequentially over a short amount of time. Once we looked at thread dumps from one of the instances that was slow to start up, we realized what was happening. When Tomcat initializes, it starts several security sub-systems, such as one responsible for generating session identifiers. These Tomcat security sub-systems use an API for random numbers from the standard Java libraries which uses the JVM configuration to choose an implementation for random numbers. On Linux, the JVM is set by default to read from the /dev/random device.

The /dev/random device is a blocking cryptographically secure pseudo random number generator (CSPRNG) that relies on the Linux kernel to gather measurements from device drivers like mice, keyboards and other sources into an entropy pool. The entropy pool is used to generate random numbers. As the /dev/random device is read, the entropy pool is drained. The /dev/random device will block if the entropy pool is empty until the entropy pool gathers enough noise to fulfill the amount of data requesting to be read. Operating systems running on virtualized servers that typically do not have keyboards, mice and other similar sources of unpredictable physical events attached therefore have lower rates of acquiring entropy.

The low amount of entropy on virtualized servers problem is exacerbated because many attendees of the road show attempt to start the application at roughly the same time. Therefore, Linux Containers running on the same host compete for a limited supply of entropy. This type of problem is sometimes referred to as a stampeding herd. The /dev/random device is a scarce shared system resource that Linux Container tenants likely have not realised they are sharing. When they all try to use it at the same time they are effectively causing a denial of service on each other.

image00

Is a stampeding herd coming for your limited supply of entropy? (photo credit via photopin cc)

Similar problems have come up before, such as Bitcoin miners trying to maximize the amount of shared CPU to generate virtual currency. To account for Bitcoin miners and other CPU heavy workloads, we use Linux cgroup CPU shares. Linux cgroups ensure a fair-share allocation was granted to the scarce resource, in this case CPU, thus preventing tenants from using more than their fair share. (when you run a public service with a free tier or trial, you will undoubtedly need to contend with Bitcoin miners at some point.)

Ok, Entropy is Low on Virtualized Cloud Servers, Now What?

There are several approaches the Java community particular have used to mitigate these types of issues. Some people advocate using /dev/urandom, which is a non-blocking and therefore much more performant implementation of a PRNG, and it is easy enough to change the JVM settings to change SecureRandom to use /dev/urandom. This approach would require changes to the default configuration of software running inside of the containers. We have found people are generally very apprehensive about changing default JVM and Tomcat approaches without strong assurances that the configuration is as secure as the default.

Another approach is to use a dameon like haveged, timer_entropyd, and randomsound to increase the rate that entropy is generated for /dev/random. This approach would be transparent to software running in Linux Containers. Our research on this approach found some concerns that haveged and timer_entropyd rely on a precision timer CPU instruction that may be virtualized, which exposes the reliance of any virtualized OS on the hypervisor capabilities and configuration. Other researchers have raised concerns about the PRNG generation techniques in /dev/random as being “not robust”.

So for the time-being we are not making changes to the Linux Containers or the hosts they are running on, we are documenting this behavior as a known issue and sharing the work-arounds to either increase the application startup timeout value or use /dev/urandom.

The Road Ahead—Improving Linux Container Isolation for /dev/random

Even if we change Java and Tomcat behavior and improve user experience for this particular scenario, there still remains the underlying problem that Linux Containers share a random device with scarcity without controlling access. For example, non-Java workloads are also able to utilize Linux’s /dev/random device. Linux cgroups allocate CPU cycles with shares, which address both noisy neighbor and denial-of-service concerns. Perhaps a resource sharing wrapper around /dev/random whereby tenants are limited to use only their fair share of allocation of data when there is resource contention is something we could consider implementing.

There may be other ideas to consider as well, such as using an entropy generator using services commonly available like NTP. We want to collaborate with others on this problem, so please reach out to us on our mailing list if you would like to work with us on this and other problems in the Linux Container space.

Challenge: How to solve multi-tenant Linux Containers having contention over /dev/random on low-entropy environments

Approach Pro Con
Adjust container software configuration to use /dev/urandom Each user is able to adjust the software configuration in the container that makes sense for them. Commonly recommended for JVMs that are not generating long-term cryptographic keys. Not dependent on generating additional entropy. Non-default configuration for most software and requires user awareness of the issue and user intervention.
Provide supplemental entropy to the Linux host /dev/random Each Linux Container transparently gets additional entropy without adjusting software configurations inside the container. Concerns about whether the mechanisms this approach depends on are as secure as users expect. For example, entropy daemons may use virtualized CPU instructions that the virtualization provider may not implement securely.
Wrap /dev/random in Linux Containers with a fair share control or similar Provides containers protection from noisy neighbors and DOS attacks. Does not address scarcity of entropy on virtualized cloud servers.
Provide alternative /dev/random implementation in Linux Containers that use non-host /dev/random source of entropy Isolates the host /dev/random from implementation in the Linux Containers. Non-trivial burden of finding a secure source of entropy and dependencies on hardware or external services.
Do Nothing. Document this as a known issue and stick with standard defaults and status quo behavior Users get the defaults they expect and are able to discover documentation with potential work-arounds. Default user experience is sometimes poor and difficult for new users to troubleshoot and identify the root cause.

Pivotal R&D Collaborators To This Issue

Mark Thomas, Ben Hale, Dieu Cao, Zach Robinson, Ryan Morgan, Matthew Kocher, Dmitriy Kalinin, Vik Rana, Andrew Shafer, Cornelia Davis, Glyn Normington, Steve Powell, James Bayer and Alex Jackson.

Thank you to VMware’s Glen McCready for providing feedback on this post.

Related Reading:

http://resources.infosecinstitute.com/random-number-generation-java/

http://www.2uo.de/myths-about-urandom/

https://www.digitalocean.com/community/tutorials/how-to-setup-additional-entropy-for-cloud-servers-using-haveged

http://en.wikipedia.org/?title=/dev/random

http://security.stackexchange.com/questions/34523/is-it-appropriate-to-use-haveged-as-a-source-of-entropy-on-virtual-machines

https://www.schneier.com/blog/archives/2013/10/insecurities_in.html

  • Eric Windisch

    I’m with Docker Security.

    I love the article. One of the great things about containers is that they spawn with high-entropy versus VMs, but it also means potentially depleting the pool faster, especially if the pool isn’t very large to begin with as may be on VMs. Personally, I have for some time recommended tools such as HAVEGED on VMs and that doesn’t change with the addition of Docker.

    Primarily, I’d like to share some hints for Docker users to force their containers to use a different random source. Since 1.2, we have supported a ‘–device’ flag. Example to follow:


    # Map host's /dev/urandom to container's /dev/random
    $ docker run --device /dev/urandom:/dev/random tomcat

    # Map a hwrng to the container's /dev/random
    $ docker run --device /dev/hwrng:/dev/random tomcat

    • Thanks for the comments and docker tips Eric! I like how this approach puts the control into the user’s hands easily with a CLI option without having to modify how the software in their containers is configured. As Stu Charlton points on in his comments, the Java configurations have been confusing and tedious for quite some time.

  • Stu Charlton

    I’ve had to deal with this in one way or another over years and almost always have the aghast “How do we even computer?!” reaction when I explain the story of /dev/urandom and the need for /dev/./urandom in the JVM.

    To me it’s ultimately the DoS problem that leads me to believe that non-blocking /dev/urandom should be the default. It’s the default with OpenSSL. The dot-syntax workaround is ludicrous. The default and internal behaviour is simply wrong in Java 7 and lower. Oracle has finally fixed SecureRandom to use /dev/urandom by default, without the dot-syntax, in Java 8: http://docs.oracle.com/javase/8/docs/technotes/guides/security/enhancements-8.html

    But as with all things TLS, people need to be educated. In this case they probably want to terminate their TLS sessions with a server that has some kind of enhanced entropy, either a TLS appliance or some kind of enhanced entropy generating daemon. We’ve (as an industry) tried to do a focused job about explaining the importance of TLS hostname verification, and certificate revocation lists, and why it’s generally a bad idea to turn them off, we likely also need to add entropy to that list of “stern warnings in shiny neon lights that most people will ignore because crypto is hard”.

  • Stu Charlton

    One correction to the above, as James pointed out on Twitter: Oracle has actually *changed* the default to securerandom.source=//dev/random in Java 8.

    tl;dr, Java 8 hasn’t fixed any of the challenges in this post… but might make it easier to convince people to switch to /dev/urandom.

    Details: Java 8 has fixed the selection of PRNG algorithm to be more reasonable – random seeds and bytes are generated natively out of /dev/urandom if you specify it in the securerandom.source property. In Java 7 and below, it would always grab blocking seeds from /dev/random even if you selected /dev/urandom, and would confusingly use the SHA1PRNG rather than the native PRNG if you selected /dev/random. If you specified anything but /dev/urandom it would use SHA1PRNG but grab seeds from the device you specified (hence the /dev/./urandom workaround).

    Many unreadable strace tests here: https://gist.github.com/svrc/5a8accc57219b9548fe1

    • Thanks for the comments Stu. The gist is very informative with the actual results of trying the various java.security configurations. Hopefully this information allows people make more informed choices. In my experience, people get very reluctant to adjust this stuff because they know they just understand it that well.

  • very interesting post, thanks! This leads me to new performance analysis of my services

  • Killian Murphy

    Another interesting source of entropy is a web service providing true random numbers such as random.org (which is run by a friend of mine). You could collect true random numbers from the service as needed or you could build up a pool of them in advance if you need them faster. The numbers are collected from physical sources of true randomness, and are tested.

    It doesn’t solve the disconnected operation use case (which would prevent access to an internet service), but might help with the more typical connected use cases.

Share This