Square Pegs & Round Holes: Giving Up Containerized Java

 Cole Brumley Senior DevOps Engineer
Cole Brumley
Senior Software Engineer

I’ve been “the container guy” for almost three years. I jumped on the Docker bandwagon early, and as the ecosystem evolved I learned a ton about the nifty kernel features that make it possible. Heck, I ended up at Maryville primarily because of my container chops. I sang its praises, argued, bullied, and berated my way into entrenching it in nearly all of our CI/CD toolchains and was generally, although it pains me to admit it, that guy about it.

But as with any technology one uses in a professional capacity, the golden rule of IT still applies: if you don’t hate it you don’t know it well enough yet.

There are still many use cases for which containers make a lot of sense, but there is a particularly common one that it’s time to let go of: Java-based microservices, especially ones using Spring, do not belong in a container.

Look, I know better than most why things get crammed into containers that don’t really belong there. I get it. As a matter of fact, I’d point you to our former colleague’s excellent talk about why it’s okay to sometimes do things with Docker that feel icky in the name of greater overall cohesion and sanity.

It’s not that you can’t cram the JVM into a container, it’s just not bringing anything functional to the party. You end up overengineering things just to make your app play along with Docker’s modus operandi.

First and foremost, the JVM does not respect Linux control group limits. Let me repeat that: THE JVM DOES NOT RECOGNIZE OR RESPECT LINUX CONTROL GROUP LIMITS. Why is that important? Oh, it’s not that important, just that it precludes use of Docker’s runtime resource constraints without additional tinkering.

Instead of throwing a --memory=512m flag onto your docker run command and having the container use 512Mb of RAM like you asked, you are immediately greeted by our old friend Mr OOMKiller. That’s because the JVM doesn’t know to look in /sys/fs/cgroup/memory/memory.limit_in_bytes and honor that as the usable memory pool, so it’ll do what it does naturally and attempt to gobble up 1/4 of the system’s total RAM. Same deal with CPU cores and swap.

Granted, you’d nearly always be specifying memory flags on the command line in a production environment anyway (unless you’ve got more RAM than sense), but now you get to track it in multiple places and hard-code it into your builds. Hooray!

You’re also making logging configuration more difficult by introducing an extra layer of abstraction. You know, the old log4j-to-stdout-to-jsonfile-to-syslog approach. Works every time.

XKCD #1319 Still Applies


Over-engineering is worse than under-engineering. You can always add more to a system you understand after all. So what are we really gaining from containerizing the JVM?screen-shot-2016-10-25-at-10-37-04-am

I need some file in a set location that I don’t have to think about and can easily control.

So, you’re saying you need to add files to your jar? I believe /usr/bin/zip would like a word with you. Perhaps a template would make more sense. If you’re only worried about config files, why not just use Spring Configuration‘s ENV vars or KV store support?

Shared lib or binary dependencies are a bit of a red herring. There are so many good config management tools out there that installing specific packages and making sure they stay at a certain version hasn’t been a legitimate issue in years (I’m not even sure configuration drift is a thing). If you’re using Docker to get around shoddy config management then you’ve probably got bigger problems to deal with and I commend your effort.

I want increased security.

This topic warrants many dedicated blog posts in itself, but in short NO. Docker does many things, but securely isolating applications from one another is not one of them. While great strides have been made towards securing it, I wouldn’t trust the Docker daemon as far as I could throw it. It’s too new, too big, and too expansive to not be full of holes.

If you want to take advantage of the underlying cgroup controls in the kernel but not expose a big, insecure, unstable, root-level daemon on your box systemd gets you a good deal of the way there by itself. That’s before accounting for overlay storage and iptables chains you can manage pretty easily without Docker, although admittedly without the slick API.

There are true container alternatives too. LXC should be the default container choice on a purely technical basis. It’s much more mature than Docker, and much less flashy. Docker itself started as a daemon around liblxc before switching to libcontainer. There’s also OpenVZ, Warden/Garden, even ye olde Solaris Zones for those fond of antiques.

I want to easily manage my application’s networking.

OK, this one is legit. It is nice to have iptables commands run for you. Personally, I wish our buddies over at Weaveworks would make Weave (or a sister project) easier to use outside of a container. Networking is still one of those things that requires a specialist and/or a ton of googling to do right. There isn’t really a nice API-driven way to control it because most of the tools out there are shell scripts around the iptables command and bash isn’t known for doing APIs very well.

I want to improve my CI/CD pipelines and unify how my app is built and deployed.

Ah, the “it makes my devs’ lives easier” argument. While normally I’d bite on that in a heartbeat (and if this were a Go, NodeJS, PHP, or Ruby app I’d say sure, let’s containerize), in the JVM world devs just don’t need to do that. In fact it’s frequently a bigger pain to shoehorn containers into well defined and battle tested Java/JVM workflows. I’ve never seen a Java dev champing at the bit to containerize, but if I met one I suppose I wouldn’t stand in her way.

If your workloads are running in the cloud (if they’re not call us, like, right now) you’re in a VM to start, then you run Docker which is providing virtualized resources, then you run the Java Virtual Machine inside of that. Seems like a lot of overhead to me. While admittedly not all that bad from a performance perspective, that extra layer of virtualization still looks like cruft. I really don’t like cruft.

Nobody ever got fired for choosing Java

I will admit that I’m no fan of the JVM. It’s a dated design that should be treated like the programming equivalent of the printer from Office Space. You shouldn’t need a list of flags this long to operate your code, and not every thing needs to be a AbstractSingletonProxyFactoryBean (yeah, I know it’s an extremely dated example).

Now that I’ve gotten that out of my system: Java ain’t going anywhere anytime soon, and there are people making new and cool things with it that have perfectly valid reasons for choosing it. Since we’ve still got to operate it, the least we can do is stop un-fixing problems that have been solved for years.

Be the first to comment “Square Pegs & Round Holes: Giving Up Containerized Java”