Photo by Bernd ๐ท Dittrich on Unsplash
So I was deploying a Node.js app at like 11pm on a Friday — classic — and the container kept dying every 20 minutes. No useful logs, no warnings, just... gone. Restarted itself, ran fine for a bit, then died again. After way too long staring at dashboards, I finally ran docker inspect on the stopped container and saw it: exit code 137. That's when I knew exactly what was going on.
If you're dealing with Docker container memory limits and your containers keep getting killed, this one's for you.
What Exit Code 137 Actually Looks Like
When the Linux kernel's OOM (Out of Memory) killer steps in and terminates your container, you'll usually see something like this when you run docker ps -a:
CONTAINER ID IMAGE COMMAND STATUS
a3f92c1b8e12 myapp:latest "node server.js" Exited (137) 3 minutes ago
And if you dig into the container details:
docker inspect a3f92c1b8e12 --format='{{.State.OOMKilled}}'
true
There it is. OOMKilled: true. The container wasn't crashing because of a bug in your code — it was getting murdered by the kernel because it crossed its memory boundary. Honestly, once you know what to look for, it's one of the easier Docker problems to diagnose. Getting there is the frustrating part.
Why This Happens
Here's the thing — Docker lets you set memory limits on containers, which is great for multi-container environments where you don't want one greedy service eating all the host's RAM. But if your app legitimately needs more memory than you've allocated, or if there's a memory leak slowly building up over time, the kernel will step in and kill the process. No warning. No graceful shutdown. Just exit 137.
The default behavior, by default, is also a little sneaky. If you haven't set any memory limits explicitly, Docker doesn't apply any hard limit — but if you're running on a host with other containers or services, the system-wide OOM killer can still kill your container when the whole machine runs low. So even "unlimited" isn't really unlimited.
I've seen this happen most often with Java apps (JVM loves RAM), data processing containers that load large files into memory, and anything running a Chromium instance headlessly. If any of those sound familiar, yeah, this is probably your problem.
Fix 1: Increase the Memory Limit
The most straightforward fix — if your app actually needs more memory — is to bump the limit up. You can do it with the --memory flag when running a container:
docker run --memory="512m" --memory-swap="1g" myapp:latest
The --memory flag sets the hard RAM limit, and --memory-swap sets the combined RAM + swap limit. Setting swap to double the memory limit is a common pattern, though honestly, relying heavily on swap in production is its own kind of pain.
If you're using Docker Compose, it looks like this in your docker-compose.yml:
services:
myapp:
image: myapp:latest
deploy:
resources:
limits:
memory: 512M
reservations:
memory: 256M
Note that the deploy.resources key only applies in Swarm mode by default. For regular Compose usage, you'd use the older-style mem_limit under the service directly if you're on an older version, or use the --compatibility flag.
Fix 2: Find and Fix the Memory Leak
Now here's where it gets tricky. Sometimes increasing the limit is just kicking the can down the road. If your container is consistently growing in memory usage until it gets killed, you've got a leak somewhere. You can watch real-time container memory stats with:
docker stats myapp_container
This gives you a live view of CPU and memory usage. If memory climbs steadily without coming back down, that's your leak. From there, it's language-specific — Node.js has tools like --inspect and clinic.js, Java has heap dumps, Python has tracemalloc. Diagnosing the actual leak is a whole separate rabbit hole, but at least now you know you're looking in the right place.
For JVM-based apps specifically, a super common culprit is not setting the -Xmx flag. Without it, the JVM will try to grab a percentage of the total host RAM — completely ignoring the Docker memory limit — and then promptly get killed for it. Always set explicit heap sizes for Java containers:
docker run --memory="1g" myapp-java:latest java -Xmx768m -Xms256m -jar app.jar
Fix 3: Set a Restart Policy (Damage Control)
Alright so sometimes you can't fix the root cause immediately — you're in production, the team's asleep, or it's Friday night and you just need the app to stay up until morning. Setting a restart policy at least keeps the container coming back automatically after the OOM kill:
docker run --restart=on-failure:5 --memory="512m" myapp:latest
The on-failure:5 means Docker will try to restart it up to 5 times if it exits with a non-zero code (which 137 is). It's not a real fix — it's more like a bandaid — but it buys you time. Just don't use --restart=always here without a limit, or you'll end up with a crash loop hammering your host. I've seen that take down an entire server before. Not fun.
Fallback Tip: Use cgroups to Monitor Memory Pressure
If you want to get ahead of OOM kills before they happen, you can watch cgroup memory stats directly on the host. Docker uses cgroups under the hood to enforce limits, and you can see the raw data here:
cat /sys/fs/cgroup/memory/docker/<container-id>/memory.usage_in_bytes
cat /sys/fs/cgroup/memory/docker/<container-id>/memory.limit_in_bytes
Divide the first by the second and you've got your usage percentage. You can script this into a simple alerting check that pings you before the container hits 90% usage. Crude, but effective. Pair it with Prometheus and cAdvisor if you want something more robust long-term.
Setting proper Docker container memory limits is one of those things that feels annoying to deal with the first time but becomes second nature once you've been OOM-killed a few times. Hope this saves you the 2am panic I went through.
Related: Docker Error 137: Why Your Container Keeps Getting Killed (And How to Fix It)
๋๊ธ
๋๊ธ ์ฐ๊ธฐ