How We Ended OOM Crashes on Self-Hosted Servers Forever

Linux is a cold-blooded killer.

Specifically, the kernel Out-Of-Memory (OOM) killer. It does not ask questions. It does not gracefully drain connections. When physical memory is exhausted, it executes your most critical process with a swift SIGKILL. If you run self-hosted applications, databases, or high-throughput servers on virtual machines, you are constantly playing Russian roulette with your memory budget.

Traditionally, sysadmins solved this by configuring swap space. However, traditional swap is broken. Modern cloud guides tell you to disable swap entirely to prevent performance degradation, while others tell you to provision a massive, static swapfile that wastes precious NVMe SSD capacity.

If a sudden traffic surge or memory leak hits your server, a static swapfile either fills up instantly or degrades your application to a snail's pace as the system thrashes. There is no middle ground.

We built the ODAC host platform to provide a zero-config, enterprise-grade cloud experience on your own hardware. To solve this fundamental self-hosting hazard, we engineered an autonomous, elastic host-swap manager directly into the ODAC v2 control plane.

Here is the story of how we eliminated OOM crashes forever, without wasting your disk space or destroying your SSD write endurance.

The Design: One Step Ahead of the Fill Frontier

Most dynamic swap scripts are reactive. They wait until your system is actively choking, then desperately try to run dd and swapon while the kernel is already freezing. This is too little, too late.

The ODAC swap manager is designed around a proactive, predictive philosophy: always keep one empty swap increment ahead of the fill frontier.

Instead of waiting for a memory crisis, the manager ensures that a pre-allocated landing pad is always online before your footprint reaches it.

Here is how the dynamic pool balances itself:

The system starts with a baseline floor of at least one swap increment. The moment your non-reclaimable memory (RAM or active swap) crosses the 80% utilization threshold (growHotPct), the engine calculates the size of the next increment. It immediately provisions a fresh, empty swapfile on disk and attaches it to the kernel.

By the time your application spikes and needs that memory, the swap space is already active. No slow allocation pauses, no blocking disk writes, and absolutely no OOM-killer intervention.

Under the Hood: Pure, Non-Blocking State Metrics

A dynamic system manager is only as good as its sensors. If it reads raw memory usage incorrectly, it will trigger false alarms or fail to act in time.

The ODAC swap manager evaluates host state with extreme care, utilizing a platform-independent pure decision engine written in Go. Every 30 seconds, an internal tick throttles the manager (checkGate) to prevent CPU thrashing and avoid constant reading of the filesystem.

During this tick, the manager reads two vital Linux kernel interfaces:

1. /proc/meminfo: To calculate genuine, non-reclaimable RAM. Traditional scripts look at "free" memory, which is a major mistake because it counts reclaimable page cache as occupied. ODAC correctly parses MemAvailable to understand the true memory headroom.
2. /proc/pressure/memory: To extract Memory Pressure Stall Information (PSI). We track the some avg10 metric, which indicates the percentage of time threads were stalled waiting for memory over the last 10 seconds.

If memory pressure spikes (PSI some avg10 crosses growPSIThreshold of 20.0), the engine flags the host RAM as "hot" regardless of literal byte utilization. This allows ODAC to preemptively open a swap increment during intense scheduling congestion, long before physical RAM is fully saturated.

Because this logic is completely decoupled from the OS actuator, we can easily test and simulate complex memory profiles. Our test suite runs unit-level simulations of memory spikes, ensuring the decision engine behaves predictably under any load.

Hysteresis: Reclaiming Space Without Disk Flapping

While allocating swap must be fast and predictive, reclaiming swap must be slow, cautious, and deliberate.

If you shrink swap too quickly, you risk entering a "flapping" loop: allocating a file, deleting it a minute later, and then immediately allocating it again as memory fluctuates. This wastes CPU cycles and kills your SSD's lifespan with constant write cycles.

To prevent this, the ODAC swap manager implements a strict cooldown state machine using two techniques:

1. The Hysteresis Gap: While growth is triggered at 80% utilization, shrinking only begins when all active swap tiers fall below 40% utilization (coolPct) and the kernel PSI average drops below 1.0.
2. The Confirmation Streak: A surplus swap increment must remain completely idle and cool for a sustained cooldown period of 10 consecutive ticks (approximately 5 minutes, tracked via shrinkStreakNeeded) before it is safely reclaimed.

Furthermore, before discarding an idle swapfile, the manager runs a critical safety check: the pages currently held in that swapfile must be small enough to comfortably fit back into the host's available physical RAM. If they fit, the manager calls swapoff to migrate those pages back to RAM, and then safely unlinks the file.

Hardening the Host: Caps and Reboot Reconciliation

Running elastic storage on a host requires strict boundaries. An out-of-control manager could easily consume your entire system disk during a catastrophic memory leak.

ODAC protects your disk health through two configurable limits defined in the swap block of config/system.json:

• maxDiskPct: Limits total managed swap space to a maximum percentage of your disk (defaulting to 25%).
• maxIncrements: Sets a hard limit on the total number of swap increments allowed (defaulting to 8).

Each new increment size is calculated dynamically as half of the system's total physical RAM, clamped between a floor of 1 GiB and a ceiling of 16 GiB. This ensures that a 4 GiB VPS gets agile 2 GiB increments, while a massive 64 GiB bare-metal server gets highly efficient 16 GiB steps.

Here is what the configuration block looks like inside your config/system.json file:

{
  "swap": {
    "autoManage": true,
    "persist": true,
    "maxDiskPct": 25,
    "maxIncrements": 8,
    "allowShrink": true
  }
}

Self-Healing Across Reboots

What happens if your physical server undergoes a hard reset or sudden power loss?

When the ODAC server orchestrator boots up, it runs a self-healing reconciliation routine. It scans the swap directory, identifies orphaned files from previous sessions, and matches them against the kernel's active swap table.

If any valid swap files exist on disk but are not active, the manager automatically reattaches them or safely purges them if they exceed the configured increment limit. This ensures the host instantly recovers its safety buffer without waiting for the 30 second tick confirm sequence to complete.

Zero Friction, Zero Panic

If you manage your servers using the ODAC Cloud dashboard at app.odac.run, you do not have to write a single line of shell script or configure systemd services. The elastic swap manager operates silently in the background, keeping your databases running and your APIs responding even during extreme traffic spikes.

By turning system swap into an intelligent, elastic, and self-cleaning resource pool, ODAC transforms self-hosted infrastructure from a fragile guessing game into a resilient, enterprise-grade application platform.