Memory leak during Socket.IO reconnection failures leads to OOM crash

[Lies188]

Summary

HAPI experiences a memory leak when Socket.IO connections fail and enter a reconnection loop. After prolonged reconnection failures, the system runs out of memory and crashes.

Environment

• HAPI Version: 0.9.2
• Installation: bun install -g @twsxtd/hapi-linux-x64
• OS: Ubuntu 24.04.3 LTS (Noble Numbat)
• Kernel: 6.8.0-90-generic x86_64
• CPU: AMD EPYC 7K62 48-Core Processor (4 vCPUs)
• Memory: 7.5 GB RAM + 1.9 GB Swap
• Node.js: v24.13.0
• Bun: 1.3.6

Running Services

The following HAPI processes were running:

• `hapi server`
• `hapi claude --yolo`
• `hapi daemon start-sync`
• `hapi codex` (spawned by daemon for remote sessions)

Timeline of Events

1. Socket Disconnection (00:36:59)

The initial disconnection was caused by a ping timeout:

```
[00:36:59.175] [API] Socket disconnected: ping timeout
```

2. Continuous Reconnection Failures (00:37 - 01:11)

Socket.IO entered a reconnection loop with 35+ consecutive failures over ~35 minutes:

```
[00:37:50.616] [API] Socket connection error: {}
[00:39:20.261] [API] Socket connection error: {}
[00:40:28.289] [API] Socket connection error: {}
... (35+ more errors)
[01:11:25.322] [API] Socket connection error: {}
```

3. Memory Pressure Begins (01:08:44)

System started experiencing memory pressure during the reconnection loop:

```
Jan 19 01:08:44 systemd-journald[74650]: Under memory pressure, flushing caches.
Jan 19 01:08:46 systemd-journald[74650]: Under memory pressure, flushing caches.
... (continuous until crash)
Jan 19 01:14:49 systemd-journald[74650]: Under memory pressure, flushing caches.
```

4. System Crash (01:17:26)

The system became unresponsive and automatically rebooted at 01:17:26.

Code Analysis

After reviewing the source code, I identified potential issues in the Socket.IO client configuration:

In `cli/src/api/apiSession.ts` (lines 74-87):

```typescript
this.socket = io(`${configuration.serverUrl}/cli`, {
auth: { ... },
path: '/socket.io/',
reconnection: true,
reconnectionAttempts: Infinity, // ⚠️ No limit on reconnection attempts
reconnectionDelay: 1000,
reconnectionDelayMax: 5000, // ⚠️ Max delay only 5 seconds
transports: ['websocket'],
autoConnect: false
})
```

In `cli/src/api/apiMachine.ts` (lines 217-229):

```typescript
this.socket = io(`${configuration.serverUrl}/cli`, {
transports: ['websocket'],
auth: { ... },
path: '/socket.io/',
reconnection: true,
reconnectionDelay: 1000,
reconnectionDelayMax: 5000 // ⚠️ Same issue
})
```

Potential Issues

1. Unlimited reconnection attempts (`reconnectionAttempts: Infinity`): The client will attempt to reconnect forever, potentially accumulating resources with each failed attempt.

2. Short max delay (`reconnectionDelayMax: 5000`): With only 5 seconds max delay, during prolonged outages the client makes frequent reconnection attempts (every 5 seconds after reaching max delay).

3. No cleanup between attempts: Looking at the disconnect handler, there's no explicit cleanup of internal buffers or event listeners that might accumulate.

4. Concurrent resource usage: The Codex session was processing a large code review task with 4,409,050 input tokens, adding to memory pressure.

Memory Usage Before Crash

Based on current measurements after restart, HAPI processes typically consume:

Process	Memory
hapi claude (2 instances)	~360 MB
hapi server	~170 MB
hapi daemon	~150 MB
hapi codex	~160 MB
hapi mcp	~140 MB
Total HAPI	~1 GB

Suggested Fixes

1. Add maximum reconnection attempts

```typescript
reconnectionAttempts: 50, // Stop after 50 failed attempts
```

2. Increase maximum reconnection delay (exponential backoff)

```typescript
reconnectionDelayMax: 30000, // 30 seconds max delay
```

3. Add reconnection failure handler

```typescript
this.socket.on('reconnect_failed', () => {
logger.error('[API] Reconnection failed after max attempts')
// Optionally: notify user, trigger graceful shutdown, or restart
})
```

4. Consider using the existing `withRetry` utility

The codebase already has a well-designed retry utility in `cli/src/utils/time.ts` with proper exponential backoff and max attempts. Consider using similar logic for Socket.IO reconnection.

5. Add memory monitoring

Consider adding a memory watchdog that logs warnings when memory usage exceeds thresholds and takes corrective action before OOM.

Logs

Full HAPI log file: `~/.hapi/logs/2026-01-18-01-44-23-pid-651323.log` (4.7 MB)

Steps to Reproduce

1. Start HAPI services (`hapi server`, `hapi claude --yolo`, `hapi daemon start-sync`)
2. Run a resource-intensive task (e.g., large code review with Codex)
3. Simulate network instability or server unavailability causing Socket.IO disconnection
4. Wait for prolonged reconnection failures (30+ minutes)
5. Observe memory pressure and eventual OOM

Impact

• Long-running HAPI services are at risk of OOM crashes
• Requires manual restart after crash
• Can cause data loss for in-progress tasks

---

Thank you for developing this excellent tool! Hope this detailed report helps identify and fix the issue. 🙏

[github-actions]

Thanks for report. Verified in repo: CLI session socket sets reconnectionAttempts: Infinity with reconnectionDelayMax: 5000; machine socket enables reconnection with same delay max; only connect_error handlers present. No reconnect_failed handler found in these clients. Leak source not confirmed from code alone.

Relevant code:

• cli/src/api/apiSession.ts:74 - Socket.IO client config includes reconnectionAttempts: Infinity, reconnectionDelayMax: 5000.
• cli/src/api/apiMachine.ts:218 - Machine-scoped client enables reconnection with reconnectionDelayMax: 5000.
• cli/src/api/apiSession.ts:126 - connect_error handler only logs + disconnect hook.
• cli/src/api/apiMachine.ts:292 - connect_error handler logs error.

Need more info:

• Which process actually grows to OOM (process name + PID from ps/journal)?
• Any per-process memory timeline during reconnect window (even 5–10 samples)?
• Log snippet around the first disconnect + a few reconnect errors (redacted ok)?
• Repro with only hapi server + one client (no daemon/codex), yes/no?

---

HAPI Bot

[Lies188]

Thank you for the quick response! After deeper investigation, I found critical new information that changes the analysis.

Key Finding: Multiple Processes Simultaneously Reconnecting

I discovered that at least 10 HAPI processes were running simultaneously and ALL of them were trying to reconnect at the same time during the outage:

=== Processes with Socket errors at 01:08-01:11 ===
PID 621588:  [01:08:08] Socket connection error
PID 638556:  [01:08:54] Socket connection error  
PID 651323:  [01:08:55] Socket connection error (main Codex process)
PID 812712:  [01:08:52] Socket connection error
PID 817868:  [01:08:18] Socket connection error
PID 931579:  [01:09:29] Socket connection error
PID 932404:  [01:09:36] Socket connection error
PID 932963:  [01:08:38] Socket connection error
PID 1005538: [01:10:08] Socket connection error
... (possibly more)

Each process was created at different times throughout the day but none were cleaned up:

• 2026-01-18-00-40-30-pid-621588.log
• 2026-01-18-01-16-45-pid-638556.log
• 2026-01-18-01-44-23-pid-651323.log
• 2026-01-18-10-01-21-pid-812712.log
• 2026-01-18-10-08-07-pid-817868.log
• ... and more

Updated Root Cause Analysis

The OOM is likely caused by process accumulation + simultaneous reconnection storm, not just single-process memory leak:

1. Process Accumulation: HAPI spawns new processes but doesn't clean up idle ones
2. Reconnection Storm: When server becomes unavailable, ALL processes try to reconnect simultaneously
3. Memory Multiplication: 10 processes × ~200MB each = ~2GB just for HAPI
4. Plus Codex Cache: The main Codex process had processed 4.4M tokens earlier, likely retaining significant memory

Answers to Your Questions

1. Which process grows to OOM?

Cannot pinpoint a single process - the issue appears to be cumulative memory from all processes. The system journal only shows Under memory pressure without OOM killer logs (possibly crashed before kernel could log).

2. Per-process memory timeline

Unfortunately I don't have per-process memory samples from before the crash. Current post-restart observation:

USER     PID  %MEM   RSS      COMMAND
ubuntu   6655  2.5   202MB    hapi claude --yolo
ubuntu   6373  2.1   170MB    hapi server
ubuntu   7326  2.1   165MB    hapi claude (remote)
ubuntu   7742  2.0   162MB    hapi codex
ubuntu   6667  1.8   147MB    hapi daemon
ubuntu   7841  1.8   144MB    hapi mcp

With 10+ processes at ~150-200MB each during reconnection, total could easily exceed available memory.

3. Log snippet around first disconnect

[00:36:59.175] [API] Socket disconnected: ping timeout
[00:37:50.616] [API] Socket connection error: {}
[00:39:20.261] [API] Socket connection error: {}
[00:40:28.289] [API] Socket connection error: {}
... (continues every ~1 minute for 35+ attempts)
[01:11:25.322] [API] Socket connection error: {}

The main Codex process (PID 651323) had completed its task at 01:54:49 the previous day and was idle for ~23 hours before the disconnect at 00:36:59.

4. Repro with only hapi server + one client?

I haven't tried this specific repro yet. Will test and report back.

Suggested Improvements

1. Process Lifecycle Management:

   • Auto-terminate idle processes after configurable timeout
   • Limit maximum concurrent HAPI processes
   • Track and clean up orphaned processes

2. Coordinated Reconnection:

   • Add jitter/randomization to prevent thundering herd
   • Implement backoff coordination across processes
   • Consider a single "connection manager" process

3. Memory Monitoring:

   • Add per-process memory limits
   • Implement graceful shutdown when memory exceeds threshold

Would you like me to test a minimal repro scenario?