High Memory Usage since v3.4.4

[BramRoets]

• Horizon Version: 3.4.4
• Laravel Version: 5.8.35
• PHP Version: 7.2
• Redis Driver & Version: predis/phpredis v1.1.1
• Database Driver & Version: Redis v5.0.0

Description:

I've updated Laravel Horizon from v3.2.2 to v3.4.4 at 21 nov 2019. I'm seeing an increase in memory usage since then.

Available Memory, free memory started dropping after 21 november

Debugging memory usage with RedisInsight show the recent keys uses almost all memory available.

Steps To Reproduce:

1. Run 50.000 jobs per hour, jobs have different models (using SerializesModels) attached to them, no tags are configured in Horizon.
2. Config horizon.trim.recent = 60 minutes
3. Notice high memory usage

I'm not 100% sure why the memory usage keeps rising over a longer period of time. We do have some additional load over the last months, but not enough to cause this.

I've looked at #715 and i've used the following piece of code:

Queue::after(function (JobProcessed $event) {
	Redis::expireat(config('horizon.prefix') . $event->job->getJobId(), Carbon::now()->addMinute()->timestamp);
});

That did not seem to make any difference. The recent keys still have a TTL of 60 minutes.

If I change horizon.trim.recent to 5 minutes. Will this solve my problem? I'm not sure what will happen with pending jobs when:

• They get release back on the queue e.g. $this->release(300)
• When the Laravel Horizon workers are down for more than 5 minutes (planned maintenance, ..)
• A burst of jobs, causing a long queue longer than 5 minutes.

[Prashank25]

Redis: v5.0.7
PhpRedis: v5.1.1
Horizon: v3.4.6

I am having this issue as well, it seems to be the horizon:recent:TAGNAME keys in my case, they seem to keep growing and never get cleared. Am I missing something?

I am able to clear some memory with redis-cli --scan --pattern horizon:recent:* | xargs redis-cli del

[BramRoets]

I've wrote a small piece of code which basically disables the tag repository. I'm not using the tags feature and I needed a quick solution before our Redis ran out of memory.

I'm still confused about the high memory usage for the recent tags. 854M for 17K of tags seems really high. I'm not sure how the tags work internally.

Temporary fix

# Add the following to AppServiceProvider@boot
app()->bind(TagRepository::class, App\Repositories\RedisTagRepository::class);

# Override RedisTagRepository

namespace App\Repositories;

use Laravel\Horizon\Repositories\RedisTagRepository as TagRepository;

class RedisTagRepository extends TagRepository
{
    public function addTemporary($minutes, $id, array $tags)
    {
        // Disabled
    }
}

Free memory after implementing this fix

[themsaid]

I can see why this is an issue. I think we should revert #665

[Prashank25]

It seems to me that the horizon:recent:TAGNAME will never actually expire since everytime a new job is pushed the expire_at is bumped up?

I am not sure about other apps, but I don't really have a use for filtering by tag on recent jobs or being able to view recent jobs for that matter since when its doing 10k jobs/minute it's not even a good idea to keep those jobs around taking up space. So at least a config option to disable StoreTagsForRecentJob would be nice.

[driesvints]

We've reverted the pr that causes this for now.

[SumitChowjar]

I am also troubling with high memory usage and my system crashed.

Laravel: v8.35.1
Horizon: v5.7

I have around 40k records. I made a chunk of 300 records and processed each chunk via job.

Users::chunk(300, function($users) {
     //dispatch a job
     MigrateUsers::dispatch($users->all());
});

My horizon config is:

'waits' => [
        'redis:default' => 60,
    ],

'trim' => [
        'recent' => 60,
        'pending' => 2880,
        'completed' => 60,
        'recent_failed' => 1440,
        'failed' => 2880,
        'monitored' => 2880,
    ],

'memory_limit' => 128,

'defaults' => [
        'supervisor-1' => [
            'connection' => 'redis',
            'queue' => ['default'],
            'balance' => 'auto',
            'minProcesses' => 1,
            'maxProcesses' => 2,
            'memory' => 128,
            'tries' => 2,
            'nice' => 0,
        ],
    ],

    'environments' => [
        'local' => [
            'supervisor-1' => [
                'maxProcesses' => 2,
                'balanceMaxShift' => 1,
                'balanceCooldown' => 3,
                'timeout' => 900 // Timeout after 15 minutes
            ],
        ],
    ]

And in this process 7 to 8 GB of RAM is consumed and the system reboots in between.
@taylorotwell