Multiple customManagers for the same file cause geometric growth in dependency count, leading to memory exhaustion #30906

vitoyucepi · 2024-08-20T06:13:54Z

vitoyucepi
Aug 20, 2024

What would you like help with?

I think I found a bug

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

GitLab SaaS
ghcr.io/renovatebot/renovate:38.44.2

Please tell us more about your question or problem

Since renovate:38.41.0, each subsequent run of Renovate causes the number of dependencies to grow geometrically, resulting in excessive memory consumption. The problem occurs when multiple customManagers process the same file while the cache persists.

Reproduction:

file
```
a=38.44.2
b=7.4.0
```

renovate.json

{
  "extends": ["config:recommended"],
  "customManagers": [
    {
      "customType": "regex",
      "fileMatch": ["file"],
      "matchStrings": ["a=(?<currentValue>.*)"],
      "datasourceTemplate": "github-tags",
      "depNameTemplate": "renovatebot/renovate"
    },
    {
      "customType": "regex",
      "fileMatch": ["file"],
      "matchStrings": ["b=(?<currentValue>.*)"],
      "datasourceTemplate": "github-tags",
      "depNameTemplate": "redis/redis"
    }
  ]
}

Minimal reproduction repositories: https://github.com/vitoyucepi/renovate-issue-30906, https://gitlab.com/vitoyucepi/renovate-issue-30906
Similar to #25120, but may actually break renovate.

Logs (if relevant)

Logs

DEBUG: Using RE2 regex engine
DEBUG: Parsing configs
DEBUG: Checking for config file in config.js
DEBUG: No config file found on disk - skipping
DEBUG: Converting GITHUB_COM_TOKEN into a global host rule
DEBUG: File config
       "config": {}
DEBUG: CLI config
       "config": {"repositories": ["vitoyucepi/renovate-issue-30906"]}
DEBUG: Env config
       "config": {
         "hostRules": [
           {"hostType": "github", "matchHost": "github.com", "token": "***********"}
         ],
         "repositoryCache": "enabled",
         "baseDir": "/builds/vito-foss/renovate-issue-30906/renovate-runner-sidecar/renovate",
         "onboardingConfig": {
           "$schema": "https://docs.renovatebot.com/renovate-schema.json",
           "extends": ["config:recommended"]
         },
         "optimizeForDisabled": true,
         "platform": "gitlab",
         "endpoint": "https://gitlab.com/api/v4",
         "token": "***********"
       }
DEBUG: Combined config
       "config": {
         "hostRules": [
           {"hostType": "github", "matchHost": "github.com", "token": "***********"}
         ],
         "repositoryCache": "enabled",
         "baseDir": "/builds/vito-foss/renovate-issue-30906/renovate-runner-sidecar/renovate",
         "onboardingConfig": {
           "$schema": "https://docs.renovatebot.com/renovate-schema.json",
           "extends": ["config:recommended"]
         },
         "optimizeForDisabled": true,
         "platform": "gitlab",
         "endpoint": "https://gitlab.com/api/v4",
         "token": "***********",
         "repositories": ["vitoyucepi/renovate-issue-30906"]
       }
DEBUG: Adding trailing slash to endpoint
DEBUG: Enabling forkProcessing while in non-autodiscover mode
DEBUG: Enabling onboardingNoDeps while in non-autodiscover mode
DEBUG: Found valid git version: 2.46.0
DEBUG: Setting global hostRules
DEBUG: Adding token authentication for github.com (hostType=github) to hostRules
DEBUG: hostRules: authentication already set for gitlab.com
DEBUG: Using queue: host=gitlab.com, concurrency=16
DEBUG: GitLab version is: 17.4.0-pre
DEBUG: Using platform gitAuthor: Vito Bot <7886889-vito-bot@users.noreply.gitlab.com>
DEBUG: Adding token authentication for gitlab.com (hostType=gitlab) to hostRules
DEBUG: Using configured baseDir: /builds/vito-foss/renovate-issue-30906/renovate-runner-sidecar/renovate
DEBUG: Using cacheDir: /builds/vito-foss/renovate-issue-30906/renovate-runner-sidecar/renovate/cache
DEBUG: Using containerbaseDir: /builds/vito-foss/renovate-issue-30906/renovate-runner-sidecar/renovate/cache/containerbase
DEBUG: Initializing Renovate internal cache into /builds/vito-foss/renovate-issue-30906/renovate-runner-sidecar/renovate/cache/renovate/renovate-cache-v1
DEBUG: Commits limit = null
DEBUG: Setting global hostRules
DEBUG: Adding token authentication for github.com (hostType=github) to hostRules
DEBUG: Adding token authentication for gitlab.com (hostType=gitlab) to hostRules
DEBUG: validatePresets()
DEBUG: Reinitializing hostRules for repo
DEBUG: Clearing hostRules
DEBUG: Adding token authentication for github.com (hostType=github) to hostRules
DEBUG: Adding token authentication for gitlab.com (hostType=gitlab) to hostRules
 INFO: Repository started (repository=vitoyucepi/renovate-issue-30906)
       "renovateVersion": "38.44.2"
DEBUG: Using localDir: /builds/vito-foss/renovate-issue-30906/renovate-runner-sidecar/renovate/repos/gitlab/vitoyucepi/renovate-issue-30906 (repository=vitoyucepi/renovate-issue-30906)
DEBUG: PackageFiles.clear() - Package files deleted (repository=vitoyucepi/renovate-issue-30906)
DEBUG: hostRules: applying Bearer authentication for gitlab.com (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Using queue: host=gitlab.com, concurrency=16 (repository=vitoyucepi/renovate-issue-30906)
DEBUG: vitoyucepi/renovate-issue-30906 default branch = main (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Enabling Git FS (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Using http URL: https://gitlab.com/vitoyucepi/renovate-issue-30906.git (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Repository cache is restored from revision 13 (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Resetting npmrc (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Resetting npmrc (repository=vitoyucepi/renovate-issue-30906)
DEBUG: checkOnboarding() (repository=vitoyucepi/renovate-issue-30906)
DEBUG: isOnboarded() (repository=vitoyucepi/renovate-issue-30906)
DEBUG: findPr(renovate/configure, Configure Renovate, !open) (repository=vitoyucepi/renovate-issue-30906)
DEBUG: findPr(renovate/configure, chore: Configure Renovate, !open) (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Checking cached config file name (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Existing config file confirmed (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Repository config (repository=vitoyucepi/renovate-issue-30906)
       "fileName": "renovate.json",
       "config": {
         "extends": ["config:recommended"],
         "customManagers": [
           {
             "customType": "regex",
             "fileMatch": ["file"],
             "matchStrings": ["a=(?<currentValue>.*)"],
             "datasourceTemplate": "github-tags",
             "depNameTemplate": "renovatebot/renovate"
           },
           {
             "customType": "regex",
             "fileMatch": ["file"],
             "matchStrings": ["b=(?<currentValue>.*)"],
             "datasourceTemplate": "github-tags",
             "depNameTemplate": "redis/redis"
           }
         ]
       }
DEBUG: Repo is onboarded (repository=vitoyucepi/renovate-issue-30906)
DEBUG: migrateAndValidate() (repository=vitoyucepi/renovate-issue-30906)
DEBUG: No config migration necessary (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Setting hostRules from config (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Found repo ignorePaths (repository=vitoyucepi/renovate-issue-30906)
       "ignorePaths": [
         "**/node_modules/**",
         "**/bower_components/**",
         "**/vendor/**",
         "**/examples/**",
         "**/__tests__/**",
         "**/test/**",
         "**/tests/**",
         "**/__fixtures__/**"
       ]
DEBUG: No vulnerability alerts found (repository=vitoyucepi/renovate-issue-30906)
DEBUG: findIssue(Dependency Dashboard) (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Using queue: host=gitlab.com, concurrency=16 (repository=vitoyucepi/renovate-issue-30906)
DEBUG: No baseBranches (repository=vitoyucepi/renovate-issue-30906)
DEBUG: extract() (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Cached extract for sha=69932bc073e9d3a79c98e45c096ede1484ca9a58 is valid and can be used (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Deleted cached dep updates (repository=vitoyucepi/renovate-issue-30906)
 INFO: Dependency extraction complete (repository=vitoyucepi/renovate-issue-30906, baseBranch=main)
       "stats": {
         "managers": {"regex": {"fileCount": 2, "depCount": 4}},
         "total": {"fileCount": 2, "depCount": 4}
       }
DEBUG: Starting package releases lookups (repository=vitoyucepi/renovate-issue-30906, baseBranch=main)
DEBUG: hostRules: authentication already set for api.github.com (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Using queue: host=api.github.com, concurrency=16 (repository=vitoyucepi/renovate-issue-30906)
DEBUG: PackageFiles.add() - Package file saved for base branch (repository=vitoyucepi/renovate-issue-30906, baseBranch=main)
DEBUG: Package releases lookups complete (repository=vitoyucepi/renovate-issue-30906, baseBranch=main)
DEBUG: branchifyUpgrades (repository=vitoyucepi/renovate-issue-30906)
DEBUG: detectSemanticCommits() (repository=vitoyucepi/renovate-issue-30906)
DEBUG: semanticCommits: returning "disabled" from cache (repository=vitoyucepi/renovate-issue-30906)
DEBUG: 0 flattened updates found:  (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Returning 0 branch(es) (repository=vitoyucepi/renovate-issue-30906)
DEBUG: config.repoIsOnboarded=true (repository=vitoyucepi/renovate-issue-30906)
DEBUG: packageFiles with updates (repository=vitoyucepi/renovate-issue-30906, baseBranch=main)
       "config": {
         "regex": [
           {
             "datasourceTemplate": "github-tags",
             "depNameTemplate": "renovatebot/renovate",
             "deps": [
               {
                 "currentValue": "38.44.2",
                 "currentVersion": "38.44.2",
                 "currentVersionTimestamp": "2024-08-20T08:37:29.000Z",
                 "datasource": "github-tags",
                 "depName": "renovatebot/renovate",
                 "fixedVersion": "38.44.2",
                 "packageName": "renovatebot/renovate",
                 "registryUrl": "https://github.com",
                 "replaceString": "a=38.44.2",
                 "sourceUrl": "https://github.com/renovatebot/renovate",
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "updates": []
               },
               {
                 "currentValue": "7.4.0",
                 "currentVersion": "7.4.0",
                 "currentVersionTimestamp": "2024-07-29T05:31:59.000Z",
                 "datasource": "github-tags",
                 "depName": "redis/redis",
                 "fixedVersion": "7.4.0",
                 "packageName": "redis/redis",
                 "registryUrl": "https://github.com",
                 "replaceString": "b=7.4.0",
                 "sourceUrl": "https://github.com/redis/redis",
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "updates": []
               },
               {
                 "currentValue": "38.44.2",
                 "currentVersion": "38.44.2",
                 "currentVersionTimestamp": "2024-08-20T08:37:29.000Z",
                 "datasource": "github-tags",
                 "depName": "renovatebot/renovate",
                 "fixedVersion": "38.44.2",
                 "packageName": "renovatebot/renovate",
                 "registryUrl": "https://github.com",
                 "replaceString": "a=38.44.2",
                 "sourceUrl": "https://github.com/renovatebot/renovate",
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "updates": []
               },
               {
                 "currentValue": "7.4.0",
                 "currentVersion": "7.4.0",
                 "currentVersionTimestamp": "2024-07-29T05:31:59.000Z",
                 "datasource": "github-tags",
                 "depName": "redis/redis",
                 "fixedVersion": "7.4.0",
                 "packageName": "redis/redis",
                 "registryUrl": "https://github.com",
                 "replaceString": "b=7.4.0",
                 "sourceUrl": "https://github.com/redis/redis",
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "updates": []
               }
             ],
             "matchStrings": ["a=(?<currentValue>.*)"],
             "packageFile": "file"
           },
           {
             "datasourceTemplate": "github-tags",
             "depNameTemplate": "redis/redis",
             "deps": [
               {
                 "currentValue": "38.44.2",
                 "currentVersion": "38.44.2",
                 "currentVersionTimestamp": "2024-08-20T08:37:29.000Z",
                 "datasource": "github-tags",
                 "depName": "renovatebot/renovate",
                 "fixedVersion": "38.44.2",
                 "packageName": "renovatebot/renovate",
                 "registryUrl": "https://github.com",
                 "replaceString": "a=38.44.2",
                 "sourceUrl": "https://github.com/renovatebot/renovate",
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "updates": []
               },
               {
                 "currentValue": "7.4.0",
                 "currentVersion": "7.4.0",
                 "currentVersionTimestamp": "2024-07-29T05:31:59.000Z",
                 "datasource": "github-tags",
                 "depName": "redis/redis",
                 "fixedVersion": "7.4.0",
                 "packageName": "redis/redis",
                 "registryUrl": "https://github.com",
                 "replaceString": "b=7.4.0",
                 "sourceUrl": "https://github.com/redis/redis",
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "updates": []
               },
               {
                 "currentValue": "38.44.2",
                 "currentVersion": "38.44.2",
                 "currentVersionTimestamp": "2024-08-20T08:37:29.000Z",
                 "datasource": "github-tags",
                 "depName": "renovatebot/renovate",
                 "fixedVersion": "38.44.2",
                 "packageName": "renovatebot/renovate",
                 "registryUrl": "https://github.com",
                 "replaceString": "a=38.44.2",
                 "sourceUrl": "https://github.com/renovatebot/renovate",
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "updates": []
               },
               {
                 "currentValue": "7.4.0",
                 "currentVersion": "7.4.0",
                 "currentVersionTimestamp": "2024-07-29T05:31:59.000Z",
                 "datasource": "github-tags",
                 "depName": "redis/redis",
                 "fixedVersion": "7.4.0",
                 "packageName": "redis/redis",
                 "registryUrl": "https://github.com",
                 "replaceString": "b=7.4.0",
                 "sourceUrl": "https://github.com/redis/redis",
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "updates": []
               }
             ],
             "matchStrings": ["b=(?<currentValue>.*)"],
             "packageFile": "file"
           }
         ]
       }
DEBUG: detectSemanticCommits() (repository=vitoyucepi/renovate-issue-30906)
DEBUG: semanticCommits: returning "disabled" from cache (repository=vitoyucepi/renovate-issue-30906)
DEBUG: processRepo() (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Processing 0 branches:  (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Calculating hourly PRs remaining (repository=vitoyucepi/renovate-issue-30906)
DEBUG: currentHourStart=2024-08-20T12:00:00.000+00:00 (repository=vitoyucepi/renovate-issue-30906)
DEBUG: PR hourly limit remaining: 2 (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Calculating prConcurrentLimit (10) (repository=vitoyucepi/renovate-issue-30906)
DEBUG: 0 PRs are currently open (repository=vitoyucepi/renovate-issue-30906)
DEBUG: PR concurrent limit remaining: 10 (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Calculated maximum PRs remaining this run: 2 (repository=vitoyucepi/renovate-issue-30906)
DEBUG: PullRequests limit = 2 (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Calculating hourly PRs remaining (repository=vitoyucepi/renovate-issue-30906)
DEBUG: currentHourStart=2024-08-20T12:00:00.000+00:00 (repository=vitoyucepi/renovate-issue-30906)
DEBUG: PR hourly limit remaining: 2 (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Calculating branchConcurrentLimit (10) (repository=vitoyucepi/renovate-issue-30906)
DEBUG: 0 already existing branches found:  (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Branch concurrent limit remaining: 10 (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Calculated maximum branches remaining this run: 2 (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Branches limit = 2 (repository=vitoyucepi/renovate-issue-30906)
DEBUG: ensureDependencyDashboard() (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Ensuring Dependency Dashboard (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Checking packageFiles for deprecated packages (repository=vitoyucepi/renovate-issue-30906)
       "packageFiles": {
         "regex": [
           {
             "datasourceTemplate": "github-tags",
             "depNameTemplate": "renovatebot/renovate",
             "deps": [
               {
                 "currentValue": "38.44.2",
                 "currentVersion": "38.44.2",
                 "currentVersionTimestamp": "2024-08-20T08:37:29.000Z",
                 "datasource": "github-tags",
                 "depName": "renovatebot/renovate",
                 "fixedVersion": "38.44.2",
                 "packageName": "renovatebot/renovate",
                 "registryUrl": "https://github.com",
                 "replaceString": "a=38.44.2",
                 "sourceUrl": "https://github.com/renovatebot/renovate",
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "updates": []
               },
               {
                 "currentValue": "7.4.0",
                 "currentVersion": "7.4.0",
                 "currentVersionTimestamp": "2024-07-29T05:31:59.000Z",
                 "datasource": "github-tags",
                 "depName": "redis/redis",
                 "fixedVersion": "7.4.0",
                 "packageName": "redis/redis",
                 "registryUrl": "https://github.com",
                 "replaceString": "b=7.4.0",
                 "sourceUrl": "https://github.com/redis/redis",
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "updates": []
               },
               {
                 "currentValue": "38.44.2",
                 "currentVersion": "38.44.2",
                 "currentVersionTimestamp": "2024-08-20T08:37:29.000Z",
                 "datasource": "github-tags",
                 "depName": "renovatebot/renovate",
                 "fixedVersion": "38.44.2",
                 "packageName": "renovatebot/renovate",
                 "registryUrl": "https://github.com",
                 "replaceString": "a=38.44.2",
                 "sourceUrl": "https://github.com/renovatebot/renovate",
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "updates": []
               },
               {
                 "currentValue": "7.4.0",
                 "currentVersion": "7.4.0",
                 "currentVersionTimestamp": "2024-07-29T05:31:59.000Z",
                 "datasource": "github-tags",
                 "depName": "redis/redis",
                 "fixedVersion": "7.4.0",
                 "packageName": "redis/redis",
                 "registryUrl": "https://github.com",
                 "replaceString": "b=7.4.0",
                 "sourceUrl": "https://github.com/redis/redis",
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "updates": []
               }
             ],
             "matchStrings": ["a=(?<currentValue>.*)"],
             "packageFile": "file"
           },
           {
             "datasourceTemplate": "github-tags",
             "depNameTemplate": "redis/redis",
             "deps": [
               {
                 "currentValue": "38.44.2",
                 "currentVersion": "38.44.2",
                 "currentVersionTimestamp": "2024-08-20T08:37:29.000Z",
                 "datasource": "github-tags",
                 "depName": "renovatebot/renovate",
                 "fixedVersion": "38.44.2",
                 "packageName": "renovatebot/renovate",
                 "registryUrl": "https://github.com",
                 "replaceString": "a=38.44.2",
                 "sourceUrl": "https://github.com/renovatebot/renovate",
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "updates": []
               },
               {
                 "currentValue": "7.4.0",
                 "currentVersion": "7.4.0",
                 "currentVersionTimestamp": "2024-07-29T05:31:59.000Z",
                 "datasource": "github-tags",
                 "depName": "redis/redis",
                 "fixedVersion": "7.4.0",
                 "packageName": "redis/redis",
                 "registryUrl": "https://github.com",
                 "replaceString": "b=7.4.0",
                 "sourceUrl": "https://github.com/redis/redis",
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "updates": []
               },
               {
                 "currentValue": "38.44.2",
                 "currentVersion": "38.44.2",
                 "currentVersionTimestamp": "2024-08-20T08:37:29.000Z",
                 "datasource": "github-tags",
                 "depName": "renovatebot/renovate",
                 "fixedVersion": "38.44.2",
                 "packageName": "renovatebot/renovate",
                 "registryUrl": "https://github.com",
                 "replaceString": "a=38.44.2",
                 "sourceUrl": "https://github.com/renovatebot/renovate",
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "updates": []
               },
               {
                 "currentValue": "7.4.0",
                 "currentVersion": "7.4.0",
                 "currentVersionTimestamp": "2024-07-29T05:31:59.000Z",
                 "datasource": "github-tags",
                 "depName": "redis/redis",
                 "fixedVersion": "7.4.0",
                 "packageName": "redis/redis",
                 "registryUrl": "https://github.com",
                 "replaceString": "b=7.4.0",
                 "sourceUrl": "https://github.com/redis/redis",
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "updates": []
               }
             ],
             "matchStrings": ["b=(?<currentValue>.*)"],
             "packageFile": "file"
           }
         ]
       }
DEBUG: ensureIssue() (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Updating issue (repository=vitoyucepi/renovate-issue-30906)
DEBUG: validateReconfigureBranch() (repository=vitoyucepi/renovate-issue-30906)
DEBUG: No reconfigure branch found (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Removing any stale branches (repository=vitoyucepi/renovate-issue-30906)
DEBUG: config.repoIsOnboarded=true (repository=vitoyucepi/renovate-issue-30906)
DEBUG: No renovate branches found (repository=vitoyucepi/renovate-issue-30906)
DEBUG: ensureIssueClosing() (repository=vitoyucepi/renovate-issue-30906)
DEBUG: ensureIssueClosing() (repository=vitoyucepi/renovate-issue-30906)
DEBUG: PackageFiles.clear() - Package files deleted (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Branch summary (repository=vitoyucepi/renovate-issue-30906)
       "cacheModified": true,
       "baseBranches": [{"branchName": "main", "sha": "69932bc073e9d3a79c98e45c096ede1484ca9a58"}],
       "branches": [],
       "defaultBranch": "main",
       "inactiveBranches": []
DEBUG: Renovate repository PR statistics (repository=vitoyucepi/renovate-issue-30906)
       "stats": {"total": 0, "open": 0, "closed": 0, "merged": 0}
DEBUG: Repository result: done, status: onboarded, enabled: true, onboarded: true (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Repository timing splits (milliseconds) (repository=vitoyucepi/renovate-issue-30906)
       "splits": {"init": 1776, "extract": 658, "lookup": 81618, "onboarding": 2, "update": 4},
       "total": 84903
DEBUG: Package cache statistics (repository=vitoyucepi/renovate-issue-30906)
       "get": {"count": 8, "avgMs": 6, "medianMs": 1, "maxMs": 19, "totalMs": 45},
       "set": {"count": 8, "avgMs": 80, "medianMs": 14, "maxMs": 455, "totalMs": 637}
DEBUG: HTTP statistics (repository=vitoyucepi/renovate-issue-30906)
       "hosts": {
         "api.github.com": {
           "count": 180,
           "reqAvgMs": 457,
           "reqMedianMs": 519,
           "reqMaxMs": 1283,
           "queueAvgMs": 0,
           "queueMedianMs": 0,
           "queueMaxMs": 1
         },
         "gitlab.com": {
           "count": 7,
           "reqAvgMs": 253,
           "reqMedianMs": 179,
           "reqMaxMs": 599,
           "queueAvgMs": 0,
           "queueMedianMs": 0,
           "queueMaxMs": 0
         }
       },
       "requests": 187
DEBUG: HTTP cache statistics (repository=vitoyucepi/renovate-issue-30906)
DEBUG: Lookup statistics (repository=vitoyucepi/renovate-issue-30906)
       "github-tags": {
         "count": 4,
         "avgMs": 42078,
         "medianMs": 81581,
         "maxMs": 81596,
         "totalMs": 168311
       }
 INFO: Repository finished (repository=vitoyucepi/renovate-issue-30906)
       "cloned": false,
       "durationMs": 84903
DEBUG: Checking file package cache for expired items
DEBUG: Deleted 0 of 8 file cached entries in 17ms

Answered by viceice

Aug 21, 2024

in case i prepared a revert

View full answer

vitoyucepi · 2024-08-21T03:05:25Z

vitoyucepi
Aug 21, 2024
Author

Discussion #30923 may be a duplicate.
From my point of view

134 error code or SIGABRT, caused by memory allocation error.

<--- Last few GCs --->
[17:0x63fdcd0]   117523 ms: Scavenge (reduce) 2046.1 (2087.2) -> 2045.0 (2083.9) MB, 21.26 / 0.00 ms  (average mu = 0.638, current mu = 0.443) allocation failure; 
[17:0x63fdcd0]   118856 ms: Mark-Compact (reduce) 2046.9 (2083.9) -> 2046.4 (2084.4) MB, 1131.87 / 0.02 ms  (+ 374.5 ms in 80 steps since start of marking, biggest step 16.2 ms, walltime since start of marking 1634 ms) (average mu = 0.512, current mu = 0.
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----
 1: 0xb82d33 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/usr/local/renovate/node]
 2: 0xef0c50 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/usr/local/renovate/node]
 3: 0xef0f37 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/usr/local/renovate/node]
 4: 0x1102ad5  [/usr/local/renovate/node]
 5: 0x111a958 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/renovate/node]
 6: 0x10f0a71 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/renovate/node]
 7: 0x10f1c05 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/renovate/node]
 8: 0x10cf256 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/local/renovate/node]
 9: 0x152b086 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/local/renovate/node]
10: 0x7db3f2699ef6 
/usr/local/sbin/renovate: line 8:    17 Aborted                 (core dumped) /usr/local/renovate/node --use-openssl-ca "${RENOVATE_NODE_ARGS[@]}" /usr/local/renovate/dist/renovate.js "$@"

That means from ~10% memory usage (of 4GB), it went up to >50%

Huge amounts of memory were allocated, like 2GB.
Somewhen after 38.40.1

The 38.41.0 is the next release after 38.40.1.

12 replies

viceice Aug 21, 2024
Maintainer

the only change which can have that result is

refactor: Flatten update lookup tasks #27369

@zharinov @rarkins should we revert it? I'll try to test with that version and see whats happening

viceice Aug 21, 2024
Maintainer

I saw same behavior here: https://code.forgejo.org/forgejo-contrib/forgejo-renovate/pulls/382

viceice Aug 21, 2024
Maintainer

in case i prepared a revert

Answer selected by viceice

florianmutter Dec 17, 2024

Was the same PR merged again? We see the same behaviour starting with version 39.59.0 which contains this change: #32930

florianmutter Dec 17, 2024

I see that the new PR should not have caused this but we can definitely see renovate working before this change (39.58.1) and failing after this change. We have a very big config so I cannot exactly tell you what is causing it to fail again. I attache the log from the first failed run renovate-logs.txt

zharinov Dec 17, 2024
Collaborator

So, can you confirm that every new run, it reports more and more dependencies are extracted?

zharinov Dec 17, 2024
Collaborator

BTW, probably resetting the repo cache could help, because it may contain the duplicated data from the period when the original bug was introduced.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple customManagers for the same file cause geometric growth in dependency count, leading to memory exhaustion #30906

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 12 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Multiple customManagers for the same file cause geometric growth in dependency count, leading to memory exhaustion #30906

vitoyucepi Aug 20, 2024

What would you like help with?

How are you running Renovate?

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

Please tell us more about your question or problem

Logs (if relevant)

Replies: 1 comment · 12 replies

vitoyucepi Aug 21, 2024 Author

viceice Aug 21, 2024 Maintainer

viceice Aug 21, 2024 Maintainer

viceice Aug 21, 2024 Maintainer

florianmutter Dec 17, 2024

florianmutter Dec 17, 2024

zharinov Dec 17, 2024 Collaborator

zharinov Dec 17, 2024 Collaborator

vitoyucepi
Aug 20, 2024

Replies: 1 comment 12 replies

vitoyucepi
Aug 21, 2024
Author

viceice Aug 21, 2024
Maintainer

viceice Aug 21, 2024
Maintainer

viceice Aug 21, 2024
Maintainer

zharinov Dec 17, 2024
Collaborator

zharinov Dec 17, 2024
Collaborator