perf(misconf): optimize work with context

[nikpivkin]

Description

• The for_each resource attribute is not added to the context because it cannot be accessed, but it can take up a lot of memory.
• If possible, use the GetAttr method to access cty.Object attributes, instead of converting the object to a map.

Unfortunately there is still the mergeVars function, which cannot be optimised due to some limitations. Because of the impossibility to modify cty.Object directly, it is necessary to convert cty.Object into a map, modify it and create the object from the map again. With a large number of resources, such actions generate a large number of objects and take a decent amount of time due to checks and conversions inside the cty package.

Test config:

locals {
  team_repos = [ for i in range(1000): "repo-${i}"]
  teams = [ for i in range(10): "team-${i}"]
  repositories = merge([for team_id in local.teams : { for repo in local.team_repos : "${team_id}-${repo}" => team_id}]...)
}

resource "aws_ecr_repository" "ecr-repository" {
  for_each = local.repositories

  name                 = each.key
  image_tag_mutability = "IMMUTABLE"
  tags = {
    "Team" : each.value
  }
}

Before:
Memory usage is increasing, scans are not completed in a reasonable amount of time.

After:

/usr/bin/time -al ./trivy conf -q -f json -o report.json /Users/nikita/projects/trivy-test/diss-6958
       29.68 real        40.85 user         0.69 sys
          1221738496  maximum resident set size

Related issues:

• Close perf(misconf): High memory usage (> 10 GB) on some repos #6959

Checklist

• I've read the guidelines for contributing to this repository.
• I've followed the conventions in the PR title.
• I've added tests that prove my fix is effective or that my feature works.
• I've updated the documentation with the relevant information (if needed).
• I've added usage information (if the PR introduces new options)
• I've included a "before" and "after" example to the description (if the PR is a user interface change).

[nikpivkin]

@simar7 Generating UUID for a large number of blocks takes an impressive amount of time, as system calls are used. Can we enable pooling, or use a lightweight way to generate id for blocks? The probability of collision is low.

[simar7]

@simar7 Generating UUID for a large number of blocks takes an impressive amount of time, as system calls are used. Can we enable pooling, or use a lightweight way to generate id for blocks? The probability of collision is low.

It makes sense but how much of an improvement is it?

[nikpivkin]

@simar7 I think improving ID generation can be done later, if it does become an issue.