Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: parallel generate #1937

Merged
merged 5 commits into from
Oct 24, 2024
Merged

refactor: parallel generate #1937

merged 5 commits into from
Oct 24, 2024

Conversation

i4ki
Copy link
Contributor

@i4ki i4ki commented Oct 21, 2024

What this PR does / why we need it:

Introduces parallel code generation.

Which issue(s) this PR fixes:

none

Special notes for your reviewer:

Does this PR introduce a user-facing change?

yes, performance

@i4ki i4ki requested a review from a team as a code owner October 21, 2024 08:46
@i4ki i4ki marked this pull request as draft October 21, 2024 08:46
Copy link

github-actions bot commented Oct 21, 2024

2024/10/23 19:12:07 Terraform detected version: 1.9.7
toolsetPath: /tmp/cmd-terramate-test-2259903456
=== RUN   TestInteropCloudSyncPreview
=== RUN   TestInteropCloudSyncPreview/preview:_basic-drift
    interoperability_test.go:46: using GITHUB_EVENT_FILE=/home/runner/work/_temp/_github_workflow/event.json
=== RUN   TestInteropCloudSyncPreview/preview:_basic-drift-uppercase-id
    interoperability_test.go:46: using GITHUB_EVENT_FILE=/home/runner/work/_temp/_github_workflow/event.json
--- PASS: TestInteropCloudSyncPreview (10.97s)
    --- PASS: TestInteropCloudSyncPreview/preview:_basic-drift (5.80s)
    --- PASS: TestInteropCloudSyncPreview/preview:_basic-drift-uppercase-id (5.17s)
=== RUN   TestInteropSyncDeployment
=== RUN   TestInteropSyncDeployment/deployment:_empty
=== RUN   TestInteropSyncDeployment/deployment:_empty-uppercase-id
--- PASS: TestInteropSyncDeployment (36.24s)
    --- PASS: TestInteropSyncDeployment/deployment:_empty (18.04s)
    --- PASS: TestInteropSyncDeployment/deployment:_empty-uppercase-id (18.20s)
=== RUN   TestInteropDrift
=== RUN   TestInteropDrift/drift:_basic-drift
=== RUN   TestInteropDrift/drift:_basic-drift-uppercase-id
--- PASS: TestInteropDrift (58.40s)
    --- PASS: TestInteropDrift/drift:_basic-drift (29.42s)
    --- PASS: TestInteropDrift/drift:_basic-drift-uppercase-id (28.98s)
PASS
ok  	github.com/terramate-io/terramate/e2etests/cloud/interop	163.813s

Copy link

github-actions bot commented Oct 21, 2024

metric: time/op
ChangeDetection-4: old 4.10ms ± 3%: new 4.03ms ± 5%: delta: 0.00%
ChangeDetectionTFAndTG-4: old 13.3ms ± 6%: new 13.4ms ± 6%: delta: 0.00%
CloudReadLines-4: old 1.02ms ± 5%: new 1.02ms ± 8%: delta: 0.00%
CloudReadLine-4: old 7.26ms ± 2%: new 7.23ms ± 2%: delta: 0.00%
ListFiles-4: old 80.8µs ± 2%: new 80.8µs ± 1%: delta: 0.00%
Generate-4: old 4.99s ± 2%: new 1.56s ± 3%: delta: -68.65%
GenerateRegex-4: old 3.27s ± 2%: new 1.06s ± 4%: delta: -67.49%
TokensForExpressionComplex-4: old 1.26ms ± 0%: new 1.26ms ± 1%: delta: 0.46%
TokensForExpressionPlainStringNoNewline-4: old 796ns ± 1%: new 792ns ± 1%: delta: -0.49%
TokensForExpressionStringWith100Newlines-4: old 23.9µs ± 2%: new 24.0µs ± 1%: delta: 0.00%
TokensForExpressionObjectWith100KeysWithNumberValues-4: old 1.44ms ± 0%: new 1.46ms ± 1%: delta: 0.87%
TokensForExpression-4: old 1.26ms ± 0%: new 1.26ms ± 1%: delta: 0.00%
PartialEvalComplex-4: old 517µs ± 1%: new 525µs ± 3%: delta: 1.64%
PartialEvalSmallString-4: old 3.73µs ± 0%: new 3.78µs ± 1%: delta: 1.33%
PartialEvalHugeString-4: old 1.92ms ± 1%: new 1.92ms ± 1%: delta: 0.00%
PartialEvalHugeInterpolatedString-4: old 4.75ms ± 3%: new 4.73ms ± 2%: delta: 0.00%
PartialEvalObject-4: old 24.7µs ± 1%: new 24.9µs ± 1%: delta: 0.90%
TmAllTrueLiteralList-4: old 620µs ± 1%: new 619µs ± 1%: delta: 0.00%
TmAllTrueFuncall-4: old 20.4µs ± 2%: new 20.8µs ± 2%: delta: 1.85%
TmAnyTrueLiteralList-4: old 4.84ms ± 0%: new 4.97ms ± 1%: delta: 2.58%
TmAnyTrueFuncall-4: old 20.8µs ± 2%: new 21.4µs ± 2%: delta: 2.78%
TmTernary-4: old 2.59µs ± 1%: new 2.64µs ± 1%: delta: 2.08%
TmTryUnknownFunc-4: old 2.30µs ± 1%: new 2.34µs ± 1%: delta: 1.71%
TmTryUnknownVariable-4: old 2.18µs ± 1%: new 2.25µs ± 1%: delta: 2.99%
TmTryUnknownObjectKey-4: old 2.50µs ± 1%: new 2.54µs ± 1%: delta: 1.70%
metric: alloc/op
ChangeDetection-4: old 357kB ± 0%: new 357kB ± 0%: delta: 0.00%
ChangeDetectionTFAndTG-4: old 4.80MB ± 0%: new 4.80MB ± 0%: delta: 0.00%
CloudReadLines-4: old 3.12MB ± 0%: new 3.12MB ± 0%: delta: 0.00%
CloudReadLine-4: old 3.37MB ± 0%: new 3.37MB ± 0%: delta: 0.00%
ListFiles-4: old 27.6kB ± 0%: new 27.6kB ± 0%: delta: 0.00%
Generate-4: old 4.47GB ± 0%: new 2.24GB ± 0%: delta: -49.84%
GenerateRegex-4: old 1.84GB ± 0%: new 0.93GB ± 0%: delta: -49.57%
TokensForExpressionComplex-4: old 394kB ± 0%: new 394kB ± 0%: delta: 0.00%
TokensForExpressionPlainStringNoNewline-4: old 512B ± 0%: new 512B ± 0%: delta: 0.00%
TokensForExpressionStringWith100Newlines-4: old 12.5kB ± 0%: new 12.5kB ± 0%: delta: 0.00%
TokensForExpressionObjectWith100KeysWithNumberValues-4: old 395kB ± 0%: new 395kB ± 0%: delta: 0.00%
TokensForExpression-4: old 394kB ± 0%: new 394kB ± 0%: delta: -0.00%
PartialEvalComplex-4: old 338kB ± 0%: new 338kB ± 0%: delta: 0.00%
PartialEvalSmallString-4: old 1.74kB ± 0%: new 1.74kB ± 0%: delta: 0.00%
PartialEvalHugeString-4: old 196kB ± 0%: new 196kB ± 0%: delta: 0.00%
PartialEvalHugeInterpolatedString-4: old 4.05MB ± 0%: new 4.05MB ± 0%: delta: 0.00%
PartialEvalObject-4: old 17.2kB ± 0%: new 17.2kB ± 0%: delta: 0.00%
TmAllTrueLiteralList-4: old 321kB ± 0%: new 321kB ± 0%: delta: 0.00%
TmAllTrueFuncall-4: old 10.6kB ± 0%: new 10.6kB ± 0%: delta: 0.00%
TmAnyTrueLiteralList-4: old 2.09MB ± 0%: new 2.09MB ± 0%: delta: 0.00%
TmAnyTrueFuncall-4: old 10.7kB ± 0%: new 10.7kB ± 0%: delta: 0.00%
TmTernary-4: old 1.20kB ± 0%: new 1.20kB ± 0%: delta: 0.00%
TmTryUnknownFunc-4: old 784B ± 0%: new 784B ± 0%: delta: 0.00%
TmTryUnknownVariable-4: old 768B ± 0%: new 768B ± 0%: delta: 0.00%
TmTryUnknownObjectKey-4: old 952B ± 0%: new 952B ± 0%: delta: 0.00%
metric: allocs/op
ChangeDetection-4: old 2.43k ± 0%: new 2.43k ± 0%: delta: -0.01%
ChangeDetectionTFAndTG-4: old 33.6k ± 0%: new 33.6k ± 0%: delta: 0.00%
CloudReadLines-4: old 5.54k ± 0%: new 5.54k ± 0%: delta: 0.00%
CloudReadLine-4: old 60.0k ± 0%: new 60.0k ± 0%: delta: 0.00%
ListFiles-4: old 335 ± 0%: new 335 ± 0%: delta: 0.00%
Generate-4: old 51.4M ± 0%: new 25.7M ± 0%: delta: -49.95%
GenerateRegex-4: old 36.8M ± 0%: new 18.4M ± 0%: delta: -49.92%
TokensForExpressionComplex-4: old 4.83k ± 0%: new 4.83k ± 0%: delta: 0.00%
TokensForExpressionPlainStringNoNewline-4: old 20.0 ± 0%: new 20.0 ± 0%: delta: 0.00%
TokensForExpressionStringWith100Newlines-4: old 227 ± 0%: new 227 ± 0%: delta: 0.00%
TokensForExpressionObjectWith100KeysWithNumberValues-4: old 3.19k ± 0%: new 3.19k ± 0%: delta: 0.00%
TokensForExpression-4: old 4.83k ± 0%: new 4.83k ± 0%: delta: 0.00%
PartialEvalComplex-4: old 3.48k ± 0%: new 3.48k ± 0%: delta: 0.00%
PartialEvalSmallString-4: old 23.0 ± 0%: new 23.0 ± 0%: delta: 0.00%
PartialEvalHugeString-4: old 35.0 ± 0%: new 35.0 ± 0%: delta: 0.00%
PartialEvalHugeInterpolatedString-4: old 23.1k ± 0%: new 23.1k ± 0%: delta: 0.00%
PartialEvalObject-4: old 152 ± 0%: new 152 ± 0%: delta: 0.00%
TmAllTrueLiteralList-4: old 5.94k ± 0%: new 5.94k ± 0%: delta: 0.00%
TmAllTrueFuncall-4: old 275 ± 0%: new 275 ± 0%: delta: 0.00%
TmAnyTrueLiteralList-4: old 59.6k ± 0%: new 59.6k ± 0%: delta: 0.00%
TmAnyTrueFuncall-4: old 277 ± 0%: new 277 ± 0%: delta: 0.00%
TmTernary-4: old 28.0 ± 0%: new 28.0 ± 0%: delta: 0.00%
TmTryUnknownFunc-4: old 21.0 ± 0%: new 21.0 ± 0%: delta: 0.00%
TmTryUnknownVariable-4: old 20.0 ± 0%: new 20.0 ± 0%: delta: 0.00%
TmTryUnknownObjectKey-4: old 23.0 ± 0%: new 23.0 ± 0%: delta: 0.00%

@i4ki i4ki force-pushed the i4k-parallel-generate branch 2 times, most recently from d4d188d to b67ed7d Compare October 22, 2024 13:30
Signed-off-by: i4k <t.nateldemoura@gmail.com>
Copy link

github-actions bot commented Oct 22, 2024

Preview of ubuntu-focal/go1.21 tests in 63c22da

🔍 View Details on Terramate Cloud

.
cmd/terramate/cli
config
generate
modvendor/download
stdlib
test/sandbox

Copy link

github-actions bot commented Oct 22, 2024

Preview of macos-ventura/go1.21 tests in 63c22da

🔍 View Details on Terramate Cloud

.
cmd/terramate/cli
config
generate
modvendor/download
stdlib
test/sandbox

Signed-off-by: i4k <t.nateldemoura@gmail.com>
Signed-off-by: i4k <t.nateldemoura@gmail.com>
@i4ki i4ki marked this pull request as ready for review October 22, 2024 16:00
soerenmartius
soerenmartius previously approved these changes Oct 22, 2024
Copy link
Contributor

@soerenmartius soerenmartius left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome, shall we get this documented by @AnnuCode ?

Signed-off-by: i4k <t.nateldemoura@gmail.com>
Copy link
Contributor

@soerenmartius soerenmartius left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@i4ki
Copy link
Contributor Author

i4ki commented Oct 22, 2024

@soerenmartius yes, please!

@soerenmartius soerenmartius self-requested a review October 23, 2024 16:42
@i4ki i4ki merged commit c309ba3 into main Oct 24, 2024
7 checks passed
@i4ki i4ki deleted the i4k-parallel-generate branch October 24, 2024 09:12
i4ki added a commit that referenced this pull request Oct 24, 2024
…ation (#1943)

## What this PR does / why we need it:

The in-place AST modifications were a nice optimization when generation
was serial but with the introduction of #1937 it's racy.

## Which issue(s) this PR fixes:
none
## Special notes for your reviewer:

## Does this PR introduce a user-facing change?
```
no
```
i4ki added a commit that referenced this pull request Oct 24, 2024
Depends on #1937 and #1943.
Please release this RC once they are merged!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants