Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Makefile performance improvements #1151

Merged
merged 29 commits into from
Nov 25, 2024
Merged

Makefile performance improvements #1151

merged 29 commits into from
Nov 25, 2024

Conversation

danielrbradley
Copy link
Member

@danielrbradley danielrbradley commented Nov 21, 2024

Allow targets to be up-to-date by using the "sentinel file" pattern where each unit of work is tracked via a file in the .make directory to indicate when it was last successfully completed. Replace dependencies on phony targets with these sentinel files.

Other improvements:

  • Add make schema target as a generic replacement for make tfgen.
  • Split out a generate phase from all the build sdk targets to allow skipping running the SDK builds locally.
  • Add make generate target for generating all things which need to be committed without having to wait on build steps.
  • Add make prepare_local_workspace which currently encompases installing plugins and preparing upstream.
  • Group .PHONY declarations near the definition of the phony target rather than in one very long line. This allows us to see if we're missing declarations for specific targets more easily.
  • Fix the .pulumi/version getting out of sync with the installed files.
  • Fix the clean target to include the bin and .make directories (remove the duplicate cleanup target alongside).
  • Replace the broken help target which hasn't been functional in many years AFAIK.
  • Move nuget setup for install_dotnet_sdk into the makefile from CI.
  • Use the new standard targets (schema & provider) instead of the internal targets (tfgen_build_only, tfgen_no_deps, provider_no_deps)
  • Save and restore makefile progress via artifacts (prerequisites.make and build_[language].make). This maintains file timestamps between jobs.

Commits are structued to address one target at a time for review and history.

Preview of new help text:

Usage: make [target]

Main Targets
  build (default)     Build the provider and all SDKs and install for testing
  generate            Generate all SDKs, documentation and schema
  provider            Build the local provider binary
  lint_provider<.fix> Run the linter on the provider (& optionally fix)
  test_provider       Run the provider tests
  test                Run the example tests (must run 'build' first)
  clean               Clean up generated files

More Precise Targets
  schema        Generate the schema
  generate_sdks Generate all SDKs
  build_sdks    Build all SDKs
  install_sdks  Install all SDKs
  provider_dist Build and package the provider for all platforms

Tool Targets
  ci-mgmt     Re-generate CI configuration from .ci-mgmt.yaml
  debug_tfgen Start a debug server for tfgen

Internal Targets (automatically run as dependencies of other targets)
  prepare_local_workspace  Prepare for building
  install_plugins          Install plugin dependencies
  upstream                 Initialize the upstream submodule, if present

Language-Specific Targets
  generate_[language]    Generate the SDK files ready for committing
  build_[language]       Build the SDK to check correctness
  install_[language]_sdk Install the SDK ready for testing

  [language] = nodejs python dotnet go java

@danielrbradley danielrbradley self-assigned this Nov 21, 2024
Base automatically changed from provider-ci-migrations to master November 22, 2024 08:47
Create a `.make` directory for tracking targets which don't generate a single file output. This should be ignored by git.
For targets which either don't generate a single file output, or the output file is committed, we use a "sentinel"
file within `.make/` to track the staleness of the target and only rebuild when needed.
For each phony target, we create an internal target with the same name, but prefixed with `.make/` where the work is performed.
At the end of each internal target we run `@touch $@` to update the file which is the name of the target.
- Depend on the binary rather than the phoney target so tfgen_no_deps can be up-to-date if we make it a file target too.
- Group tfgen .PHONY targets near where they're declared.
- Upstream depends on the patches and upstream HEAD
- Remove confusing whitespace around `XrunUpstreamTools` block.
- Move .PHONY entry adjacent to target.
Add dependency on sources files (including the upstream HEAD) so we rebuild when they're updated.

Add an alias of "schema" which is more generic that "tfgen" and can be standardised for all other providers too.
If the target it running but we're already on the right version, we still need to bump the timestamp to avoid re-running this whole check.
- Remove "cleanup" so we only have "clean" for consistency.
- The schema.go hasn't existed for a while now so can be removed.
It doesn't output anything useful. More comments & better structure would probably be more helpful.
- Add dependency on bin/$(TFGEN).
If we make the file target depend on _no_deps, it will never be up-to-date. If we make _no_deps depend on the file target it will pull in all dependencies. Instead we extract the build provider command to a variable so it's re-used from both the file target and _no_deps versions.
- Separate generate from build for an option for faster local iteration. We don't actually need to run the full SDK build to be able to commit and push to CI.
- Extract env vars to a makefile variable to reduce duplication and improve visibility in the logs. Don't include the PATH in the logs though as that's too noisy.
- Auto-generate generate and install_sdks aliases from supported languages.
- Group all install_*_sdk targets together after the build_* targets.
- Move .PHONY declaration next to the target definition.
Let's keep this consistent and make it not the first thing you read in each block.
blampe added a commit to pulumi/pulumi-artifactory that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-aws that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This PR removes the `make upstream` step from `preTest` and updates
`examples/webserver-go/go.mod` to no longer break `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-tailscale that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-sdwan that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-hcloud that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-azuredevops that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-harness that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-digitalocean that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-datadog that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-meraki that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-postgresql that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-gitlab that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-cloudflare that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-dbtcloud that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-okta that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-ise that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-cloudngfwaws that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-newrelic that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-random that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-scm that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-pagerduty that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-auth0 that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-azure that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-snowflake that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-gcp that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-vault that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-ise that referenced this pull request Dec 17, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-artifactory that referenced this pull request Dec 18, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
blampe added a commit to pulumi/pulumi-junipermist that referenced this pull request Dec 18, 2024
After pulumi/ci-mgmt#1151 our `make` targets
should now all correctly call `make upstream` if they need it.

This updates `.ci-mgmt.yaml` and re-runs `make ci-mgmt`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants