Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PythonMutator: propagate source locations #1783

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

kanterov
Copy link
Contributor

@kanterov kanterov commented Sep 20, 2024

Changes

Add a mechanism to load Python source locations in the Python mutator. Previously, locations pointed to generated YAML. Now, they point to Python sources instead. Python process outputs "locations.json" containing locations of bundle paths, examples:

{"path": "resources.jobs.job_0", "file": "resources/job_0.py", "line": 3, "column": 5}
{"path": "resources.jobs.job_0.tasks[0].task_key", "file": "resources/job_0.py", "line": 10, "column": 5}
{"path": "resources.jobs.job_1", "file": "resources/job_1.py", "line": 5, "column": 7}

Such locations form a tree, and we assign locations of the closest ancestor to each dyn.Value based on its path. For example, resources.jobs.job_0.tasks[0].task_key is located at job_0.py:10:5 and resources.jobs.job_0.tasks[0].email_notifications is located at job_0.py:3:5, because we use the location of the job as the most precise approximation.

This feature is only enabled if experimental/python is used.

Note: for now, we don't update locations with relative paths, because it has a side effect in changing how these paths are resolved

Example

% databricks bundle validate

Warning: job_cluster_key abc is not defined
  at resources.jobs.examples.tasks[0].job_cluster_key
  in resources/example.py:10:1

Tests

Unit tests and manually

@kanterov kanterov requested a review from pietern September 20, 2024 16:09
Copy link
Contributor

@pietern pietern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

The line protocol in the locations file looks fine.

bundle/config/experimental.go Outdated Show resolved Hide resolved
bundle/config/mutator/python/python_locations.go Outdated Show resolved Hide resolved
bundle/config/mutator/python/python_locations.go Outdated Show resolved Hide resolved
bundle/config/mutator/python/python_locations.go Outdated Show resolved Hide resolved
bundle/config/mutator/python/python_locations.go Outdated Show resolved Hide resolved

currentNode.location = location
currentNode.exists = true
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this can be a member function of the type.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked into changing but it doesn't make it obviously better. putPythonLocation/findPythonLocation is not intended to be used outside of merging code, so if we could limit visibility, we would only export mergePythonLocations and parsePythonLocations

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack, let's leave it

bundle/config/mutator/python/python_mutator.go Outdated Show resolved Hide resolved
@kanterov kanterov force-pushed the kanterov/python-locations branch from def4744 to 43ce278 Compare October 8, 2024 08:24
@kanterov kanterov force-pushed the kanterov/python-locations branch from 96a6cef to d9bf157 Compare January 8, 2025 15:12
Copy link

github-actions bot commented Jan 8, 2025

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

  • PR number: 1783
  • Commit SHA: 4ec0f52b10b1fe961be757704db33fca8e981d1d

Checks will be approved automatically on success.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants