Skip to content

Commit

Permalink
Add --no-cache option to run_pyright script (#24212)
Browse files Browse the repository at this point in the history
## Summary & Motivation

The `uv` cache can cause subtle problems sometimes when rebuilding
pyright environments. Rebuilding envs without using the cache can fix an
apparently intractable problem. This PR adds `--no-cache` as an arg to
`run-pyright.py`. This is intended for rare use as an escape hatch for
pyright env problems-- it makes rebuilding the envs much more slow.

See this issue I posted over at `uv` for some info about why you might
want to use this: astral-sh/uv#7028

## How I Tested These Changes

My pyright was (unresolvable imports for all dagster packages). After
rebuilding my env with `--no-cache` the problem went away. The issue was
that the cache seemed to be holding `pth` files for our dagster
editables that did not respect `editable_mode=compat` and were
incorrectly formatted for pyright.

## Changelog [New | Bug | Docs]

NOCHANGELOG
  • Loading branch information
smackesey authored Sep 4, 2024
1 parent 5fd25f0 commit de3b1af
Show file tree
Hide file tree
Showing 2 changed files with 18 additions and 17 deletions.
13 changes: 0 additions & 13 deletions pyright/master/requirements-pinned.txt
Original file line number Diff line number Diff line change
Expand Up @@ -375,18 +375,6 @@ noteable-origami==1.1.5
notebook==7.2.2
notebook-shim==0.2.4
numpy==1.26.4
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.6.20
nvidia-nvtx-cu12==12.1.105
oauth2client==4.1.3
oauthlib==3.2.2
objgraph==3.6.1
Expand Down Expand Up @@ -588,7 +576,6 @@ tqdm==4.66.5
traitlets==5.14.3
trio==0.26.2
trio-websocket==0.11.1
triton==3.0.0
-e examples/experimental/dagster-airlift/examples/tutorial-example
-e examples/tutorial_notebook_assets
twilio==9.2.4
Expand Down
22 changes: 18 additions & 4 deletions scripts/run-pyright.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,13 @@
help="Output results in JSON format.",
)

parser.add_argument(
"--no-cache",
action="store_true",
default=False,
help="If rebuilding pyright environments, do not use the uv cache. This is much slower but can be useful for debugging.",
)

parser.add_argument(
"--rebuild",
"-r",
Expand Down Expand Up @@ -120,6 +127,7 @@ class Params(TypedDict):
mode: Literal["env", "path"]
targets: Sequence[str]
json: bool
no_cache: bool
rebuild: bool
update_pins: bool
venv_python: str
Expand Down Expand Up @@ -232,6 +240,7 @@ def get_params(args: argparse.Namespace) -> Params:
json=args.json,
rebuild=args.rebuild,
unannotated=args.unannotated,
no_cache=args.no_cache,
venv_python=venv_python,
skip_typecheck=args.skip_typecheck,
)
Expand Down Expand Up @@ -273,7 +282,9 @@ def map_paths_to_envs(paths: Sequence[str]) -> Mapping[str, Sequence[str]]:
return env_path_map


def normalize_env(env: str, rebuild: bool, update_pins: bool, venv_python: str) -> None:
def normalize_env(
env: str, rebuild: bool, update_pins: bool, venv_python: str, no_cache: bool
) -> None:
venv_path = os.path.join(get_env_path(env), ".venv")
python_path = f"{venv_path}/bin/python"
if (rebuild or update_pins) and os.path.exists(venv_path):
Expand All @@ -299,13 +310,14 @@ def normalize_env(env: str, rebuild: bool, update_pins: bool, venv_python: str)
"install",
"--python",
python_path,
"-r",
dest_requirements_path,
# editable-mode=compat ensures dagster-internal editable installs are done
# in a way that is legible to pyright (i.e. not using import hooks). See:
# https://github.com/microsoft/pyright/blob/main/docs/import-resolution.md#editable-installs
"--config-settings",
"editable-mode=compat",
"-r",
dest_requirements_path,
"--no-cache" if no_cache else "",
*extra_pip_install_args,
]
),
Expand Down Expand Up @@ -520,7 +532,9 @@ def print_report(result: RunResult) -> None:
env_path_map = {env: None for env in params["targets"]}

for env in env_path_map:
normalize_env(env, params["rebuild"], params["update_pins"], params["venv_python"])
normalize_env(
env, params["rebuild"], params["update_pins"], params["venv_python"], params["no_cache"]
)
if params["skip_typecheck"]:
print("Successfully built environments. Skipping typecheck.")
elif len(env_path_map) == 0:
Expand Down

0 comments on commit de3b1af

Please sign in to comment.