[To merge AFTER flashinfer-ci changes updated] Reduce test time by moving compilation off-line #2089
Conversation
WalkthroughAdds a non-dry-run initialization block to Changes
Sequence Diagram(s)sequenceDiagram
participant Script as task_test_blackwell_kernels.sh
participant Env as Environment
participant FS as Filesystem (dist/)
participant Pip as pip
participant Python as python
rect rgb(230,240,255)
Note over Script: Start (only if DRY_RUN unset)
Script->>Env: Check DRY_RUN
alt DRY_RUN is not set
Script->>Env: Read & echo CUDA_VERSION
Script->>Script: Set DIST_CUBIN_DIR = dist/${CUDA_VERSION}/cubin
Script->>Script: Set DIST_JIT_CACHE_DIR = dist/${CUDA_VERSION}/jit-cache
else DRY_RUN set
Script->>Env: Exit / skip install
end
end
rect rgb(220,255,230)
Note over Script,FS: Attempt install of prebuilt wheels (warn if missing)
Script->>FS: Check DIST_CUBIN_DIR for wheels
alt cubin wheels present
Script->>Pip: pip install <wheels from DIST_CUBIN_DIR>
Pip-->>Script: success
else missing
Script-->>Script: emit warning and continue
end
Script->>FS: Check DIST_JIT_CACHE_DIR for wheels
alt jit-cache wheels present
Script->>Pip: pip install <wheels from DIST_JIT_CACHE_DIR>
Pip-->>Script: success
else missing
Script-->>Script: emit warning and continue
end
end
rect rgb(255,245,220)
Note over Script,Python: Install & verify local package
Script->>Pip: pip install -e . -v --no-deps
Pip-->>Script: installation complete
Script->>Python: (cd /tmp && python -m flashinfer show-config)
Python-->>Script: configuration output / verification
end
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes
Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
Tip 📝 Customizable high-level summaries are now available in beta!You can now customize how CodeRabbit generates the high-level summary in your pull requests — including its content, structure, tone, and formatting.
Example:
Note: This feature is currently in beta for Pro-tier users, and pricing will be announced later. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
kahyunnam
force-pushed
the
knam/unit-testing-move-compilation-offline
branch
from
c9c6768 to
375ca18
Compare
|
/bot run |
kahyunnam
requested review from
bkryu,
nvmbreughe,
yongwww and
yzh119
as code owners
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
scripts/task_test_blackwell_kernels.sh (2)
41-50: Inconsistent verbosity flags in sequential pip installations.Lines 43 and 45 use
-q(quiet) for kernel installations, while line 49 uses-v(verbose) for local source installation. This inconsistency makes it unclear whether the verbosity change is intentional and may make output harder to parse in CI logs.Standardize the verbosity flags across all pip installations in this initialization block:
# Install precompiled kernels echo "Installing flashinfer-cubin from PyPI/index..." - pip install -q flashinfer-cubin + pip install -q flashinfer-cubin echo "Installing flashinfer-jit-cache for ${CUDA_STREAM} from https://flashinfer.ai/whl/${CUDA_STREAM} ..." - pip install -q --extra-index-url "https://flashinfer.ai/whl/${CUDA_STREAM}" flashinfer-jit-cache + pip install -q --extra-index-url "https://flashinfer.ai/whl/${CUDA_STREAM}" flashinfer-jit-cache echo "" # Install local python sources - pip install -e . -v --no-deps + pip install -e . -q --no-depsAlternatively, if verbose output is intentional for debugging local installs, add a comment explaining the choice.
41-50: Verify that the custom PyPI index URL forflashinfer-jit-cacheis reliable.The script hardcodes the index URL
https://flashinfer.ai/whl/${CUDA_STREAM}and expects it to always be available and contain theflashinfer-jit-cachepackage for the detected CUDA stream. If this URL becomes unavailable or if a CUDA stream version is not published, the pip install will fail and halt all subsequent tests.Add error handling and diagnostics to surface issues clearly:
echo "Installing flashinfer-jit-cache for ${CUDA_STREAM} from https://flashinfer.ai/whl/${CUDA_STREAM} ..." - pip install -q --extra-index-url "https://flashinfer.ai/whl/${CUDA_STREAM}" flashinfer-jit-cache + if ! pip install -q --extra-index-url "https://flashinfer.ai/whl/${CUDA_STREAM}" flashinfer-jit-cache; then + echo "❌ ERROR: Failed to install flashinfer-jit-cache for CUDA stream ${CUDA_STREAM}" + echo " Index URL: https://flashinfer.ai/whl/${CUDA_STREAM}" + exit 1 + fiCan you confirm that the custom index URL is stable and that all supported CUDA streams (cu128, cu129, cu130) are consistently published with the corresponding flashinfer-jit-cache package?
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
scripts/task_test_blackwell_kernels.sh(1 hunks)
🔇 Additional comments (1)
scripts/task_test_blackwell_kernels.sh (1)
52-55: Verify thatpython -m flashinfer show-configis an appropriate verification step.The verification runs
python -m flashinfer show-configto confirm successful installation. However, this assumes:
- The
show-configsubcommand exists in the flashinfer module- The command is idempotent and doesn't modify the environment
- The command completes quickly without external dependencies
If this command fails (e.g., due to missing dependencies, invalid environment, or a transient issue), the entire test run is aborted before any tests can run, which may be overly strict for a verification step.
Can you confirm:
- That
python -m flashinfer show-configis a lightweight, read-only command that verifies the installation without side effects?- What the expected output is and whether it should be validated beyond the exit code?
- Whether a failed verification should block all tests or only warn/skip?
|
[FAILED] Pipeline #38459095: 3/17 passed |
kahyunnam
changed the title
There was a problem hiding this comment.
Thanks @kahyunnam! Left a comment about the behavior with jit cache & cubin wheels are not found.
4 participants
📌 Description
Download
flashinfer-cubinandflashinfer-jit-cacheto avoid compilation. (Unless the JIT kernel is not in theflashinfer-jit-cache; then it will still JIT compile during test runtime. We could setexport FLASHINFER_DISABLE_JIT = 1to avoid this, but then it will "skip" a lot of tests that use JIT kernels that aren't found inflashinfer-jit-cache.)🔍 Related Issues
Issue was discussed on slack. "Ideally, we would move that compilation off-line which would reduce test time & make kernel hang detection much easier. "
🚀 Pull Request Checklist
Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete.
✅ Pre-commit Checks
pre-commitby runningpip install pre-commit(or used your preferred method).pre-commit install.pre-commit run --all-filesand fixed any reported issues.🧪 Tests
unittest, etc.).Summary by CodeRabbit