Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enable oneflow_iree(python package) and corresponding test works in ci #8431

Merged
merged 52 commits into from
Jun 22, 2022
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
2a8da82
update test.yml
howin98 Jun 16, 2022
21c39f6
add pytest for oneflow_iree examples
howin98 Jun 16, 2022
28a8f4e
Merge branch 'master' of github.com:Oneflow-Inc/oneflow into support-…
howin98 Jun 17, 2022
76e5872
add oneflow frontend test
howin98 Jun 17, 2022
142a15f
abs path for oneflow_src
howin98 Jun 17, 2022
bb61267
push
howin98 Jun 17, 2022
84fecaa
update llvm version
howin98 Jun 20, 2022
b0977aa
auto format by CI
oneflow-ci-bot Jun 20, 2022
544ab0f
remove iree test python script
howin98 Jun 20, 2022
99afce5
Merge branch 'support-iree-python-ci' of github.com:Oneflow-Inc/onefl…
howin98 Jun 20, 2022
21acbde
rm line
howin98 Jun 20, 2022
2eae04d
pub
howin98 Jun 20, 2022
8020155
fix
jackalcooper Jun 20, 2022
c8847fe
auto format by CI
oneflow-ci-bot Jun 20, 2022
687739e
refine
jackalcooper Jun 20, 2022
ba0ffbc
add node
jackalcooper Jun 20, 2022
b6b263d
auto format by CI
oneflow-ci-bot Jun 20, 2022
db72a13
Merge branch 'master' into support-iree-python-ci
jackalcooper Jun 20, 2022
d559774
Merge branch 'master' into support-iree-python-ci
mergify[bot] Jun 20, 2022
fa72e66
pub
howin98 Jun 20, 2022
ce292be
pub
howin98 Jun 20, 2022
eace554
Merge branch 'master' into support-iree-python-ci
mergify[bot] Jun 20, 2022
d568015
pub
howin98 Jun 21, 2022
aabb99c
Merge branch 'support-iree-python-ci' of github.com:Oneflow-Inc/onefl…
howin98 Jun 21, 2022
e5b4f31
pub
howin98 Jun 21, 2022
fa97852
hardcoded proxy
jackalcooper Jun 21, 2022
db05558
refine code
jackalcooper Jun 21, 2022
7179924
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow into …
jackalcooper Jun 21, 2022
38d8d71
pub
howin98 Jun 21, 2022
f453af3
pub
howin98 Jun 21, 2022
5acd5fa
revert
jackalcooper Jun 21, 2022
f9a91d3
try fix
jackalcooper Jun 21, 2022
accf210
refine
jackalcooper Jun 21, 2022
41e34fc
fix
jackalcooper Jun 21, 2022
163e28a
pub
howin98 Jun 21, 2022
a72fd73
revert
howin98 Jun 21, 2022
0d59646
revert
howin98 Jun 21, 2022
53b0c13
Update test.yml
howin98 Jun 21, 2022
cade2c8
pub
howin98 Jun 21, 2022
e9e6d04
Merge branch 'support-iree-python-ci' of github.com:Oneflow-Inc/onefl…
howin98 Jun 21, 2022
e11a3c1
pub
howin98 Jun 21, 2022
302f7ed
larger timeut
jackalcooper Jun 21, 2022
c9d7ed8
mv code to safer place
jackalcooper Jun 21, 2022
a547cf7
rm useless print
jackalcooper Jun 21, 2022
b09eacb
fix
jackalcooper Jun 21, 2022
0a25013
pub
howin98 Jun 21, 2022
92132bb
add returns
jackalcooper Jun 21, 2022
9c4b14e
fix crash when dialect type printer not found
jackalcooper Jun 22, 2022
5b1d691
fix
jackalcooper Jun 22, 2022
e355174
Merge branch 'master' into support-iree-python-ci
jackalcooper Jun 22, 2022
40c0608
pub
howin98 Jun 22, 2022
8bdb59f
Merge branch 'support-iree-python-ci' of github.com:Oneflow-Inc/onefl…
howin98 Jun 22, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ env:
FLOW_VISION_COMMIT: ca8ebc663b58667cf8cd1b6ef0c861522780b7bb
LIBAI_SRC: libai
LIBAI_COMMIT: 7d31d9781e5f2d559dc0820f599e0bed798488ca
ONEFLOW_IREE_SRC: oneflow_iree
ONEFLOW_IREE_COMMIT: bec7304ca74b76e550b864e90202d7520a5759e8
TEST_WITH_TORCH_IMG_TAG: registry.cn-beijing.aliyuncs.com/oneflow/test-with-pytorch-1.10.0-cuda11.3-cudnn8-runtime:afaf913e02a4ba02db92260daee22f99121cef62
MLIR_DOCKER_ARGS: "-e ONEFLOW_MLIR_ENABLE_ROUND_TRIP=1 -e ONEFLOW_MLIR_PREFER_NHWC=0 -e ONEFLOW_MLIR_ENABLE_INFERENCE_OPTIMIZATION=1"

Expand Down Expand Up @@ -456,6 +458,14 @@ jobs:
# please use a commit here
ref: ${{ env.LIBAI_COMMIT}}
path: ${{ env.LIBAI_SRC}}
- name: Checkout Oneflow-Inc/oneflow_iree
if: ${{ !fromJson(matrix.cache-hit) && contains(matrix.runs-on, 'self-hosted') }}
uses: actions/checkout@v2
with:
repository: Oneflow-Inc/oneflow_iree
# please use a commit here
ref: ${{ env.ONEFLOW_IREE_COMMIT}}
path: ${{ env.ONEFLOW_IREE_SRC}}
- name: Remove container
timeout-minutes: 45
if: ${{ contains(matrix.runs-on, 'self-hosted') }}
Expand Down Expand Up @@ -560,6 +570,7 @@ jobs:
docker exec ${TEST_CONTAINER_NAME} python3 -m pip install -e ${{ env.FLOW_VISION_SRC}}
docker exec ${TEST_CONTAINER_NAME} python3 -m pip install pybind11 --user
docker exec ${TEST_CONTAINER_NAME} python3 -m pip install -e ${{ env.LIBAI_SRC}}
docker exec ${TEST_CONTAINER_NAME} python3 -m pip install -e ${{ env.ONEFLOW_IREE_SRC}}
- name: Module API test (distributed)
timeout-minutes: 90
if: ${{ !fromJson(matrix.cache-hit) && matrix.test-type == 'module' && matrix.device == 'cuda' && fromJson(matrix.is-distributed) }}
Expand Down Expand Up @@ -649,6 +660,14 @@ jobs:
# please use a commit here
ref: ${{ env.LIBAI_COMMIT}}
path: ${{ env.LIBAI_SRC}}
- name: Checkout Oneflow-Inc/oneflow_iree
if: ${{ !fromJson(matrix.cache-hit) && contains(matrix.runs-on, 'self-hosted') }}
uses: actions/checkout@v2
with:
repository: Oneflow-Inc/oneflow_iree
# please use a commit here
ref: ${{ env.ONEFLOW_IREE_COMMIT}}
path: ${{ env.ONEFLOW_IREE_SRC}}
- name: Remove container
timeout-minutes: 45
if: ${{ contains(matrix.runs-on, 'self-hosted') }}
Expand Down Expand Up @@ -782,6 +801,7 @@ jobs:
docker exec ${TEST_CONTAINER_NAME} python3 -m pip install -e ${{ env.FLOW_VISION_SRC}}
docker exec ${TEST_CONTAINER_NAME} python3 -m pip install pybind11 --user
docker exec ${TEST_CONTAINER_NAME} python3 -m pip install -e ${{ env.LIBAI_SRC}}
docker exec ${TEST_CONTAINER_NAME} python3 -m pip install -e ${{ env.ONEFLOW_IREE_SRC}}
- name: Run OneFlow doctor
if: ${{ !fromJson(matrix.cache-hit) && contains(matrix.runs-on, 'self-hosted') }}
run: |
Expand Down Expand Up @@ -884,6 +904,11 @@ jobs:
docker exec -e ONEFLOW_TEST_DEVICE_NUM=4 -w $PWD/${{ env.LIBAI_SRC }} ${{ env.TEST_CONTAINER_NAME }} python3 -m oneflow.distributed.launch --nproc_per_node 4 -m unittest -f tests/models/test_gpt.py
docker exec -e ONEFLOW_TEST_DEVICE_NUM=4 -w $PWD/${{ env.LIBAI_SRC }} ${{ env.TEST_CONTAINER_NAME }} python3 -m oneflow.distributed.launch --nproc_per_node 4 -m unittest -f tests/models/test_t5.py
docker exec -e ONEFLOW_TEST_DEVICE_NUM=4 -w $PWD/${{ env.LIBAI_SRC }} ${{ env.TEST_CONTAINER_NAME }} python3 -m oneflow.distributed.launch --nproc_per_node 4 -m unittest -f tests/models/test_vit.py
- name: oneflow_iree test
timeout-minutes: 45
if: ${{ !fromJson(matrix.cache-hit) && matrix.test-type == 'misc' }}
run: |
docker exec -w $PWD/${{ env.ONEFLOW_IREE_SRC }} ${{ env.TEST_CONTAINER_NAME }} python3 -m pytest examples
howin98 marked this conversation as resolved.
Show resolved Hide resolved
- name: Expensive tests (models, cases require exclusive access to GPU)
timeout-minutes: 45
if: ${{ !fromJson(matrix.cache-hit) && (matrix.test-type == 'speed-test' || (matrix.test-type == 'misc' && matrix.device == 'cpu')) && !fromJson(matrix.is-distributed) }}
Expand Down
7 changes: 5 additions & 2 deletions cmake/oneflow.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -250,18 +250,21 @@ if("${LLVM_MONO_REPO_URL}" STREQUAL
"https://github.com/llvm/llvm-project/archive/7eaa84eac3ba935d13f4267d3d533a6c3e1283ed.zip"
OR "${LLVM_MONO_REPO_URL}" STREQUAL
"https://github.com/llvm/llvm-project/archive/35e60f5de180aea55ed478298f4b40f04dcc57d1.zip"
OR "${LLVM_MONO_REPO_URL}" STREQUAL
"https://github.com/llvm/llvm-project/archive/6a9bbd9f20dcd700e28738788bb63a160c6c088c.zip"
OR "${LLVM_MONO_REPO_MD5}" STREQUAL "f2f17229cf21049663b8ef4f2b6b8062"
OR "${LLVM_MONO_REPO_MD5}" STREQUAL "6b7c6506d5922de9632c8ff012b2f945"
OR "${LLVM_MONO_REPO_MD5}" STREQUAL "e0ea669a9f0872d35bffda5ec6c5ac6f"
OR "${LLVM_MONO_REPO_MD5}" STREQUAL "241a333828bba1efa35aff4c4fc2ce87"
OR "${LLVM_MONO_REPO_MD5}" STREQUAL "075fbfdf06cb3f02373ea44971af7b03")
unset(LLVM_MONO_REPO_URL CACHE)
unset(LLVM_MONO_REPO_MD5 CACHE)
endif()
set(LLVM_MONO_REPO_URL
"https://github.com/llvm/llvm-project/archive/6a9bbd9f20dcd700e28738788bb63a160c6c088c.zip"
"https://github.com/llvm/llvm-project/archive/32805e60c9de1f82887cd2af30d247dcabd2e1d3.zip"
CACHE STRING "")
use_mirror(VARIABLE LLVM_MONO_REPO_URL URL ${LLVM_MONO_REPO_URL})
set(LLVM_MONO_REPO_MD5 "241a333828bba1efa35aff4c4fc2ce87" CACHE STRING "")
set(LLVM_MONO_REPO_MD5 "e412dc61159b5e929b0c94e44b11feb2" CACHE STRING "")
set(ONEFLOW_BUILD_ROOT_DIR "${PROJECT_BINARY_DIR}")
add_subdirectory(${PROJECT_SOURCE_DIR}/oneflow/ir)
if(WITH_MLIR)
Expand Down
31 changes: 0 additions & 31 deletions oneflow/ir/include/OneFlow/Conversion/SCFToGPU.h

This file was deleted.

6 changes: 0 additions & 6 deletions oneflow/ir/include/OneFlow/OneFlowOps.td
Original file line number Diff line number Diff line change
Expand Up @@ -288,12 +288,6 @@ def LowerOneFlowToTosaPass : Pass<"lower-oneflow-to-tosa", "ModuleOp"> {
];
}

def MapSCFToGPUPass : Pass<"gpu-greedy-parallel-loop-mapping", "ModuleOp"> {
let summary = "Greedily maps all parallel loops to gpu hardware ids";
let constructor = "mlir::oneflow::createMapSCFToGPUPass()";
let dependentDialects = ["scf::SCFDialect"];
}

def BufferHostRegisterPass : Pass<"buffer-host-register", "func::FuncOp"> {
let summary = "";
let constructor = "mlir::oneflow::createBufferHostRegisterPass()";
Expand Down
2 changes: 1 addition & 1 deletion oneflow/ir/include/OneFlow/OneFlowPatterns.td
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
include "mlir/IR/PatternBase.td"
include "OneFlow/OneFlowOps.td"
include "mlir/Dialect/MemRef/IR/MemRefOps.td"
include "mlir/Dialect/GPU/GPUOps.td"
include "mlir/Dialect/GPU/IR/GPUOps.td"

def IsNotNestedInJit: Constraint<CPred<"($0.getDefiningOp()->getParentOfType<::mlir::oneflow::Job>())">, "">;
def IsScalarTensor: Constraint<CPred<"::mlir::oneflow::IsScalarTensor($0)">, "">;
Expand Down
3 changes: 1 addition & 2 deletions oneflow/ir/include/OneFlow/Passes.h
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,12 @@ limitations under the License.
#include "mlir/Dialect/MemRef/IR/MemRef.h"
#include "mlir/Dialect/Tosa/IR/TosaOps.h"
#include "mlir/Dialect/SCF/SCF.h"
#include "mlir/Dialect/GPU/GPUDialect.h"
#include "mlir/Dialect/GPU/IR/GPUDialect.h"
#include "mlir/Dialect/LLVMIR/NVVMDialect.h"
#include "mlir/Dialect/Linalg/IR/Linalg.h"
#include "mlir/Pass/Pass.h"
#include "mlir/Dialect/Func/IR/FuncOps.h"
#include "OneFlow/Conversion/OneFlowToTosa.h"
#include "OneFlow/Conversion/SCFToGPU.h"
#include "OneFlow/Transform/BufferHostRegister.h"
#include "OneFlow/Transform/ConvertInferenceOp.h"
#include "OneFlow/Transform/OutlineAndFuse.h"
Expand Down
1 change: 1 addition & 0 deletions oneflow/ir/install-llvm.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ if(NOT llvm_monorepo_POPULATED)
execute_process(
COMMAND
"${CMAKE_COMMAND}" ${llvm_monorepo_SOURCE_DIR}/llvm
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} # this is required in newer version of LLVM
-DCMAKE_C_COMPILER_LAUNCHER=${CMAKE_C_COMPILER_LAUNCHER}
-DCMAKE_CXX_COMPILER_LAUNCHER=${CMAKE_CXX_COMPILER_LAUNCHER}
-DCMAKE_CUDA_COMPILER_LAUNCHER=${CMAKE_CUDA_COMPILER_LAUNCHER}
Expand Down
4 changes: 2 additions & 2 deletions oneflow/ir/lib/OneFlow/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
message(STATUS "MLIR_DIALECT_LIBS: ${dialect_libs}")
if(WITH_MLIR_CUDA_CODEGEN)
set(MLIR_GPU_LIBS MLIRSCFToGPU MLIRGPUToNVVMTransforms MLIRNVVMToLLVMIRTranslation)
set(MLIR_GPU_LIBS MLIRGPUToNVVMTransforms MLIRNVVMToLLVMIRTranslation)
endif(WITH_MLIR_CUDA_CODEGEN)

set(ONEFLOW_OP_GROUPS
Expand All @@ -24,7 +24,6 @@ oneflow_add_mlir_dialect_library(
OneFlowSupport.cpp
OneFlowOpFolders.cpp
Conversion/OneFlowToTosa.cpp
Conversion/SCFToGPU.cpp
Conversion/PTXToCubin.cpp
Transform/BufferHostRegister.cpp
Transform/OutlineAndFuse.cpp
Expand All @@ -43,6 +42,7 @@ oneflow_add_mlir_dialect_library(
MLIRTosaToLinalg
MLIRMemRefToLLVM
MLIRLinalgToLLVM
MLIRSCFToGPU
MLIRReconcileUnrealizedCasts
${MLIR_GPU_LIBS}
MLIRIR
Expand Down
2 changes: 1 addition & 1 deletion oneflow/ir/lib/OneFlow/Conversion/PTXToCubin.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ limitations under the License.
This file is ported from mlir/lib/Dialect/GPU/Transforms/SerializeToCubin.cpp
*/

#include "mlir/Dialect/GPU/Passes.h"
#include "mlir/Dialect/GPU/Transforms/Passes.h"
#ifdef WITH_MLIR_CUDA_CODEGEN

#include "mlir/Pass/Pass.h"
Expand Down
70 changes: 0 additions & 70 deletions oneflow/ir/lib/OneFlow/Conversion/SCFToGPU.cpp

This file was deleted.

10 changes: 6 additions & 4 deletions oneflow/ir/lib/OneFlow/Passes.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ limitations under the License.
#include "mlir/Conversion/AffineToStandard/AffineToStandard.h"
#include "mlir/Conversion/GPUCommon/GPUCommonPass.h"
#include "mlir/Conversion/GPUToNVVM/GPUToNVVMPass.h"
#include "mlir/Dialect/GPU/Passes.h"
#include "mlir/Dialect/GPU/Transforms/Passes.h"
#include "mlir/Conversion/SCFToGPU/SCFToGPUPass.h"
#endif // WITH_MLIR_CUDA_CODEGEN

Expand Down Expand Up @@ -769,9 +769,10 @@ LogicalResult LowerModuleToCUDALLVM(mlir::MLIRContext* context, ModuleOp module)
AddLowerToLinalgMemRefPasses(pm);
pm.addNestedPass<func::FuncOp>(
createConvertLinalgToParallelLoopsPass()); // convert-linalg-to-parallel-loops
pm.addPass(createMapSCFToGPUPass()); // gpu-greedy-parallel-loop-mapping
pm.addPass(createParallelLoopToGpuPass()); // convert-parallel-loops-to-gpu
pm.addPass(createGpuKernelOutliningPass()); // gpu-kernel-outlining
pm.addNestedPass<func::FuncOp>(createGpuMapParallelLoopsPass()); // gpu-map-parallel-loops
pm.addPass(createParallelLoopToGpuPass()); // convert-parallel-loops-to-gpu
pm.addPass(createGpuLauchSinkIndexComputationsPass());
pm.addPass(createGpuKernelOutliningPass()); // gpu-kernel-outlining
pm.addNestedPass<func::FuncOp>(createBufferHostRegisterPass()); // buffer-host-register
pm.addPass(createCanonicalizerPass()); // canonicalize
// -pass-pipeline='gpu.module([PASS1][PASS2]...)'
Expand All @@ -781,6 +782,7 @@ LogicalResult LowerModuleToCUDALLVM(mlir::MLIRContext* context, ModuleOp module)
pm.addNestedPass<gpu::GPUModuleOp>(createSerializeToCubinPass()); // out-of-tree-gpu-to-cubin
pm.addNestedPass<func::FuncOp>(createGpuCopyArgPass()); // buffer-host-register
pm.addPass(createGpuToLLVMConversionPass());
pm.addPass(createReconcileUnrealizedCastsPass()); // reconcile-unrealized-casts
if (enable_ir_printing) pm.enableIRPrinting();
return pm.run(module);
}
Expand Down
2 changes: 1 addition & 1 deletion oneflow/ir/oneflow-extension/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ oneflow_add_mlir_library(
MLIRIR
MLIRParser
MLIRPass
MLIRSPIRV
MLIRSPIRVDialect
MLIRTranslateLib
MLIRSupport
MLIROneFlow
Expand Down
2 changes: 1 addition & 1 deletion oneflow/ir/oneflow-opt/oneflow-opt.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ int32_t main(int32_t argc, char** argv) {
mlir::registerAllPasses();
mlir::registerTestOneFlowTraitsPass();
mlir::registerLowerOneFlowToTosaPassPass();
mlir::registerMapSCFToGPUPassPass();
mlir::registerGpuMapParallelLoopsPassPass();
mlir::registerBufferHostRegisterPassPass();
mlir::registerGpuCopyArgPassPass();
#ifdef WITH_MLIR_CUDA_CODEGEN
Expand Down
2 changes: 1 addition & 1 deletion oneflow/ir/oneflow-runner/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ target_link_libraries(
MLIRExecutionEngine
MLIRIR
MLIRJitRunner
MLIRLLVMIR
MLIRLLVMIRTransforms
MLIRLLVMToLLVMIRTranslation
MLIRToLLVMIRTranslationRegistration
MLIRParser
Expand Down
2 changes: 1 addition & 1 deletion oneflow/ir/oneflow-translate/lib/OneFlow/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ oneflow_add_mlir_library(
MLIRIR
MLIRParser
MLIRPass
MLIRSPIRV
MLIRSPIRVDialect
MLIRTranslateLib
MLIRSupport
MLIROneFlow
Expand Down
Loading