-
Notifications
You must be signed in to change notification settings - Fork 793
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Implement all User Ops with Op Schema (#7032)
* add oneflow-tblgen: generate op schema (OpInterpCtx) from ods * cmake: add inja * tblgen: add oneflow_datatype * tblgen: use option cat * tblgen: fix error * tblgen: put impl in .cpp * tblgen: fix null attrs * tblgen: fix null ops * refine * refine * reifne * Refine op schema template and compilation * add base OpInterpCtx to finish compilation * fix * refine * fix * add custom infer code * generate op registrants automatically * refine * fix * update user op ods and fix shape attr * refine * refine * add custom code in op base * refine comments * add same_output_regst_num and infer * support declare hasxx * update op schema emitter * refine * emit output regist num * refine * refine * migrate acc op * migrate onerec_reader, ones_like, send, pack and padding ops * add has_sbp_signature_infer_fn * refine * migrate pad, parallel_cast, partial_fc and pooling ops * rm redundant has_device_infer_fn * migrate prelu, quantization, randperm, reduce and repeat ops * migrate reshape, reshape_like, roi_align, same_pad, selu and scalar related ops * back port * backport * migrate ops * refine * refine * refine * refine * add new op * fix llvm not found * fix mlir headers * fix mlir headers * fix llvm not found * irefine * mark override * fix merge * fix * fix * set op schema as obj lib to speed up * rewrite ops * add addn * add grdi * refien * add more def (#7051) * affine grid * refien * refine * refine * refine * fix * refien * refine * refine * refine * refine * refine * refien * refine * refine * refein * refine * refine * refine * refine * refien * refine * refine * refine * refien * refien * refien * refine * refine * refien * refine * refine * refine * refein * refine * refine * refine * refine * refine * refien * refine * refine * refine * refine * refine * refine * refine * refine * refine * refine * refine * refine * refine * refine * refine * refine * refine * refein * refine * refine * refine * move more ops * fix math_binary_broadcast/elementwise_ops * fix hardtanh * add norm * rename file and add CpuOnly no_grad * fix ir & fix norm op * fix oneflow-tblgen * fix math_unary_elementwise_op * fix norm * fix bn * fix op schema * refine * fix * refine physical_tensor_desc_infer_fn * refine * add ScalarLogicalNotEqualOp & RecvOp * refine * auto format by CI * fix fmt * add cuda only trait * delete unused inja * del inja_copy_headers_to_destination * delete unused inja * del inja_copy_headers_to_destination * add cuda only to tblgen * fix json inja url and md5 not used * fix json inja url and md5 not used * refine * revert * add with cuda * refine * delete GenUserOpODS * remove cuda only * revert cuda only after meeting * fix Co-authored-by: PragmaTwice <i@twice.moe> Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
- Loading branch information
4 people
authored
Dec 24, 2021
1 parent
f278606
commit 120ecad
Showing
236 changed files
with
22,777 additions
and
13,222 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| add_subdirectory(test) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| set(PYTHON_EXECUTABLE python3 CACHE STRING "python3 exe to run test, usually is the python3 installation oneflow is linked to") | ||
| set(ONEFLOW_SRC_DIR ${CMAKE_SOURCE_DIR} CACHE STRING "source dir of oneflow") | ||
| set(IS_DEV ON CACHE BOOL "") | ||
| set(CTEST_RESOURCE_SPEC_FILE "${CMAKE_CURRENT_SOURCE_DIR}/resource-spec/2x-rtx-2080.json" CACHE STRING "") | ||
|
|
||
| # CTEST_OUTPUT_ON_FAILURE=1 CTEST_PARALLEL_LEVEL=20 ninja test | ||
|
|
||
| file(GLOB_RECURSE PYTHON_TEST_FILES LIST_DIRECTORIES false RELATIVE ${ONEFLOW_SRC_DIR} "${ONEFLOW_SRC_DIR}/python/oneflow/test_*.py") | ||
| foreach(PYTHON_TEST_FILE ${PYTHON_TEST_FILES}) | ||
| set(TEST_NAME ${PYTHON_TEST_FILE}) | ||
| add_test(NAME ${TEST_NAME} | ||
| COMMAND ${PYTHON_EXECUTABLE} ${ONEFLOW_SRC_DIR}/${PYTHON_TEST_FILE} --failfast --verbose | ||
| ) | ||
| set_tests_properties(${TEST_NAME} | ||
| PROPERTIES | ||
| ENVIRONMENT "$<$<NOT:$<BOOL:${BUILD_CUDA}>>:ONEFLOW_TEST_CPU_ONLY=1>;$<$<BOOL:${IS_DEV}>:PYTHONPATH=${ONEFLOW_SRC_DIR}/python:$ENV{PYTHONPATH}>" | ||
| RESOURCE_GROUPS | ||
| "vram:2000" | ||
| ) | ||
| endforeach() | ||
| set_tests_properties(python/oneflow/test/modules/test_rnn.py | ||
| PROPERTIES | ||
| RESOURCE_GROUPS | ||
| "vram:4000" | ||
| ) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| { | ||
| "version": { | ||
| "major": 1, | ||
| "minor": 0 | ||
| }, | ||
| "local": [ | ||
| { | ||
| "vram": [ | ||
| { | ||
| "id": "0", | ||
| "slots": 8117 | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| { | ||
| "version": { | ||
| "major": 1, | ||
| "minor": 0 | ||
| }, | ||
| "local": [ | ||
| { | ||
| "vram": [ | ||
| { | ||
| "id": "0", | ||
| "slots": 7982 | ||
| }, | ||
| { | ||
| "id": "1", | ||
| "slots": 7982 | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| { | ||
| "version": { | ||
| "major": 1, | ||
| "minor": 0 | ||
| }, | ||
| "local": [ | ||
| { | ||
| "vram": [ | ||
| { | ||
| "id": "0", | ||
| "slots": 11019 | ||
| }, | ||
| { | ||
| "id": "1", | ||
| "slots": 11019 | ||
| }, | ||
| { | ||
| "id": "2", | ||
| "slots": 11019 | ||
| }, | ||
| { | ||
| "id": "3", | ||
| "slots": 11019 | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,90 @@ | ||
| get_property(LLVM_INSTALL_DIR GLOBAL PROPERTY LLVM_INSTALL_DIR) | ||
| set(LLVM_INSTALL_DIR ${THIRD_PARTY_DIR}/llvm) | ||
| set(LLVM_DIR ${LLVM_INSTALL_DIR}/lib/cmake/llvm) | ||
| set(ONEFLOW_OP_GROUPS | ||
| "ASSIGN" | ||
| "BINARY" | ||
| "BROADCAST" | ||
| "CONV" | ||
| "CROSS_ENTROPY" | ||
| "CUDA" | ||
| "DATASET" | ||
| "DETECTION" | ||
| "EAGER" | ||
| "FUSED" | ||
| "IDEMPOTENT" | ||
| "IDENTITY" | ||
| "IMAGE" | ||
| "INDICES" | ||
| "INVOLUTION" | ||
| "LOSS" | ||
| "MATH" | ||
| "MATMUL" | ||
| "MISC" | ||
| "NCCL" | ||
| "NORMALIZATION" | ||
| "OPTIMIZER" | ||
| "PADDING" | ||
| "PARALLEL_CAST" | ||
| "POOL" | ||
| "QUANTIZATION" | ||
| "REDUCE" | ||
| "RESHAPE" | ||
| "SCALAR" | ||
| "SOFTMAX" | ||
| "SUMMARY" | ||
| "TENSOR_BUFFER" | ||
| "TEST" | ||
| "TRIGONOMETRIC" | ||
| "UNARY" | ||
| "UPSAMPLE" | ||
| ) | ||
| foreach (OP_GROUP_NAME IN LISTS ONEFLOW_OP_GROUPS) | ||
| list(APPEND ONEFLOW_SCHEMA_TABLEGEN_FLAGS "-DGET_ONEFLOW_${OP_GROUP_NAME}_OP_DEFINITIONS") | ||
| endforeach() | ||
| list(APPEND ONEFLOW_SCHEMA_TABLEGEN_FLAGS "-DREMOVE_ONEFLOW_MLIR_ONLY_OP_DEFINITIONS") | ||
|
|
||
| set(GENERATED_OP_SCHEMA_DIR oneflow/core/framework) | ||
| set(GENERATED_IR_INCLUDE_DIR oneflow/ir/include) | ||
| set(SOURCE_IR_INCLUDE_DIR ${PROJECT_SOURCE_DIR}/oneflow/ir/include) | ||
| set(ONEFLOW_ODS ${SOURCE_IR_INCLUDE_DIR}/OneFlow/OneFlowOps.td) | ||
|
|
||
| list(APPEND ONEFLOW_SCHEMA_TABLEGEN_FLAGS "-I${GENERATED_IR_INCLUDE_DIR}") | ||
| list(APPEND ONEFLOW_SCHEMA_TABLEGEN_FLAGS "-I${SOURCE_IR_INCLUDE_DIR}") | ||
| list(APPEND ONEFLOW_SCHEMA_TABLEGEN_FLAGS "-I${LLVM_INSTALL_DIR}/include") | ||
|
|
||
| set(GENERATED_OP_SCHEMA_H "${GENERATED_OP_SCHEMA_DIR}/op_generated.h") | ||
| set(GENERATED_OP_SCHEMA_CPP "${GENERATED_OP_SCHEMA_DIR}/op_generated.cpp") | ||
|
|
||
|
|
||
| set(ONEFLOW_TABLE_GEN_EXE ${LLVM_INSTALL_DIR}/bin/oneflow_tblgen) | ||
| if(LLVM_PROVIDER STREQUAL "in-tree") | ||
| set(ONEFLOW_TABLE_GEN_TARGET oneflow_tblgen install-oneflow-tblgen install-mlir-headers) | ||
| elseif(LLVM_PROVIDER STREQUAL "install") | ||
| set(ONEFLOW_TABLE_GEN_TARGET ${ONEFLOW_TABLE_GEN_EXE}) | ||
| endif() | ||
|
|
||
| file(GLOB_RECURSE ODS_FILES LIST_DIRECTORIES false "${SOURCE_IR_INCLUDE_DIR}/*.td") | ||
| if(NOT ODS_FILES) | ||
| message(FATAL_ERROR "ODS_FILES not found: ${ODS_FILES}") | ||
| endif() | ||
| add_custom_command( | ||
| OUTPUT ${GENERATED_OP_SCHEMA_H} ${GENERATED_OP_SCHEMA_CPP} | ||
| COMMAND ${CMAKE_COMMAND} | ||
| ARGS -E make_directory ${GENERATED_OP_SCHEMA_DIR} | ||
| COMMAND ${ONEFLOW_TABLE_GEN_EXE} | ||
| ARGS --gen-op-schema-h ${ONEFLOW_ODS} ${ONEFLOW_SCHEMA_TABLEGEN_FLAGS} -o ${GENERATED_OP_SCHEMA_H} | ||
| COMMAND ${ONEFLOW_TABLE_GEN_EXE} | ||
| ARGS --gen-op-schema-cpp ${ONEFLOW_ODS} ${ONEFLOW_SCHEMA_TABLEGEN_FLAGS} | ||
| --op-include ${GENERATED_OP_SCHEMA_H} -o ${GENERATED_OP_SCHEMA_CPP} | ||
| DEPENDS ${ONEFLOW_TABLE_GEN_TARGET} | ||
| ${ODS_FILES} | ||
| VERBATIM | ||
| ) | ||
| set_source_files_properties( | ||
| ${GENERATED_OP_SCHEMA_H} ${GENERATED_OP_SCHEMA_CPP} PROPERTIES GENERATED TRUE | ||
| ) | ||
|
|
||
| oneflow_add_library(of_op_schema OBJECT ${GENERATED_OP_SCHEMA_H} ${GENERATED_OP_SCHEMA_CPP}) | ||
| add_dependencies(of_op_schema of_cfgobj) | ||
| add_dependencies(of_op_schema prepare_oneflow_third_party) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.