[Relay] External codegen

[zhiics]

Part of #4258 to make the review process easier.

This PR adds the external codegen for Relay. It contains the following changes

• C source style codegen to generate TVM compatible C library wrapper that can be compiled and library together with DSOModule
• DNNL codegen that generates DNNL kernel wrappers to execute part of a Relay program
• DNNL library kernel that initializes dnnl engine for execution.
• graphruntime execution
• All external functions are collected for codegen and an array of external runtime modules are returned and imported to the DSOModule.
• Various unit tests

Followup PRs will to send separately to cover the following aspects:

• VM execution
• Move comprehensive dnnl kernels
• Annotation and partitioning

CC @tqchen @soiferj @masahi @jroesch @icemelon9 @u99127 @comaniac

[masahi]

LGTM modulo minor issues. I've also verified the dnnl example.
Looking forward to upcoming PRs :)

[tqchen]

I see see several places that contains "extern" keyword. It would be great if we can avoid the "extern" terminology and instead focus on generate c code, which is more clear.

e.g. contrib/codegen_c/codgen_c.h

[comaniac]

Thanks @tqchen, we've addressed your comments. Please take another look when you get a chance.

[tqchen]

Some additional comments on API design:

• The keyword IsExternal is still quite confusing. I would suggest make the attribute "compiler", alternatively we can call it a "backend". When it is set, we will look up for the customized compiler hook. The value "default" invokes the default compilation pipeline.

• Let us deliberate a bit on the FuncName keyword. While I understand it is necessary to attach such attribute before lowering to that ensure consistent symbol lookup, is it the right name?

More broadly, as we start to introduce more attributes to functions, it would be great to come up with a naming convention and document them

[zhiics]

@tqchen how about this?

The keyword IsExternal is still quite confusing. I would suggest make the attribute "compiler", alternatively we can call it a "backend". When it is set, we will look up for the customized compiler hook. The value "default" invokes the default compilation pipeline.

Let us deliberate a bit on the FuncName keyword. While I understand it is necessary to attach such attribute before lowering to that ensure consistent symbol lookup, is it the right name?

Change "external" to "compiler". The value is compiler+id, e.g. "dnnl_1" and "dnnl_2" so that we can have a unique name for symbol lookup. And then we can just remove the "FuncName" attribute. Or do you have other suggestions?

More broadly, as we start to introduce more attributes to functions, it would be great to come up

We have moved them to relay::attrs and documented.

[tqchen]

compiler attr sounds good, I think we want something like FuncName to indicate the symbol name, but not sure about the choice of the naming

[zhiics]

@tqchen How about CustomFuncSymbol?

[tqchen]

Let us also hear about others' opinions, perhaps create a discuss thread in the forum and list candidates, i think something along the direction of symbol, export_func_name, unique_id might makes sense. Note the same attribute might be used for compiling non-custom functions(in the case when the user wants to force a certain name)

[zhiics]

@tqchen We've resolved the naming issues, could you please take another look? Thanks.

[zhiics]

@tqchen Any other outstanding issues or concerns?

[tqchen]

Thanks @comaniac @zhiics @liangfu @masahi !