[Model Support] Add support for wav2vec

[yaoyaoding]

Major:

1. Enhance the hidet dynamo backend to support the pytorch model that used weight_norm.
2. Add resolve rule for conv1d.

The example code that use hidet could be found at https://github.com/egorsmkv/wav2vec2-hidet.

Currently, hidet is still slower than pytorch (hidet 30ms vs. pytorch 25ms on RTX 4090). Will optimize these models when we have more hands.

[egorsmkv]

Nice, thanks

[egorsmkv]

@yaoyaoding hi

My little feedback on this issue - https://github.com/egorsmkv/wav2vec2-hidet/blob/main/logs_hidet_0.3.0_and_python3.11.txt

[yaoyaoding]

Hi @Aalanli, could you take a look at this? Thanks!

[egorsmkv]

Hello! Any update on it? @Aalanli

[egorsmkv]

Just tried hidet with wav2vec2-bert model recently released. Also I have some problems.

I made a colab to share what's going wrong: https://colab.research.google.com/drive/1qsw0Tu9ENS_q3ao0YUbh4CDTyWqUxLWq?usp=sharing

In short:

BackendCompilerFailed: backend='hidet' raised:
NotImplementedError: The following modules/functions are not supported by hidet yet:
  torch.rand
  torch.nn.GLU
  torch.softmax

[Aalanli]

Hi @egorsmkv, sorry for the long delay. It turns out that implementing rand would require some invasive changes to the runtime, which we held off doing until our AMD support is ready.
As a potential work-around, one can pass any random state as an input to the model, kind of like what is done in Jax.
The other functions should be a quick fix.

[egorsmkv]

An update with hidet 0.4.1:

https://github.com/egorsmkv/wav2vec2-hidet-0.4.0/blob/master/logs_hidet_0.4.1_and_python3.12.txt

In short:

NotImplementedError: The following operators are not supported or mapped by hidet yet:
  torch.nn.GLU

[vadiklyutiy]

An update with hidet 0.4.1:

https://github.com/egorsmkv/wav2vec2-hidet-0.4.0/blob/master/logs_hidet_0.4.1_and_python3.12.txt

In short:

NotImplementedError: The following operators are not supported or mapped by hidet yet:
  torch.nn.GLU

How to reproduce?

PS actually it should be supported.

[egorsmkv]

An update with hidet 0.4.1:
https://github.com/egorsmkv/wav2vec2-hidet-0.4.0/blob/master/logs_hidet_0.4.1_and_python3.12.txt
In short:

NotImplementedError: The following operators are not supported or mapped by hidet yet:
  torch.nn.GLU

How to reproduce?

PS actually it should be supported.

https://github.com/egorsmkv/wav2vec2-hidet-0.4.0?tab=readme-ov-file#install here is the instruction to reproduce that issue.

[vadiklyutiy]

@egorsmkv
Sorry, it was my foul. torch.nn.GLU is not supported yet.

[egorsmkv]

@egorsmkv
Sorry, it was my foul. torch.nn.GLU is not supported yet.

Oh, okay. I'm waiting for final support of it. Luck to the team :) You do awesome work 👏