Add LayoutLMv2 to models exportable with ONNX #14555
Conversation
|
@michaelbenayoun Can you please review this PR and highlight why these circleci tests are failing ? |
|
|
||
|
|
||
| class LayoutLMv2OnnxConfig(OnnxConfig): | ||
| def __init__(self, config: PretrainedConfig, task: str = "default", patching_specs: List[PatchingSpec] = None): |
There was a problem hiding this comment.
You do not need to override this as you are simply calling the base clase __init__ method.
| ] | ||
| ) | ||
|
|
||
| def generate_dummy_inputs( |
There was a problem hiding this comment.
You do not need to rewrite everything, I think you can inspire from what was done for LayoutLM
There was a problem hiding this comment.
@michaelbenayoun input for LayoutLMv2 requires image too and bboxes tile so I had to write this method. If I am missing anything please highlight what changes can I make,
There was a problem hiding this comment.
Yes, you need to, but you can use the base class generate_dummy_inputs method, and then complete the method with the additional stuff, just as it was done for LayoutLM.
| @@ -855,10 +855,10 @@ def forward( | |||
|
|
|||
| device = input_ids.device if input_ids is not None else inputs_embeds.device | |||
|
|
|||
| visual_shape = list(input_shape) | |||
| visual_shape = list(torch.empty(size=input_shape).size()) | |||
There was a problem hiding this comment.
What's the reason for this?
There was a problem hiding this comment.
Yes @linchpinlin exactly. Thanks
@michaelbenayoun I get tensor size error if I don't apply these changes
| visual_shape[1] = self.config.image_feature_pool_shape[0] * self.config.image_feature_pool_shape[1] | ||
| visual_shape = torch.Size(visual_shape) | ||
| final_shape = list(input_shape) | ||
| final_shape = list(torch.empty(size=input_shape).size()) |
| @@ -110,6 +112,19 @@ def export( | |||
| if not inputs_match: | |||
| raise ValueError("Model and config inputs doesn't match") | |||
|
|
|||
| if isinstance(model.base_model, LayoutLMv2Model): | |||
There was a problem hiding this comment.
This is problematic because export should be somehow agnostic to the model to export.
If I understand correctly you want to replace nn.AdaptiveAvgPool2d by nn.AvgPool2d.
What you can do instead of patching the modules (also is it wanted to have hardcoded values for kernel size and stride?), is to patch the underlying op: torch.nn.functional.adaptive_avg_pool2d -> torch.nn.functional.avg_pool2d.
You can do this by providing a PatchingSpec in the LayoutLMv2OnnxConfig.__init__ method.
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
|
Hi! Thanks for working on this @fadi212 🙏 I am interested in this work, but I see it has been quite some time since you've had an opportunity to work on it. Are you still working on this? If you no longer have the time or resources to do so, would you be able to provide any next steps or advice on what is necessary for completion? Thank you for your time and effort 🤗 |
This PR adds the code for converting LayoutLMv2 model to onnx format.