Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add parameters to make custom backbone for detr #14933

Closed
wants to merge 9 commits into from
Closed

Add parameters to make custom backbone for detr #14933

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
8146214
add parameters to make custom backbone for detr
Zhylkaaa Dec 25, 2021
15ba331
rename `replace_batch_norm` to `fix_batch_norm`
Zhylkaaa Dec 26, 2021
2815479
reformat
Zhylkaaa Dec 26, 2021
42624b2
rename parameters and resolve tensor device placement issue
Zhylkaaa Dec 27, 2021
9880a19
bugfix: change names in constructor
Zhylkaaa Dec 27, 2021
23986a3
bugfix
Zhylkaaa Dec 27, 2021
aaacc2b
fix PIL grayscale handling
Zhylkaaa Dec 27, 2021
f267747
fix mask for grayscale images
Zhylkaaa Dec 27, 2021
8ac8b66
bugfix
Zhylkaaa Dec 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions src/transformers/models/detr/configuration_detr.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,10 @@ def __init__(
bbox_loss_coefficient=5,
giou_loss_coefficient=2,
eos_coefficient=0.1,
in_chans=3,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
in_chans=3,
num_channels=3,

I would rename this to num_channels to be consistent with other models in the library (like ViT).

pretrained=True,
Copy link
Contributor

@NielsRogge NielsRogge Dec 26, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pretrained=True,
use_pretrained_backbone=True,

This can be renamed to use_pretrained_backbone, for clarity.

freeze_layers=True,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would leave away this one, as there's already a method one can call on DetrModel, called freeze_backbone as seen here.

Maybe we can improve its documentation for visibility.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this parameter is more specific, it disables freezing 2-4 layer of resnet-50 (this was just hard coded into encoder initialisation), but I can change it if you think freeze_backbone is better anyway

fix_batch_norm=True,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reason you want to replace the frozen batch norm layers?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Training from scratch, I want to try training from fully randomly initialised model. Also I don't have pretrained backbone for my problem anyway, so I think this parameter won't harm

**kwargs
):
self.num_queries = num_queries
Expand Down Expand Up @@ -190,6 +194,10 @@ def __init__(
self.bbox_loss_coefficient = bbox_loss_coefficient
self.giou_loss_coefficient = giou_loss_coefficient
self.eos_coefficient = eos_coefficient
self.in_chans = in_chans
self.pretrained = pretrained
self.freeze_layers = freeze_layers
self.fix_batch_norm = fix_batch_norm
super().__init__(is_encoder_decoder=is_encoder_decoder, **kwargs)

@property
Expand Down
24 changes: 17 additions & 7 deletions src/transformers/models/detr/modeling_detr.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -315,23 +315,26 @@ class DetrTimmConvEncoder(nn.Module):

"""

def __init__(self, name: str, dilation: bool):
def __init__(
self, name: str, dilation: bool, in_chans: int, pretrained=True, freeze_layers=True, fix_batch_norm=True
):
super().__init__()

kwargs = {}
kwargs = {"in_chans": in_chans}
if dilation:
kwargs["output_stride"] = 16

requires_backends(self, ["timm"])

backbone = create_model(name, pretrained=True, features_only=True, out_indices=(1, 2, 3, 4), **kwargs)
backbone = create_model(name, pretrained=pretrained, features_only=True, out_indices=(1, 2, 3, 4), **kwargs)
# replace batch norm by frozen batch norm
with torch.no_grad():
replace_batch_norm(backbone)
if fix_batch_norm:
with torch.no_grad():
replace_batch_norm(backbone)
self.model = backbone
self.intermediate_channel_sizes = self.model.feature_info.channels()

if "resnet" in name:
if "resnet" in name and freeze_layers:
for name, parameter in self.model.named_parameters():
if "layer2" not in name and "layer3" not in name and "layer4" not in name:
parameter.requires_grad_(False)
Expand Down Expand Up @@ -1159,7 +1162,14 @@ def __init__(self, config: DetrConfig):
super().__init__(config)

# Create backbone + positional encoding
backbone = DetrTimmConvEncoder(config.backbone, config.dilation)
backbone = DetrTimmConvEncoder(
config.backbone,
config.dilation,
config.in_chans,
pretrained=config.pretrained,
freeze_layers=config.freeze_layers,
fix_batch_norm=config.fix_batch_norm,
)
position_embeddings = build_position_encoding(config)
self.backbone = DetrConvModel(backbone, position_embeddings)

Expand Down