Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Segment Anything Model (SAM) #22654

Merged
merged 184 commits into from
Apr 19, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
184 commits
Select commit Hold shift + click to select a range
0d59baa
initial commit
ArthurZucker Apr 7, 2023
8798e62
keys match
younesbelkada Apr 10, 2023
4acf7a2
update, fix conversion
younesbelkada Apr 10, 2023
b0e2994
fixes, inference working
younesbelkada Apr 10, 2023
cd5cd0a
fix
younesbelkada Apr 10, 2023
589a9fc
more fixes
younesbelkada Apr 10, 2023
17e624c
more fixes
younesbelkada Apr 10, 2023
f43619a
clean up
younesbelkada Apr 10, 2023
8c30ac3
more clean up
younesbelkada Apr 10, 2023
dd19b15
fix copies and add convext copied layer norm
ArthurZucker Apr 11, 2023
17aeb1b
stash
ArthurZucker Apr 11, 2023
f02408f
pretty big upfate
ArthurZucker Apr 14, 2023
a062a2a
cleaning
ArthurZucker Apr 14, 2023
3572298
more cleaning
ArthurZucker Apr 14, 2023
878c9c5
fixup stuffs
ArthurZucker Apr 14, 2023
4eb3eb1
fix copies
ArthurZucker Apr 14, 2023
15937e7
fix iinit
ArthurZucker Apr 14, 2023
bd7d2b4
update test removing tokenizer
ArthurZucker Apr 14, 2023
5370e35
nits
ArthurZucker Apr 14, 2023
51be4cc
add pretrained
ArthurZucker Apr 14, 2023
5d4ea59
more nits
ArthurZucker Apr 14, 2023
e740463
remove tracking of pipeline
ArthurZucker Apr 14, 2023
5e2c4ea
few fixes
younesbelkada Apr 14, 2023
3840fcd
update san and conversion script
ArthurZucker Apr 14, 2023
63a6f75
fix mask decoder and prompt encoder conversion
younesbelkada Apr 14, 2023
71c5377
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
younesbelkada Apr 14, 2023
12232d7
fixes
younesbelkada Apr 14, 2023
272a470
small update
ArthurZucker Apr 14, 2023
89d8280
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
ArthurZucker Apr 14, 2023
01e6b57
fix order
ArthurZucker Apr 14, 2023
93c5185
fix
younesbelkada Apr 14, 2023
b179571
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
younesbelkada Apr 14, 2023
2ad8e6d
fix image embeddings
younesbelkada Apr 14, 2023
1d503cb
nites
ArthurZucker Apr 14, 2023
e754719
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
ArthurZucker Apr 14, 2023
dba2be3
few fixes
younesbelkada Apr 14, 2023
854b01b
fix logits
younesbelkada Apr 14, 2023
7409029
clean up
younesbelkada Apr 15, 2023
23d358a
fixes boxes inference
younesbelkada Apr 15, 2023
2e89bed
v1 AMG
younesbelkada Apr 15, 2023
a770a51
clean up
younesbelkada Apr 15, 2023
bb40cb0
some clean up
younesbelkada Apr 15, 2023
30da508
multi points support
younesbelkada Apr 16, 2023
c4408c7
amg working
younesbelkada Apr 16, 2023
4a3b75f
fixup
younesbelkada Apr 16, 2023
88757f3
clean up
younesbelkada Apr 16, 2023
e2e88d0
readme
younesbelkada Apr 16, 2023
a246c96
update toctree
younesbelkada Apr 16, 2023
a447563
fix type hint
younesbelkada Apr 16, 2023
b9fcddd
multiple fixes
younesbelkada Apr 17, 2023
98bf922
fixup
younesbelkada Apr 17, 2023
2a78a31
fixes
younesbelkada Apr 17, 2023
70c7c61
updates
younesbelkada Apr 17, 2023
6b21e68
updates
younesbelkada Apr 17, 2023
bc8c1b8
more tests
younesbelkada Apr 17, 2023
96e320f
few fixes
younesbelkada Apr 17, 2023
09ccaea
Merge remote-tracking branch 'upstream/main' into HEAD
younesbelkada Apr 17, 2023
f382b8d
change to `SamForMaskGeneration`
younesbelkada Apr 17, 2023
1c317db
doc
younesbelkada Apr 17, 2023
2729dbc
fixup
ArthurZucker Apr 17, 2023
39d611c
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
ArthurZucker Apr 17, 2023
5de1b08
fix more tests
younesbelkada Apr 17, 2023
073e730
multiple fixes
younesbelkada Apr 17, 2023
2d27b47
fix CI tests
younesbelkada Apr 17, 2023
7efbba2
refactor processor
younesbelkada Apr 17, 2023
d467dbd
renamings
ArthurZucker Apr 17, 2023
0d9e20f
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
ArthurZucker Apr 17, 2023
25c569b
draft the pipeline
ArthurZucker Apr 17, 2023
7c896b0
refactor
younesbelkada Apr 17, 2023
8810145
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
younesbelkada Apr 17, 2023
9ffb2e1
fix tests
younesbelkada Apr 17, 2023
47b12d1
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
ArthurZucker Apr 17, 2023
20b90b9
fix test
younesbelkada Apr 17, 2023
aea7c6a
few cleanings
younesbelkada Apr 17, 2023
ba7a851
fix test
younesbelkada Apr 17, 2023
2135c85
edit pipelien support chunking
ArthurZucker Apr 17, 2023
5720970
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
ArthurZucker Apr 17, 2023
4cfc51d
udate
ArthurZucker Apr 17, 2023
c995f9f
add slow tests
younesbelkada Apr 17, 2023
2a8399d
fix nit
younesbelkada Apr 17, 2023
2c60040
fixup
younesbelkada Apr 17, 2023
de963b8
fix nit
younesbelkada Apr 17, 2023
5247ee2
current chunk pipleine
ArthurZucker Apr 17, 2023
80158a4
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
ArthurZucker Apr 17, 2023
be3fa6c
cast boxes in fp32
younesbelkada Apr 17, 2023
812359e
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
younesbelkada Apr 17, 2023
8504910
nit
younesbelkada Apr 17, 2023
b515a62
current updates
ArthurZucker Apr 17, 2023
403cf44
piepleine works
ArthurZucker Apr 17, 2023
e426f3b
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
ArthurZucker Apr 17, 2023
af4a7e3
fixup
ArthurZucker Apr 17, 2023
bc2a2da
clean up config
younesbelkada Apr 18, 2023
116b13a
fix slow tests
younesbelkada Apr 18, 2023
ad34e9d
fix slow tests
younesbelkada Apr 18, 2023
1926bc1
clean up
younesbelkada Apr 18, 2023
4a0532e
update doc and pipeline
ArthurZucker Apr 18, 2023
c896253
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
ArthurZucker Apr 18, 2023
88f2463
adds more slow tests
younesbelkada Apr 18, 2023
7b123aa
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
younesbelkada Apr 18, 2023
3ff5bfc
fix slow tests
younesbelkada Apr 18, 2023
ff87e20
cleaning
ArthurZucker Apr 18, 2023
97fb32b
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
ArthurZucker Apr 18, 2023
88f8453
tests pass
younesbelkada Apr 18, 2023
cf5158e
add docstring
younesbelkada Apr 18, 2023
1f17915
fix copies
younesbelkada Apr 18, 2023
bb48be9
clean up
younesbelkada Apr 18, 2023
dfccc0d
support batch of images
ArthurZucker Apr 18, 2023
46e0ff1
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
ArthurZucker Apr 18, 2023
6df9500
style
ArthurZucker Apr 18, 2023
ed99b54
dummy is needed, add tests
ArthurZucker Apr 18, 2023
c98e4c0
fix slow tests
younesbelkada Apr 18, 2023
dd676be
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
younesbelkada Apr 18, 2023
577d285
fix CI
younesbelkada Apr 18, 2023
afb449f
update
ArthurZucker Apr 18, 2023
3a4f5e8
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
ArthurZucker Apr 18, 2023
fd4dc82
adds more tests
younesbelkada Apr 18, 2023
6435187
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
younesbelkada Apr 18, 2023
c32f43a
fixes
younesbelkada Apr 18, 2023
9886a53
fixes
younesbelkada Apr 18, 2023
8b313a0
fixup
younesbelkada Apr 18, 2023
21e2f7e
fixes
younesbelkada Apr 18, 2023
b4a7e8b
few fixes
younesbelkada Apr 18, 2023
9ffbc69
filter
younesbelkada Apr 18, 2023
bbdd2ee
few fixes
younesbelkada Apr 18, 2023
1950a16
some refactor
younesbelkada Apr 18, 2023
1320e36
touches finales
younesbelkada Apr 18, 2023
0fc8680
fix
younesbelkada Apr 18, 2023
e996e01
style
ArthurZucker Apr 18, 2023
a5d8e92
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
ArthurZucker Apr 18, 2023
53499e3
remove pipeline files
ArthurZucker Apr 18, 2023
d4c4de9
fixes nits
younesbelkada Apr 18, 2023
0115cd9
revert pipeline changes
ArthurZucker Apr 18, 2023
75448c3
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
ArthurZucker Apr 18, 2023
8b60f3a
fix test
younesbelkada Apr 18, 2023
f326dfa
fixup
ArthurZucker Apr 18, 2023
53a9f0e
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
ArthurZucker Apr 18, 2023
0b57836
remove automodel for automatic mask generation
ArthurZucker Apr 18, 2023
9f80285
fix failing torch tests
ArthurZucker Apr 18, 2023
88b7ac9
update mdx
ArthurZucker Apr 19, 2023
fc898ca
revert removal of `MODEL_FOR_AUTOMATIC_MASK_GENERATION_MAPPING`
ArthurZucker Apr 19, 2023
8e86426
update sam config based on review
ArthurZucker Apr 19, 2023
818f476
update low_resolution_masks -> pred_masks
ArthurZucker Apr 19, 2023
480b8e3
update processor docstring
ArthurZucker Apr 19, 2023
ae79546
remove image processor import empty
ArthurZucker Apr 19, 2023
6cccfb3
update for testing
ArthurZucker Apr 19, 2023
2cb2d9e
output vision hidden states + clean recomm
ArthurZucker Apr 19, 2023
2d1b3bc
fixup
ArthurZucker Apr 19, 2023
e5b0af9
fixup
ArthurZucker Apr 19, 2023
61a9ef5
remove unused
ArthurZucker Apr 19, 2023
e121384
Update src/transformers/models/sam/modeling_sam.py
ArthurZucker Apr 19, 2023
cdb8c07
Update src/transformers/models/sam/image_processing_sam.py
ArthurZucker Apr 19, 2023
f9508f1
nits
ArthurZucker Apr 19, 2023
cd9a8e8
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
ArthurZucker Apr 19, 2023
db41554
fix
younesbelkada Apr 19, 2023
8eae6ef
fix CI tests and slow tests
younesbelkada Apr 19, 2023
93a504c
replace with Amy's processor
younesbelkada Apr 19, 2023
8440530
clearer docstring
younesbelkada Apr 19, 2023
93e1364
add `SamVisionNeck`
younesbelkada Apr 19, 2023
5800114
refactor - all CI tests should pass
younesbelkada Apr 19, 2023
7228cb8
fix broken import on Gcolab
younesbelkada Apr 19, 2023
0c61d3b
few fixes here and there
younesbelkada Apr 19, 2023
d7e805d
fix another bug
younesbelkada Apr 19, 2023
731d416
fix more bugs
younesbelkada Apr 19, 2023
08eaaf9
update and merge
ArthurZucker Apr 19, 2023
81be02c
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
ArthurZucker Apr 19, 2023
c931b55
correct ckpt
younesbelkada Apr 19, 2023
932a479
address comments
younesbelkada Apr 19, 2023
d71cabf
add tips
younesbelkada Apr 19, 2023
bbcbad1
revert
younesbelkada Apr 19, 2023
4000130
fix docstring
younesbelkada Apr 19, 2023
b921294
replace with `SamModel`
younesbelkada Apr 19, 2023
5810f28
make fixup
younesbelkada Apr 19, 2023
44542dc
add support for bathed images and batch ed points
ArthurZucker Apr 19, 2023
cdb6aba
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
ArthurZucker Apr 19, 2023
b1f1686
make fixup this time, really
younesbelkada Apr 19, 2023
b391539
Merge branch 'add-sam' of https://github.com/ArthurZucker/transformer…
younesbelkada Apr 19, 2023
2e44dd2
make fixup again and again
younesbelkada Apr 19, 2023
46c8937
few fixes here and there, this should be the touche finale
younesbelkada Apr 19, 2023
ce8b35f
Update docs/source/en/model_doc/sam.mdx
younesbelkada Apr 19, 2023
533d637
fixup
younesbelkada Apr 19, 2023
8308f3f
correct checkpoints
younesbelkada Apr 19, 2023
d958f14
correct name
younesbelkada Apr 19, 2023
8422b8f
rm unneeded file
younesbelkada Apr 19, 2023
fb3827a
add notebook
younesbelkada Apr 19, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions docs/source/en/model_doc/sam.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,6 @@ The original code can be found [here](<INSERT LINK TO GITHUB REPO HERE>).

[[autodoc]] SamVisionConfig

## SamQFormerConfig

[[autodoc]] SamQFormerConfig

## SamProcessor

Expand Down
14 changes: 2 additions & 12 deletions src/transformers/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,6 @@
"SAM_PRETRAINED_CONFIG_ARCHIVE_MAP",
"SamConfig",
"SamProcessor",
"SamQFormerConfig",
"SamVisionConfig",
],
"models.bloom": ["BLOOM_PRETRAINED_CONFIG_ARCHIVE_MAP", "BloomConfig"],
Expand Down Expand Up @@ -1196,11 +1195,7 @@
_import_structure["models.sam"].extend(
[
"SAM_PRETRAINED_MODEL_ARCHIVE_LIST",
"SamForConditionalGeneration",
"SamModel",
"SamPreTrainedModel",
"SamQFormerModel",
"SamVisionModel",
"SamForImageSegmentation",
]
)
_import_structure["models.bloom"].extend(
Expand Down Expand Up @@ -3905,7 +3900,6 @@
SAM_PRETRAINED_CONFIG_ARCHIVE_MAP,
SamConfig,
SamProcessor,
SamQFormerConfig,
SamVisionConfig,
)
from .models.bloom import BLOOM_PRETRAINED_CONFIG_ARCHIVE_MAP, BloomConfig
Expand Down Expand Up @@ -4790,11 +4784,7 @@
)
from .models.sam import (
SAM_PRETRAINED_MODEL_ARCHIVE_LIST,
SamForConditionalGeneration,
SamModel,
SamPreTrainedModel,
SamQFormerModel,
SamVisionModel,
SamForImageSegmentation,
)
from .models.bloom import (
BLOOM_PRETRAINED_MODEL_ARCHIVE_LIST,
Expand Down
14 changes: 2 additions & 12 deletions src/transformers/models/sam/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@
"configuration_sam": [
"SAM_PRETRAINED_CONFIG_ARCHIVE_MAP",
"SamConfig",
"SamQFormerConfig",
"SamVisionConfig",
],
"processing_sam": ["SamProcessor"],
Expand All @@ -34,18 +33,13 @@
else:
_import_structure["modeling_sam"] = [
"SAM_PRETRAINED_MODEL_ARCHIVE_LIST",
"SamModel",
"SamQFormerModel",
"SamPreTrainedModel",
"SamForConditionalGeneration",
"SamVisionModel",
"SamForImageSegmentation",
]

if TYPE_CHECKING:
from .configuration_sam import (
SAM_PRETRAINED_CONFIG_ARCHIVE_MAP,
SamConfig,
SamQFormerConfig,
SamVisionConfig,
)
from .processing_sam import SamProcessor
Expand All @@ -58,11 +52,7 @@
else:
from .modeling_sam import (
SAM_PRETRAINED_MODEL_ARCHIVE_LIST,
SamForConditionalGeneration,
SamModel,
SamPreTrainedModel,
SamQFormerModel,
SamVisionModel,
SamForImageSegmentation
)

else:
Expand Down
Loading