Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new models (Janus, Qwen2-VL, JinaCLIP, LLaVA-OneVision, ViTPose, MGP-STR) & refactor processors. #1001

Merged
merged 38 commits into from
Nov 26, 2024
Merged
Changes from 1 commit
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
bef6361
Extract processor classes into separate folders
xenova Oct 28, 2024
628d59f
Fix typo
xenova Oct 28, 2024
03f6662
Define which classes use `processor_config.json`
xenova Oct 30, 2024
d040e81
[WIP] Add support for `deepseek-ai/Janus-1.3B`
xenova Oct 30, 2024
ae5a29d
Fix unit tests
xenova Oct 30, 2024
931579e
Remove redundant `extends` JSDoc
xenova Oct 31, 2024
2ade0ba
Fix JSDoc
xenova Oct 31, 2024
a4ecf08
Update Janus JSDoc
xenova Oct 31, 2024
29097c0
Improve `VLChatProcessor` processor types
xenova Nov 14, 2024
76f8c33
Expose ImageFeatureExtractor as copy of ImageProcessor
xenova Nov 14, 2024
c2005c9
Add support for `LLaVA-OneVision`
xenova Nov 19, 2024
f00456c
Add support for ViTPose
xenova Nov 19, 2024
c681ed9
Add ViTPose to README
xenova Nov 19, 2024
c5164f3
Merge branch 'main' into add-janus
xenova Nov 19, 2024
545354f
Bump dependencies
xenova Nov 19, 2024
5ac2cb9
Add support for `MGP-STR` models
xenova Nov 20, 2024
c4b1d63
Documentation fixes
xenova Nov 20, 2024
a06fbc6
Add support for `Qwen2VLImageProcessor`
xenova Nov 20, 2024
155bb9d
Format tests folder
xenova Nov 20, 2024
76c132f
Use `AutoImageProcessor` for image processors
xenova Nov 20, 2024
6146f0b
Add support for `Qwen2VLProcessor`
xenova Nov 20, 2024
8138b23
Fix `image_grid_thw` dtype
xenova Nov 22, 2024
29de4b0
Fix bigint product
xenova Nov 22, 2024
54073c8
[WIP] Support for qwen2vl models
xenova Nov 22, 2024
cb0e09b
Add support for JinaCLIP models
xenova Nov 22, 2024
dba3b2f
Add listed support for Janus
xenova Nov 22, 2024
cf0714f
Fix qwen2vl processor unit test
xenova Nov 23, 2024
41a0755
Update dependency versions
xenova Nov 23, 2024
7d60bfe
Export logits processors
xenova Nov 24, 2024
95688fc
Expose batch_decode for processor
xenova Nov 24, 2024
2e945a5
Qwen2VL - Implement `get_rope_index`
xenova Nov 24, 2024
e28ac7a
Add `Qwen2VLForConditionalGeneration` unit tests
xenova Nov 24, 2024
79fe412
Update dependencies
xenova Nov 25, 2024
83da94b
Update `onnxslim==0.1.42`
xenova Nov 25, 2024
3019e55
`tokenizer.default_chat_template` has been removed
xenova Nov 25, 2024
a3992a0
Merge branch 'main' into add-janus
xenova Nov 26, 2024
3017402
Add listed support for Qwen2-VL
xenova Nov 26, 2024
cbfdc07
Fix `.from_pretrained` function type
xenova Nov 26, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix unit tests
  • Loading branch information
xenova committed Oct 30, 2024
commit ae5a29d23e139aa3373b232e98e5f9ca7af10c67
2 changes: 1 addition & 1 deletion src/models.js
Original file line number Diff line number Diff line change
@@ -1726,7 +1726,7 @@ export class PreTrainedModel extends Callable {
const dtype = session?.config?.kv_cache_dtype ?? 'float32';
const empty = (dtype === 'float16') ? new Uint16Array() : [];

const batch_size = decoderFeeds[this.main_input_name].dims[0];
const batch_size = (decoderFeeds[this.main_input_name] ?? decoderFeeds.attention_mask).dims?.[0] ?? 1;
const shapes = getKeyValueShapes(this.config, { batch_size });

for (const name in shapes) {
Loading