Skip to content

Releases: embeddings-benchmark/mteb

1.34.4

06 Feb 12:50
@KennethEnevoldsen KennethEnevoldsen
1.34.4
4a37967
Compare
Choose a tag to compare
1.34.4 Latest
Latest

1.34.4 (2025-02-06)

Fix

  • fix: Added description and resolved bug in rangeslider (#1990)

  • fix: Added description and resolved bug in rangeslider

addresses #1987
fixes #1954

  • format

  • remove certificate (8583383)

Assets 6
Loading

1.34.3

06 Feb 11:34
@KennethEnevoldsen KennethEnevoldsen
1.34.3
56f90a7
Compare
Choose a tag to compare
1.34.3

1.34.3 (2025-02-06)

Fix

  • fix: Meta information ru_sentence_models (#1991)

cointegrated/rubert-tiny
sergeyzh/rubert-tiny-turbo
sergeyzh/LaBSE-ru-turbo (370b26c)

Unknown

  • Fix HagridRetrieval load_dataset "trust_remote_code" (#1989) (9618505)

  • Add mini-gte model (#1906)

  • add mini-gte model

  • fix formatting

  • Update mteb/models/qtack_models.py

add adapted from metadata

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>

  • add datasets to mini-gte

  • linting for qtack_models.py

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (1520979)

Assets 6
Loading

1.34.2

05 Feb 16:34
@KennethEnevoldsen KennethEnevoldsen
1.34.2
adc06ab
Compare
Choose a tag to compare
1.34.2

1.34.2 (2025-02-05)

Fix

  • fix: leaderboard and benchmark.py inconstiencies (#1956)

  • fix: fix task types in MIEB

  • fix: French leaderboard inconsistencies

fixes #1919

Unknown

  • Fix: added link for text-embedding-005 and other google models (#1962)

Add reference link for text-embedding-005, text-embedding-004 and text-multilingual-embedding-002 (2a8c25a)

  • Update tasks table (fc6696f)

  • Merge MIEB into main 🎉 (#1968)

  • mieb ZeroshotClassification

  • mieb docs

  • mieb implementation demo

  • model meta; abstask column names; linear probe clf

  • model meta; abstask column names; linear probe clf

  • fix: update naming as candidate_labels

  • Update README.md

  • Update README.md

  • i2tretrieval

  • test load data ignore i2tretrieval

  • [MIEB] Add image clustering (#1088)

  • make lint

  • wip

  • add TinyImageNet and run

  • type hints

  • add accuracy

  • lint

  • remove unused & fix typos

  • T2I Retrieval

  • Any2AnyRetrieval

  • fix tests from merge

  • [MIEB] Add image text pair classification and tests (#1099)

  • add ImageTextPairClassification abstask and evaluator

  • dataset transform into sequence of images for each sample

  • fix processing logic; list of list images compatability

  • lint and docstrings

  • make lint

  • fix failing tests in TaskMetadata

  • add tests for mieb

  • skip gated repo

Co-authored-by: gowitheflow-1998 <jsbs54@durham.ac.uk>

  • [MIEB] Add image classification and zero shot classification tasks (#1101)

  • fix task metadata

  • use overrideable column names

  • add CIFAR datasets

  • add caltech101 dataset

  • add FGVC aircraft dataset

  • add food 101 dataset

  • add OxfordPets dataset

  • remove comments

  • correct cifar100 path

  • update cifar100 classification results

  • cifar zero shot results

  • add caltech101 zero shot

  • matching CLIP paper implementation

  • add aircraft and food zero shot

  • add oxford pets zero shot

  • [MIEB] Add CIFAR clustering (#1104)

add CIFAR clustering

  • [MIEB] Add more image classification and zero shot classification datasets (#1103)

  • update category to i2t

  • add MNIST linear probe and zero shot

  • add FER2013 linear probe and zero shot

  • add stanford cars linear probe and zero shot

  • add birdsnap linear probe and zero shot

  • add eurosat linear probe and zero shot

  • lint

  • correct eurosat zero shot labels

  • add abstask for image multilable and voc2007

  • make lint

  • [MIEB] Add more image classification and zero shot datasets (#1105)

  • add STL10 linear probe and zero shot

  • add RESISC45 linear probe and zeor shot

  • add Describable textures linear probe and zero shot

  • fix spacing lint

  • add SUN397 linear probe and zero shot

  • correct SUN397 zero shot captions

  • add baai bge vista

  • add e5-v

  • linting

  • memory issues for image linear probe & zeroshot

  • kknn linear probe arguments

  • del comments

  • Add some classification and ZeroShot classification tasks (#1107)

  • Add Country211 classification task

  • Add imagenet1k classification task

  • Add UCF101 classification task

  • Add PatchCamelyon Classification task

  • Add GTSRB classification task

  • Add GSTRB Zero Shot Classification

  • Add country211 zero shot classification

  • Add results for classification tasks

  • Add zero shot classification tasks

  • Add PatchCamelyon tasks and results

  • Add linting

  • Add results and fix prompts for zero shot

  • Add results

  • Add results and linting

  • fix dependency & clip mock test

  • [MIEB] Add jina clip (#1120)

  • add jina clip and mscoco i2t and t2i results

  • make lint

  • [MIEB] Update mieb with the main branch and some fixes (#1126)

  • fix instruction retrival (#1072)

  • fix instruction retrival

  • fix test

  • add points

  • make nested results

  • add test

  • skip instruction test

  • fix instruction passes

  • fix unions

  • move do_length_ablation

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

  • Update points table

  • fix: fix bug-causing spelling error in function name of e5-mistral-instruct (#1106)

found bug

  • 1.12.85

Automatically generated by python-semantic-release

  • fix: MultilingualSentimentClassification (#1109)

  • Update points table

  • fix: Avoid spaces in dataset name for CQADupstack and ignore speed tasks

  • 1.12.86

Automatically generated by python-semantic-release

  • fix: Ensure that MLSUMClusteringP2P.v2 use the fast implementation as was intended (#1112)

  • fix: Ensure that MLSUMClusteringP2P.v2 use the fast implementation as was intended

  • fix: fixed formatting for cli

  • docs: improve searchability in the advanced usage documentation

  • 1.12.87

Automatically generated by python-semantic-release

  • docs: improve searchability in the advanced usage documentation (#1113)

  • docs: improve searchability in the advanced usage documentation

  • docs: update based on corrections

  • fix: export type for mteb create_meta (#1114)

  • fix export type

  • fix dataset version too

  • 1.12.88

Automatically generated by python-semantic-release

  • fix: Simplify models implementations (#1085)

  • Merge

  • Adapt

  • Simplify

  • Check for rev again

  • Rmv cmmnt

  • Simplify

  • simplify

  • Rmv comment

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

  • Use logging; change try except; add info

  • Lint

  • Rmv results

  • Update rev

  • format

  • Simplify models; Allow instructions

  • Jobs

  • Fix merge

  • Format

  • Adapt models

  • fix: ensure that e5 ignores the NQ

  • format

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

  • 1.12.89

Automatically generated by python-semantic-release

  • fix: nomic models using prefix correctly (#1125)

  • fix: nomic models using prefix correctly

  • chore: remove comment

  • fix: handling in case not torch tensor

  • Fix typo

Co-authored-by: Niklas Muennighoff <n.muennighoff@gmail.com>

  • 1.12.90

Automatically generated by python-semantic-release

  • refactor vista model wrapper to contain lib import

  • python 38 type hints

Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com>
Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: anpalmak2003 <73543260+anpalmak2003@users.noreply.github.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Niklas Muennighoff <n.muennighoff@gmail.com>
Co-authored-by: Zach Nussbaum <zanussbaum@gmail.com>
Co-authored-by: chenghao xiao <85804993+gowitheflow-1998@users.noreply.github.com>

  • image memoery issues for all retrieval Abstasks

  • Add CLEVR and SciMMIR Image-Text Understanding tasks (#1127)

  • Add CLEVER and SciMMIR

  • Update metadata

  • remove useless comment

  • Add linting

  • fix typo and tests

  • Add CLEVR count task

  • add linting

  • add fashion200k & fashionIQ test passed

  • clip text max seq truncation

  • add WebQA, NIGHTS, OVEN

  • any2any retrieval chunk encoding

  • add nomic vision model; any2any topk bug

  • add cv recall

  • add InfoSeek; VisualNews

  • [MIEB] Add Stanford Cars i2i Retrieval (#1147)

  • wip

  • add results

  • make lint

  • change back the order

  • [MIEB] Add CUB200 i2i retrieval (#1154)

  • add cub200 and results

  • add skip_first_result

  • skipped self and rerun results

  • consolidate i2t and t2i to any2any

  • remove abstask and evaluators

  • remove references from test

  • tu-add berlin sketch retrieval

  • XM3600; XFlickr30kCO; mutilingual

  • wit multilingual retrieval t2i

  • correct multilingual t2i meta

  • meta

  • add dinov2 model; 4 sizes

  • cls evaluator channel bug fix

  • add ALIGN model

  • add FORBI2IRetrieval

  • forb & tuberlin new revision

  • disable tokenization parallelism

  • add hateful meme retrieval i2tt2i

  • add memotion retrieval t2ii2t

  • add SciMMIR Retrieval i2tt2i

  • ruff update

  • Visual STS Abstask&evaluator

  • add visual STS17

  • add visual STS 12-16

  • [mieb] Add blip and blip2 models, and ImageNetDog15Clustering task (#1226)

  • wip: start adding BLIP models

  • add other blip variants

  • wip: add blip2_models.py

  • make lint

  • wip: implement blip2 wrapper

  • feat: add blip2 models, still mismatched names

  • fix: remove projections from image and text embeddings

  • make lint

  • wip: add coco BLIP2

  • fix: BLIP2 better zero-shot classification without text_proj and vision_proj

  • tidy blip2

  • add imagenet-dog-15 dataset

  • tidy and lint

  • remove unused import

  • add cluster_accuracy, ari and nmi to Image.ClusteringEvaluator

  • add imagenet-10 clustering task

  • add results forclip on ImageNet10Clustering and ImageNetDog15Clustering

  • [mieb] add 3 compositionality evaluation tasks (#1229)

  • linting & update unavailable dataset path

  • add aro visual relation&attribution; sugarcrepe

  • correct reference

  • add SOPI2IRetrieval dataset/task (#1232)

  • wip: start adding BLIP models

  • add other blip variants

  • wip: add blip2_models.py

  • make lint

  • wip: implement blip2 wrapper

  • feat: add blip2 models, still mismatched names

  • fix: remove projections from image and text embeddi...

Read more

Contributors

mrshu and Samoed
Assets 6
Loading

1.34.1

05 Feb 09:09
@KennethEnevoldsen KennethEnevoldsen
1.34.1
29713ce
Compare
Choose a tag to compare
1.34.1

1.34.1 (2025-02-05)

Documentation

  • docs: Add instruction for running leaderboard (#1925)

  • fix: Add instruction for running leaderboard

@x-tabdeveloping should we rename it from demo to app?

Fixes #1923

Fix

  • fix: Changed callback for slider, accounted for None input (#1969)

Changed callback for slider, accounted for None input (4232427)

Unknown

  • Fix: Corrected model size for UAE (#1965) (a37e559)

  • Revert "feat: Merge MIEB into main 🎉 " (#1957)

  • Revert "feat: Merge MIEB into main 🎉 (#1944)"

This reverts commit 6d63d06.

  • Update extract_model_names.py

  • make lint

  • trigger CI

  • make lint (2c62d21)

  • misc: add warnings (#1945)

  • add warnings

  • update warns

  • add warning to metadata dict

  • Update mteb/evaluation/MTEB.py

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> (195e688)

  • misc: lint main (#1961)

  • lint main

  • try 3.10

  • make lint (cde11ae)

  • Update ModelMeta of inf-retriever-v1 (#1964)

Update inf-retriever-v1 of bf16 version and max_tokens (c22f14d)

Contributors

x-tabdeveloping
Assets 6
Loading

1.34.0

04 Feb 16:56
@KennethEnevoldsen KennethEnevoldsen
1.34.0
9a03325
Compare
Choose a tag to compare
1.34.0

1.34.0 (2025-02-04)

Feature

  • feat: Add new benchmark BEIR-NL (#1909)

  • BEIR-NL datasets

  • BEIR-NL added to benchmarks

  • BEIR-NL annotations_creators changed to derived

  • BEIR-NL sample_creation clarified

  • Update mteb/tasks/Retrieval/nld/MMARCONLRetrieval.py

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

  • Update mteb/tasks/Retrieval/nld/FEVERNLRetrieval.py

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

  • Update mteb/tasks/Retrieval/nld/ClimateFEVERNLRetrieval.py

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

  • descriptions of models are changed to include BEIR-NL

  • dates for BEIR-NL fixed

  • more metadata annotations for BEIR-NL

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> (de8f384)

Unknown

Assets 6
Loading

1.33.1

04 Feb 15:19
@KennethEnevoldsen KennethEnevoldsen
1.33.1
635b204
Compare
Choose a tag to compare
1.33.1

1.33.1 (2025-02-04)

Fix

Assets 6
Loading

1.33.0

04 Feb 12:41
@KennethEnevoldsen KennethEnevoldsen
1.33.0
cd657f5
Compare
Choose a tag to compare
1.33.0

1.33.0 (2025-02-04)

Feature

  • feat: Merge MIEB into main 🎉 (#1944)

  • mieb ZeroshotClassification

  • mieb docs

  • mieb implementation demo

  • model meta; abstask column names; linear probe clf

  • model meta; abstask column names; linear probe clf

  • fix: update naming as candidate_labels

  • Update README.md

  • Update README.md

  • i2tretrieval

  • test load data ignore i2tretrieval

  • [MIEB] Add image clustering (#1088)

  • make lint

  • wip

  • add TinyImageNet and run

  • type hints

  • add accuracy

  • lint

  • remove unused & fix typos

  • T2I Retrieval

  • Any2AnyRetrieval

  • fix tests from merge

  • [MIEB] Add image text pair classification and tests (#1099)

  • add ImageTextPairClassification abstask and evaluator

  • dataset transform into sequence of images for each sample

  • fix processing logic; list of list images compatability

  • lint and docstrings

  • make lint

  • fix failing tests in TaskMetadata

  • add tests for mieb

  • skip gated repo

Co-authored-by: gowitheflow-1998 <jsbs54@durham.ac.uk>

  • [MIEB] Add image classification and zero shot classification tasks (#1101)
  • fix task metadata
  • use overrideable column names
  • add CIFAR datasets
  • add caltech101 dataset
  • add FGVC aircraft dataset
  • add food 101 dataset
  • add OxfordPets dataset
  • remove comments
  • correct cifar100 path
  • update cifar100 classification results
  • cifar zero shot results
  • add caltech101 zero shot
  • matching CLIP paper implementation
  • add aircraft and food zero shot
  • add oxford pets zero shot
  • [MIEB] Add CIFAR clustering (#1104)
    add CIFAR clustering
  • [MIEB] Add more image classification and zero shot classification datasets (#1103)
  • update category to i2t
  • add MNIST linear probe and zero shot
  • add FER2013 linear probe and zero shot
  • add stanford cars linear probe and zero shot
  • add birdsnap linear probe and zero shot
  • add eurosat linear probe and zero shot
  • lint
  • correct eurosat zero shot labels
  • add abstask for image multilable and voc2007
  • make lint
  • [MIEB] Add more image classification and zero shot datasets (#1105)
  • add STL10 linear probe and zero shot
  • add RESISC45 linear probe and zeor shot
  • add Describable textures linear probe and zero shot
  • fix spacing lint
  • add SUN397 linear probe and zero shot
  • correct SUN397 zero shot captions
  • add baai bge vista
  • add e5-v
  • linting
  • memory issues for image linear probe & zeroshot
  • kknn linear probe arguments
  • del comments
  • Add some classification and ZeroShot classification tasks (#1107)
  • Add Country211 classification task
  • Add imagenet1k classification task
  • Add UCF101 classification task
  • Add PatchCamelyon Classification task
  • Add GTSRB classification task
  • Add GSTRB Zero Shot Classification
  • Add country211 zero shot classification
  • Add results for classification tasks
  • Add zero shot classification tasks
  • Add PatchCamelyon tasks and results
  • Add linting
  • Add results and fix prompts for zero shot
  • Add results
  • Add results and linting
  • fix dependency & clip mock test
  • [MIEB] Add jina clip (#1120)
  • add jina clip and mscoco i2t and t2i results
  • make lint
  • [MIEB] Update mieb with the main branch and some fixes (#1126)
  • fix instruction retrival (#1072)
  • fix instruction retrival
  • fix test
  • add points
  • make nested results
  • add test
  • skip instruction test
  • fix instruction passes
  • fix unions
  • move do_length_ablation
    Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

  • Update points table
  • fix: fix bug-causing spelling error in function name of e5-mistral-instruct (#1106)
    found bug
  • 1.12.85
    Automatically generated by python-semantic-release
  • fix: MultilingualSentimentClassification (#1109)
  • Update points table
  • fix: Avoid spaces in dataset name for CQADupstack and ignore speed tasks
  • 1.12.86
    Automatically generated by python-semantic-release
  • fix: Ensure that MLSUMClusteringP2P.v2 use the fast implementation as was intended (#1112)
  • fix: Ensure that MLSUMClusteringP2P.v2 use the fast implementation as was intended
  • fix: fixed formatting for cli
  • docs: improve searchability in the advanced usage documentation
  • 1.12.87
    Automatically generated by python-semantic-release
  • docs: improve searchability in the advanced usage documentation (#1113)
  • docs: improve searchability in the advanced usage documentation
  • docs: update based on corrections
  • fix: export type for mteb create_meta (#1114)
  • fix export type
  • fix dataset version too
  • 1.12.88
    Automatically generated by python-semantic-release
  • fix: Simplify models implementations (#1085)
  • Merge
  • Adapt
  • Simplify
  • Check for rev again
  • Rmv cmmnt
  • Simplify
  • simplify
  • Rmv comment
    Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>
  • Use logging; change try except; add info
  • Lint
  • Rmv results
  • Update rev
  • format
  • Simplify models; Allow instructions
  • Jobs
  • Fix merge
  • Format
  • Adapt models
  • fix: ensure that e5 ignores the NQ
  • format

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

  • 1.12.89
    Automatically generated by python-semantic-release
  • fix: nomic models using prefix correctly (#1125)
  • fix: nomic models using prefix correctly
  • chore: remove comment
  • fix: handling in case not torch tensor
  • Fix typo

Co-authored-by: Niklas Muennighoff <n.muennighoff@gmail.com>

  • 1.12.90
    Automatically generated by python-semantic-release
  • refactor vista model wrapper to contain lib import
  • python 38 type hints

Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com>
Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: anpalmak2003 <73543260+anpalmak2003@users.noreply.github.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Niklas Muennighoff <n.muennighoff@gmail.com>
Co-authored-by: Zach Nussbaum <zanussbaum@gmail.com>
Co-authored-by: chenghao xiao <85804993+gowitheflow-1998@users.noreply.github.com>

  • image memoery issues for all retrieval Abstasks
  • Add CLEVR and SciMMIR Image-Text Understanding tasks (#1127)
  • Add CLEVER and SciMMIR
  • Update metadata
  • remove useless comment
  • Add linting
  • fix typo and tests
  • Add CLEVR count task
  • add linting
  • add fashion200k & fashionIQ test passed
  • clip text max seq truncation
  • add WebQA, NIGHTS, OVEN
  • any2any retrieval chunk encoding
  • add nomic vision model; any2any topk bug
  • add cv recall
  • add InfoSeek; VisualNews
  • [MIEB] Add Stanford Cars i2i Retrieval (#1147)
  • wip
  • add results
  • make lint
  • change back the order
  • [MIEB] Add CUB200 i2i retrieval (#1154)
  • add cub200 and results
  • add skip_first_result
  • skipped self and rerun results
  • consolidate i2t and t2i to any2any
  • remove abstask and evaluators
  • remove references from test
  • tu-add berlin sketch retrieval
  • XM3600; XFlickr30kCO; mutilingual
  • wit multilingual retrieval t2i
  • correct multilingual t2i meta
  • meta
  • add dinov2 model; 4 sizes
  • cls evaluator channel bug fix
  • add ALIGN model
  • add FORBI2IRetrieval
  • forb & tuberlin new revision
  • disable tokenization parallelism
  • add hateful meme retrieval i2tt2i
  • add memotion retrieval t2ii2t
  • add SciMMIR Retrieval i2tt2i
  • ruff update
  • Visual STS Abstask&evaluator
  • add visual STS17
  • add visual STS 12-16
  • [mieb] Add blip and blip2 models, and ImageNetDog15Clustering task (#1226)
  • wip: start adding BLIP models
  • add other blip variants
  • wip: add blip2_models.py
  • make lint
  • wip: implement blip2 wrapper
  • feat: add blip2 models, still mismatched names
  • fix: remove projections from image and text embeddings
  • make lint
  • wip: add coco BLIP2
  • fix: BLIP2 better zero-shot classification without text_proj and vision_proj
  • tidy blip2
  • add imagenet-dog-15 dataset
  • tidy and lint
  • remove unused import
  • add cluster_accuracy, ari and nmi to Image.ClusteringEvaluator
  • add imagenet-10 clustering task
  • add results forclip on ImageNet10Clustering and ImageNetDog15Clustering
  • [mieb] add 3 compositionality evaluation tasks (#1229)
  • linting & update unavailable dataset path
  • add aro visual relation&attribution; sugarcrepe
  • correct reference
  • add SOPI2IRetrieval dataset/task (#1232)
  • wip: start adding BLIP models
  • add other blip variants
  • wip: add blip2_models.py
  • make lint
  • wip: implement blip2 wrapper
  • feat: add blip2 models, still mismatched names
  • fix: remove projections from image and text embeddings
  • make lint
  • wip: add coco BLIP2
  • fix: BLIP2 better zero-shot classification without text_proj and vision_proj
  • tidy blip2
  • add imagenet-dog-15 dataset
  • tidy and lint
  • remove unused import
  • add cluster_accuracy, ari and nmi to Image.ClusteringEvaluator
  • add imagenet-10 clustering task
  • add SOPI2IRetrieval
  • add results forclip on ImageNet10Clustering and ImageNetDog15Clustering
  • add SOPI2IRetrieval results for clip 32
  • add results for clip vit 32/SOPI2IRetrieval
  • resolve conflict
  • change reference
  • Image text pair cls (#1233)
  • fix ImageTextPair dataloading for large datasets; more compositionality evaluation datasets
  • fix meta data
  • fix validate points

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>

  • Add RP2kI2IRetrieval and METI2IRetrieval (#1239)
  • wip: start adding BLIP models
  • add other blip variants
  • wip: add blip2_models.py
  • make lint
  • wip: implement blip2 wrapper
  • feat: add blip2 models, still mismatched names
  • fix: remove projections from image and text embeddings
  • make lint
  • wip: add coco BLIP2
  • fix: BLIP2 better zero-shot classification without text_proj and...
Read more

Contributors

mrshu and Samoed
Assets 6
Loading

1.32.0

04 Feb 10:41
@KennethEnevoldsen KennethEnevoldsen
1.32.0
4e453e5
Compare
Choose a tag to compare
1.32.0

1.32.0 (2025-02-04)

Feature

add beir (7ef3a90)

Unknown

  • Updated links in MTEB(eng) and eng,classic (#1948) (3cf2bed)

  • misc: add bgev1 models (#1928)

  • add bgev1 models

  • add bge-*-en

  • fix naming (e16acf8)

  • misc: add warning for save_suffix removal from AbsTask (#1940)

add warning for param removal (07c489d)

  • Leaderboard: Acks (#1930)

Add acs (476afc7)

Assets 6
Loading

1.31.8

01 Feb 16:03
@KennethEnevoldsen KennethEnevoldsen
1.31.8
dba7a95
Compare
Choose a tag to compare
1.31.8

1.31.8 (2025-02-01)

Documentation

  • docs: Updated citation for mteb(scandinavian) (#1914)

fix: Updated citation for mteb(scandinavian) (f3526fc)

Fix

  • fix: Add datasets in CodeRAG-Bench (#1595)

  • add three out of four datasets in CodeRAG-Bench

  • add verified CodeRAGStackoverflowPostsRetrieval dataset

  • clean up code and make some comments

  • fixed lint errors

  • addressed comments about code-rag datasets: fixed grammar and remove unnessary code and loop

  • roll back files which is not supposed to change

  • fixed the comments in split_by_first_newline() and make the methods private by adding a underscore prefix

  • refactor to use common args

  • update task descriptions

  • add entry in benchmarks

  • correct the alphanumeric order for the dataset

  • add in tasks.md

  • add in tasks.md

  • update task metadata

  • update importing path

  • fix lint errors

  • correct CodeRAG task metadata description field and id for stackoverflow-posts

  • fix error in test

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> (9c762da)

Unknown

Assets 6
Loading

1.31.7

01 Feb 15:31
@KennethEnevoldsen KennethEnevoldsen
1.31.7
23f626d
Compare
Choose a tag to compare
1.31.7

1.31.7 (2025-02-01)

Documentation

  • docs: Add sort to domains for task metadata (#1922)

Tests currently go into an infinite loop. This should prevent that. (6f673ba)

Fix

Unknown

Read more
Assets 6
Loading