You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: .github/pull_request_template.md
+4-4
Original file line number
Diff line number
Diff line change
@@ -29,12 +29,12 @@
29
29
-[ ] I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
30
30
-[ ] If the dataset is too big (e.g. >2048 examples), considering using `self.stratified_subsampling() under dataset_transform()`
31
31
-[ ] I have filled out the metadata object in the dataset file (find documentation on it [here](https://github.com/embeddings-benchmark/mteb/blob/main/docs/adding_a_dataset.md#2-creating-the-metadata-object)).
32
-
-[ ] Run tests locally to make sure nothing is broken using `make test`.
33
-
-[ ] Run the formatter to format the code using `make lint`.
32
+
-[ ] Run tests locally to make sure nothing is broken using `make test`.
33
+
-[ ] Run the formatter to format the code using `make lint`.
34
34
35
35
36
36
### Adding a model checklist
37
-
<!--
37
+
<!--
38
38
When adding a model to the model registry
39
39
see also https://github.com/embeddings-benchmark/mteb/blob/main/docs/reproducible_workflow.md
40
40
-->
@@ -43,4 +43,4 @@ see also https://github.com/embeddings-benchmark/mteb/blob/main/docs/reproducibl
43
43
-[ ] I have ensured that my model can be loaded using
44
44
-[ ]`mteb.get_model(model_name, revision)` and
45
45
-[ ]`mteb.get_model_meta(model_name, revision)`
46
-
-[ ] I have tested the implementation works on a representative set of tasks.
46
+
-[ ] I have tested the implementation works on a representative set of tasks.
Copy file name to clipboardexpand all lines: CONTRIBUTING.md
+3-3
Original file line number
Diff line number
Diff line change
@@ -1,4 +1,4 @@
1
-
## Contributing to mteb
1
+
## Contributing to MTEB
2
2
3
3
We welcome contributions to `mteb` such as new tasks, code optimization or benchmarks.
4
4
@@ -18,7 +18,7 @@ cd mteb
18
18
make install
19
19
```
20
20
21
-
This uses [make](https://www.gnu.org/software/make/) to define the install command. You can see what each command does in the [makefile](https://github.com/embeddings-benchmark/mteb/blob/main/Makefile).
21
+
This uses [make](https://www.gnu.org/software/make/) to define the install command. You can see what each command does in the [makefile](https://github.com/embeddings-benchmark/mteb/blob/main/Makefile).
22
22
23
23
### Running Tests
24
24
@@ -52,4 +52,4 @@ This command is equivalent to the command run during CI. It will check for code
52
52
53
53
Any commit with one of these prefixes will trigger a version bump upon merging to the main branch as long as tests pass. A version bump will then trigger a new release on PyPI as well as a new release on GitHub.
54
54
55
-
Other prefixes will not trigger a version bump. For example, `docs:`, `chore:`, `refactor:`, etc., however they will structure the commit history and the changelog. You can find more information about this in the [python-semantic-release documentation](https://python-semantic-release.readthedocs.io/en/latest/). If you do not intend to trigger a version bump you're not required to follow this convention when contributing to `mteb`.
55
+
Other prefixes will not trigger a version bump. For example, `docs:`, `chore:`, `refactor:`, etc., however they will structure the commit history and the changelog. You can find more information about this in the [python-semantic-release documentation](https://python-semantic-release.readthedocs.io/en/latest/). If you do not intend to trigger a version bump you're not required to follow this convention when contributing to MTEB.
# for an example of a HF subset see "Subset" in the dataset viewer at: https://huggingface.co/datasets/mteb/bucc-bitext-mining
152
152
```
153
153
154
+
* by their modalities
155
+
156
+
```python
157
+
tasks = mteb.get_tasks(modalities=["text", "image"]) # Only select tasks with text or image modalities
158
+
```
159
+
160
+
You can also specify exclusive modality filtering to only get tasks with exactly the requested modalities (default behavior with exclusive_modality_filter=False):
161
+
```python
162
+
# Get tasks with text modality, this will also include tasks having both text and image modalities
`mteb` comes with a set of predefined benchmarks. These can be fetched using `get_benchmark` and run in a similar fashion to other sets of tasks.
176
+
`mteb` comes with a set of predefined benchmarks. These can be fetched using `get_benchmark` and run in a similar fashion to other sets of tasks.
162
177
For instance to select the 56 English datasets that form the "Overall MTEB English leaderboard":
163
178
164
179
```python
@@ -248,13 +263,13 @@ class CustomModel:
248
263
**kwargs,
249
264
) -> np.ndarray:
250
265
"""Encodes the given sentences using the encoder.
251
-
266
+
252
267
Args:
253
268
sentences: The sentences to encode.
254
269
task_name: The name of the task.
255
270
prompt_type: The prompt type to use.
256
271
**kwargs: Additional arguments to pass to the encoder.
257
-
272
+
258
273
Returns:
259
274
The encoded sentences.
260
275
"""
@@ -298,7 +313,7 @@ evaluation.run(model)
298
313
299
314
### Using a cross encoder for reranking
300
315
301
-
To use a cross encoder for reranking, you can directly use a CrossEncoder from SentenceTransformers. The following code shows a two-stage run with the second stage reading results saved from the first stage.
316
+
To use a cross encoder for reranking, you can directly use a CrossEncoder from SentenceTransformers. The following code shows a two-stage run with the second stage reading results saved from the first stage.
0 commit comments