Making IR dataset compatible with MTEB

[Muennighoff]

@conceptofmind was asking:

I was looking to make an IR dataset compatible with MTEB. The dataset consists of a query and an article. Are there any guides on making the dataset work directly with MTEB evaluation?
Beir has https://github.com/beir-cellar/beir/wiki/Load-your-custom-dataset

I think the mteb reference would be: https://github.com/embeddings-benchmark/mteb/blob/main/docs/adding_a_dataset.md
It may help to reference one of the retrieval datasets like SciFact: https://github.com/embeddings-benchmark/mteb/blob/main/mteb/tasks/Retrieval/eng/SciFactRetrieval.py

[conceptofmind]

Thank you for the input. I will review everything.

[KennethEnevoldsen]

Sounds great! Let us know if there is any questions

[conceptofmind]

@KennethEnevoldsen @Muennighoff

Is there an upper limit to the number of samples that should be added? For example if one of the subsets contains 1m documents.

[KennethEnevoldsen]

@conceptofmind there is no upper limit - I would generally recommend keeping the number of samples as small as needed (often people overestimate the samples required for evaluation)

[conceptofmind]

@conceptofmind there is no upper limit - I would generally recommend keeping the number of samples as small as needed (often people overestimate the samples required for evaluation)

Thank you for the input. I will keep that in mind and scale it appropriately