Why do not you add query prefix to model when evaluating STS?

[daegonYu]

Hello.

Can you tell me why you don't add query prefix to the model when evaluating sts? I think you should add query prefix for more accurate evaluation, but I wonder why you don't add it in mteb benchmark.

thank you

[KennethEnevoldsen]

Hi @daegonYu

Different models handle STS differently e.g. Jinas model would use text-matching as it prompts for STS tasks.

We do allow the model to know the task (we give the task name as input) so the implementation can be implemented however you like, but by default, we do not give the model a query prompt.

Is there specific models where you believe it should be changed?

[Samoed]

does each model use the specified query prefix during evaluation for STS tasks?

If model added specific prompt to thier model_prompts, then yes.

Additionally, can I submit a model by defining a different prefix for each task and have it evaluated accordingly?

Yes, you can documentation and example

[daegonYu]

Thanks for your reply!

[OnAnd0n]

@KennethEnevoldsen @Samoed
I believe that applying a query prefix in the following manner may not work well for the STS task evaluation.

-Applying the "query:" prefix to Sentence 1
-Not applying the "query:" prefix to Sentence 2

Would this be right?

I am asking just in case there are any users who might prefer this approach.

[Samoed]

Same prefix will be applied to both sentences

[OnAnd0n]

thanks for your reply!!