evaluate

Evaluate candidates matchs.

Parameters

• scores (list[list[dict]])

• qrels (dict)

  Qrels.

• queries (list[str])

  index of queries of qrels.

• metrics (list | None) – defaults to None

  Metrics to compute.

Examples

>>> from pylate import evaluation

>>> scores = [
...     [{"id": "1", "score": 0.9}, {"id": "2", "score": 0.8}],
...     [{"id": "3", "score": 0.7}, {"id": "4", "score": 0.6}],
... ]

>>> qrels = {
...     "query1": {"1": True, "2": True},
...     "query2": {"3": True, "4": True},
... }

>>> queries = ["query1", "query2"]

>>> results = evaluation.evaluate(
...     scores=scores,
...     qrels=qrels,
...     queries=queries,
...     metrics=["ndcg@10", "hits@1"],
... )