Benchmarks
Benchmarks¶
Ducksearch and BM25s¶
While DuckSearch provide advanced filtering features / updates on the index, DuckSearch only score top_k_token document per query token. Benchmark might evolve with DuckDB improvements and DuckSearch updates.
| Dataset | Metric | Ducksearch | BM25s | Difference (Ducksearch - BM25s) |
|---|---|---|---|---|
| arguana | ndcg@10 | 0.3779 | 0.3663 | +0.0116 |
| hits@1 | 0.0 | 0.0 | 0.0 | |
| mrr@10 | 0.2491 | 0.2443 | +0.0048 | |
| map@10 | 0.2528 | 0.2430 | +0.0098 | |
| qps | 117.80 | 2113.50 | -1995.70 | |
| Index Time(s) | 1.42 | 0.48 | +0.94 | |
| climate-fever | ndcg@10 | 0.1184 | 0.1313 | -0.0129 |
| hits@1 | 0.1068 | 0.1186 | -0.0118 | |
| mrr@10 | 0.1644 | 0.1809 | -0.0165 | |
| map@10 | 0.0803 | 0.0907 | -0.0104 | |
| qps | 5.88 | 99.49 | -93.61 | |
| Index Time(s) | 302.39 | 209.97 | +92.42 | |
| dbpedia-entity | ndcg@10 | 0.6046 | 0.6172 | -0.0126 |
| hits@1 | 0.7669 | 0.7744 | -0.0075 | |
| mrr@10 | 0.8311 | 0.8382 | -0.0071 | |
| map@10 | 0.0649 | 0.0672 | -0.0023 | |
| qps | 113.20 | 182.79 | -69.59 | |
| Index Time(s) | 181.42 | 119.18 | +62.24 | |
| fever | ndcg@10 | 0.3861 | 0.4825 | -0.0964 |
| hits@1 | 0.2583 | 0.3312 | -0.0729 | |
| mrr@10 | 0.3525 | 0.4423 | -0.0898 | |
| map@10 | 0.3329 | 0.4212 | -0.0883 | |
| qps | 74.40 | 104.97 | -30.57 | |
| Index Time(s) | 329.70 | 207.52 | +122.18 | |
| fiqa | ndcg@10 | 0.2445 | 0.2326 | +0.0119 |
| hits@1 | 0.2207 | 0.2160 | +0.0047 | |
| mrr@10 | 0.3002 | 0.2875 | +0.0127 | |
| map@10 | 0.1848 | 0.1726 | +0.0122 | |
| qps | 545.77 | 2157.35 | -1611.58 | |
| Index Time(s) | 6.04 | 4.27 | +1.77 | |
| hotpotqa | ndcg@10 | 0.4487 | 0.5630 | -0.1143 |
| hits@1 | 0.5059 | 0.6523 | -0.1464 | |
| mrr@10 | 0.5846 | 0.7249 | -0.1403 | |
| map@10 | 0.3642 | 0.4697 | -0.1055 | |
| qps | 48.15 | 104.43 | -56.28 | |
| Index Time(s) | 163.14 | 123.39 | +39.75 | |
| msmarco | ndcg@10 | 0.8951 | 0.9705 | -0.0754 |
| hits@1 | 1.0 | 1.0 | 0.0 | |
| mrr@10 | 1.0 | 1.0 | 0.0 | |
| map@10 | 0.0459 | 0.0532 | -0.0073 | |
| qps | 35.11 | 71.26 | -36.15 | |
| Index Time(s) | 202.37 | 229.22 | -26.85 | |
| nfcorpus | ndcg@10 | 0.3301 | 0.3059 | +0.0242 |
| hits@1 | 0.4396 | 0.4458 | -0.0062 | |
| mrr@10 | 0.5292 | 0.5205 | +0.0087 | |
| map@10 | 0.1233 | 0.1168 | +0.0065 | |
| qps | 3464.66 | 3933.12 | -468.46 | |
| Index Time(s) | 0.99 | 1.67 | -0.68 | |
| nq | ndcg@10 | 0.2451 | 0.2735 | -0.0284 |
| hits@1 | 0.1272 | 0.1460 | -0.0188 | |
| mrr@10 | 0.2099 | 0.2366 | -0.0267 | |
| map@10 | 0.1934 | 0.2177 | -0.0243 | |
| qps | 150.23 | 272.62 | -122.39 | |
| Index Time(s) | 71.43 | 87.98 | -16.55 | |
| quora | ndcg@10 | 0.7705 | 0.7491 | +0.0214 |
| hits@1 | 0.6783 | 0.6622 | +0.0161 | |
| mrr@10 | 0.7606 | 0.7433 | +0.0173 | |
| map@10 | 0.7206 | 0.6988 | +0.0218 | |
| qps | 741.13 | 1004.44 | -263.31 | |
| Index Time(s) | 3.78 | 6.57 | -2.79 | |
| scidocs | ndcg@10 | 0.1025 | 0.0993 | +0.0032 |
| hits@1 | 0.1790 | 0.1910 | -0.0120 | |
| mrr@10 | 0.2754 | 0.2765 | -0.0011 | |
| map@10 | 0.0154 | 0.0147 | +0.0007 | |
| qps | 879.11 | 3570.06 | -2690.95 | |
| Index Time(s) | 4.46 | 1.64 | +2.82 | |
| scifact | ndcg@10 | 0.6908 | 0.6617 | +0.0291 |
| hits@1 | 0.5533 | 0.5433 | +0.0100 | |
| mrr@10 | 0. | |||
| 6527 | 0.6312 | +0.0215 | ||
| map@10 | 0.6416 | 0.6199 | +0.0217 | |
| qps | 2153.64 | 3708.28 | -1554.64 | |
| Index Time(s) | 1.22 | 0.41 | +0.81 | |
| trec-covid | ndcg@10 | 0.9533 | 0.8983 | +0.0550 |
| hits@1 | 1.0 | 0.92 | +0.08 | |
| mrr@10 | 1.0 | 0.96 | +0.04 | |
| map@10 | 0.0074 | 0.0069 | +0.0005 | |
| qps | 112.38 | 1275.41 | -1163.03 | |
| Index Time(s) | 22.15 | 10.15 | +12.00 | |
| webis-touche2020 | ndcg@10 | 0.4130 | 0.4671 | -0.0541 |
| hits@1 | 0.5510 | 0.6122 | -0.0612 | |
| mrr@10 | 0.7114 | 0.7541 | -0.0427 | |
| map@10 | 0.0564 | 0.0659 | -0.0095 | |
| qps | 104.65 | 961.73 | -857.08 | |
| Index Time(s) | 44.14 | 34.89 | +9.25 |