RankGPT: 作为重排序代理的 LLM

论文“ChatGPT 是否擅长搜索？研究大语言模型作为重排序代理的作用”的官方代码实现。

本项目旨在探索 ChatGPT 和 GPT-4 等生成式 LLM 在信息检索（IR）领域的相关性排序能力。

新闻

• [2023.12.10] 我们的 RankGPT 论文荣获 EMNLP 2023 杰出论文奖！🎉🎉🎉
• [2023.11.06] 引入 指令蒸馏 (Instruction Distillation)：通过简化复杂的排序指令来提升 LLM 的效率。仅使用开源 LLM 即可实现 SOTA 排序性能！
• [2023.10.08] 我们的论文已被 EMNLP 2023 主会接收。查看更新版本请访问：https://arxiv.org/pdf/2304.09542.pdf
• [2023.08.05] 现在通过 LiteLLM 支持 Azure、Claude、Cohere 和 Llama2！
• [2023.07.11] 发布了全新的测试集 NovelEval，其中包含未被最新 LLM（如 GPT-4）污染的新颖搜索问题和段落。详情请参阅 NovelEval。
• [2023.04.23] 在 此处 分享了 10 万条 ChatGPT 在 MS MARCO 训练集上预测的排列结果。
• [2023.04.19] 论文现已上线：https://arxiv.org/abs/2304.09542

快速示例

以下定义了一个查询（Query）和三个候选段落：

item = {
    'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
    'hits': [
        {'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
        {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'},
        {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'}
    ]
}

我们可以使用 ChatGPT 通过指令式排列生成（instructional permutation generation）对这些段落进行重排序。