-
Notifications
You must be signed in to change notification settings - Fork 0
/
local-search.xml
167 lines (80 loc) · 76.2 KB
/
local-search.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
<?xml version="1.0" encoding="utf-8"?>
<search>
<entry>
<title>project</title>
<link href="/2024/01/11/project/"/>
<url>/2024/01/11/project/</url>
<content type="html"><![CDATA[<h3 id="即插即用的视频编辑适配器"><a class="markdownIt-Anchor" href="#即插即用的视频编辑适配器"></a> 即插即用的视频编辑适配器</h3><h5 id="baseline"><a class="markdownIt-Anchor" href="#baseline"></a> baseline</h5><img src="pipeline1.jpeg" style="zoom:50%;" /><h3 id="llm-引导的图像局部编辑"><a class="markdownIt-Anchor" href="#llm-引导的图像局部编辑"></a> LLM 引导的图像局部编辑</h3><h5 id="pipeline"><a class="markdownIt-Anchor" href="#pipeline"></a> pipeline</h5><img src="pipeline2.jpg" style="zoom:50%;" />]]></content>
<categories>
<category>project</category>
</categories>
<tags>
<tag>project</tag>
</tags>
</entry>
<entry>
<title>Stable Diffusion</title>
<link href="/2023/10/10/sd/"/>
<url>/2023/10/10/sd/</url>
<content type="html"><![CDATA[<p>咱张老师建议我先学Stable Diffusion，速速学起！持续更新。</p><hr /><p>Stable Diffusion是基于Latent Diffusion Models (LDMs, paper: <a href="https://arxiv.org/abs/2112.10752">https://arxiv.org/abs/2112.10752</a>) 的文生图模型，所以先学LDMs：<a href="https://zhuanlan.zhihu.com/p/582693939">https://zhuanlan.zhihu.com/p/582693939</a></p>]]></content>
<categories>
<category>Diffusion Model</category>
</categories>
<tags>
<tag>Stable Diffusion</tag>
<tag>Latent Diffusion Models</tag>
</tags>
</entry>
<entry>
<title>DDIM</title>
<link href="/2023/10/05/DDIM/"/>
<url>/2023/10/05/DDIM/</url>
<content type="html"><![CDATA[<p>跟着张老师学扩散模型系列2——DDIM，会持续更新。</p><p><strong>paper</strong>：<a href="https://arxiv.org/abs/2010.02502">https://arxiv.org/abs/2010.02502</a></p><p>DDIM通过去马尔可夫化的方式来实现加速采样，且生成样本的过程是确定的：<a href="https://zhuanlan.zhihu.com/p/627616358">https://zhuanlan.zhihu.com/p/627616358</a></p>]]></content>
<categories>
<category>Diffusion Model</category>
</categories>
<tags>
<tag>DDIM</tag>
</tags>
</entry>
<entry>
<title>[论文阅读] RALM 2</title>
<link href="/2023/10/04/RALM2/"/>
<url>/2023/10/04/RALM2/</url>
<content type="html"><![CDATA[<p>这是第二篇有关检索增强语言模型 (<strong>R</strong>etrieval-<strong>A</strong>ugmented <strong>L</strong>anguage <strong>M</strong>odeling)的笔记，这篇笔记记录的文章大多针对检索源和检索器的限制，提出了有效的改进或替代方法。</p><h3 id="promptagator-few-shot-dense-retrieval-from-8-examples"><a class="markdownIt-Anchor" href="#promptagator-few-shot-dense-retrieval-from-8-examples"></a> Promptagator: Few-shot Dense Retrieval From 8 Examples</h3><p><strong>paper:</strong> <a href="https://arxiv.org/abs/2209.11755">https://arxiv.org/abs/2209.11755</a></p><p><strong>Motivation.</strong> 最近关于信息检索的许多研究都集中在如何从一个任务（通常具有丰富的监督数据）转移到其他各种检索任务上（监督数据有限），并隐含地假设可以从一个任务推广到其他所有任务。但它们忽略了一个事实： 对于不同的检索问题，都针对不同的搜索意图、queries和搜索域。</p><p><strong>Framework.</strong> 由三部分组成：</p><ol><li><p><strong>prompt-based query generation</strong>：将特定于任务的少量示例输入给LLM，为<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mi>T</mi></msub></mrow><annotation encoding="application/x-tex">D_T</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.83333em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em;">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.32833099999999993em;"><span style="top:-2.5500000000000003em;margin-left:-0.02778em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.13889em;">T</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>中的所有文档生成query以得到合成示例<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mover accent="true"><mi>q</mi><mo>^</mo></mover><mo separator="true">,</mo><mi>d</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(\hat q, d)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord accent"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.69444em;"><span style="top:-3em;"><span class="pstrut" style="height:3em;"></span><span class="mord mathnormal" style="margin-right:0.03588em;">q</span></span><span style="top:-3em;"><span class="pstrut" style="height:3em;"></span><span class="accent-body" style="left:-0.16666em;"><span class="mord">^</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.19444em;"><span></span></span></span></span></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="mord mathnormal">d</span><span class="mclose">)</span></span></span></span>；</p></li><li><p><strong>round-trip filtering generated data</strong>：从所有未过滤的合成示例中训练一个初始检索器，然后对于给定的<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>q</mi><mo>^</mo></mover></mrow><annotation encoding="application/x-tex">\hat q</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8888799999999999em;vertical-align:-0.19444em;"></span><span class="mord accent"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.69444em;"><span style="top:-3em;"><span class="pstrut" style="height:3em;"></span><span class="mord mathnormal" style="margin-right:0.03588em;">q</span></span><span style="top:-3em;"><span class="pstrut" style="height:3em;"></span><span class="accent-body" style="left:-0.16666em;"><span class="mord">^</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.19444em;"><span></span></span></span></span></span></span></span></span>，让初始检索器检索与其最相关的k个文档，只有当d出现在这k个文档中时，才保留<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>q</mi><mo>^</mo></mover></mrow><annotation encoding="application/x-tex">\hat q</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8888799999999999em;vertical-align:-0.19444em;"></span><span class="mord accent"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.69444em;"><span style="top:-3em;"><span class="pstrut" style="height:3em;"></span><span class="mord mathnormal" style="margin-right:0.03588em;">q</span></span><span style="top:-3em;"><span class="pstrut" style="height:3em;"></span><span class="accent-body" style="left:-0.16666em;"><span class="mord">^</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.19444em;"><span></span></span></span></span></span></span></span></span>；</p></li><li><p><strong>few-shot promptagator retriever</strong>：先使用Contriever在C4上预训练检索器，然后使用合成示例对其进行微调得到初始检索器，在一定epoch后应用round-trip filtering并继续在过滤的数据上微调检索器。</p><img src="Promptagator.png" style="zoom:50%;" /></li></ol><h3 id="generate-rather-than-retrieve-large-language-models-are-strong-context-generators"><a class="markdownIt-Anchor" href="#generate-rather-than-retrieve-large-language-models-are-strong-context-generators"></a> Generate rather than Retrieve: Large Language Models are Strong Context Generators</h3><p><strong>paper:</strong> <a href="https://arxiv.org/abs/2209.10063">https://arxiv.org/abs/2209.10063</a></p><p><strong>code:</strong> <a href="https://github.com/wyu97/GenRead">https://github.com/wyu97/GenRead</a></p><p><strong>Motivation.</strong> 知识密集型任务现在的主流方法是使用retrieve-then-read pipeline，但这些方法有三个缺点：1. 候选的文档是一个块（比如有100个单词），这可能包含与问题无关的噪声信息；2. 在双塔检索模型中，问题和文档的表示是独立获得的，导致它们之间只捕获了浅层的交互；3. 在大型语料库上检索文档需要先对所有文档进行表示与存储，这限制了检索器的参数。</p><p>论文提出了一个解决知识密集型任务的新视角，即<strong>用大语言模型生成器取代文档检索器</strong>。它首先提示一个大语言模型生成与给定query相关的上下文文档，然后读取生成的文档以预测最终答案。</p><p><strong>Framework.</strong> 给出了两种设置：</p><ol><li><p><strong>Zero-shot setting.</strong></p><p><strong>Step 1. generate</strong> 提示LLM根据给定的问题生成上下文文档；</p><p><strong>Step 2. read</strong> 使用生成的文档和输入的问题从LLM中生成答案。</p></li><li><p><strong>Supervised setting.</strong></p><p><strong>Step 1.</strong> 使用LLM为每个问题生成一个上下文文档或使用无监督检索器从Wikipedia中检索一个文档；</p><p><strong>Step 2.</strong> 对每个问题-文档进行编码，并使用k-means进行聚类；</p><p><strong>Step 3.</strong> 从每个聚类中抽取n个问题-文档对，使用in-context learning生成一个上下文文档，共生成k个上下文文档。然后读取生成的k个文档来生成答案。</p><img src="genread.png" style="zoom:50%;" /></li></ol><h3 id="improving-retrieval-augmented-large-language-models-via-data-importance-learning"><a class="markdownIt-Anchor" href="#improving-retrieval-augmented-large-language-models-via-data-importance-learning"></a> Improving Retrieval-Augmented Large Language Models via Data Importance Learning</h3><p><strong>paper:</strong> <a href="https://arxiv.org/abs/2307.03027">https://arxiv.org/abs/2307.03027</a></p><p><strong>code:</strong> <a href="https://github.com/amsterdata/ragbooster">https://github.com/amsterdata/ragbooster</a></p><p><strong>Motivation.</strong> 检索语料库在现实中很少是干净的(特别是如果底层数据来自网络)，且数据中的噪声和错误来源很难追踪的，针对这一数据质量问题，作者提出通过<strong>学习检索源的数据重要性</strong>来改进检索增强模型。</p><p><strong>思想&贡献.</strong> 提出了一种基于多元线性扩展的算法来评估检索数据的重要性。在多元线性扩展中有指数级多的项，论文的一个<strong>关键贡献</strong>是提出了一种能在多项式时间内进行精确计算的算法，该算法在给定具有可加效用函数和验证集的检索增强模型的情况下，使用模型效用函数的多元线性扩展来精确计算检索语料库中数据的重要性。且进一步提出了一个更有效的<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>ϵ</mi><mo separator="true">,</mo><mi>δ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(ϵ, δ)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord mathnormal">ϵ</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="mord mathnormal" style="margin-right:0.03785em;">δ</span><span class="mclose">)</span></span></span></span>近似算法。</p><p>数据重要性评估器会根据验证集上的性能来学习检索语料库中数据源的权重，这些权重随后被用来重新加权或修剪数据源，并在没有进一步训练的情况下提高模型的性能。</p><img src="ragbooster.png" style="zoom:50%;" /><h3 id="lift-yourself-up-retrieval-augmented-text-generation-with-self-memory"><a class="markdownIt-Anchor" href="#lift-yourself-up-retrieval-augmented-text-generation-with-self-memory"></a> Lift Yourself Up: Retrieval-augmented Text Generation with Self-Memory</h3><p><strong>paper:</strong> <a href="https://arxiv.org/abs/2305.02437">https://arxiv.org/abs/2305.02437</a></p><p><strong>code:</strong> <a href="https://github.com/hannibal046/selfmemory">https://github.com/hannibal046/selfmemory</a></p><p><strong>Motivation.</strong> 许多研究都集中在如何更好地进行检索，但受到有限检索空间、有限内存以及语料库质量的影响，内存增强生成模型的潜力受到限制。</p><p><strong>Framework.</strong> 提出的framework基于观察：在推理过程中与数据分布更相似的内存不是训练数据，而是模型自己在无界生成空间内的输出。</p><ol><li><p><strong>Retrieval-augmented Generator</strong>：</p><ol><li>对于LLM， 使用标准的in-context learning格式给出(x, y)作为演示示例；</li><li>对于可微调的generator，仅保留top-1检索结果进行微调，考虑Joint-Encoder或Dual-Encoder generator的架构。</li></ol></li><li><p><strong>Memory Selector</strong>：最小化selector的预测<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>p</mi><msub><mi>S</mi><mi>θ</mi></msub></msub><mo stretchy="false">(</mo><msub><mi>c</mi><mi>i</mi></msub><mo>∣</mo><mi>x</mi><mo stretchy="false">)</mo><mo>=</mo><mfrac><mrow><mi>e</mi><mi>x</mi><mi>p</mi><mo stretchy="false">(</mo><msub><mi>S</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mtext> </mtext><mo stretchy="false">[</mo><mi>S</mi><mi>E</mi><mi>P</mi><mo stretchy="false">]</mo><mtext> </mtext><msub><mi>c</mi><mi>i</mi></msub><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow><mrow><msubsup><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mrow><mo stretchy="false">∣</mo><mi mathvariant="double-struck">C</mi><mo stretchy="false">∣</mo></mrow></msubsup><mi>e</mi><mi>x</mi><mi>p</mi><mo stretchy="false">(</mo><msub><mi>S</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mtext> </mtext><mo stretchy="false">[</mo><mi>S</mi><mi>E</mi><mi>P</mi><mo stretchy="false">]</mo><mtext> </mtext><msub><mi>c</mi><mi>j</mi></msub><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow></mfrac></mrow><annotation encoding="application/x-tex">p_{S_{\theta}}(c_{i}\mid x) = \frac{exp(S_{\theta}(x\ [SEP]\ c_{i}))}{\sum_{j=1}^{\lvert \mathbb{C} \rvert}exp(S_{\theta}(x\ [SEP]\ c_{j}))}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.00586em;vertical-align:-0.25586em;"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.32833099999999993em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.05764em;">S</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em;"><span style="top:-2.3487714285714287em;margin-left:-0.05764em;margin-right:0.07142857142857144em;"><span class="pstrut" style="height:2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em;">θ</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15122857142857138em;"><span></span></span></span></span></span></span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.25586em;"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal">c</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.31166399999999994em;"><span style="top:-2.5500000000000003em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">∣</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal">x</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:1.8841799999999997em;vertical-align:-0.8741799999999998em;"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.01em;"><span style="top:-2.44864em;"><span class="pstrut" style="height:3em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mop mtight"><span class="mop op-symbol small-op mtight" style="position:relative;top:-0.0000050000000000050004em;">∑</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.0590857142857142em;"><span style="top:-2.2134285714285715em;margin-left:0em;margin-right:0.07142857142857144em;"><span class="pstrut" style="height:2.5357142857142856em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.05724em;">j</span><span class="mrel mtight">=</span><span class="mord mtight">1</span></span></span></span><span style="top:-3.059085714285714em;margin-right:0.07142857142857144em;"><span class="pstrut" style="height:2.5357142857142856em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mopen mtight">∣</span><span class="mord mtight"><span class="mord mathbb mtight">C</span></span><span class="mclose mtight">∣</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.46117142857142857em;"><span></span></span></span></span></span></span><span class="mspace mtight" style="margin-right:0.19516666666666668em;"></span><span class="mord mathnormal mtight">e</span><span class="mord mathnormal mtight">x</span><span class="mord mathnormal mtight">p</span><span class="mopen mtight">(</span><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.05764em;">S</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em;"><span style="top:-2.3487714285714287em;margin-left:-0.05764em;margin-right:0.07142857142857144em;"><span class="pstrut" style="height:2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em;">θ</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15122857142857138em;"><span></span></span></span></span></span></span><span class="mopen mtight">(</span><span class="mord mathnormal mtight">x</span><span class="mspace mtight"><span class="mtight"> </span></span><span class="mopen mtight">[</span><span class="mord mathnormal mtight" style="margin-right:0.05764em;">S</span><span class="mord mathnormal mtight" style="margin-right:0.05764em;">E</span><span class="mord mathnormal mtight" style="margin-right:0.13889em;">P</span><span class="mclose mtight">]</span><span class="mspace mtight"><span class="mtight"> </span></span><span class="mord mtight"><span class="mord mathnormal mtight">c</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3280857142857143em;"><span style="top:-2.357em;margin-left:0em;margin-right:0.07142857142857144em;"><span class="pstrut" style="height:2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.05724em;">j</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.2818857142857143em;"><span></span></span></span></span></span></span><span class="mclose mtight">)</span><span class="mclose mtight">)</span></span></span></span><span style="top:-3.23em;"><span class="pstrut" style="height:3em;"></span><span class="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.485em;"><span class="pstrut" style="height:3em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">e</span><span class="mord mathnormal mtight">x</span><span class="mord mathnormal mtight">p</span><span class="mopen mtight">(</span><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.05764em;">S</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em;"><span style="top:-2.3487714285714287em;margin-left:-0.05764em;margin-right:0.07142857142857144em;"><span class="pstrut" style="height:2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em;">θ</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15122857142857138em;"><span></span></span></span></span></span></span><span class="mopen mtight">(</span><span class="mord mathnormal mtight">x</span><span class="mspace mtight"><span class="mtight"> </span></span><span class="mopen mtight">[</span><span class="mord mathnormal mtight" style="margin-right:0.05764em;">S</span><span class="mord mathnormal mtight" style="margin-right:0.05764em;">E</span><span class="mord mathnormal mtight" style="margin-right:0.13889em;">P</span><span class="mclose mtight">]</span><span class="mspace mtight"><span class="mtight"> </span></span><span class="mord mtight"><span class="mord mathnormal mtight">c</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3280857142857143em;"><span style="top:-2.357em;margin-left:0em;margin-right:0.07142857142857144em;"><span class="pstrut" style="height:2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.143em;"><span></span></span></span></span></span></span><span class="mclose mtight">)</span><span class="mclose mtight">)</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.8741799999999998em;"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span></span></span></span>和候选的得分<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>p</mi><mi>M</mi></msub><mo stretchy="false">(</mo><msub><mi>c</mi><mi>i</mi></msub><mo stretchy="false">)</mo><mo>=</mo><mfrac><mrow><mi>e</mi><mi>x</mi><mi>p</mi><mo stretchy="false">(</mo><mi mathvariant="normal">Δ</mi><mo stretchy="false">(</mo><msub><mi>c</mi><mi>i</mi></msub><mo separator="true">,</mo><mi>y</mi><mo stretchy="false">)</mo><mi mathvariant="normal">/</mi><mi>τ</mi><mo stretchy="false">)</mo></mrow><mrow><msubsup><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mrow><mo stretchy="false">∣</mo><mi mathvariant="double-struck">C</mi><mo stretchy="false">∣</mo></mrow></msubsup><mi>e</mi><mi>x</mi><mi>p</mi><mo stretchy="false">(</mo><mi mathvariant="normal">Δ</mi><mo stretchy="false">(</mo><msub><mi>c</mi><mi>j</mi></msub><mo separator="true">,</mo><mi>y</mi><mo stretchy="false">)</mo><mi mathvariant="normal">/</mi><mi>τ</mi><mo stretchy="false">)</mo></mrow></mfrac></mrow><annotation encoding="application/x-tex">p_{M}(c_{i})= \frac{exp(\Delta(c_{i},y)/ \tau)}{\sum_{j=1}^{\lvert \mathbb{C} \rvert}exp(\Delta(c_{j},y)/ \tau)}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.32833099999999993em;"><span style="top:-2.5500000000000003em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.10903em;">M</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal">c</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.31166399999999994em;"><span style="top:-2.5500000000000003em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:1.8841799999999997em;vertical-align:-0.8741799999999998em;"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.01em;"><span style="top:-2.44864em;"><span class="pstrut" style="height:3em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mop mtight"><span class="mop op-symbol small-op mtight" style="position:relative;top:-0.0000050000000000050004em;">∑</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.0590857142857142em;"><span style="top:-2.2134285714285715em;margin-left:0em;margin-right:0.07142857142857144em;"><span class="pstrut" style="height:2.5357142857142856em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.05724em;">j</span><span class="mrel mtight">=</span><span class="mord mtight">1</span></span></span></span><span style="top:-3.059085714285714em;margin-right:0.07142857142857144em;"><span class="pstrut" style="height:2.5357142857142856em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mopen mtight">∣</span><span class="mord mtight"><span class="mord mathbb mtight">C</span></span><span class="mclose mtight">∣</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.46117142857142857em;"><span></span></span></span></span></span></span><span class="mspace mtight" style="margin-right:0.19516666666666668em;"></span><span class="mord mathnormal mtight">e</span><span class="mord mathnormal mtight">x</span><span class="mord mathnormal mtight">p</span><span class="mopen mtight">(</span><span class="mord mtight">Δ</span><span class="mopen mtight">(</span><span class="mord mtight"><span class="mord mathnormal mtight">c</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3280857142857143em;"><span style="top:-2.357em;margin-left:0em;margin-right:0.07142857142857144em;"><span class="pstrut" style="height:2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.05724em;">j</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.2818857142857143em;"><span></span></span></span></span></span></span><span class="mpunct mtight">,</span><span class="mord mathnormal mtight" style="margin-right:0.03588em;">y</span><span class="mclose mtight">)</span><span class="mord mtight">/</span><span class="mord mathnormal mtight" style="margin-right:0.1132em;">τ</span><span class="mclose mtight">)</span></span></span></span><span style="top:-3.23em;"><span class="pstrut" style="height:3em;"></span><span class="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.485em;"><span class="pstrut" style="height:3em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">e</span><span class="mord mathnormal mtight">x</span><span class="mord mathnormal mtight">p</span><span class="mopen mtight">(</span><span class="mord mtight">Δ</span><span class="mopen mtight">(</span><span class="mord mtight"><span class="mord mathnormal mtight">c</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3280857142857143em;"><span style="top:-2.357em;margin-left:0em;margin-right:0.07142857142857144em;"><span class="pstrut" style="height:2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.143em;"><span></span></span></span></span></span></span><span class="mpunct mtight">,</span><span class="mord mathnormal mtight" style="margin-right:0.03588em;">y</span><span class="mclose mtight">)</span><span class="mord mtight">/</span><span class="mord mathnormal mtight" style="margin-right:0.1132em;">τ</span><span class="mclose mtight">)</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.8741799999999998em;"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span></span></span></span>之间的KL散度。</p><img src="selfmem1.png" style="zoom:50%;" /><img src="selfmem2.png" style="zoom:50%;" /></li></ol><h3 id="selective-annotation-makes-language-models-better-few-shot-learners"><a class="markdownIt-Anchor" href="#selective-annotation-makes-language-models-better-few-shot-learners"></a> Selective Annotation Makes Language Models Better Few-Shot Learners</h3><p><strong>paper:</strong> <a href="https://arxiv.org/abs/2209.01975">https://arxiv.org/abs/2209.01975</a></p><p><strong>code:</strong> <a href="https://github.com/xlang-ai/icl-selective-annotation">https://github.com/xlang-ai/icl-selective-annotation</a></p><p><strong>Motivation.</strong> In-context learning仅需要每个测试示例的几个注释示例，但通常假设所有带注释的训练数据都可以用于快速检索。论文研究了如何在保持in-context learning性能的同时降低人工注释成本。</p><p><strong>Framework.</strong></p><ol><li><p>在测试前选择示例进行注释（选择具有多样性和代表性的示例），论文提出了Vote-k选择注释方法：这是一种基于图的方法，图G = (V, E)中的顶点v为所有未被注释示例的嵌入向量，对每个顶点v，基于cosine为它创建k条到它最近的顶点的边。定义L和U分别为已被注释的示例和剩下的示例的集合，每次迭代时，取得分最高的顶点将其从U移动到L（得分的计算公式为<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi><mi>c</mi><mi>o</mi><mi>r</mi><mi>e</mi><mo stretchy="false">(</mo><mi>u</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mo>∑</mo><mrow><mi>v</mi><mo>∈</mo><mo stretchy="false">{</mo><mi>v</mi><mo>∣</mo><mo stretchy="false">(</mo><mi>v</mi><mo separator="true">,</mo><mi>u</mi><mo stretchy="false">)</mo><mo>∈</mo><mi>E</mi><mo separator="true">,</mo><mi>v</mi><mo>∈</mo><mi>U</mi><mo stretchy="false">}</mo></mrow></msub><mi>s</mi><mo stretchy="false">(</mo><mi>v</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">score(u)=\sum_{v \in \{ v \mid (v,u) \in E, v \in U \}} s(v)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">c</span><span class="mord mathnormal">o</span><span class="mord mathnormal" style="margin-right:0.02778em;">r</span><span class="mord mathnormal">e</span><span class="mopen">(</span><span class="mord mathnormal">u</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:1.22471em;vertical-align:-0.47471em;"></span><span class="mop"><span class="mop op-symbol small-op" style="position:relative;top:-0.0000050000000000050004em;">∑</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.22528999999999993em;"><span style="top:-2.40029em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em;">v</span><span class="mrel mtight">∈</span><span class="mopen mtight">{</span><span class="mord mathnormal mtight" style="margin-right:0.03588em;">v</span><span class="mrel mtight">∣</span><span class="mopen mtight">(</span><span class="mord mathnormal mtight" style="margin-right:0.03588em;">v</span><span class="mpunct mtight">,</span><span class="mord mathnormal mtight">u</span><span class="mclose mtight">)</span><span class="mrel mtight">∈</span><span class="mord mathnormal mtight" style="margin-right:0.05764em;">E</span><span class="mpunct mtight">,</span><span class="mord mathnormal mtight" style="margin-right:0.03588em;">v</span><span class="mrel mtight">∈</span><span class="mord mathnormal mtight" style="margin-right:0.10903em;">U</span><span class="mclose mtight">}</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.47471em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="mord mathnormal">s</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em;">v</span><span class="mclose">)</span></span></span></span>，其中<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi><mo stretchy="false">(</mo><mi>v</mi><mo stretchy="false">)</mo><mo>=</mo><msup><mi>ρ</mi><mrow><mo>−</mo><mo stretchy="false">∣</mo><mo stretchy="false">{</mo><mi>l</mi><mo>∈</mo><mi>L</mi><mo>∣</mo><mo stretchy="false">(</mo><mi>v</mi><mo separator="true">,</mo><mi>l</mi><mo stretchy="false">)</mo><mo>∈</mo><mi>E</mi><mo stretchy="false">}</mo><mo stretchy="false">∣</mo></mrow></msup><mo separator="true">,</mo><mi>ρ</mi><mo>></mo><mn>1</mn></mrow><annotation encoding="application/x-tex">s(v)=\rho ^ {-\lvert \{ l \in L \mid (v,l) \in E \} \rvert}, \rho > 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal">s</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em;">v</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:1.0824399999999998em;vertical-align:-0.19444em;"></span><span class="mord"><span class="mord mathnormal">ρ</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8879999999999999em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">−</span><span class="mopen mtight">∣</span><span class="mopen mtight">{</span><span class="mord mathnormal mtight" style="margin-right:0.01968em;">l</span><span class="mrel mtight">∈</span><span class="mord mathnormal mtight">L</span><span class="mrel mtight">∣</span><span class="mopen mtight">(</span><span class="mord mathnormal mtight" style="margin-right:0.03588em;">v</span><span class="mpunct mtight">,</span><span class="mord mathnormal mtight" style="margin-right:0.01968em;">l</span><span class="mclose mtight">)</span><span class="mrel mtight">∈</span><span class="mord mathnormal mtight" style="margin-right:0.05764em;">E</span><span class="mclose mtight">}</span><span class="mclose mtight">∣</span></span></span></span></span></span></span></span></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="mord mathnormal">ρ</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">></span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:0.64444em;vertical-align:0em;"></span><span class="mord">1</span></span></span></span>），共运行M/10次迭代，然后使用L作为LLM的in-context learning示例，并为U中的每个示例生成预测。计算生成输出的平均对数概率作为模型的置信度得分，根据置信度得分将U划分为M个大小相等的桶，将前9M/10个桶中每个桶的具有最高得分的顶点添加到L中。</p></li><li><p>测试时从带注释的集合中检索一些示例作为每个测试示例的in-context示例。</p><img src="selective-annotation.png" style="zoom:50%;" /></li></ol>]]></content>
<categories>
<category>paper-reading</category>
</categories>
<tags>
<tag>RALM</tag>
<tag>NLP</tag>
<tag>IR</tag>
</tags>
</entry>
<entry>
<title>DDPM</title>
<link href="/2023/09/30/DDPM/"/>
<url>/2023/09/30/DDPM/</url>
<content type="html"><![CDATA[<p>zxc同学提供的DDPM学习路线（yyds）！</p><p><s>由于自己一直以来疏于整理，在查找一些读过的对自己很有帮助的文章上效率十分低下</s>，所以想以这种形式来方便自己学习/复习（也是搭博客的初衷了）。</p><p>希望能早点学完DDPM！会持续更新。</p><p><strong>前向过程和反向过程</strong>：<a href="https://zhuanlan.zhihu.com/p/610012156">https://zhuanlan.zhihu.com/p/610012156</a></p><p><strong>优化目标（损失函数）</strong>：</p><ul><li><p><a href="https://zhuanlan.zhihu.com/p/563661713">https://zhuanlan.zhihu.com/p/563661713</a></p></li><li><p><a href="https://lilianweng.github.io/posts/2021-07-11-diffusion-models/">https://lilianweng.github.io/posts/2021-07-11-diffusion-models/</a></p></li></ul><p><strong>整体框架、流程</strong>：<a href="https://zhuanlan.zhihu.com/p/652684897">https://zhuanlan.zhihu.com/p/652684897</a></p><p>补充知识：<a href="https://zhuanlan.zhihu.com/p/542478018">重参数技巧</a>、<a href="https://blog.csdn.net/m0_46385527/article/details/121149701">变分推断</a></p>]]></content>
<categories>
<category>Diffusion Model</category>
</categories>
<tags>
<tag>DDPM</tag>
</tags>
</entry>
<entry>
<title>[论文阅读] RALM 1</title>
<link href="/2023/09/28/RALM1/"/>
<url>/2023/09/28/RALM1/</url>
<content type="html"><![CDATA[<p>笔者近期快速刷了几篇与检索增强语言模型 (<strong>R</strong>etrieval-<strong>A</strong>ugmented <strong>L</strong>anguage <strong>M</strong>odeling) 相关的文章，并<em>自行</em>将其归为了三类。这篇笔记记录的文章大多基于LLM自身的推理能力来对检索方式进行设计。</p><h3 id="investigating-the-factual-knowledge-boundary-of-large-language-models-with-retrieval-augmentation"><a class="markdownIt-Anchor" href="#investigating-the-factual-knowledge-boundary-of-large-language-models-with-retrieval-augmentation"></a> Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation</h3><p><strong>paper:</strong> <a href="https://arxiv.org/abs/2307.11019">https://arxiv.org/abs/2307.11019</a></p><p><strong>code:</strong> <a href="https://github.com/RUCAIBox/LLM-Knowledge-Boundary">https://github.com/RUCAIBox/LLM-Knowledge-Boundary</a></p><p><strong>Motivation.</strong> 目前，领域内缺乏对LLM事实知识边界的深入了解。</p><p><strong>内容.</strong> 文章回答了如下问题：</p><ul><li><p>LLM感知其事实知识边界的能力如何？ LLM对事实知识边界的感知是不准确的，并对自身结果过度自信。</p></li><li><p>检索增强对LLM有什么影响？ LLM不能充分利用它们所拥有的知识，而检索增强可以一定程度上弥补这一缺陷。</p></li><li><p>具有不同特征的辅助文档如何影响LLM？ 提供高质量的辅助文档时，LLM性能更佳且更加自信；LLM倾向于依赖所提供的辅助文档生成反馈。辅助文档与问题的相关性越强，LLM越自信，也更加依赖辅助文档。</p></li></ul><p><strong>思考.</strong> 文章发现<strong>基于先验判断动态地引入检索能很好地提升性能</strong>。即先要求LLM判断它们是否能够提供问题的答案能很好地提升性能，这能给检索增强模型的设计带来启发。</p><h3 id="interleaving-retrieval-with-chain-of-thought-reasoning-for-knowledge-intensive-multi-step-questions"><a class="markdownIt-Anchor" href="#interleaving-retrieval-with-chain-of-thought-reasoning-for-knowledge-intensive-multi-step-questions"></a> Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions</h3><p><strong>paper:</strong> <a href="https://arxiv.org/abs/2212.10509">https://arxiv.org/abs/2212.10509</a></p><p><strong>code:</strong> <a href="https://github.com/stonybrooknlp/ircot">https://github.com/stonybrooknlp/ircot</a></p><p><strong>Motivation.</strong> 对于复杂的多步骤推理问题，仅对知识库进行一次检索是不够的。</p><p><strong>Framework.</strong> IRCoT利用LMs的CoT生成能力来指导检索，并反过来使用检索来改进CoT推理：</p><p>首先，将问题作为query检索一组基本段落；然后，交替进行以下两个步骤:</p><ol><li>扩展CoT: 使用问题、迄今为止收集到的段落和迄今为止生成的CoT句子来生成下一个CoT句子;</li><li>扩展检索到的信息: 使用最后一个CoT句子作为query来检索其他段落。</li></ol><p>重复上述步骤，直到CoT报告答案，或达到所允许的最大推理步骤数。终止时，将所有收集到的段落作为检索结果返回。</p><img src="IRCoT.png" style="zoom: 50%;" /><h3 id="active-retrieval-augmented-generation"><a class="markdownIt-Anchor" href="#active-retrieval-augmented-generation"></a> Active Retrieval Augmented Generation</h3><p><strong>paper:</strong> <a href="https://arxiv.org/abs/2305.06983">https://arxiv.org/abs/2305.06983</a></p><p><strong>code:</strong> <a href="https://github.com/jzbjyb/FLARE">https://github.com/jzbjyb/FLARE</a></p><p><strong>Motivation.</strong> 1、大多现有的检索增强LLMs仅根据输入检索一次信息，但在涉及生成长文本的更一般的场景中，在生成过程中不断收集信息是必不可少的。2、过去尝试在生成输出时进行多次检索的工作主要以固定的间隔使用前面的文本作为query。</p><p><strong>Framework.</strong> 文章提出的FLARE能在整个生成过程中<strong>主动决定</strong>何时检索和检索什么，并给出了<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>F</mi><mi>L</mi><mi>A</mi><mi>R</mi><msub><mi>E</mi><mrow><mi>i</mi><mi>n</mi><mi>s</mi><mi>t</mi><mi>r</mi><mi>u</mi><mi>c</mi><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">FLARE_{instruct}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.83333em;vertical-align:-0.15em;"></span><span class="mord mathnormal" style="margin-right:0.13889em;">F</span><span class="mord mathnormal">L</span><span class="mord mathnormal">A</span><span class="mord mathnormal" style="margin-right:0.00773em;">R</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em;">E</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.31166399999999994em;"><span style="top:-2.5500000000000003em;margin-left:-0.05764em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">s</span><span class="mord mathnormal mtight">t</span><span class="mord mathnormal mtight" style="margin-right:0.02778em;">r</span><span class="mord mathnormal mtight">u</span><span class="mord mathnormal mtight">c</span><span class="mord mathnormal mtight">t</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>和<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>F</mi><mi>L</mi><mi>A</mi><mi>R</mi><msub><mi>E</mi><mrow><mi>d</mi><mi>i</mi><mi>r</mi><mi>e</mi><mi>c</mi><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">FLARE_{direct}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.83333em;vertical-align:-0.15em;"></span><span class="mord mathnormal" style="margin-right:0.13889em;">F</span><span class="mord mathnormal">L</span><span class="mord mathnormal">A</span><span class="mord mathnormal" style="margin-right:0.00773em;">R</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em;">E</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.33610799999999996em;"><span style="top:-2.5500000000000003em;margin-left:-0.05764em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">d</span><span class="mord mathnormal mtight">i</span><span class="mord mathnormal mtight" style="margin-right:0.02778em;">r</span><span class="mord mathnormal mtight">e</span><span class="mord mathnormal mtight">c</span><span class="mord mathnormal mtight">t</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>两种方法。</p><ol><li><p><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>F</mi><mi>L</mi><mi>A</mi><mi>R</mi><msub><mi>E</mi><mrow><mi>i</mi><mi>n</mi><mi>s</mi><mi>t</mi><mi>r</mi><mi>u</mi><mi>c</mi><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">FLARE_{instruct}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.83333em;vertical-align:-0.15em;"></span><span class="mord mathnormal" style="margin-right:0.13889em;">F</span><span class="mord mathnormal">L</span><span class="mord mathnormal">A</span><span class="mord mathnormal" style="margin-right:0.00773em;">R</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em;">E</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.31166399999999994em;"><span style="top:-2.5500000000000003em;margin-left:-0.05764em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">s</span><span class="mord mathnormal mtight">t</span><span class="mord mathnormal mtight" style="margin-right:0.02778em;">r</span><span class="mord mathnormal mtight">u</span><span class="mord mathnormal mtight">c</span><span class="mord mathnormal mtight">t</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>: 提示LM在必要时生成retrieval queries，同时使用retrieval-encouraging指令生成回答: 让LM在需要额外信息时生成“[Search(query)]”。但可能会出现以下问题：</p><ol><li>LMs产生的search queries比需要的少：对token “[” 的logit加上2.0来解决。</li><li>生成过多的search queries可能会破坏答案的生成、对性能产生影响：用search queries检索相关信息时，及时将“[Search(query)]”从生成中删除，并向“[” 的logit添加一个大的负值来禁止“[” 。</li></ol><img src="FLARE1.png" style="zoom: 33%;" /></li><li><p><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>F</mi><mi>L</mi><mi>A</mi><mi>R</mi><msub><mi>E</mi><mrow><mi>d</mi><mi>i</mi><mi>r</mi><mi>e</mi><mi>c</mi><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">FLARE_{direct}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.83333em;vertical-align:-0.15em;"></span><span class="mord mathnormal" style="margin-right:0.13889em;">F</span><span class="mord mathnormal">L</span><span class="mord mathnormal">A</span><span class="mord mathnormal" style="margin-right:0.00773em;">R</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em;">E</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.33610799999999996em;"><span style="top:-2.5500000000000003em;margin-left:-0.05764em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">d</span><span class="mord mathnormal mtight">i</span><span class="mord mathnormal mtight" style="margin-right:0.02778em;">r</span><span class="mord mathnormal mtight">e</span><span class="mord mathnormal mtight">c</span><span class="mord mathnormal mtight">t</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>: 直接使用LM的生成作为queries。由两部分组成：</p><ol><li><p>基于置信度的主动检索：令<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>s</mi><mo>^</mo></mover><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">\hat{s}_t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.84444em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord accent"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.69444em;"><span style="top:-3em;"><span class="pstrut" style="height:3em;"></span><span class="mord"><span class="mord mathnormal">s</span></span></span><span style="top:-3em;"><span class="pstrut" style="height:3em;"></span><span class="accent-body" style="left:-0.19444em;"><span class="mord">^</span></span></span></span></span></span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2805559999999999em;"><span style="top:-2.5500000000000003em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>为LM在第t步临时生成的句子、<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>θ</mi></mrow><annotation encoding="application/x-tex">θ</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.69444em;vertical-align:0em;"></span><span class="mord mathnormal" style="margin-right:0.02778em;">θ</span></span></span></span>为阈值，当<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>s</mi><mo>^</mo></mover><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">\hat{s}_t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.84444em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord accent"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.69444em;"><span style="top:-3em;"><span class="pstrut" style="height:3em;"></span><span class="mord"><span class="mord mathnormal">s</span></span></span><span style="top:-3em;"><span class="pstrut" style="height:3em;"></span><span class="accent-body" style="left:-0.19444em;"><span class="mord">^</span></span></span></span></span></span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2805559999999999em;"><span style="top:-2.5500000000000003em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>中有token的概率小于<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>θ</mi></mrow><annotation encoding="application/x-tex">θ</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.69444em;vertical-align:0em;"></span><span class="mord mathnormal" style="margin-right:0.02778em;">θ</span></span></span></span>时，则触发检索；</p></li><li><p>基于置信度的query表示：给出了两种query表示方法：1) Masked sentences作为隐式query：mask概率低于置信度的tokens；2) 生成问题作为显示query：提取概率低于置信度的所有区间，对于每个提取的区间z，提示gpt-3.5-turbo生成一个问题，这个问题可以用z来回答。</p><img src="FLARE2.png" style="zoom:33%;" /></li></ol></li></ol><h3 id="decomposed-prompting-a-modular-approach-for-solving-complex-tasks"><a class="markdownIt-Anchor" href="#decomposed-prompting-a-modular-approach-for-solving-complex-tasks"></a> Decomposed Prompting: A Modular Approach for Solving Complex Tasks</h3><p><strong>paper:</strong> <a href="https://arxiv.org/abs/2210.02406">https://arxiv.org/abs/2210.02406</a></p><p><strong>code:</strong> <a href="https://github.com/allenai/DecomP">https://github.com/allenai/DecomP</a></p><p><strong>Motivation.</strong> Few-shot prompting是使用LLMs解决各种任务的一种强大方法，但当任务的复杂性增加或任务本身的单个推理步骤难以学习时，这种方法就会出现问题。</p><p><strong>Method.</strong> 如图，将复杂问题提供给decomposer prompt以获得第一个子问题Q1，并将其提供给split prompt，由这个prompt生成的答案随后被附加到decomposer prompt中，以获得第二个子问题Q2，因为该问题中含有foreach操作符，所以会生成两个问题，并将它们提供给str_pos prompt以得到关于两个回答的数组，这用于生成第三个子问题Q3，并提供给merge prompt以获得最终答案。因为问题已经被解决，所以decomposer prompt会产生结束标记[EOQ]并将前一个回答作为最终答案返回。</p><img src="Decomposed_Prompting.png" style="zoom:50%;" /><h3 id="search-in-the-chain-towards-accurate-credible-and-traceable-large-language-models-for-knowledge-intensive-tasks"><a class="markdownIt-Anchor" href="#search-in-the-chain-towards-accurate-credible-and-traceable-large-language-models-for-knowledge-intensive-tasks"></a> Search-in-the-Chain: Towards Accurate, Credible and Traceable Large Language Models for Knowledge-intensive Tasks</h3><p><strong>paper:</strong> <a href="https://arxiv.org/abs/2304.14732v6">https://arxiv.org/abs/2304.14732v6</a></p><p><strong>code:</strong> <a href="https://github.com/xsc1234/Search-in-the-Chain">https://github.com/xsc1234/Search-in-the-Chain</a></p><p><strong>Motivation.</strong> 之前的工作存在着由IR检索到错误的知识对LLM产生误导或破坏LLM推理链的缺点。此外，以前的方法假设推理链中生成的答案总是正确的，不能在必要时及时修改推理方向。</p><p><strong>Framework.</strong> SearChain关注推理链的全局规划：</p><p>首先，让LLM构建一个全局推理链C，其中每个节点是一个query-answer对，如果对于某个节点，LLM不知道答案，则将query标记为[Unsolved Query]. 在每一轮交互中，IR对C上每个节点的信息进行<em>验证和补全</em>，直到C上所有query都不需要更正或达到最大交互轮数时结束。</p><p><em>验证</em>：如果<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>a</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">a_i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.58056em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathnormal">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.31166399999999994em;"><span style="top:-2.5500000000000003em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>与检索到的文档<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">d_i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.84444em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathnormal">d</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.31166399999999994em;"><span style="top:-2.5500000000000003em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>信息不一致，则构建一个prompt帮助LLM更正<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>a</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">a_i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.58056em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathnormal">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.31166399999999994em;"><span style="top:-2.5500000000000003em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>得到<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>a</mi><mi>i</mi><mtext>′</mtext></msubsup></mrow><annotation encoding="application/x-tex">a_i^′</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.010556em;vertical-align:-0.258664em;"></span><span class="mord"><span class="mord mathnormal">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.751892em;"><span style="top:-2.441336em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">′</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.258664em;"><span></span></span></span></span></span></span></span></span></span>，并生成一个以<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><msub><mi>q</mi><mi>i</mi></msub><mo separator="true">,</mo><msubsup><mi>a</mi><mi>i</mi><mtext>′</mtext></msubsup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(q_i,a_i^′)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.010556em;vertical-align:-0.258664em;"></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em;">q</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.31166399999999994em;"><span style="top:-2.5500000000000003em;margin-left:-0.03588em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="mord"><span class="mord mathnormal">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.751892em;"><span style="top:-2.441336em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">′</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.258664em;"><span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>为根节点的推理链CoQ.</p><p><em>补全</em>：对于被标记为[Unsolved Query]的query <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>q</mi><mi>i</mi><mtext>∗</mtext></msubsup></mrow><annotation encoding="application/x-tex">q_i^∗</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.94736em;vertical-align:-0.258664em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em;">q</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.688696em;"><span style="top:-2.441336em;margin-left:-0.03588em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">∗</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.258664em;"><span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>d</mi><mi>i</mi><mtext>∗</mtext></msubsup></mrow><annotation encoding="application/x-tex">d_i^∗</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.953104em;vertical-align:-0.258664em;"></span><span class="mord"><span class="mord mathnormal">d</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.688696em;"><span style="top:-2.441336em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">∗</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.258664em;"><span></span></span></span></span></span></span></span></span></span> 为检索到的文档，<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>g</mi><mtext>∗</mtext></msup></mrow><annotation encoding="application/x-tex">g^∗</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8831359999999999em;vertical-align:-0.19444em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em;">g</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.688696em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">∗</span></span></span></span></span></span></span></span></span></span></span>为从文档中提取到的答案，<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>g</mi><mtext>∗</mtext></msup></mrow><annotation encoding="application/x-tex">g^∗</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8831359999999999em;vertical-align:-0.19444em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em;">g</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.688696em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">∗</span></span></span></span></span></span></span></span></span></span></span>和<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>d</mi><mi>i</mi><mtext>∗</mtext></msubsup></mrow><annotation encoding="application/x-tex">d_i^∗</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.953104em;vertical-align:-0.258664em;"></span><span class="mord"><span class="mord mathnormal">d</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.688696em;"><span style="top:-2.441336em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">∗</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.258664em;"><span></span></span></span></span></span></span></span></span></span>将以prompt的形式反馈给LLM，LLM收到反馈给出答案<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>a</mi><mi>i</mi><mtext>∗</mtext></msubsup></mrow><annotation encoding="application/x-tex">a_i^∗</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.94736em;vertical-align:-0.258664em;"></span><span class="mord"><span class="mord mathnormal">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.688696em;"><span style="top:-2.441336em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">∗</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.258664em;"><span></span></span></span></span></span></span></span></span></span>，并生成一个以<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><msubsup><mi>q</mi><mi>i</mi><mtext>∗</mtext></msubsup><mo separator="true">,</mo><msubsup><mi>a</mi><mi>i</mi><mtext>∗</mtext></msubsup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(q_i^∗,a_i^∗)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.008664em;vertical-align:-0.258664em;"></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em;">q</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.688696em;"><span style="top:-2.441336em;margin-left:-0.03588em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">∗</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.258664em;"><span></span></span></span></span></span></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="mord"><span class="mord mathnormal">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.688696em;"><span style="top:-2.441336em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">∗</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.258664em;"><span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>为根节点的推理链CoQ.</p><img src="Searchain.png" style="zoom:50%;" /><h3 id="in-context-retrieval-augmented-language-models"><a class="markdownIt-Anchor" href="#in-context-retrieval-augmented-language-models"></a> In-Context Retrieval-Augmented Language Models</h3><p><strong>paper:</strong> <a href="https://arxiv.org/abs/2302.00083">https://arxiv.org/abs/2302.00083</a></p><p><strong>code:</strong> <a href="https://github.com/AI21Labs/in-context-ralm">https://github.com/AI21Labs/in-context-ralm</a></p><p><strong>内容.</strong> 论文提出In-Context RALM。论文还针对 (1) 哪种现成检索器最适合语言建模；(2) 检索操作的频率；(3) 最佳的query长度 进行了分析，并引入了两种重排方法。</p><img src="In-context_RALM.png" style="zoom:50%;" />]]></content>
<categories>
<category>paper-reading</category>
</categories>
<tags>
<tag>RALM</tag>
<tag>NLP</tag>
<tag>IR</tag>
</tags>
</entry>
</search>