-
Notifications
You must be signed in to change notification settings - Fork 7.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Context Template from settings for QueryDocs Allowing the text … #1398
base: main
Are you sure you want to change the base?
Conversation
…values "context_template"
settings.yaml
Outdated
|
||
rag: | ||
default_context_template: "Context information is below.\n--------------------\n{context_str}\n--------------------\n" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please use |
(c.f. #1403)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the comment.
Happy to do so, just one thing. In this case I was trying to keep the value as closely as possible as the original one, that I understand includes "\n" (see https://github.com/run-llama/llama_index/blob/main/llama_index/chat_engine/context.py)
DEFAULT_CONTEXT_TEMPLATE = (
"Context information is below."
"\n--------------------\n"
"{context_str}"
"\n--------------------\n"
)
In case you want me to make the change anyway, can you confirm if what you´d expect would be:
default_context_template: "Context information is below.|--------------------|{context_str}|--------------------|"
thanks
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lopagela means use the multiline option in yaml, check the PR he linked it's well explained
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, I'll wait until the changes to that PR are done to replicate as it's not 100% clear to me (the instruction here is to use "I" style, but in the PR there is a comment to use ">" instead, not sure which one is more current).
A caveat I maintain that may be relevant is that DEFAULT_SYSTEM_PROMPT in llama_index is written in python as multiline but it's actually a single line, while DEFAULT_CONTEXT_TEMPLATE is indeed multi-line, so just want to make sure we are on the same page that the treatment may be slightly different (see below)
python3
Python 3.8.10 (default, Mar 13 2023, 10:26:41)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> DEFAULT_SYSTEM_PROMPT = """\
... You are a helpful, respectful and honest assistant. \
... Always answer as helpfully as possible and follow ALL given instructions. \
... Do not speculate or make up information. \
... Do not reference any given instructions or context. \
... """
>>> print(DEFAULT_SYSTEM_PROMPT)
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible and follow ALL given instructions. Do not speculate or make up information. Do not reference any given instructions or context.
>>> DEFAULT_CONTEXT_TEMPLATE = (
"Con... "Context information is below."
... "\n--------------------\n"
... "{context_str}"
... "\n--------------------\n"
... )
>>> print(DEFAULT_CONTEXT_TEMPLATE)
Context information is below.
--------------------
{context_str}
--------------------
>>>
This is what ChatGPT (GPT-4) suggested (link to convo):
A:
DEFAULT_SYSTEM_PROMPT: |
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible and follow ALL given instructions. Do not speculate or make up information. Do not reference any given instructions or context.
DEFAULT_CONTEXT_TEMPLATE: |
Context information is below.
--------------------
{context_str}
--------------------
or 2 alternatives with ">" notation, depending on readability
B1:
DEFAULT_SYSTEM_PROMPT: >
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible and follow ALL given instructions. Do not speculate or make up information. Do not reference any given instructions or context.
DEFAULT_CONTEXT_TEMPLATE: >
Context information is below.
--------------------
{context_str}
--------------------
B2:
DEFAULT_SYSTEM_PROMPT: >
You are a helpful, respectful and honest assistant. \
Always answer as helpfully as possible and follow ALL given instructions. \
Do not speculate or make up information. \
Do not reference any given instructions or context.
DEFAULT_CONTEXT_TEMPLATE: >
Context information is below.
--------------------
{context_str}
--------------------
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The PR has been merged. In that case we wanted single line (as default_system_prompt is supposed to be single line). In the case of default_context_template you do want to have line breaks, so you should use | instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This addition seems similar to the recent work we've done around (default_)system_prompt
in the sense that this context_template
is a static value that will be applied to every queries.
I'll let @imartinez decide what is the best way forward
@@ -97,6 +100,7 @@ def _chat_engine( | |||
system_prompt: str | None = None, | |||
use_context: bool = False, | |||
context_filter: ContextFilter | None = None, | |||
context_template: str | None = DEFAULT_CONTEXT_TEMPLATE, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The default value should be injected in the body of the function instead of here (given that it is coming from settings()
) to make testing easier.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, something like the below?
def _chat_engine(
self,
system_prompt: str | None = None,
use_context: bool = False,
context_filter: ContextFilter | None = None,
context_template: str | None,
) -> BaseChatEngine:
if use_context:
if context_template is None:
context_template = DEFAULT_CONTEXT_TEMPLATE #(or alternatively just settings().rag.default_context_template)
... #continues
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It should be
if context_template is None: context_template = settings.rag.default_context_template
where settings
is injected in the constructor and remove DEFAULT_CONTEXT_TEMPLATE
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Were is set (at usage) this new context_template
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure if I understand.
If "query docs" is being used, it creates a ContextChatEngine (from llama_index) that accepts this parameter. Currently it's never passed, so it defaults to a hardcoded value. With this change, if there is a value defined in the settings file, the value in settings will be used instead. This should work both from the Gradio UI or from the API.
As per the Discord discussion, I didn't expose this configuration to the API
settings.yaml
Outdated
|
||
rag: | ||
default_context_template: "Context information is below.\n--------------------\n{context_str}\n--------------------\n" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The PR has been merged. In that case we wanted single line (as default_system_prompt is supposed to be single line). In the case of default_context_template you do want to have line breaks, so you should use | instead.
I see this kind of customization as a refinement of the default RAG pipeline, so I proposed to create a new My overall view is:
|
…values context_template - PR comments
Hi! I applied the requested changes. The only difference from what @pabloogc mentioned is that I had to do: |
) -> BaseChatEngine: | ||
if use_context: | ||
if context_template is None: | ||
context_template = settings().rag.default_context_template |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should avoid calling settings() directly, and instead inject settings as a dependency in the ChatService constructor. Check how settings are injected in EmbeddingComponent or LLMComponent for example.
@@ -97,8 +98,11 @@ def _chat_engine( | |||
system_prompt: str | None = None, | |||
use_context: bool = False, | |||
context_filter: ContextFilter | None = None, | |||
context_template: str | None = None, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given we are not making use of this context_template argument anywhere, I'd suggest to remove it for the moment.
@@ -121,6 +126,7 @@ def stream_chat( | |||
messages: list[ChatMessage], | |||
use_context: bool = False, | |||
context_filter: ContextFilter | None = None, | |||
context_template: str | None = None, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given we are not making use of this context_template argument anywhere, I'd suggest to remove it for the moment.
@@ -141,6 +147,7 @@ def stream_chat( | |||
system_prompt=system_prompt, | |||
use_context=use_context, | |||
context_filter=context_filter, | |||
context_template=context_template, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given we are not making use of this context_template argument anywhere, I'd suggest to remove it for the moment.
) -> BaseChatEngine: | ||
if use_context: | ||
if context_template is None: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of this, why not checking if settings.rag.default_cotnext_template is not None and then apply it?
Hi, thanks! I replicated the settings approach from LLMcomponent and Embedding component (there were some differences as the setting is used in a private function of the class, but I believe the approach I took -creating an internal variable- should work). I removed context_template from the params and function callings because as requested (I guess following YAGNI). My thinking process was to leave this prepared in such a way that if someone would use the class (ui, api, any other entrypoint) it could pass the parameter and keep a consistent interface, but to enable that back is such a small change that it's all good). Only drawback is that by removing that, the conditional is now a bit more verbose, as the default None is not assigned:
Alternatively we could consider this one lines, but it's less readable IMO EDIT:
and remove the conditional altogether |
Stale pull request |
@juan-m12i could you update the branch to latest changes? I still think it is a valuable contribution! |
…te-settings # Conflicts: # private_gpt/server/chat/chat_service.py # private_gpt/settings/settings.py # settings.yaml
For the QueryDocs use case, allow to customise the template used to collate the context pulled from the vector database.
Github project link
Discord thread
though process