Token Limit

[Gowtham-Sivam]

1. Consider I have added some KB file where the content is very huge that exceeds the LLM limit in this case. Would the token limit be a problem when the LLM is called with a user question ?

Use case Description :
Consider there is a sql table with millions of rows where the token size is very huge and consider I had added to KB folder and there is a column called profit for each day of some 5 years. If I ask what is the total profit would the LLM be able to add up all the values from those millions of Rows and get me the final correct value ? or would it hallucinate or would it bring up token size limit.

[drazvan]

Hi @Gowtham-Sivam !

Only the top 3 "relevant chunks" are included in the prompt, and their maximum length is limited (default 400 characters - https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/nemoguardrails/kb/utils.py#L22). So, the short answer is no, the token limit should not be reached.

As per the above, that type of query would not work out of the box. Most likely, only 3 rows would be included in the prompt and you would get an incorrect result.