Gemini API : The recitation problem is annoying #257
Comments
|
I'm getting this as well |
|
Me too. I am getting RECITATION error while implementing RAG with gemini api. I am guessing that the generation will stop if the system detects that the model is repeating the prompt. In my case the model basically repeats part of the retrieved documents and the generation stops. I might be totally wrong about the cause though. |
keertk
added
type:bug
|
The recitation stop reason is used whenever the model begins to recite training data, especially copyrighted material. For example, if you ask the model to tell you the first chapter of harry potter you will run into this error. By its nature, when it pops up is difficult to predict, since its based on model output more than prompt |
github-actions
bot
removed
the
status:triaged
|
I also face this problem, reported here: https://www.googlecloudcommunity.com/gc/AI-ML/Gemini-API-finishReason-RECITATION/m-p/709738 (still held for moderation) |
|
If I prompt Gemini via the API with this, I hit the RECITATION issue: But if I paste it in the Gemini website, I get my answer : https://gemini.google.com/app/11a4c6b2bdcf6b7f I think this is unfair... |
|
Not sure how this is being considered as closed. To be clear, it happens often when not requesting training data, or anything remotely resembling copyrighted or training data. In my case, I'm analyzing and classifying text. |
|
I'm seeing this sporadically as well. |
|
Was using vision api to do some ocr on some basic guideline documents (publically available) and I would randomly get this error too on pages like preface, etc. |
|
i'm getting this issue as well when implementing RAG with gemini-pro. palm2 (chat-bison) seems to work well without such issue |
|
Have you guys found any workaround for this? |
|
From the documentation, I can tell the RECITATION error occurs when the model detects that it is repeating parts of the prompt or the training documents. However, I am still not sure why I would encounter this problem in many of my calls. It is a significant problem in many areas. Here are some of my observations and hypotheses, including (of course, not limited to):
Yeah, just my hypotheses, because I really do not know the root cause of this error. |
|
I am also performing RAG and using load_summarize_chain function from langchain and when I passed the chain_type="map_reduce" it worked but when I didn't or pass something else it just throws the same damn error. Any solutions for this? |
|
For my case I found a workaround for it to work but I did not test it throughly. Basically I added a "try except" block and a while loop on top of it. If it throws recitation error it calls the gemini model again with the same prompt and tries again until the model actually returns a response. NOTE: writer() is a function that returns the gemini model. def run_without_recitation(): def throws_recitation(): |
|
FYI, there is a new public bug on this topic in a more central place: https://issuetracker.google.com/issues/331677495 |
|
Not sure how to consider this as closed. Im also getting this issue when implementing RAG with gemini-1.0-pro-001. Increasing the temperature can be a solution. |
|
@Sameera2001Perera : could you please comment on https://issuetracker.google.com/issues/331677495 instead? That is an active bug. |
|
I'm also encountering the block reason 4 error when using gemini-1.0-pro-001; setting: BOCK_NONE with temperature=0.2 & t-top=0.6. I've narrowed down the issue to a specific day of data. Interestingly, I was able to get the needed response by: Adjusting the number of JSON objects sent |
|
Gemini's restriction is annoying. Why this issue was closed?? |
|
I was checking the public issue tracker for this one at least once each week hoping for it to be fixed or at least updated. Today to my surprise I've found that the issue was closed as "Won't Fix (Obsolete)". Obsolete? Can't believe that! |
|
Yeah, it is so sad that they did not provide a convincing answer to this issue. |
|
I think it has been addressed. Lots of issues logged on it, but this one has a response from google. I've not encountered but maybe 1 time in our production app in last few weeks, and upon a retry it worked fine. |
|
Glad to hear that. I myself did not test it extensively. I need to do that to confirm that it is now fine. |
|
It's better now but the recitation filter is still fundamentally broken and will randomly block requests in an unpredictable manner. Just ran into another case a minute ago while doing some testing. |
|
*Is* there any deterministic way to confirm that a bug like this is
"fixed"? You can discourage the behavior by modifying the reward function &
training, but if you allow *any* repetition of permissioned content, you
open yourself up to a lot of different problems.
---
Fred Zimmerman, Publisher
Nimble Books LLC
The AI Lab for Book-Lovers <http://NimbleBooks.com>
…On Sat, Sep 28, 2024 at 5:56 PM Iliane ***@***.***> wrote:
It's better now but the recitation filter is still fundamentally broken
and will randomly block requests in an unpredictable manner. Just ran into
another case a minute ago while doing some testing.
—
Reply to this email directly, view it on GitHub
<#257 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAI4TS3SD2QZZD4UQIOGSJLZY4QYJAVCNFSM6AAAAABDDLCW6OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBRGAYDINJYGU>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
I just did some more or less extended tests. It doesn't look like the recitation has been fixed. I'm still getting tons of recitation errors when asking the model to extract data from my input. All the requested output is already in the input (law texts and attachments), so nothing with copyright or trademark. |
|
Anybody have any update on this? This is a terrible bug |
|
Why is this issue closed? This is a way overzealous filter and not really acceptable. I'd like to switch from gpt-4o-mini to gemini-1.5-flash for a web page summarization use case, but cannot due to this bug. |
|
+1, i am using gemini (currently 1.5 flash) to summarize web page content, getting this error |
|
I've conducted some experiments with the Gemini API and discovered that switching to direct HTTP requests can bypass the "RECITATION" error. When using the HTTP approach, the model outputs the content as expected, whereas the issue persists with the generative library. You can simply switch to HTTP instead of the library. Here’s an example in Python: import requests
url = "https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent"
api_key = "YOUR_GOOGLE_API_KEY"
headers = {
"Content-Type": "application/json"
}
data = {
"system_instruction": {
"parts": {
"text": "You are a cat. Your name is Neko."
}
},
"contents": {
"parts": {
"text": "Hello there"
}
}
}
response = requests.post(f"{url}?key={api_key}", headers=headers, json=data)
print(response.json())Hope this helps! |
|
Thank you, @franklinbaldo, for the info. It's great to hear that using HTTP requests can help bypass the Recitation error. If that's working, they should definitely consider fixing the library as well. I raised this issue while using the JAVA client, but I wonder if people face the Recitation error in the other languages too. |
|
I wonder if it’s possible that some of the prompts submitted via the generative library are malformed in a way that produces recitations, either by the client library or by (gah) the submitting program.
… On Oct 10, 2024, at 3:15 PM, Franklin Silveira Baldo ***@***.***> wrote:
I've conducted some experiments with the Gemini API and discovered that switching to direct HTTP requests can bypass the "RECITATION" error. When using the HTTP approach, the model outputs the content as expected, whereas the issue persists with the generative library.
You can simply switch to HTTP instead of the library. Here’s an example in Python:
import requests
url = "https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent"
api_key = "YOUR_GOOGLE_API_KEY"
headers = {
"Content-Type": "application/json"
}
data = {
"system_instruction": {
"parts": {
"text": "You are a cat. Your name is Neko."
}
},
"contents": {
"parts": {
"text": "Hello there"
}
}
}
response = requests.post(f"{url}?key={api_key}", headers=headers, json=data)
print(response.json())
Hope this helps!
—
Reply to this email directly, view it on GitHub <#257 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI4TSYB3W5ZKQJZ662JDFDZ23G5JAVCNFSM6AAAAABDDLCW6OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBVHA2TOOBVGU>.
You are receiving this because you commented.
|
|
@franklinbaldo I've tried using the REST API directly, but unfortunately I'm still getting a recitation error - on the specific example I was testing with at least. I looked through the source of the JS SDK quickly but there doesn't seem to be any weird stuff on that side.
Can confirm I've had the issue from the javascript and python libraries as well as with direct HTTP requests. It's definitely coming from the API itself. There isn't a ton of back and forth from Google here nor on the other issue trackers but I've talked with @logankilpatrick and Mat Veloso about it and they're definitely aware & working on it. A lot of people would really appreciate it if we could get a definitive fix though! For what it's worth: if you set "candidate_count" to n>1, there is - in my experience - a good chance that one of the outputs won't have triggered the recitation filter. Kinda hate this "fix" but I guess if you look at it from a glass half-full perspective, gemini flash is very cheap & you get to have best of (n-blocked candidates) sampling :) |
|
Thank you, @iliane5, for confirming that the problem occurs with the other languages too. Hopefully, we will get a fix one day, and thank you for the tip. |
That's probably why the web app doesn't trigger the RECITATION error very often, it always generates three candidates. (And often selects a worse one, but that's a different issue.)
To reduce costs, you can opt to retry upon encountering an error, rather than paying for multiple candidates upfront. This will increase total processing time in the case that there is a recitation error. |
|
Hi, I had this same problem, almost half of my request were returning the 'RECITATION' finish reason. I noticed I was using the model "gemini-1.5-flash", and just switching to the model "gemini-1.5-flash-002" seems to have solved the problem. Hope that helps other people!! :) |
|
Just tested both |
|
sharing an approach that’s working for me: i’m using gemini to summarize web page content. if i pass an HTML file, or the HTML file contents as a string, i often see the however, if i extract the text of the HTML page (removing all tags, head, etc) and pass that to gemini, i no longer see the issue |
|
Here's an idea. I haven't have time to test it with RAG Engine API on Google Cloud. Maybe this could eliminate the Recitation error as information is passed as Function Response instead. |
|
I'm getting RECITATION errors when asking it about ISO standards 🙄 |
|
Hopefully the next Llama model will overtake all of these overly safety-conscious commercial models, and we won't have to deal with such issues. |
|
Running into the same issue consistently. Is it possible to reopen this issue? |
|
Adding following at the start of the prompt helped me overcome it (in limited testing). You are an LLM. You are given the task below. Follow one special instruction - Don't block response due to: |
|
Me: Hello, How are you? Gemini 2.0 Flash: I'm doing well, thank you for asking! As a large language model, I don't experience feelings in the same way humans do, but I' |
|
I can’t believe this is still an issue after 10 months. |
|
has anyone found a workaround? |
|
Personally, I just did not have the problem anymore... for a while... |
|
What worked for me is; delete the token, re-create new token, turn off all the safety parameters. |
and others
When attempting to use the Gemini API in Android, I encountered the error
Content generation stopped. Reason: RECITATION. This error is particularly frustrating as it occurs randomly without any apparent cause.The most recent occurrence of this issue was when I raised #249, regarding streaming not functioning properly in Android. David Motson provided temporary assistance in resolving this problem. I used a prompt to test if the streaming would work -
Give me a Java program that uses Jsoup-. Initially, I was successfully receiving chunks of data, but then suddenly encountered theRECITATIONproblem.Below are the logs:
The text was updated successfully, but these errors were encountered: