Please upgrade the KV cache size yes using --ctx-size

[enn-nafnlaus]

          Please upgrade the KV cache size yes using `--ctx-size`

Originally posted by in #6603 (comment)

This is not an appropriate response to people having this problem. --ctx-size is a memory-limited operation; of course we'd set it higher if we could. Mine is at 16k and I still hit this problem.

The appropriate response to running out of tokens is to fail the query. It's not for the server to go into an infinite loop and stop all further processing. I've lost days worth of processing time to this bug, when I log into my server and discover that it's no longer running because of this.

The server should never go into an infinite loop; I mean, obviously? If it can't handle a query, it should just reject it and move on.

EDIT: The folk was just running a very outdated server version. Always use --n-predict N to avoid infinite loop.

[phymbert]

Do you have an appropriate response then ?

[phymbert]

You are welcome to submit a new scenario using the server test framework to reproduce any issue you identified. It will help the maintainers understand it and eventually submit a patch.

Contributions are welcomed, and helping the community requires a non null effort.

Regarding the closed discussion, some questions may require more attention than others. I did not notice here that the server was infinite looping because the KV cache limit is reached. I am not sure this is unexpected.

@ggerganov, probably I missed something

[enn-nafnlaus]

Do you have an appropriate response then ?

As stated: " If it can't handle a query, it should just reject it and move on." It shouldn't go into an infinite loop and stop serving requests.

As for reproducing it: I don't know which of my queries are triggering it, as they're randomly generated (within given frameworks) and it queues up many queries in advance (perhaps others might have a specific one?). But you do have the error message. The error is in examples/server/server.cpp, in update_slots(), in response to a failure in llama_decode. The symptoms are that the GPU activity drops to zero and instead the server just rapidly repeats that message while cycling through batch sizes.

I don't think saying "helping the community requires a non null effort" to someone who has never refused a request to help is productive - and nor am I the only person who has brought up this problem here.

My commandline:

. venv/bin/activate
CUDA_VISIBLE_DEVICES=1 ./server --model /path/to/TheBloke_Mixtral-8x7B-Instruct-v0.1-GGUF/mixtral-8x7b-instruct-v0.1.Q3_K_M.gguf --port 1234 --batch-size 2048 --threads 4 --threads-batch 1 --numa --mlock --ctx-size 16384 -cb

So yes, if there's some more datagathering you want from me, given the above information, please let me know how you would like me to go about doing so.

[phymbert]

So yes, if there's some more datagathering you want from me, given the above information, please let me know how you would like me to go about doing so.

You need to provide detailed steps to reproduce your issue. For the server, there is a dedicated framework for that.

Again, if your kv cache is full , it is just not adapted for your usage. You did not provide information to prove that the server goes to an infinite loop in this situation.

If you see failed to find free space in the KV cache, you can try to enable defrag. --defrag-thold 0.1.

I advise to change the way you are requesting for help, we are a team of volunteers here, feedback like that is not appreciable and especially it does not help at all.

[enn-nafnlaus]

All anyone here cares about is the server not going into an infinite loop, which should be considered a de minimis requirement for a server. Closing fixing an infinite loop bug on everyone who brings it up (aka, not just me) as "not planned" is beyond bizarre for a server maintainer. Especially when the server runs a non-deterministic process of which the user has little clue of what the output will be in advance (it's not like users ask the server "please write 20 million tokens", it's going to be "do some simple task" but then the server gets stuck in a loop repeating itself).

This (server #2 in this case) is not okay for a server.

Again: I have never refused a request to gather more data. All I ask is that you not close this bug that's hitting multiple people and paralyzing their servers is "not planned". In the meantime, I'll keep working on seeing if I can figure out a consistently repeatable way to reproduce it. It happens on average once every 15 hours or so to me.

[ggerganov]

@phymbert #6603 was closed reasonably - the user asked how to increase context size and an answer was provided. Similar topic was discussed in #5737 (comment)

@enn-nafnlaus First try to update to latest master. If the problem still persists, you can try to set an upper limit for your queries using the --n-predict argument. By default it is set to -1 which means infinite generation (using context shift) unless EOS token is encountered. My guess is that there are some degenerate cases in which EOS is never emitted which causes the issue that is observed, but without a way to reproduce it's hard to figure out what to do.

[enn-nafnlaus]

@phymbert #6603 was closed reasonably - the user asked how to increase context size and an answer was provided. Similar topic was discussed in #5737 (comment)

@enn-nafnlaus First try to update to latest master. If the problem still persists, you can try to set an upper limit for your queries using the --n-predict argument. By default it is set to -1 which means infinite generation (using context shift) unless EOS token is encountered. My guess is that there are some degenerate cases in which EOS is never emitted which causes the issue that is observed, but without a way to reproduce it's hard to figure out what to do.

Thanks for this :) Can't update to the latest update at the moment (I've moved and the server has no net access), but will try --n_predict to see if that works as a workaround (note: I don't know the number of input tokens in advance on the server, so I'll have to set --n_predict to the max context). And will keep trying to see if I can figure out how to reproduce it. Is there any way to have the server print out what queries it's processing, so I can know what one(s) is/are causing the problem? I see it seems to be creating a file called "llama.log", but I can't specify the filename, so I assume that all running servers write to the same logfile...

[enn-nafnlaus]

You did not provide information to prove that the server goes to an infinite loop in this situation.

I didn't realize that bug reports were about having to "prove" that you're actually experiencing the bug. :Þ

See the size of llama.log.bak (moved from llama.log before I restarted the server) above:

That's from the spam from when it goes into an infinite loop.

Obviously I can't attach a 73 gig file. :) Here's a version truncated after it goes into an infinite loop:

llama.log.bz2.txt

(Remove the .txt extension before uploading)

[phymbert]

@ggerganov, thanks. We have a test feature wrong_usage for this use case where the user forgot to pass n_predict. But I am wondering if it should not be defaulted to n_ctx. Thoughts ?

[slaren]

Are you sure that this is wrong usage? I think the logs hint very strongly that there is an infinite loop in update_slots.

In any case, it is also not ok to generate indefinitely with repeated context shifts if the model never generates an eos. At the very least, that's a denial of service vulnerability.

[phymbert]

Yes, either n_ctx is too small (n_batch=512), either the model hallucinates indefinitely without n_predict set.

[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 256
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 128
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 64
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 32
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 16
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 8
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 4
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 2
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 1
[1712885022] update_slots : failed to decode the batch, n_batch = 1, ret = 1

It is not an infinite loop inside update_slots as slots are beeing released when n_batch == 1.

https://github.com/ggerganov/llama.cpp/blob/91c736015b66ba1d0b82cbae6313b6d5eaa61b68/examples/server/server.cpp#L2187-L2193

[slaren]

Wouldn't you see log messages such as this if it is was truly the model generating indefinitely with context shifts?

{"tid":"140358648172544","timestamp":1712918141,"level":"INFO","function":"update_slots","line":1811,"msg":"slot context shift","id_slot":0,"id_task":404,"n_keep":29,"n_left":2,"n_discard":1,"n_ctx":32,"n_past":31,"n_system_tokens":0,"n_cache_tokens":31}

If the slot was released properly, that should be the end of it, but the same thing happens again and again.

[enn-nafnlaus]

Yes, either n_ctx is too small (512), either the model hallucinates indefinitely without n_predict set.

[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 256
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 128
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 64
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 32
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 16
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 8
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 4
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 2
[1712885022] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 1
[1712885022] update_slots : failed to decode the batch, n_batch = 1, ret = 1

It is not an infinite loop inside update_slots as slots are beeing released when n_batch == 1.

https://github.com/ggerganov/llama.cpp/blob/91c736015b66ba1d0b82cbae6313b6d5eaa61b68/examples/server/server.cpp#L2187-L2193

n_ctx is 16k - see my commandline above.

It very much is an infinite loop if it gets stuck on processing a single query indefinitely, all incoming queries just sit there and the GPU usage drops to 0 while the server spits out spam until I restart it.

The instant it hits n_batch == 1 it jumps back to n_batch 512 failures, continuing over and over in a loop. Spam is pumped out at an immense speed (note that the logfile hit 73 gigs overnight), cycling through this over and over, until I notice that it's gotten into this state and kill the server

[phymbert]

Wouldn't you see log messages such as this if it is was truly the model generating indefinitely with context shifts?

These logs unfortunately are only in stdout not in llama.log.

[slaren]

@enn-nafnlaus please try again with the current version of the server, and if it happens again, try to collect logs that include both stdout and stderr. Your client may be ignoring error responses from the server.

[ggerganov]

I've reconstructed the generated text using the provided log and in the end the generation indeed falls into EOS-less generation:

dispatchEvent(new Custom Event   quot transitionendquot ); e &&
(n=null)});})}})(e.target), n        	     = null)}, Number("25" )*Math .
random () + 30 ); break ; caseflexq: n&&((e=>{if (!e)return void document .
dispatchEvent(new Custom Event  & " transitionend   quot;); switch (n){case
this:break; default : e.target ===this &&   	        	     ((n =
null), setTimeout(()=>{},Number("25")*Math random ()+30))}})(e)); break ;
caseflexq-sep: n&&((e=>{if (!e)return void document . dispatchEvent(new Custom
Event & "transitionend   quot;); switch (n){case this:break; default :
e.target ===this &&   	        	     ((n = null),
setTimeout(()=>{},Number("25")*Math random ()+30))}})(e)); break ; caseit
under to: n&&((()=>{for(var t of document .querySelectorAll (&
#91[data-animate="voice"] [data```vbnet animate="window"
]))t.classList["toggle"]("hidden")})()), e => {if (!e) return void
document   	     dispatchEvent(new Custom Event & "transitionend   q
uot;&#x26amp;; switch (n){case this:break ; default : n .
addEventListener("focus",
function(){this.classList["toggle"]("hidden")})}})(e)); break }); setTimeout(()
=> { Array . from(document.querySelectorAllspan)).forEach((el) =???> {if
(!document[“selection”] || !window.getSelection().selectAllContainingRange &&
window        	       . get Selection()["rangeCount"]<1 )return setTimeout(()
=> { Array . from(document 	      querySelectorAllspan)).forEach((el)
=???> {if (!docu ment[“selec tion”] || !window.getSelection().selectA
llContainingRange && window        	        get Selection()["rangeCount"]<1
))return setTimeout(()=&gt;{},Number("25")*Math . random ()+30); el.classList
["add" ](&#x60;"clickable&#x60;&nbsp;;if (!document[“selec tion”] ||
!window.getSelection().selectAllContainingRange && window
get Selection()["rangeCount"]<1 ))el . dispatchEvent(new Custom Event& "focus"
)}), 5e3)})(), Number("20")*Math random ()+3e4); //]]> </script><p>&nbsp;She
had said.</div></blockquote class="flexq"><wbr id=	           ""
/></span>, <a href="/">speaking</a> as they are, perhaps. Waiting to return<wbr
/>to the window.<br/>Where will he ever find a woman&nbsp;</p><blockquote
cite="#" style="background: rgb(254, 178, 63); border-left: 4px solid  	rgb(209
, 108, 35 ); padding : 1em ; margin - inline : none ; color :
white;"><p>he</p></blockquote><span id="can" style="display:         none"> can
</at find a woman<wbr /> he is able to share this with.</div >. He thinks.<sup
classjf>&nbsp;</sup></p> <p>"Do you think&gt;Can</div><span id=	     "think"
style="display: none">find </at find a woman<wbr/>he is able to share this
with.</div >. He thinks.<br /      &#8239;&#x0 26 ; do&gt;Can</div><span id=
"do" style="display: none">think </at find a woman<wbr/>he is able to share
this with.</div >. He thinks.<sup classjf>&nbsp;</ sup>      you think we have
any idea how lucky&gt;Do</ div><span id=	     "any" style="display:
none">lucky </at find a woman<wbr/>he is able to share this with.</div >. He
thinks.<sup classjf>&nbsp;</ sup></p>  <blockquote cite="#"><p>"Do you think we
have any idea how lucky&gt;Any</ div><span id=	     "idea" style="display:
none">lucky </at find a woman<wbr/>he is able to share this with.</div >. He
Thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#"><p>"Do you think we
have any idea how lucky&gt;Luck y</ div><span id=	     "we"
style="display: none">lucky </at find a woman<wbr/>he is able to share this
with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#"><p>"Do you think we have any idea how lucky&gt;Have</ div><span id=
"do" style="display: none">lucky </at find a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do you think we have any idea how lucky&gt;Idea</ div><span id=
"any" style="display: none">lucky </ at find a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do you think we have any idea how&gt;How</ div><span id=
"lucky" style="display: none">any </at find a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do you think we&gt;Think</ div><span id=	     "any"
style="display: none">how </at find a woman<wbr/>he is able to share this
with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#"><
p>"Do you&gt;You</ div><span id=	     "think" style="display: none">we
</at find a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Any</ div><span id=
"you" style="display: none">You </at find a woman<wbr/>he is able to share this
with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#"><
p>"Do you&gt;Have</ div><span id=	     "you" style="display: none">Any
</at find a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</ div><span id=
"do" style="display: none">You </at find a woman<wbr/>he is able to share this
with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#"><
p>"Do you&gt;Idea</ div><span id=	     "do" style="display: none">Have </
at find a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Idea</ div><span id=
"do" style="display: none">You </ at find a woman<wbr/>he is able to share this
with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#"><
p>"Do&gt;How</ div><span id=	     "do" style="display: none">Idea </ at find
a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</ div><span id=
"do" style="display: none">How </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do you&gt;Luck y</ div><span id=	     "you" style="display:
none">How </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</
div><span id=	     "do" style="display: none">You </ at find  a woman<wbr/>he
is able to share this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p>
<blockquote cite="#">< p>"Do&gt;Luck y</ div><span id=	     "do"
style="display: none">You </ at find  a woman<wbr/>he is able to share this
with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#"><
p>"Do&gt;Think</ div><span id=	     "do" style="display: none">Luck y </at
find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Think</ div><span id=
"do" style="display: none">Luck y </at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Think</ div><span id=	     "do" style="display:
none">Any </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</
div><span id=	     "do" style="display: none">Think </at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"do" style="display: none">Any </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;How</ div><span id=	     "do" style="display: none">Have </
at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"do" style="display: none">Idea </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;How</ div><span id=	     "do" style="display: none">Luck y
</at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do you&gt;How</ div><span
id=	     "you" style="display: none">Luck y </at find  a woman<wbr/>he is
able to share this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p>
<blockquote cite="#">< p>"Do&gt;Any</ div><span id=	     "you"
style="display: none">How </ at find  a woman<wbr/>he is able to share this
with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#"><
p>"Do&gt;Any</ div><span id=	     "do" style="display: none">How </ at find
a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Any</ div><span id=
"do" style="display: none">You </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Any</ div><span id=	     "do" style="display: none">You </
at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"do" style="display: none">Any </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;How</ div><span id=	     "do" style="display: none">Luck y
</at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"do" style="display: none">Have </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;How</ div><span id=	     "do" style="display: none">Idea </
at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</ div><span id=
"do" style="display: none">How </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do you&gt;Have</ div><span id=	     "you" style="display:
none">Luck y </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do
you&gt;Idea</ div><span id=	     "you" style="display: none">Have </ at
find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Idea</ div><span id=
"do" style="display: none">You </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;How</ div><span id=	     "do" style="display: none">Idea </
at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"do" style="display: none">Luck y </at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;How</ div><span id=	     "do" style="display: none">Have </
at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do you&gt;Luck y</ div><span
id=	     "you" style="display: none">How </ at find  a woman<wbr/>he is
able to share this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p>
<blockquote cite="#">< p>"Do&gt;Think</ div><span id=	     "do"
style="display: none">Luck y </at find  a woman<wbr/>he is able to share this
with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#"><
p>"Do&gt;Think</ div><span id=	     "do" style="display: none">Any </ at find
a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Think</ div><span id=
"do" style="display: none">Luck y </at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Have</ div><span id=	     "you" style="display:
none">Think </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</
div><span id=	     "do" style="display: none">Think </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Any</ div><span id=
"do" style="display: none">Think </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Any</ div><span id=	     "do" style="display: none">Luck y
</at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Any</ div><span id=
"do" style="display: none">Have </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Any</ div><span id=	     "do" style="display: none">Idea </
at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Think</ div><span id=
"do" style="display: none">Have </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Think</ div><span id=	     "do" style="display:
none">Any </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</
div><span id=	     "do" style="display: none">Any </ at find  a woman<wbr/>he
is able to share this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p>
<blockquote cite="#">< p>"Do&gt;How</ div><span id=	     "do"
style="display: none">Idea </ at find  a woman<wbr/>he is able to share this
with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#"><
p>"Do&gt;Luck y</ div><span id=	     "do" style="display: none">Idea </ at find
a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</ div><span id=
"do" style="display: none">How </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Have</ div><span id=	     "you" style="display:
none">How </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</
div><span id=	     "you" style="display: none">Luck y </at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</ div><span id=
"you" style="display: none">Idea </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Have</ div><span id=	     "you" style="display:
none">How </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Idea</
div><span id=	     "you" style="display: none">How </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Idea</ div><span id=
"you" style="display: none">Luck y </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Idea</ div><span id=	     "you" style="display:
none">Have </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</
div><span id=	     "you" style="display: none">Idea </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</ div><span id=
"you" style="display: none">How </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Luck y</ div><span id=	     "you" style="display:
none">Have </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Any</
div><span id=	     "you" style="display: none">How </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Any</ div><span id=
"you" style="display: none">Luck y </at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Any</ div><span id=	     "you" style="display: none">Idea
</ at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Any</ div><span id=
"you" style="display: none">How </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Think</ div><span id=	     "you" style="display:
none">How </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Think</
div><span id=	     "you" style="display: none">Luck y </at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Think</ div><span id=
"you" style="display: none">Idea </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;How</ div><span id=	     "you" style="display: none">Idea
</ at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"you" style="display: none">Luck y </at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;How</ div><span id=	     "you" style="display: none">Have
</ at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"you" style="display: none">Any </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Luck y</ div><span id=	     "you" style="display:
none">Have </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Idea</
div><span id=	     "you" style="display: none">Have </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Idea</ div><span id=
"you" style="display: none">Luck y </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Idea</ div><span id=	     "you" style="display:
none">How </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</
div><span id=	     "you" style="display: none">Luck y </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</ div><span id=
"you" style="display: none">How </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Have</ div><span id=	     "you" style="display:
none">Idea </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</
div><span id=	     "you" style="display: none">Any </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;You</ div><span id=
"have" style="display: none">Have </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;You</ div><span id=	     "have" style="display: none">Luck
y </ at find  a woman<wbr/>he is able to share this with.</div >. He thinks.<
sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;You</ div><span
id=	     "have" style="display: none">Idea </ at find  a woman<wbr/>he is
able to share this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p>
<blockquote cite="#">< p>"Do&gt;You</ div><span id=	     "have"
style="display: none">How </ at find  a woman<wbr/>he is able to share this
with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#"><
p>"Do&gt;You</ div><span id=	     "have" style="display: none">Any </ at
find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do you&gt;You</ div><span
id=	     "have" style="display: none">Think </ at find  a woman<wbr/>he is
able to share this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p>
<blockquote cite="#">< p>"Do&gt;You</ div><span id=	     "have"
style="display: none">Think </ at find  a woman<wbr/>he is able to share this
with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#"><
p>"Do&gt;How</ div><span id=	     "you" style="display: none">Think </ at
find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"you" style="display: none">Luck y </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;How</ div><span id=	     "you" style="display: none">Have
</ at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"you" style="display: none">Idea </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;How</ div><span id=	     "you" style="display: none">Any </
at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</ div><span id=
"you" style="display: none">How </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Luck y</ div><span id=	     "you" style="display:
none">Idea </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</
div><span id=	     "you" style="display: none">Have </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</ div><span id=
"you" style="display: none">Any </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Have</ div><span id=	     "you" style="display:
none">Any </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</
div><span id=	     "you" style="display: none">Idea </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</ div><span id=
"you" style="display: none">How </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Have</ div><span id=	     "you" style="display:
none">Luck y </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;You</
div><span id=	     "have" style="display: none">Luck y </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;You</ div><span id=
"have" style="display: none">Idea </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;You</ div><span id=	     "have" style="display: none">How
</ at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;You</ div><span id=
"have" style="display: none">Any </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do you&gt;You</ div><span id=	     "have" style="display:
none">Think </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;You</
div><span id=	     "have" style="display: none">Think </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"you" style="display: none">Think </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;How</ div><span id=	     "you" style="display: none">Luck y
</ at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"you" style="display: none">Have </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;How</ div><span id=	     "you" style="display: none">Idea
</ at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"you" style="display: none">Any </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Luck y</ div><span id=	     "you" style="display:
none">How </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</
div><span id=	     "you" style="display: none">Idea </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</ div><span id=
"you" style="display: none">Have </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Luck y</ div><span id=	     "you" style="display:
none">Any </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</
div><span id=	     "you" style="display: none">Any </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</ div><span id=
"you" style="display: none">Idea </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Have</ div><span id=	     "you" style="display:
none">How </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</
div><span id=	     "you" style="display: none">Luck y </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;You</ div><span id=
"have" style="display: none">Luck y </ at find  a woman<wbr/>he is able to
share this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;You</ div><span id=	     "have" style="display: none">Idea
</ at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;You</ div><span id=
"have" style="display: none">How </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;You</ div><span id=	     "have" style="display: none">Any
</ at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do you&gt;You</ div><span
id=	     "have" style="display: none">Think </ at find  a woman<wbr/>he is
able to share this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p>
<blockquote cite="#">< p>"Do&gt;You</ div><span id=	     "have"
style="display: none">Think </ at find  a woman<wbr/>he is able to share this
with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#"><
p>"Do&gt;How</ div><span id=	     "you" style="display: none">Think </ at
find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"you" style="display: none">Luck y </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;How</ div><span id=	     "you" style="display: none">Have
</ at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"you" style="display: none">Idea </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;How</ div><span id=	     "you" style="display: none">Any </
at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</ div><span id=
"you" style="display: none">How </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Luck y</ div><span id=	     "you" style="display:
none">Idea </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</
div><span id=	     "you" style="display: none">Have </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</ div><span id=
"you" style="display: none">Any </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Have</ div><span id=	     "you" style="display:
none">Any </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</
div><span id=	     "you" style="display: none">Idea </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</ div><span id=
"you" style="display: none">How </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Have</ div><span id=	     "you" style="display:
none">Luck y </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;You</
div><span id=	     "have" style="display: none">Luck y </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;You</ div><span id=
"have" style="display: none">Idea </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;You</ div><span id=	     "have" style="display: none">How
</ at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;You</ div><span id=
"have" style="display: none">Any </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do you&gt;You</ div><span id=	     "have" style="display:
none">Think </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;You</
div><span id=	     "have" style="display: none">Think </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"you" style="display: none">Think </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;How</ div><span id=	     "you" style="display: none">Luck y
</ at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"you" style="display: none">Have </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;How</ div><span id=	     "you" style="display: none">Idea
</ at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"you" style="display: none">Any </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Luck y</ div><span id=	     "you" style="display:
none">How </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</
div><span id=	     "you" style="display: none">Idea </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</ div><span id=
"you" style="display: none">Have </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Luck y</ div><span id=	     "you" style="display:
none">Any </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</
div><span id=	     "you" style="display: none">Any </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</ div><span id=
"you" style="display: none">Idea </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Have</ div><span id=	     "you" style="display:
none">How </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</
div><span id=	     "you" style="display: none">Luck y </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;You</ div><span id=
"have" style="display: none">Luck y </ at find  a woman<wbr/>he is able to
share this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;You</ div><span id=	     "have" style="display: none">Idea
</ at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;You</ div><span id=
"have" style="display: none">How </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;You</ div><span id=	     "have" style="display: none">Any
</ at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do you&gt;You</ div><span
id=	     "have" style="display: none">Think </ at find  a woman<wbr/>he is
able to share this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p>
<blockquote cite="#">< p>"Do&gt;You</ div><span id=	     "have"
style="display: none">Think </ at find  a woman<wbr/>he is able to share this
with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#"><
p>"Do&gt;How</ div><span id=	     "you" style="display: none">Think </ at
find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"you" style="display: none">Luck y </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;How</ div><span id=	     "you" style="display: none">Have
</ at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;How</ div><span id=
"you" style="display: none">Idea </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;How</ div><span id=	     "you" style="display: none">Any </
at find  a woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</ div><span id=
"you" style="display: none">How </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Luck y</ div><span id=	     "you" style="display:
none">Idea </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</
div><span id=	     "you" style="display: none">Have </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Luck y</ div><span id=
"you" style="display: none">Any </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Have</ div><span id=	     "you" style="display:
none">Any </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</
div><span id=	     "you" style="display: none">Idea </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;Have</ div><span id=
"you" style="display: none">How </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;Have</ div><span id=	     "you" style="display:
none">Luck y </ at find  a woman<wbr/>he is able to share this with.</div >. He
thinks.< sup classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;You</
div><span id=	     "have" style="display: none">Luck y </ at find  a
woman<wbr/>he is able to share this with.</div >. He thinks.< sup
classjf>&nbsp;</sup></p> <blockquote cite="#">< p>"Do&gt;You</ div><span id=
"have" style="display: none">Idea </ at find  a woman<wbr/>he is able to share
this with.</div >. He thinks.< sup classjf>&nbsp;</sup></p> <blockquote
cite="#">< p>"Do&gt;You</ div><span id=	     "have" style="display: none">How
</

[update_slots errors begin]

Even so, the update_slots : failed to decode the batch, n_batch = 1, ret = 1 errors are not expected for single-slot usage since there should always be space for at least 1 new token thanks to the context shifts. Observing this indicates a bug.

But it could be something that has already been fixed since there was some work recently to improve server, so before looking more into this, try to reproduce using the latest version from master.

[enn-nafnlaus]

I've reconstructed the generated text using the provided log and in the end the generation indeed falls into EOS-less generation:

Even so, the update_slots : failed to decode the batch, n_batch = 1, ret = 1 errors are not expected for single-slot usage since there should always be space for at least 1 new token thanks to the context shifts. Observing this indicates a bug.

But it could be something that has already been fixed since there was some work recently to improve server, so before looking more into this, try to reproduce using the latest version from master.

Client sees only timeouts. Server is truly idle, as you can see in the nvtop screenshot, while it spits out megs-per-second of spam.

(Do note that, as can be seen, I have two instances of the server running at once, one for each GPU, in case that might be related)

Will update as soon as the server gets net connected and will report back. In the meantime I've added in -v --n-predict 16384 and 2>&1 | tee server.log.

Thanks for your help - hopefully we'll ultimately figure out what's sending it into a loop. :)

[enn-nafnlaus]

I'll note that I don't think this is a hallucination / lack of EOS generation issue, because GPU usage drops to 0 when this happens, whereas if it's generating, but just fails to generate an EOS, GPU usage should still continue to be pegged out. The error messages also come way too fast (megs per second) for there to be actual token generations during this time.

I will note one possibility worth considering, which is that I'm running (as noted earlier) two servers, one on each GPU. And I think it's only been server #2 that's been failing. Now, they're also each running separate tasks. To isolate whether it's a server + GPU issue, or a task issue, that's triggering this bug, I've also swapped which task is running on which server/GPU. So when it fails next (that is, assuming --n-predict doesn't prevent failures), it should isolate which aspect is the problem.

[slaren]

The command line that you showed before does not include -ngl to offload any layers, and in this case the GPU would only be used for prompt processing.

[slaren]

I have been trying to reproduce the conditions necessary for the retrying with smaller n_batch error, but I have not been able to. There is always some context available for the slot, so instead of producing this error the server does a context shift. I suspect that this logic does not always work as expected because the batch splitting in llama_decode means that a part of the batch may have been evaluated even if an error is returned.

[enn-nafnlaus]

The command line that you showed before does not include -ngl to offload any layers, and in this case the GPU would only be used for prompt processing.

Sorry, I had to type that in by hand, because I wasn't connected to the server (the lack of net access is really frustrating); part of the command line got lost. I also edited the paths and ports for security reasons.

I have my laptop on the local network now, and it's mobile tethered, so here's the current commandline, copied directly (but again, with paths and port changed):

/server -v --model /path/to/TheBloke_Mixtral-8x7B-Instruct-v0.1-GGUF/mixtral-8x7b-instruct-v0.1.Q3_K_M.gguf --port 1234 --n-gpu-layers 9999 --batch-size 2048 --threads 4 --threads-batch 1 --numa --mlock --ctx-size 16384 -cb --n-predict 16384

Note that the -v and --n-predict are new.

Here's how things look normally when the bug hasn't hit- GPUs pegged.

Compare to the earlier image after the bug hits.

[enn-nafnlaus]

Maybe I should try direct mobile tethering the server so I can update. The current version of llama.cpp is from 29 January.

ED: Success. Got the server tethered and updated llama.cpp to the most recent commit on master.

[phymbert]

It is probably this:

https://github.com/ggerganov/llama.cpp/pull/5708/files#r1501799090

And I just lost 2h more here. No, now we have better logs :)

[enn-nafnlaus]

It is probably this:

https://github.com/ggerganov/llama.cpp/pull/5708/files#r1501799090

And I just lost 2h more here. No, now we have better logs :)

If my understanding is correct and that's committed (it looks like it is), hopefully that'll do the trick. Will update when either the issue happens again, or it fails to happen again. :)

[enn-nafnlaus]

Nope, another fail sadly. This time on the first server / GPU. So it's not bound to the second server or second GPU.

Unfortunately, I don't think the output of tee from the server is very useful. Here's what happens at the transition point between token generation and infinite loop:

I had captured (in theory) a log of all queries sent to the server, via the client side, but I accidentally deleted it when restarting the server (facepalm). Will need to wait for the next infinite loop. That said, I'm not sure even that would be of use unless one were to replay all queries from the start, due to the nature of batching. And it's even worse if the problem has something to do with having two servers running.

Anyway, this rules out the following solutions:

• Updating (did that)
• --n_predict (did that, 16k, same as context length)
• Swapping servers (did that)

What might I try next?

ED: Thinking about it last night, I decided to try to make it deterministic:

• Only one client thread that waits for completion before sending the next task
• Fixed seed 0
• Removed continuous batching from the server commandline

This should help ensure that infinite loops can be tracked down to a single query (assuming it still loops under these conditions) - albeit at the cost of slowing down my work.

[FSSRepo]

I'm dealing with something similar to this, but mainly to stress-test the server processing requests from 4 clients simultaneously (documents of up to 3000 tokens and relatively short questions of 8 tokens) and generating a maximum of 512 tokens n_predict: 512, although it can stop when it encounters a stop word. In the master branch, I haven't had any issues, at least when using more than 1 client (from what I see, @enn-nafnlaus is only using 1 slot with a context of 16K, a case I haven't tested).

@enn-nafnlaus According to the command you claim to be using to start the server:

. venv/bin/activate
CUDA_VISIBLE_DEVICES=1 ./server --model /path/to/TheBloke_Mixtral-8x7B-Instruct-v0.1-GGUF/mixtral-8x7b-instruct-v0.1.Q3_K_M.gguf --port 1234 --batch-size 2048 --threads 4 --threads-batch 1 --numa --mlock --ctx-size 16384 -cb

I notice that you don't have the -ngl parameter, which explains why your GPU isn't being utilized as you expect (perhaps I'm wrong, and you simply forgot to include it when commenting), if your GPU has enough VRAM set -ngl 99.

Perhaps what I'm commenting on may not be relevant, but while I'm also actively conducting tests on the server, I may encounter strange behaviors and will try to fix them.

[enn-nafnlaus]

I'm dealing with something similar to this, but mainly to stress-test the server processing requests from 4 clients simultaneously (documents of up to 3000 tokens and relatively short questions of 8 tokens) and generating a maximum of 512 tokens n_predict: 512, although it can stop when it encounters a stop word. In the master branch, I haven't had any issues, at least when using more than 1 client (from what I see, @enn-nafnlaus is only using 1 slot with a context of 16K, a case I haven't tested).

@enn-nafnlaus According to the command you claim to be using to start the server:

. venv/bin/activate
CUDA_VISIBLE_DEVICES=1 ./server --model /path/to/TheBloke_Mixtral-8x7B-Instruct-v0.1-GGUF/mixtral-8x7b-instruct-v0.1.Q3_K_M.gguf --port 1234 --batch-size 2048 --threads 4 --threads-batch 1 --numa --mlock --ctx-size 16384 -cb

I notice that you don't have the -ngl parameter, which explains why your GPU isn't being utilized as you expect (perhaps I'm wrong, and you simply forgot to include it when commenting), if your GPU has enough VRAM set -ngl 99.

Perhaps what I'm commenting on may not be relevant, but while I'm also actively conducting tests on the server, I may encounter strange behaviors and will try to fix them.

See further down in the thread - that command was hand-typed, not copy-pasted, due to net access issues; I accidentally forgot to type that part.

[enn-nafnlaus]

Well, this is annoying . After everything I did to try to make it fully deterministic (fixed seed, no continuous batching, one query submitted at a time and waiting for the response)... it's still not deterministic) (unless I'm doing something wrong here). I printed out an exact URI and json for the final query, and it just completed normally.

The only other thing I can think to try, and it's not pleasant, is to try shutting down one of my GPUs and its respective server, to see if the problem relates to having two GPUs and servers. Though that will obviously cut my output in half :(

I'll be fully open here, I'll take any solution, even if it's an ugly hack. Like, for example, a timeout would be just fine. Or detecting that, "Hey, it's spit out the same messages at a rate of several megs per second, maybe things aren't okay at home". We don't actually have to solve the problem, just detect the failure.

[enn-nafnlaus]

Can confirm that it still goes into an infinite loop when only one server is running.

[ggerganov]

I suspect that this logic does not always work as expected because the batch splitting in llama_decode means that a part of the batch may have been evaluated even if an error is returned.

This seems plausible. @enn-nafnlaus Could you try running with --batch-size 512 --ubatch-size 512 and see if it still occurs

[enn-nafnlaus]

I suspect that this logic does not always work as expected because the batch splitting in llama_decode means that a part of the batch may have been evaluated even if an error is returned.

This seems plausible. @enn-nafnlaus Could you try running with --batch-size 512 --ubatch-size 512 and see if it still occurs

Will do!

[enn-nafnlaus]

Okay, this is the point where I discover, and then sheepishly admit, that - having gotten so used to working with Python projects - I forgot to run "make" after doing my last update. :Þ So I haven't actually yet ruled out that it was a version issue.

Anyway, ball is in my court now. Will report back.

ED: Huh, the numa flag went away?

[enn-nafnlaus]

Have been running for two days now, and it hasn't gotten stuck in an infinite loop. :)

Now I'm going to need to backtrack and see if I can narrow down whether it was the update or one (or both) of the two new flags you had me add that did the trick.

[phymbert]

So please remove my name from the issue summary. Saying "apologies for my mistake" has never been hurting anyone. You are welcome.
I am happy to have fixed your bug 2 months ago.

[enn-nafnlaus]

So please remove my name from the issue summary. Saying "apologies for my mistake" has never been hurting anyone. You are welcome. I am happy to have fixed your bug 2 months ago.

Why have you been so consistently rude about this? Georgi has been nothing but helpful and respectful. You've done nothing but gaslight, complain, insist that an infinite loop isn't a bug, demand proof that they're actually experiencing the bug that they're reporting, and close other peoples' issues before waiting to see if they were actually fixed.

Do you want me to actually find out whether it was an update or either of the two added flags that fixed it? Because I've spent the past days trying to help you track down this bug that's been hitting plenty of your users, and I was planning to continue trying to figure out what the fix was even though I now have a workaround for myself. But if you don't actually care...?

[phymbert]

Edited for you. Yes I don't care.

[pekeso924]

You may be able to resolve the issue by setting --ctx-size to a value larger than the -b value you specified, and by setting the prompt value with -c. This could be a temporary solution and I'm not sure about the underlying principle, so please don't put too much trust in it. However, I think it's worth trying.