KCPP - Idle Responses - Can this be improved?

[wh33t]

I feel like several versions ago "Idle Responses" used to work slightly differently (and better imo). I feel like it used to just continue outputting responses as if the token output limit wasn't reached. Now it seems to trigger differently and it rarely seems to continue the initial response.

Take the following excerpt from koboldcpp/tiny-llama-1.1b-chat-medical.q8_0. You can see that I start with the input = "Can you please describe in detail how the digestive system works?"

Ps. This is instruct mode.

{{[INPUT]}}
Can you please describe in detail how the digestive system works?
{{[OUTPUT]}}
The digestive system is responsible for breaking down food into its components. It starts with the mouth, where saliva helps break down food particles, and then moves to the esophagus, where acid is formed. This acid helps break down food into smaller particles that are absorbed through the small intestine. The small intestine acts as a reservoir for food, storing
{{[OUTPUT]}} <-- AI Talking to itself - Idle Response 1
Can you explain the different functions of the colon and rectum?
{{[OUTPUT]}} <-- AI answering itself - Idle Response 2
The colon is responsible for digesting fats, proteins, carbohydrates, and fiber. The rectum is responsible for detoxifying the body of waste products. It breaks down fats, proteins, and fiber.
{{[OUTPUT]}} <-- AI Talking to itself - Idle Response 3 
Can you explain the role of the stomach in digestion?
{{[OUTPUT]}} <-- AI answering itself - Idle Response 4
The Stomach plays an essential role in digestion. It regulates the body's energy levels and helps to regulate digestive processes. It also plays a part in defecation.
{{[OUTPUT]}} <-- AI Talking to itself - Idle Response 5
Can you describe the process of digestion during childbirth?

The first {{[OUTPUT]}} is triggered by me submitting the initial {{[INPUT]}}, every subsequent follow up is from the Idle Response triggering.

You can see that the second {{[OUTPUT]}} = "Can you explain the different functions of the colon and rectum?" is the AI NOT continuing the first first output, it's now posing as me (or rather talking to itself). It then answers itself in the next {{[OUTPUT]}} and then once again asks itself another question.

Is there a way to correct this behavior? I feel like I used to be able to use Idle Responses to generate lengthy responses. I would ask a question, go do something else for a few minutes and come back to a several hundred token length response that was fully comprehensible.

What has changed or what can I do to achieve this behavior?

[LostRuins]

Thanks for the feedback. I've added a change that will allow you to correct the behavior by selecting the "Continue Bot Replies" toggle. You can try it on https://lite.koboldai.net/

[wh33t]

It worked flawlessly! I don't believe my "Continue Bot Replies" option works this way in my version. I'm using the latest cuda standalone linux executable. How do I get this update?

{{[INPUT]}}
Can you explain the human digestive system in vivid detail?
{{[OUTPUT]}}
The human digestive system is a complex and fascinating process that begins with food entering the mouth and ends with the elimination of waste products. Here's a detailed breakdown:

1. Mouth: The digestive process starts when food enters the mouth. The teeth mechanically break down the food into smaller pieces through chewing (mastication). The salivary glands produce saliva, which contains an enzyme called amylase that begins to chemically break down carbohydrates. This mixture of chewed food and saliva is known as a bolus.
2. Esophagus: The bolus travels down the esophagus, a muscular tube that connects the mouth to the stomach. Swallowing or deglutition involves coordinated contractions of muscles in the esophagus (peristalsis) to move the food along.
3. Stomach: The stomach is a J-shaped organ located on the left side of the abdomen. It receives the partially digested food from the esophagus and mixes it with gastric juices containing hydrochloric acid and enzymes like pepsin. These substances further break down proteins and kill bacteria. The churning action in the stomach (peristalsis) helps to homogenize the mixture, making it easier for the small intestine to absorb nutrients.
4. Small Intestine: The small intestine is the primary site of digestion and absorption. It is long and coiled (about 20 feet in adults) to increase its surface area for better absorption. The partially digested food from the stomach enters the small intestine in the form of chyme. Here, bile and pancreatic juices aid in breaking down fats and proteins. Enzymes produced by the small intestine continue to break down carbohydrates, proteins, and fats.

[LostRuins]

Update will be available in next version of KoboldCpp.

[wh33t]

Terrific! Tyvm.

While I've got you here, suggestions:

• would it be difficult to give a 1s option to trigger idle responses? 5s is currently the lowest.

• would it be difficult to bring this idle response UI to the main window near the generate button to save us from going through the menus to enable/disable? It's super useful for the GPU poor who have to wait several minutes to generate long outputs. Having a near by toggle or something would be an amazing UX upgrade.

And would you be able to explain how the "Auto" option works versus a timed delay for idle response?