feat: Threads can override and inherit default params from Assistants, Models, and Engine

[dan-homebrew]

Tasks

• Eng Spec @dan-jan
  • Local Models
  • Remote Models (KIV)
• janhq/internal#66
• Functionality
  • Each Recommended Model should have default values defined in respective model.json
• Documentation
  • Architecture section should contain page on relationship between Threads, Assistants, Models, Engines.
  • Docs page on Models Extension,
  • Docs page on Inference Extension
  • Docs page on Engine extensions
• API Reference (i.e. Swaggerfile)
  • model.json params are well documented in Swaggerfile
  • assistant.json params are well documented in Swaggerfile
  • engine.json params are well documented in Swaggerfile

Spec

Overview

• Threads are a core primitive in Jan, and the main means by which a user will interact with AI
  • A Thread is defined as an exchange between a User and an Assistant
• A thread uses default params that are specified for that Assistant (and its associated Model(s) and Engine(s))
  • A user may choose to override these default params
  • Assistant params are inherited from assistant.json
  • Model params are inherited from the assistant.model's model.json
  • Engine params are inherited from model.engine's engine.json
• The Thread UI offers an optional, slide-in "Right Panel" that allows the user to tinker and change params
  • Any user-specified, thread-level params are persisted in an associated thread.json
  • thread.json is usually empty, as it auto-inherits properties from the associated defaults
• The Thread UI should prompt the user to "Reload" the model + engine if required
  • Model Params and Engine Params will usually require reload
  • We should find way to "automatically" reload it, if user sends new message to Assistant

Wireframes

Excalidraw:
https://link.excalidraw.com/l/kFY0dI05mm/3WHim5p2OLu

[dan-homebrew]

We should ideally capture most if not all of the Model and Inference Engine settings, that Oobabooga, and etc avail.

[dan-homebrew]

See LiteLLM's approach to prefix, suffix and roles and prompt templates

abetlen/llama-cpp-python#717 (comment)

[dan-homebrew]

One question is also whether there should be a "Hardware Settings", e.g. for Apple Silicon users? Is this also present for other types of Hardware - e.g. Intel, etc.

https://www.reddit.com/r/LocalLLaMA/comments/186phti/m1m2m3_increase_vram_allocation_with_sudo_sysctl/

[tikikun]

@dan-jan for current release there will be no different in settings for different hardwares, mostly already automatically handled at build time

[tikikun]

important asset reference for prompt template:
https://github.com/mounta11n/plusplus-caMalL/blob/4fa32ad0e3b61a6e13cdb6261723b6fd608a0fad/examples/server/public/promptFormats.js#L15

[tikikun]

Q:Where are these prompt template coming from?
Most of community projects now using TheBloke metadata as referrence, example.

https://huggingface.co/TheBloke/LLaMA2-13B-Psyfighter2-AWQ#prompt-template-alpaca-tiefighter

[tikikun]

We will have 2 main modes: LOCAL and REMOTE

LOCAL

REMOTE

[tikikun]

Link to excalid draw:
https://excalidraw.com/#json=kOBPg9OoLTCLAm3JO7FHn,qV29wMh7fLvGkFXf5HRYNA

[0xSage]

Asks:

1. Can we the settings properties in swagger/json schema (with validation, etc).
2. A list of prompt templates (so users can have a dropdown selector)

schema fields

id: 
type: 
required: boolean
data_validation: {
  min: 
  max:
}
default:
information: // (i) that tells users what this field actually does

[0xSage]

Discussion updates, mostly as a reminder/reference for @imtuyethan

Based on: https://app.excalidraw.com/s/kFY0dI05mm/3WHim5p2OLu

1. Assistant Section: Add back the model selector

• Users should be able to choose a model to chat with when using Jan assistant.
• Remember: users can only change models at the beginning of a thread, not after thread has started
• So we need to also show a change-model-disabled-state in designs.
• Remember: the model determines the variables in the next sections: inference parameters, model params engine params content may change depending on the model chosen.

2. Users can change Assistant Instructions aka the system message

• Users can edit assistant instructions after conversation starts.
• We briefly discussed sending a new “system message updated... etc.” In the MainPanel. Ashley will play around with a potential design for it.
• The system message may show up in MainPane differently, not as a regular message sent from the user, but as a little box/styled differently?
• Again, this is a nice to have. It's possible we don't show anything related to it in the MainView at all

3. We shouldn't auto save when users edit each field. We should make users click a global save button, which saves all sections at once.

• Reason: reduce fs writes to thread.json
• See (4)

4. Users must restart model after editting model_params or engine_parameters

• If users edit model_parameters & engine_parameters, then they have to save & restart model, a two step process. @imtuyethan will play around with how the designs communicate this.
• These 2 sections should say “requires model restart” somewhere...

5. Alan is working on parameter fields, like mins/max/etc.

6. Model Parameters section UI

• “Prompt Template” is a drop down where ppl can choose from different templates
• Example 1: https://huggingface.co/TheBloke/meditron-7B-GPTQ#prompt-template-chatml
• Example 2: https://huggingface.co/TheBloke/SlimOrca-13B-GGUF#prompt-template-ajibawa-assistant
• UI wise, just show the prompt template as in a plain text box. Advanced users can directly edit the text. Do not highlight : “{prompt}” “{system_message}”, or do anything fancy like Daniel suggested XD.
• Have a reset button so ppl can undo if they mess up. Reset button goes back to assistant.json defaults

7. Actually play around with having a reset button for each sections, at the section level (not global)

8. @rex make sure every model has a prompt template filled in

9. Alan TODO later: low fidelity mock on how “model.json settings is different than thread.model.json settings”

10. Remote model side panel - we didn't discuss this as much. Punt till later

[0xSage]

@imtuyethan

A note on the menu items, I saw that we currently open up model.json assistant.json when users click "Reveal in Finder/JSON" in all the sections.

It should be

• Every section: "Reveal in finder/View as JSON": opens `threads.json
• "Change Default*** Model/Assistant Settings": opens Model/Assistant settings page

[namchuai]

@imtuyethan , please help to verify this.
cc @louis-jan