gguf: Add types for LLM architectures

[ngxson]

Resolve #566

This PR introduces a generator script that pulls llama.cpp file, then use regex to extract needed information.

It also addresses some smaller issues:

• Some places (for example Attention, Rope, MOE,...) use type intersection while it should be union
• TokenizerModel should be limited to 4 models, see llm_load_vocab(): https://github.com/ggerganov/llama.cpp/blob/928e0b7013c862cf10701957b3d654aa70f11bd8/llama.cpp#L4198

While it's fun to have the script & these type definitions, I think it would be nice to have some clearer directions on how to use them in the future.

Demo:

[julien-c]

very cool work @ngxson

I've launched the CI run, let me know if any help is needed, LGTM otherwise

[mishig25]

@ngxson locally you can check the errors since the CI uses same commands:

pnpm format
pnpm lint

[ngxson]

@julien-c @mishig25 Regarding how to actually use these types, I'm thinking about having a list of functions like this:

getArchLlama(metadata: GGUFMetadata): ArchLlama?
getArchFalcon(metadata: GGUFMetadata): ArchFalcon?
...

Which returns undefined if the model is not the one being checked.

The idea is to prevent user from doing something like const arch = model as ArchLlama, but I'm not sure if my proposal is the right way to do.

Do you have any other ideas? Thanks.

---

P/s: I'm not planning to do that in this PR, it should be in another PR with more considerations before actually implement it.

[mishig25]

the proposal sounds good to me.

but I'm not sure if my proposal is the right way to do

You can start with a draft PR and we can iterate if needed

[mishig25]

If you can write a small section on readme about this change, that would be great.

I've left one last nit here #640 (comment)

LGTM! Unless @julien-c comments in the next few hours, I will merge this PR soon

[ngxson]

Thanks for the careful review. For now it's not very clear how to use these types, so I don't think I can add new section to readme right now. Maybe with the next PR (the proposal that you mentioned), I will add an example to readme.

[julien-c]

Suggested change

	const ARCHITECTURES = [...LLM_ARCHITECTURES, "rwkv", "whisper"];
	const ARCHITECTURES = [...LLM_ARCHITECTURES, "rwkv", "whisper"] as const;

I think you need to add this here otherwise the type inference is string[] – unless we're not using the same version of typescript?

[julien-c]

re. your proposal, please note that I am not super good with TS type inference, but IMO the ideal way we would use this is to enable like this:

if (metadata["general.architecture"] === "grok") {
	const x = metadata; // type = TransformerLLMBase<"grok">
	const foo = x["grok.attention.alibi_bias_max"]; // type = number
	// and acess all other grok specific properties, with the right typing
}

do you think it's possible at all?

[mishig25]

checked it manually. As of right now, ts server is just giving generic MetadatValue rather than specific type (e.g. number)

cc: @coyotte508 @ngxson do you know if above request is possible #640 (comment) ?

[julien-c]

maybe by doing some generics stuff in

export type GGUFMetadata<K extends Architecture = Architecture> = {
	version: Version;
	tensor_count: bigint;
	kv_count: bigint;
} & Partial<General<K>> &
	Partial<ModelBase<K>>;

maybe too complex though

[coyotte508]

It's maybe more readable to have:

Suggested change

	> = { [K in `${TArchitecture}.layer_count`]: number } & { [K in `${TArchitecture}.feed_forward_length`]: number } & {
	[K in `${TArchitecture}.context_length`]: number;
	} & { [K in `${TArchitecture}.embedding_length`]: number } & { [K in `${TArchitecture}.block_count`]: number };
	> = Record<
	| `${TArchitecture}.layer_count`
	| `${TArchitecture}.feed_forward_length`
	| `${TArchitecture}.context_length`
	| `${TArchitecture}.embedding_length`
	| `${TArchitecture}.block_count`,
	number
	>

[ngxson]

Actually it's being formatted by pnpm run format. Can we maybe ignore specific lines from the formatter?

Sorry I misunderstood the proposal. Yeah seems like a good idea, I'll try that (and also take a bit more time to clean up other parts of the PR)

[julien-c]

maybe we merge this one and improve in later PRs?

[ngxson]

Yeah I think we can merge now. I'm not having much time for doing code atm, but I'll open another pull to resolve this and the also the proposal related to usage of these types (hopefully I'll have time this weekend).

[coyotte508]

checked it manually. As of right now, ts server is just giving generic MetadatValue rather than specific type (e.g. number)

cc: @coyotte508 @ngxson do you know if above request is possible #640 (comment) ?

You can do this:

const x: TransformerLLM = {} as any;

if ("grok.layer_count" in x) {
	// below is typed correctly
	console.log(x["grok.attention.layer_norm_epsilon"])
}

maybe by doing some generics stuff in

export type GGUFMetadata<K extends Architecture = Architecture> = {
	version: Version;
	tensor_count: bigint;
	kv_count: bigint;
} & Partial<General<K>> &
	Partial<ModelBase<K>>;

maybe too complex though

Yes it's possible probably

You would need a mapped type:

type Archs = {
  "grok": ArchGrok,
  ...
} satisfies Record<Architecture, unknown>

And then call Archs[T] somewhere in your Model<T> or GGufMetaData<T> type.

Since we have code generation it's also possible to generate every possible GGufMetadataXXX and do GGufMetadata = GgufMetadaGrok | .... And have less complex generics code (but more verbose)

This can totally be a follow up PR

[coyotte508]

LGTM, although would appreciate #640 (comment) :)

[julien-c]

And have less complex generics code (but more verbose)

i tend to favor verbose in those situations:)

[mishig25]

Let's merge ?

[mishig25]

@ngxson thanks a lot for the contribution !
looking forward to your future contrib 🔥 #640 (comment)