Universal (generic) argument/parameter interface

[pudepiedj]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

Please provide a detailed written description of what you were trying to do, and what you expected llama.cpp to do.

Current Behavior

Please provide a detailed written description of what llama.cpp did, instead.

Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

• Physical (or virtual) hardware you are using, e.g. for Linux:

$ lscpu

• Operating System, e.g. for Linux:

$ uname -a

• SDK version, e.g. for Linux:

$ python3 --version
$ make --version
$ g++ --version

Failure Information (for bugs)

Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.

Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

1. step 1
2. step 2
3. step 3
4. etc.

Failure Logs

Please include any relevant log snippets or files. If it works under one configuration but not under another, please provide logs for both configurations and their corresponding outputs so it is easy to see where behavior changes.

Also, please try to avoid using screenshots if at all possible. Instead, copy/paste the console output and use Github's markdown to cleanly format your logs for easy readability.

Example environment info:

llama.cpp$ git log | head -1
commit 2af23d30434a677c6416812eea52ccc0af65119c

llama.cpp$ lscpu | egrep "AMD|Flags"
Vendor ID:                       AuthenticAMD
Model name:                      AMD Ryzen Threadripper 1950X 16-Core Processor
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca sme sev
Virtualization:                  AMD-V

llama.cpp$ python3 --version
Python 3.10.9

llama.cpp$ pip list | egrep "torch|numpy|sentencepiece"
numpy                         1.24.2
numpydoc                      1.5.0
sentencepiece                 0.1.97
torch                         1.13.1
torchvision                   0.14.1

llama.cpp$ make --version | head -1
GNU Make 4.3

$ md5sum ./models/65B/ggml-model-q4_0.bin
dbdd682cce80e2d6e93cefc7449df487  ./models/65B/ggml-model-q4_0.bin

Example run with the Linux command perf

llama.cpp$ perf stat ./main -m ./models/65B/ggml-model-q4_0.bin -t 16 -n 1024 -p "Please close your issue when it has been answered."
main: seed = 1679149377
llama_model_load: loading model from './models/65B/ggml-model-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 8192
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 64
llama_model_load: n_layer = 80
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 22016
llama_model_load: n_parts = 8
llama_model_load: ggml ctx size = 41477.73 MB
llama_model_load: memory_size =  2560.00 MB, n_mem = 40960
llama_model_load: loading model part 1/8 from './models/65B/ggml-model-q4_0.bin'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 2/8 from './models/65B/ggml-model-q4_0.bin.1'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 3/8 from './models/65B/ggml-model-q4_0.bin.2'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 4/8 from './models/65B/ggml-model-q4_0.bin.3'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 5/8 from './models/65B/ggml-model-q4_0.bin.4'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 6/8 from './models/65B/ggml-model-q4_0.bin.5'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 7/8 from './models/65B/ggml-model-q4_0.bin.6'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 8/8 from './models/65B/ggml-model-q4_0.bin.7'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723

system_info: n_threads = 16 / 32 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |

main: prompt: 'Please close your issue when it has been answered.'
main: number of tokens in prompt = 11
     1 -> ''
 12148 -> 'Please'
  3802 -> ' close'
   596 -> ' your'
  2228 -> ' issue'
   746 -> ' when'
   372 -> ' it'
   756 -> ' has'
  1063 -> ' been'
  7699 -> ' answered'
 29889 -> '.'

sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000


Please close your issue when it has been answered.
@duncan-donut: I'm trying to figure out what kind of "support" you need for this script and why, exactly? Is there a question about how the code works that hasn't already been addressed in one or more comments below this ticket, or are we talking something else entirely like some sorta bugfixing job because your server setup is different from mine??
I can understand if your site needs to be running smoothly and you need help with a fix of sorts but there should really be nothing wrong here that the code itself could not handle. And given that I'm getting reports about how it works perfectly well on some other servers, what exactly are we talking? A detailed report will do wonders in helping us get this resolved for ya quickly so please take your time and describe the issue(s) you see as clearly & concisely as possible!!
@duncan-donut: I'm not sure if you have access to cPanel but you could try these instructions. It is worth a shot! Let me know how it goes (or what error message, exactly!) when/if ya give that code a go? [end of text]


main: mem per token = 71159620 bytes
main:     load time = 19309.95 ms
main:   sample time =   168.62 ms
main:  predict time = 223895.61 ms / 888.47 ms per token
main:    total time = 246406.42 ms

 Performance counter stats for './main -m ./models/65B/ggml-model-q4_0.bin -t 16 -n 1024 -p Please close your issue when it has been answered.':

        3636882.89 msec task-clock                #   14.677 CPUs utilized
             13509      context-switches          #    3.714 /sec
              2436      cpu-migrations            #    0.670 /sec
          10476679      page-faults               #    2.881 K/sec
    13133115082869      cycles                    #    3.611 GHz                      (16.77%)
       29314462753      stalled-cycles-frontend   #    0.22% frontend cycles idle     (16.76%)
    10294402631459      stalled-cycles-backend    #   78.39% backend cycles idle      (16.74%)
    23479217109614      instructions              #    1.79  insn per cycle
                                                  #    0.44  stalled cycles per insn  (16.76%)
     2353072268027      branches                  #  647.002 M/sec                    (16.77%)
        1998682780      branch-misses             #    0.08% of all branches          (16.76%)

     247.802177522 seconds time elapsed

    3618.573072000 seconds user
      18.491698000 seconds sys

[pudepiedj]

It seems to me that we need a universal "help" that ensures that everything in the --help or -h response to any command-executable is available OR which identifies which argument/parameter functions are available (or not). I think this requires what I would call an "application-contextual help" which knows what is possible from each app. The proliferation of apps is now creating confusion about what is possible and what is a local installation error and what is not possible (for example, because the argument/parameter options like -f or -tb have not yet been incorporated).

[pudepiedj]

How about a universal help file processed in common.cpp where each line/element/argument of the help file has a unique binary code with just one bit set. Each app then has an appcode made up of the sum of these binaries according to whether a feature has been implemented. So appcode 3 will only display helps 1 and 2; appcode 17 will display helps 1 and 4. Mocked-up examples for six different cases:

Appcode   2 00000010      -f FNAME --file FNAME read the prompts from an external text file

Appcode   5 00000001      -h --help print this help list and exit
Appcode   5 00000100      -n     N --n-predict     N number of tokens to predict in generating a completion

Appcode  11 00000001      -h --help print this help list and exit
Appcode  11 00000010      -f FNAME --file FNAME read the prompts from an external text file
Appcode  11 00001000      -t     N --threads     N number of threads to use

Appcode  17 00000001      -h --help print this help list and exit
Appcode  17 00010000      -m MODELPATH --model MODELPATH path to llama model to use

Appcode  23 00000001      -h --help print this help list and exit
Appcode  23 00000010      -f FNAME --file FNAME read the prompts from an external text file
Appcode  23 00000100      -n     N --n-predict     N number of tokens to predict in generating a completion
Appcode  23 00010000      -m MODELPATH --model MODELPATH path to llama model to use

Appcode  31 00000001      -h --help print this help list and exit
Appcode  31 00000010      -f FNAME --file FNAME read the prompts from an external text file
Appcode  31 00000100      -n     N --n-predict     N number of tokens to predict in generating a completion
Appcode  31 00001000      -t     N --threads     N number of threads to use
Appcode  31 00010000      -m MODELPATH --model MODELPATH path to llama model to use

[pudepiedj]

After more thought the suggestion to use some sort of binary coding is unnecessary. It's possible to write search programs that look for command-line arguments and gpt-params from common.h in each of the .cpp files in examples/ and process them to produce a context-sensitive help. To make the process more robust we would need to adopt some sort of standard encoding because as this exercise shows, there are things that are missing or don't show up because they have slightly variant codings. This would need approval from the boss @ggerganov!

The work-in-progress code for this is dynamic, so it reads and parses the common.h and *.cpp files afresh on each run. There are three versions for each .cpp file, and if there are different numbers of rows for the three versions, it indicates that something needs attention, usually a variant coding. Also prompt and n_threads don't have explanatory comments in the master common.h file, which is why they are blank.

Here is an output from a beta- python3 attempt to generate the context-sensitive help by this method, based on the actual .cpp files in examples/. There are two anomalies: "logit-bias" is declared differently and needs a fix; there also needs to be a fix to the multiple-line helps which currently get separated; some of these attributes lacked lines in the common.cpp help list, which I have added in a version that will be part of the eventual PR (not yet opened):

######################## find help context parameters #################################

########################
Filename: baby-llama.cpp
########################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  -h, --help            show this help message and exit\n"

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         

Searching the other way round is more efficient:

 1 key:                      help; role: // universal help parameter                                 ; default: ""        

No unresolved help-list incompatibilities with baby-llama.cpp


#####################
Filename: batched.cpp
#####################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  -tb N, --threads-batch N\n"  
 2 help:  -p PROMPT, --prompt PROMPT\n"
 3 help:  -t N, --threads N     number of threads to use during generation (default: %d)\n", params.n-threads
 4 help:  --numa                attempt optimizations that help on some NUMA systems\n"
 5 help:  -m FNAME, --model FNAME\n"   
 6 help:  -h, --help            show this help message and exit\n"

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:           n_threads; role: // user-defined or num of internal physical cores           ;  default: get_num_physical_cores() 
 2 gpt_param:     n_threads_batch; role: // num threads for batch proc (-1 use n_threads)            ;  default: -1         
 3 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         
 4 gpt_param:               model; role: // model path                                               ;  default: "models/7B/ggml-model-f16.gguf" 
 5 gpt_param:              prompt; role: // user-provided single prompt                              ;  default: ""         
 6 gpt_param:                numa; role: // attempt optimizations that help on some NUMA systems     ;  default: false      

Searching the other way round is more efficient:

 1 key:                     model; role: // model path                                               ; default: "models/7B/ggml-model-f16.gguf"
 2 key:                    prompt; role: // user-provided single prompt                              ; default: ""        
 3 key:                 n_threads; role: // user-defined or num of internal physical cores           ; default: get_num_physical_cores()
 4 key:                      numa; role: // attempt optimizations that help on some NUMA systems     ; default: false     
 5 key:                      help; role: // universal help parameter                                 ; default: ""        
 6 key:           n_threads_batch; role: // num threads for batch proc (-1 use n_threads)            ; default: -1        

No unresolved help-list incompatibilities with batched.cpp


#########################
Filename: beam-search.cpp
#########################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  -p PROMPT, --prompt PROMPT\n"
 2 help:  --numa                attempt optimizations that help on some NUMA systems\n"
 3 help:  --beams N             0 (default): if non-zero use beam search of given width N.\n"
 4 help:  -ngl N, --n-gpu-layers N\n"  
 5 help:  -m FNAME, --model FNAME\n"   
 6 help:  -h, --help            show this help message and exit\n"

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:        n_gpu_layers; role: // num layers stored in VRAM (-1 for default)               ;  default: -1         
 2 gpt_param:             n_beams; role: // if non-zero then use beam search of given width.         ;  default: 0          
 3 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         
 4 gpt_param:               model; role: // model path                                               ;  default: "models/7B/ggml-model-f16.gguf" 
 5 gpt_param:              prompt; role: // user-provided single prompt                              ;  default: ""         
 6 gpt_param:                numa; role: // attempt optimizations that help on some NUMA systems     ;  default: false      

Searching the other way round is more efficient:

 1 key:                     model; role: // model path                                               ; default: "models/7B/ggml-model-f16.gguf"
 2 key:                    prompt; role: // user-provided single prompt                              ; default: ""        
 3 key:                      numa; role: // attempt optimizations that help on some NUMA systems     ; default: false     
 4 key:                   n_beams; role: // if non-zero then use beam search of given width.         ; default: 0         
 5 key:                      help; role: // universal help parameter                                 ; default: ""        
 6 key:              n_gpu_layers; role: // num layers stored in VRAM (-1 for default)               ; default: -1        

No unresolved help-list incompatibilities with beam-search.cpp


###############################
Filename: benchmark-matmult.cpp
###############################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  -h, --help            show this help message and exit\n"
 2 help:  -t N, --threads N     number of threads to use during generation (default: %d)\n", params.n-threads

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:           n_threads; role: // user-defined or num of internal physical cores           ;  default: get_num_physical_cores() 
 2 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         

Searching the other way round is more efficient:

 1 key:                      help; role: // universal help parameter                                 ; default: ""        
 2 key:                 n_threads; role: // user-defined or num of internal physical cores           ; default: get_num_physical_cores()

No unresolved help-list incompatibilities with benchmark-matmult.cpp


##########################
Filename: cmap-example.cpp
##########################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  -h, --help            show this help message and exit\n"

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         

Searching the other way round is more efficient:

 1 key:                      help; role: // universal help parameter                                 ; default: ""        

No unresolved help-list incompatibilities with cmap-example.cpp


#####################################
Filename: convert-llama2c-to-ggml.cpp
#####################################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  -n N, --n-predict N   number of tokens to predict (default: %d, -1 = infinity, -2 = until context filled)\n", params.n-predict
 2 help:  -b N, --batch-size N  batch size for prompt processing (default: %d)\n", params.n-batch
 3 help:  -s SEED, --seed SEED  RNG seed (default: -1, use random seed for < 0)\n"
 4 help:  -t N, --threads N     number of threads to use during generation (default: %d)\n", params.n-threads
 5 help:  -h, --help            show this help message and exit\n"
 6 help:  -c N, --ctx-size N    size of the prompt context (default: %d, 0 = loaded from model)\n", params.n-ctx

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:                seed; role: // RNG seed                                                 ;  default: -1         
 2 gpt_param:           n_threads; role: // user-defined or num of internal physical cores           ;  default: get_num_physical_cores() 
 3 gpt_param:           n_predict; role: // new tokens to predict                                    ;  default: -1         
 4 gpt_param:               n_ctx; role: // context size                                             ;  default: 512        
 5 gpt_param:             n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ;  default: 512        
 6 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         

Searching the other way round is more efficient:

 1 key:                 n_threads; role: // user-defined or num of internal physical cores           ; default: get_num_physical_cores()
 2 key:                     n_ctx; role: // context size                                             ; default: 512       
 3 key:                      seed; role: // RNG seed                                                 ; default: -1        
 4 key:                   n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ; default: 512       
 5 key:                      help; role: // universal help parameter                                 ; default: ""        
 6 key:                 n_predict; role: // new tokens to predict                                    ; default: -1        

No unresolved help-list incompatibilities with convert-llama2c-to-ggml.cpp


############################
Filename: embd-input-lib.cpp
############################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  --top-p N             top-p sampling (default: %.1f, 1.0 = disabled)\n", (double)params.top-p
 2 help:  -b N, --batch-size N  batch size for prompt processing (default: %d)\n", params.n-batch
 3 help:  -l T, --logit-bias T  T = TOKEN-ID(plus/minus)BIAS\n"
 4 help:  --typical N           locally typical sampling, parameter p (default: %.1f, 1.0 = disabled)\n", (double)params.typical-p
 5 help:  --presence-penalty N  repeat alpha presence penalty (default: %.1f, 0.0 = disabled)\n", (double)params.presence-penalty
 6 help:  --mirostat N          use Mirostat sampling.\n"
   help:                        i.e. `--logit-bias 15043+1` to increase likelihood of token ' Hello',\n"
 7 help:  --tfs N               tail free sampling, parameter z (default: %.1f, 1.0 = disabled)\n", (double)params.tfs-z
 8 help:  --frequency-penalty N repeat alpha frequency penalty (default: %.1f, 0.0 = disabled)\n", (double)params.frequency-penalty
 9 help:  -h, --help            show this help message and exit\n"
   help:                        or `--logit-bias 15043-1` to decrease likelihood of token ' Hello'\n"
10 help:  -s SEED, --seed SEED  RNG seed (default: -1, use random seed for < 0)\n"
11 help:  --repeat-last-n N     last n tokens to consider for penalize (default: %d, 0 = disabled, -1 = ctx-size)\n", params.repeat-last-n
12 help:  --temp N              temperature (default: %.1f)\n", (double)params.temp
13 help:  --numa                attempt optimizations that help on some NUMA systems\n"
14 help:  --top-k N             top-k sampling (default: %d, 0 = disabled)\n", params.top-k
15 help:  --repeat-penalty N    penalize repeat sequence of tokens (default: %.1f, 1.0 = disabled)\n", (double)params.repeat-penalty
16 help:  --no-penalize-nl      do not penalize newline token (default is DO penalise nl token)\n"
17 help:  --mirostat-ent N      Mirostat target entropy, parameter tau (default: %.1f)\n", (double)params.mirostat-tau
18 help:  --mirostat-lr N       Mirostat learning rate, parameter eta (default: %.1f)\n", (double)params.mirostat-eta

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:          logit_bias; role: // way to alter prob of particular words                    ;  default: 0          
 2 gpt_param:                seed; role: // RNG seed                                                 ;  default: -1         
 3 gpt_param:             n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ;  default: 512        
 4 gpt_param:               top_k; role: // < 0 to use vocab size                                    ;  default: 40         
 5 gpt_param:               top_p; role: // 1.0 disabled                                             ;  default: 0.95f      
 6 gpt_param:               tfs_z; role: // 1.0 disabled                                             ;  default: 1.00f      
 7 gpt_param:           typical_p; role: // 1.0 disabled                                             ;  default: 1.00f      
 8 gpt_param:                temp; role: // 1.0 disabled                                             ;  default: 0.80f      
 9 gpt_param:      repeat_penalty; role: // 1.0 disabled                                             ;  default: 1.10f      
10 gpt_param:       repeat_last_n; role: // last n tokens to penalize (0 disable, -1 cxt size)       ;  default: 64         
11 gpt_param:   frequency_penalty; role: // 0.0 disabled                                             ;  default: 0.00f      
12 gpt_param:    presence_penalty; role: // 0.0 disabled                                             ;  default: 0.00f      
13 gpt_param:            mirostat; role: // 0 disabled, 1 mirostat, 2 mirostat 2.0                   ;  default: 0          
14 gpt_param:        mirostat_tau; role: // target entropy                                           ;  default: 5.00f      
15 gpt_param:        mirostat_eta; role: // learning rate                                            ;  default: 0.10f      
16 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         
17 gpt_param:         penalize_nl; role: // consider newlines as a repeatable token                  ;  default: true       
18 gpt_param:                numa; role: // attempt optimizations that help on some NUMA systems     ;  default: false      

Searching the other way round is more efficient:

 1 key:          presence_penalty; role: // 0.0 disabled                                             ; default: 0.00f     
 2 key:                  mirostat; role: // 0 disabled, 1 mirostat, 2 mirostat 2.0                   ; default: 0         
 3 key:                      numa; role: // attempt optimizations that help on some NUMA systems     ; default: false     
 4 key:                     tfs_z; role: // 1.0 disabled                                             ; default: 1.00f     
 5 key:                     top_k; role: // < 0 to use vocab size                                    ; default: 40        
 6 key:             repeat_last_n; role: // last n tokens to penalize (0 disable, -1 cxt size)       ; default: 64        
 7 key:                      seed; role: // RNG seed                                                 ; default: -1        
 8 key:                   n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ; default: 512       
 9 key:                 typical_p; role: // 1.0 disabled                                             ; default: 1.00f     
10 key:               penalize_nl; role: // consider newlines as a repeatable token                  ; default: true      
11 key:                     top_p; role: // 1.0 disabled                                             ; default: 0.95f     
12 key:            repeat_penalty; role: // 1.0 disabled                                             ; default: 1.10f     
13 key:         frequency_penalty; role: // 0.0 disabled                                             ; default: 0.00f     
14 key:              mirostat_eta; role: // learning rate                                            ; default: 0.10f     
15 key:                      temp; role: // 1.0 disabled                                             ; default: 0.80f     
16 key:              mirostat_tau; role: // target entropy                                           ; default: 5.00f     
17 key:                      help; role: // universal help parameter                                 ; default: ""        
18 key:                logit_bias; role: // way to alter prob of particular words                    ; default: 0         

No unresolved help-list incompatibilities with embd-input-lib.cpp


#############################
Filename: embd-input-test.cpp
#############################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  -h, --help            show this help message and exit\n"
 2 help:  -p PROMPT, --prompt PROMPT\n"

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         
 2 gpt_param:              prompt; role: // user-provided single prompt                              ;  default: ""         

Searching the other way round is more efficient:

 1 key:                    prompt; role: // user-provided single prompt                              ; default: ""        
 2 key:                      help; role: // universal help parameter                                 ; default: ""        

No unresolved help-list incompatibilities with embd-input-test.cpp


#######################
Filename: embedding.cpp
#######################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  --embedding           0 (default): get only sentence embedding\n"
 2 help:  --random-prompt       start with a randomized prompt.\n"
 3 help:  -b N, --batch-size N  batch size for prompt processing (default: %d)\n", params.n-batch
 4 help:  -s SEED, --seed SEED  RNG seed (default: -1, use random seed for < 0)\n"
 5 help:  -p PROMPT, --prompt PROMPT\n"
 6 help:  --numa                attempt optimizations that help on some NUMA systems\n"
 7 help:  -h, --help            show this help message and exit\n"
 8 help:  --verbose-prompt      print prompt before generation\n"

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:                seed; role: // RNG seed                                                 ;  default: -1         
 2 gpt_param:             n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ;  default: 512        
 3 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         
 4 gpt_param:              prompt; role: // user-provided single prompt                              ;  default: ""         
 5 gpt_param:       random_prompt; role: // do not randomize prompt if none provided                 ;  default: false      
 6 gpt_param:           embedding; role: // get only sentence embedding                              ;  default: false      
 7 gpt_param:                numa; role: // attempt optimizations that help on some NUMA systems     ;  default: false      
 8 gpt_param:      verbose_prompt; role: // print prompt tokens before generation                    ;  default: false      

Searching the other way round is more efficient:

 1 key:                    prompt; role: // user-provided single prompt                              ; default: ""        
 2 key:                 embedding; role: // get only sentence embedding                              ; default: false     
 3 key:                      seed; role: // RNG seed                                                 ; default: -1        
 4 key:                   n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ; default: 512       
 5 key:                      numa; role: // attempt optimizations that help on some NUMA systems     ; default: false     
 6 key:                      help; role: // universal help parameter                                 ; default: ""        
 7 key:             random_prompt; role: // do not randomize prompt if none provided                 ; default: false     
 8 key:            verbose_prompt; role: // print prompt tokens before generation                    ; default: false     

No unresolved help-list incompatibilities with embedding.cpp


#########################
Filename: export-lora.cpp
#########################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  -h, --help            show this help message and exit\n"

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         

Searching the other way round is more efficient:

 1 key:                      help; role: // universal help parameter                                 ; default: ""        

No unresolved help-list incompatibilities with export-lora.cpp


######################
Filename: finetune.cpp
######################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  --rope-freq-scale N   RoPE frequency linear scaling factor (default: loaded from model)\n"
 2 help:  -h, --help            show this help message and exit\n"
 3 help:  --rope-freq-base N    RoPE base frequency, used by NTK-aware scaling (default: loaded from model)\n"
 4 help:  -t N, --threads N     number of threads to use during generation (default: %d)\n", params.n-threads

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:           n_threads; role: // user-defined or num of internal physical cores           ;  default: get_num_physical_cores() 
 2 gpt_param:      rope_freq_base; role: // RoPE base frequency                                      ;  default: 0.0f       
 3 gpt_param:     rope_freq_scale; role: // RoPE frequency scaling factor                            ;  default: 0.0f       
 4 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         

Searching the other way round is more efficient:

 1 key:                 n_threads; role: // user-defined or num of internal physical cores           ; default: get_num_physical_cores()
 2 key:            rope_freq_base; role: // RoPE base frequency                                      ; default: 0.0f      
 3 key:                      help; role: // universal help parameter                                 ; default: ""        
 4 key:           rope_freq_scale; role: // RoPE frequency scaling factor                            ; default: 0.0f      

No unresolved help-list incompatibilities with finetune.cpp


##################
Filename: gguf.cpp
##################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  -h, --help            show this help message and exit\n"

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         

Searching the other way round is more efficient:

 1 key:                      help; role: // universal help parameter                                 ; default: ""        

No unresolved help-list incompatibilities with gguf.cpp


#########################
Filename: falcon-main.cpp
#########################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  --top-p N             top-p sampling (default: %.1f, 1.0 = disabled)\n", (double)params.top-p
 2 help:  -i, --interactive     run in interactive mode\n"
 3 help:  --repeat-penalty N    penalize repeat sequence of tokens (default: %.1f, 1.0 = disabled)\n", (double)params.repeat-penalty
 4 help:  -n N, --n-predict N   number of tokens to predict (default: %d, -1 = infinity, -2 = until context filled)\n", params.n-predict
 5 help:  -b N, --batch-size N  batch size for prompt processing (default: %d)\n", params.n-batch
 6 help:  -s SEED, --seed SEED  RNG seed (default: -1, use random seed for < 0)\n"
 7 help:  -p PROMPT, --prompt PROMPT\n"
 8 help:  -t N, --threads N     number of threads to use during generation (default: %d)\n", params.n-threads
 9 help:  --repeat-last-n N     last n tokens to consider for penalize (default: %d, 0 = disabled, -1 = ctx-size)\n", params.repeat-last-n
10 help:  --temp N              temperature (default: %.1f)\n", (double)params.temp
11 help:  --top-k N             top-k sampling (default: %d, 0 = disabled)\n", params.top-k
12 help:  -ngl N, --n-gpu-layers N\n"  
13 help:  -m FNAME, --model FNAME\n"   
14 help:  -h, --help            show this help message and exit\n"

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:                seed; role: // RNG seed                                                 ;  default: -1         
 2 gpt_param:           n_threads; role: // user-defined or num of internal physical cores           ;  default: get_num_physical_cores() 
 3 gpt_param:           n_predict; role: // new tokens to predict                                    ;  default: -1         
 4 gpt_param:             n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ;  default: 512        
 5 gpt_param:        n_gpu_layers; role: // num layers stored in VRAM (-1 for default)               ;  default: -1         
 6 gpt_param:               top_k; role: // < 0 to use vocab size                                    ;  default: 40         
 7 gpt_param:               top_p; role: // 1.0 disabled                                             ;  default: 0.95f      
 8 gpt_param:                temp; role: // 1.0 disabled                                             ;  default: 0.80f      
 9 gpt_param:      repeat_penalty; role: // 1.0 disabled                                             ;  default: 1.10f      
10 gpt_param:       repeat_last_n; role: // last n tokens to penalize (0 disable, -1 cxt size)       ;  default: 64         
11 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         
12 gpt_param:               model; role: // model path                                               ;  default: "models/7B/ggml-model-f16.gguf" 
13 gpt_param:              prompt; role: // user-provided single prompt                              ;  default: ""         
14 gpt_param:         interactive; role: // interactive mode                                         ;  default: false      

Searching the other way round is more efficient:

 1 key:                     model; role: // model path                                               ; default: "models/7B/ggml-model-f16.gguf"
 2 key:            repeat_penalty; role: // 1.0 disabled                                             ; default: 1.10f     
 3 key:                    prompt; role: // user-provided single prompt                              ; default: ""        
 4 key:                 n_threads; role: // user-defined or num of internal physical cores           ; default: get_num_physical_cores()
 5 key:                      temp; role: // 1.0 disabled                                             ; default: 0.80f     
 6 key:                      seed; role: // RNG seed                                                 ; default: -1        
 7 key:               interactive; role: // interactive mode                                         ; default: false     
 8 key:                   n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ; default: 512       
 9 key:                      help; role: // universal help parameter                                 ; default: ""        
10 key:                 n_predict; role: // new tokens to predict                                    ; default: -1        
11 key:                     top_k; role: // < 0 to use vocab size                                    ; default: 40        
12 key:                     top_p; role: // 1.0 disabled                                             ; default: 0.95f     
13 key:              n_gpu_layers; role: // num layers stored in VRAM (-1 for default)               ; default: -1        
14 key:             repeat_last_n; role: // last n tokens to penalize (0 disable, -1 cxt size)       ; default: 64        

No unresolved help-list incompatibilities with falcon-main.cpp


##########################
Filename: gptneox-main.cpp
##########################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  --top-p N             top-p sampling (default: %.1f, 1.0 = disabled)\n", (double)params.top-p
 2 help:  -i, --interactive     run in interactive mode\n"
 3 help:  --repeat-penalty N    penalize repeat sequence of tokens (default: %.1f, 1.0 = disabled)\n", (double)params.repeat-penalty
 4 help:  -n N, --n-predict N   number of tokens to predict (default: %d, -1 = infinity, -2 = until context filled)\n", params.n-predict
 5 help:  -b N, --batch-size N  batch size for prompt processing (default: %d)\n", params.n-batch
 6 help:  -s SEED, --seed SEED  RNG seed (default: -1, use random seed for < 0)\n"
 7 help:  -p PROMPT, --prompt PROMPT\n"
 8 help:  -t N, --threads N     number of threads to use during generation (default: %d)\n", params.n-threads
 9 help:  --repeat-last-n N     last n tokens to consider for penalize (default: %d, 0 = disabled, -1 = ctx-size)\n", params.repeat-last-n
10 help:  --temp N              temperature (default: %.1f)\n", (double)params.temp
11 help:  --top-k N             top-k sampling (default: %d, 0 = disabled)\n", params.top-k
12 help:  -ngl N, --n-gpu-layers N\n"  
13 help:  -m FNAME, --model FNAME\n"   
14 help:  -h, --help            show this help message and exit\n"

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:                seed; role: // RNG seed                                                 ;  default: -1         
 2 gpt_param:           n_threads; role: // user-defined or num of internal physical cores           ;  default: get_num_physical_cores() 
 3 gpt_param:           n_predict; role: // new tokens to predict                                    ;  default: -1         
 4 gpt_param:             n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ;  default: 512        
 5 gpt_param:        n_gpu_layers; role: // num layers stored in VRAM (-1 for default)               ;  default: -1         
 6 gpt_param:               top_k; role: // < 0 to use vocab size                                    ;  default: 40         
 7 gpt_param:               top_p; role: // 1.0 disabled                                             ;  default: 0.95f      
 8 gpt_param:                temp; role: // 1.0 disabled                                             ;  default: 0.80f      
 9 gpt_param:      repeat_penalty; role: // 1.0 disabled                                             ;  default: 1.10f      
10 gpt_param:       repeat_last_n; role: // last n tokens to penalize (0 disable, -1 cxt size)       ;  default: 64         
11 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         
12 gpt_param:               model; role: // model path                                               ;  default: "models/7B/ggml-model-f16.gguf" 
13 gpt_param:              prompt; role: // user-provided single prompt                              ;  default: ""         
14 gpt_param:         interactive; role: // interactive mode                                         ;  default: false      

Searching the other way round is more efficient:

 1 key:                     model; role: // model path                                               ; default: "models/7B/ggml-model-f16.gguf"
 2 key:            repeat_penalty; role: // 1.0 disabled                                             ; default: 1.10f     
 3 key:                    prompt; role: // user-provided single prompt                              ; default: ""        
 4 key:                 n_threads; role: // user-defined or num of internal physical cores           ; default: get_num_physical_cores()
 5 key:                      temp; role: // 1.0 disabled                                             ; default: 0.80f     
 6 key:                      seed; role: // RNG seed                                                 ; default: -1        
 7 key:               interactive; role: // interactive mode                                         ; default: false     
 8 key:                   n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ; default: 512       
 9 key:                      help; role: // universal help parameter                                 ; default: ""        
10 key:                 n_predict; role: // new tokens to predict                                    ; default: -1        
11 key:                     top_k; role: // < 0 to use vocab size                                    ; default: 40        
12 key:                     top_p; role: // 1.0 disabled                                             ; default: 0.95f     
13 key:              n_gpu_layers; role: // num layers stored in VRAM (-1 for default)               ; default: -1        
14 key:             repeat_last_n; role: // last n tokens to penalize (0 disable, -1 cxt size)       ; default: 64        

No unresolved help-list incompatibilities with gptneox-main.cpp


####################
Filename: infill.cpp
####################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  --top-p N             top-p sampling (default: %.1f, 1.0 = disabled)\n", (double)params.top-p
 2 help:  -i, --interactive     run in interactive mode\n"
 3 help:  --embedding           0 (default): get only sentence embedding\n"
 4 help:  -n N, --n-predict N   number of tokens to predict (default: %d, -1 = infinity, -2 = until context filled)\n", params.n-predict
 5 help:  -b N, --batch-size N  batch size for prompt processing (default: %d)\n", params.n-batch
 6 help:  --interactive-first   run in interactive mode and wait for input right away\n"
 7 help:  --in-suffix STRING    string to suffix after user inputs with (default: empty)\n"
 8 help:  -p PROMPT, --prompt PROMPT\n"
 9 help:  -r PROMPT, --reverse-prompt PROMPT\n"
10 help:  --rope-freq-scale N   RoPE frequency linear scaling factor (default: loaded from model)\n"
11 help:  -l T, --logit-bias T  T = TOKEN-ID(plus/minus)BIAS\n"
12 help:  --typical N           locally typical sampling, parameter p (default: %.1f, 1.0 = disabled)\n", (double)params.typical-p
13 help:  --color               colorise output to distinguish prompt and user input from generations\n"
14 help:  --presence-penalty N  repeat alpha presence penalty (default: %.1f, 0.0 = disabled)\n", (double)params.presence-penalty
15 help:  -c N, --ctx-size N    size of the prompt context (default: %d, 0 = loaded from model)\n", params.n-ctx
16 help:  --multiline-input     allows you to write or paste multiple lines without ending each in '\\'\n"
17 help:  --mirostat N          use Mirostat sampling.\n"
18 help:  --verbose-prompt      print prompt before generation\n"
   help:                        i.e. `--logit-bias 15043+1` to increase likelihood of token ' Hello',\n"
19 help:  --in-prefix-bos       prefix BOS to user inputs, preceding the `--in-prefix` string\n"
20 help:  --use-color           0 (default): use color to distinguish generations from inputs\n"
21 help:  --tfs N               tail free sampling, parameter z (default: %.1f, 1.0 = disabled)\n", (double)params.tfs-z
22 help:  --prompt-cache FNAME  file to cache prompt state for faster startup (default: none)\n"
23 help:  --frequency-penalty N repeat alpha frequency penalty (default: %.1f, 0.0 = disabled)\n", (double)params.frequency-penalty
24 help:  -h, --help            show this help message and exit\n"
25 help:  --grammar GRAMMAR     BNF-like grammar to constrain generations (see samples in grammars/ dir)\n"
26 help:  --logits-all          return logits for all tokens in the batch (default: disabled)\n"
   help:                        or `--logit-bias 15043-1` to decrease likelihood of token ' Hello'\n"
27 help:  --cfg-scale N         strength of guidance (default: %f, 1.0 = disable)\n", params.cfg-scale
28 help:  -s SEED, --seed SEED  RNG seed (default: -1, use random seed for < 0)\n"
29 help:  --cfg-negative-prompt PROMPT\n"
30 help:  --repeat-last-n N     last n tokens to consider for penalize (default: %d, 0 = disabled, -1 = ctx-size)\n", params.repeat-last-n
31 help:  --temp N              temperature (default: %.1f)\n", (double)params.temp
32 help:  --numa                attempt optimizations that help on some NUMA systems\n"
33 help:  -ld LOGDIR, --logdir LOGDIR\n"
34 help:  --top-k N             top-k sampling (default: %d, 0 = disabled)\n", params.top-k
35 help:  -ins, --instruct      run in instruction mode (use with Alpaca models)\n"
36 help:  --random-prompt       start with a randomized prompt.\n"
37 help:  --repeat-penalty N    penalize repeat sequence of tokens (default: %.1f, 1.0 = disabled)\n", (double)params.repeat-penalty
38 help:  --infill              0 (defaut) use infill mode\n"
39 help:  --in-prefix STRING    string to prefix user inputs with (default: empty)\n"
40 help:  --rope-freq-base N    RoPE base frequency, used by NTK-aware scaling (default: loaded from model)\n"
41 help:  --mirostat-ent N      Mirostat target entropy, parameter tau (default: %.1f)\n", (double)params.mirostat-tau
42 help:  --mirostat-lr N       Mirostat learning rate, parameter eta (default: %.1f)\n", (double)params.mirostat-eta
43 help:  --keep N              number of tokens to keep from the initial prompt (default: %d, -1 = all)\n", params.n-keep

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:          logit_bias; role: // way to alter prob of particular words                    ;  default: 0          
 2 gpt_param:                seed; role: // RNG seed                                                 ;  default: -1         
 3 gpt_param:           n_predict; role: // new tokens to predict                                    ;  default: -1         
 4 gpt_param:               n_ctx; role: // context size                                             ;  default: 512        
 5 gpt_param:             n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ;  default: 512        
 6 gpt_param:              n_keep; role: // number of tokens to keep from initial prompt             ;  default: 0          
 7 gpt_param:      rope_freq_base; role: // RoPE base frequency                                      ;  default: 0.0f       
 8 gpt_param:     rope_freq_scale; role: // RoPE frequency scaling factor                            ;  default: 0.0f       
 9 gpt_param:               top_k; role: // < 0 to use vocab size                                    ;  default: 40         
10 gpt_param:               top_p; role: // 1.0 disabled                                             ;  default: 0.95f      
11 gpt_param:               tfs_z; role: // 1.0 disabled                                             ;  default: 1.00f      
12 gpt_param:           typical_p; role: // 1.0 disabled                                             ;  default: 1.00f      
13 gpt_param:                temp; role: // 1.0 disabled                                             ;  default: 0.80f      
14 gpt_param:      repeat_penalty; role: // 1.0 disabled                                             ;  default: 1.10f      
15 gpt_param:       repeat_last_n; role: // last n tokens to penalize (0 disable, -1 cxt size)       ;  default: 64         
16 gpt_param:   frequency_penalty; role: // 0.0 disabled                                             ;  default: 0.00f      
17 gpt_param:    presence_penalty; role: // 0.0 disabled                                             ;  default: 0.00f      
18 gpt_param:            mirostat; role: // 0 disabled, 1 mirostat, 2 mirostat 2.0                   ;  default: 0          
19 gpt_param:        mirostat_tau; role: // target entropy                                           ;  default: 5.00f      
20 gpt_param:        mirostat_eta; role: // learning rate                                            ;  default: 0.10f      
21 gpt_param: cfg_negative_prompt; role: // string to help guidance                                  ;  default: //         
22 gpt_param:           cfg_scale; role: // How strong is guidance                                   ;  default: 1.f        
23 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         
24 gpt_param:              prompt; role: // user-provided single prompt                              ;  default: ""         
25 gpt_param:   path_prompt_cache; role: // path to file for saving/loading prompt eval state        ;  default: ""         
26 gpt_param:        input_prefix; role: // string to prefix user inputs with                        ;  default: ""         
27 gpt_param:        input_suffix; role: // string to suffix user inputs with                        ;  default: ""         
28 gpt_param:             grammar; role: // optional BNF-like grammar to constrain sampling          ;  default: ""         
29 gpt_param:          antiprompt; role: // string upon seeing which more user input is prompted     ;  default: //         
30 gpt_param:              logdir; role: // directory in which to save YAML log files                ;  default: ""         
31 gpt_param:       random_prompt; role: // do not randomize prompt if none provided                 ;  default: false      
32 gpt_param:           use_color; role: // use color to distinguish generations and inputs          ;  default: false      
33 gpt_param:         interactive; role: // interactive mode                                         ;  default: false      
34 gpt_param:           embedding; role: // get only sentence embedding                              ;  default: false      
35 gpt_param:   interactive_first; role: // wait for user input immediately                          ;  default: false      
36 gpt_param:     multiline_input; role: // reverse the usage of `\`                                 ;  default: false      
37 gpt_param:           simple_io; role: // improves compat'y with subprocs and ltd consoles         ;  default: false      
38 gpt_param:    input_prefix_bos; role: // prefix BOS to user inputs, preceding input_prefix        ;  default: false      
39 gpt_param:            instruct; role: // instruction mode (used for Alpaca models)                ;  default: false      
40 gpt_param:          logits_all; role: // return logits for all tokens in the batch                ;  default: false      
41 gpt_param:                numa; role: // attempt optimizations that help on some NUMA systems     ;  default: false      
42 gpt_param:      verbose_prompt; role: // print prompt tokens before generation                    ;  default: false      
43 gpt_param:              infill; role: // use infill mode                                          ;  default: false      

Searching the other way round is more efficient:

 1 key:                 embedding; role: // get only sentence embedding                              ; default: false     
 2 key:                    infill; role: // use infill mode                                          ; default: false     
 3 key:               interactive; role: // interactive mode                                         ; default: false     
 4 key:                  instruct; role: // instruction mode (used for Alpaca models)                ; default: false     
 5 key:          presence_penalty; role: // 0.0 disabled                                             ; default: 0.00f     
 6 key:           multiline_input; role: // reverse the usage of `\`                                 ; default: false     
 7 key:                logits_all; role: // return logits for all tokens in the batch                ; default: false     
 8 key:                    prompt; role: // user-provided single prompt                              ; default: ""        
 9 key:                 cfg_scale; role: // How strong is guidance                                   ; default: 1.f       
10 key:                  mirostat; role: // 0 disabled, 1 mirostat, 2 mirostat 2.0                   ; default: 0         
11 key:         path_prompt_cache; role: // path to file for saving/loading prompt eval state        ; default: ""        
12 key:                      numa; role: // attempt optimizations that help on some NUMA systems     ; default: false     
13 key:            rope_freq_base; role: // RoPE base frequency                                      ; default: 0.0f      
14 key:                     tfs_z; role: // 1.0 disabled                                             ; default: 1.00f     
15 key:                     top_k; role: // < 0 to use vocab size                                    ; default: 40        
16 key:            verbose_prompt; role: // print prompt tokens before generation                    ; default: false     
17 key:                 simple_io; role: // improves compat'y with subprocs and ltd consoles         ; default: false     
18 key:             repeat_last_n; role: // last n tokens to penalize (0 disable, -1 cxt size)       ; default: 64        
19 key:       cfg_negative_prompt; role: // string to help guidance                                  ; default: //        
20 key:                     n_ctx; role: // context size                                             ; default: 512       
21 key:                      seed; role: // RNG seed                                                 ; default: -1        
22 key:                   n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ; default: 512       
23 key:         interactive_first; role: // wait for user input immediately                          ; default: false     
24 key:                 typical_p; role: // 1.0 disabled                                             ; default: 1.00f     
25 key:              input_prefix; role: // string to prefix user inputs with                        ; default: ""        
26 key:                     top_p; role: // 1.0 disabled                                             ; default: 0.95f     
27 key:              input_suffix; role: // string to suffix user inputs with                        ; default: ""        
28 key:          input_prefix_bos; role: // prefix BOS to user inputs, preceding input_prefix        ; default: false     
29 key:                 use_color; role: // use color to distinguish generations and inputs          ; default: false     
30 key:            repeat_penalty; role: // 1.0 disabled                                             ; default: 1.10f     
31 key:         frequency_penalty; role: // 0.0 disabled                                             ; default: 0.00f     
32 key:              mirostat_eta; role: // learning rate                                            ; default: 0.10f     
33 key:                      temp; role: // 1.0 disabled                                             ; default: 0.80f     
34 key:                   grammar; role: // optional BNF-like grammar to constrain sampling          ; default: ""        
35 key:                    logdir; role: // directory in which to save YAML log files                ; default: ""        
36 key:              mirostat_tau; role: // target entropy                                           ; default: 5.00f     
37 key:                      help; role: // universal help parameter                                 ; default: ""        
38 key:                    n_keep; role: // number of tokens to keep from initial prompt             ; default: 0         
39 key:           rope_freq_scale; role: // RoPE frequency scaling factor                            ; default: 0.0f      
40 key:                 n_predict; role: // new tokens to predict                                    ; default: -1        
41 key:             random_prompt; role: // do not randomize prompt if none provided                 ; default: false     
42 key:                logit_bias; role: // way to alter prob of particular words                    ; default: 0         
43 key:                antiprompt; role: // string upon seeing which more user input is prompted     ; default: //        

No unresolved help-list incompatibilities with infill.cpp


#########################
Filename: llama-bench.cpp
#########################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  -ts SPLIT --tensor-split SPLIT\n"
 2 help:  -mg i, --main-gpu i   the GPU to use for scratch and small tensors\n"
 3 help:  -b N, --batch-size N  batch size for prompt processing (default: %d)\n", params.n-batch
 4 help:  -t N, --threads N     number of threads to use during generation (default: %d)\n", params.n-threads
 5 help:  -ngl N, --n-gpu-layers N\n"  
 6 help:  -m FNAME, --model FNAME\n"   
 7 help:  -h, --help            show this help message and exit\n"

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:           n_threads; role: // user-defined or num of internal physical cores           ;  default: get_num_physical_cores() 
 2 gpt_param:             n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ;  default: 512        
 3 gpt_param:        n_gpu_layers; role: // num layers stored in VRAM (-1 for default)               ;  default: -1         
 4 gpt_param:            main_gpu; role: // the GPU that is used for scratch and small tensors       ;  default: 0          
 5 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         
 6 gpt_param:               model; role: // model path                                               ;  default: "models/7B/ggml-model-f16.gguf" 
 7 gpt_param:           mul_mat_q; role: // if true, use mul_mat_q kernels instead of cuBLAS         ;  default: true       

Searching the other way round is more efficient:

 1 key:                     model; role: // model path                                               ; default: "models/7B/ggml-model-f16.gguf"
 2 key:                 mul_mat_q; role: // if true, use mul_mat_q kernels instead of cuBLAS         ; default: true      
 3 key:                 n_threads; role: // user-defined or num of internal physical cores           ; default: get_num_physical_cores()
 4 key:                   n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ; default: 512       
 5 key:                      help; role: // universal help parameter                                 ; default: ""        
 6 key:              n_gpu_layers; role: // num layers stored in VRAM (-1 for default)               ; default: -1        
 7 key:                  main_gpu; role: // the GPU that is used for scratch and small tensors       ; default: 0         

No unresolved help-list incompatibilities with llama-bench.cpp


##################
Filename: main.cpp
##################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  --top-p N             top-p sampling (default: %.1f, 1.0 = disabled)\n", (double)params.top-p
 2 help:  -i, --interactive     run in interactive mode\n"
 3 help:  --embedding           0 (default): get only sentence embedding\n"
 4 help:  -n N, --n-predict N   number of tokens to predict (default: %d, -1 = infinity, -2 = until context filled)\n", params.n-predict
 5 help:  -b N, --batch-size N  batch size for prompt processing (default: %d)\n", params.n-batch
 6 help:  --interactive-first   run in interactive mode and wait for input right away\n"
 7 help:  --in-suffix STRING    string to suffix after user inputs with (default: empty)\n"
 8 help:  -p PROMPT, --prompt PROMPT\n"
 9 help:  -r PROMPT, --reverse-prompt PROMPT\n"
10 help:  --rope-freq-scale N   RoPE frequency linear scaling factor (default: loaded from model)\n"
11 help:  -l T, --logit-bias T  T = TOKEN-ID(plus/minus)BIAS\n"
12 help:  --typical N           locally typical sampling, parameter p (default: %.1f, 1.0 = disabled)\n", (double)params.typical-p
13 help:  --color               colorise output to distinguish prompt and user input from generations\n"
14 help:  --presence-penalty N  repeat alpha presence penalty (default: %.1f, 0.0 = disabled)\n", (double)params.presence-penalty
15 help:  -c N, --ctx-size N    size of the prompt context (default: %d, 0 = loaded from model)\n", params.n-ctx
16 help:  --multiline-input     allows you to write or paste multiple lines without ending each in '\\'\n"
17 help:  --mirostat N          use Mirostat sampling.\n"
18 help:  --verbose-prompt      print prompt before generation\n"
   help:                        i.e. `--logit-bias 15043+1` to increase likelihood of token ' Hello',\n"
19 help:  --in-prefix-bos       prefix BOS to user inputs, preceding the `--in-prefix` string\n"
20 help:  --use-color           0 (default): use color to distinguish generations from inputs\n"
21 help:  --tfs N               tail free sampling, parameter z (default: %.1f, 1.0 = disabled)\n", (double)params.tfs-z
22 help:  --prompt-cache-ro     if specified, uses the prompt cache but does not update it.\n"
23 help:  --prompt-cache FNAME  file to cache prompt state for faster startup (default: none)\n"
24 help:  --frequency-penalty N repeat alpha frequency penalty (default: %.1f, 0.0 = disabled)\n", (double)params.frequency-penalty
25 help:  -h, --help            show this help message and exit\n"
26 help:  --grammar GRAMMAR     BNF-like grammar to constrain generations (see samples in grammars/ dir)\n"
27 help:  --logits-all          return logits for all tokens in the batch (default: disabled)\n"
   help:                        or `--logit-bias 15043-1` to decrease likelihood of token ' Hello'\n"
28 help:  --cfg-scale N         strength of guidance (default: %f, 1.0 = disable)\n", params.cfg-scale
29 help:  -s SEED, --seed SEED  RNG seed (default: -1, use random seed for < 0)\n"
30 help:  --cfg-negative-prompt PROMPT\n"
31 help:  --repeat-last-n N     last n tokens to consider for penalize (default: %d, 0 = disabled, -1 = ctx-size)\n", params.repeat-last-n
32 help:  --temp N              temperature (default: %.1f)\n", (double)params.temp
33 help:  --numa                attempt optimizations that help on some NUMA systems\n"
34 help:  -ld LOGDIR, --logdir LOGDIR\n"
35 help:  --top-k N             top-k sampling (default: %d, 0 = disabled)\n", params.top-k
36 help:  -ins, --instruct      run in instruction mode (use with Alpaca models)\n"
37 help:  --prompt-cache-all    if specified, saves user input and generations to cache as well.\n"
38 help:  --random-prompt       start with a randomized prompt.\n"
39 help:  --repeat-penalty N    penalize repeat sequence of tokens (default: %.1f, 1.0 = disabled)\n", (double)params.repeat-penalty
40 help:  --in-prefix STRING    string to prefix user inputs with (default: empty)\n"
41 help:  --rope-freq-base N    RoPE base frequency, used by NTK-aware scaling (default: loaded from model)\n"
42 help:  --mirostat-ent N      Mirostat target entropy, parameter tau (default: %.1f)\n", (double)params.mirostat-tau
43 help:  --mirostat-lr N       Mirostat learning rate, parameter eta (default: %.1f)\n", (double)params.mirostat-eta
44 help:  --keep N              number of tokens to keep from the initial prompt (default: %d, -1 = all)\n", params.n-keep

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:          logit_bias; role: // way to alter prob of particular words                    ;  default: 0          
 2 gpt_param:                seed; role: // RNG seed                                                 ;  default: -1         
 3 gpt_param:           n_predict; role: // new tokens to predict                                    ;  default: -1         
 4 gpt_param:               n_ctx; role: // context size                                             ;  default: 512        
 5 gpt_param:             n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ;  default: 512        
 6 gpt_param:              n_keep; role: // number of tokens to keep from initial prompt             ;  default: 0          
 7 gpt_param:      rope_freq_base; role: // RoPE base frequency                                      ;  default: 0.0f       
 8 gpt_param:     rope_freq_scale; role: // RoPE frequency scaling factor                            ;  default: 0.0f       
 9 gpt_param:               top_k; role: // < 0 to use vocab size                                    ;  default: 40         
10 gpt_param:               top_p; role: // 1.0 disabled                                             ;  default: 0.95f      
11 gpt_param:               tfs_z; role: // 1.0 disabled                                             ;  default: 1.00f      
12 gpt_param:           typical_p; role: // 1.0 disabled                                             ;  default: 1.00f      
13 gpt_param:                temp; role: // 1.0 disabled                                             ;  default: 0.80f      
14 gpt_param:      repeat_penalty; role: // 1.0 disabled                                             ;  default: 1.10f      
15 gpt_param:       repeat_last_n; role: // last n tokens to penalize (0 disable, -1 cxt size)       ;  default: 64         
16 gpt_param:   frequency_penalty; role: // 0.0 disabled                                             ;  default: 0.00f      
17 gpt_param:    presence_penalty; role: // 0.0 disabled                                             ;  default: 0.00f      
18 gpt_param:            mirostat; role: // 0 disabled, 1 mirostat, 2 mirostat 2.0                   ;  default: 0          
19 gpt_param:        mirostat_tau; role: // target entropy                                           ;  default: 5.00f      
20 gpt_param:        mirostat_eta; role: // learning rate                                            ;  default: 0.10f      
21 gpt_param: cfg_negative_prompt; role: // string to help guidance                                  ;  default: //         
22 gpt_param:           cfg_scale; role: // How strong is guidance                                   ;  default: 1.f        
23 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         
24 gpt_param:              prompt; role: // user-provided single prompt                              ;  default: ""         
25 gpt_param:   path_prompt_cache; role: // path to file for saving/loading prompt eval state        ;  default: ""         
26 gpt_param:        input_prefix; role: // string to prefix user inputs with                        ;  default: ""         
27 gpt_param:        input_suffix; role: // string to suffix user inputs with                        ;  default: ""         
28 gpt_param:             grammar; role: // optional BNF-like grammar to constrain sampling          ;  default: ""         
29 gpt_param:          antiprompt; role: // string upon seeing which more user input is prompted     ;  default: //         
30 gpt_param:              logdir; role: // directory in which to save YAML log files                ;  default: ""         
31 gpt_param:       random_prompt; role: // do not randomize prompt if none provided                 ;  default: false      
32 gpt_param:           use_color; role: // use color to distinguish generations and inputs          ;  default: false      
33 gpt_param:         interactive; role: // interactive mode                                         ;  default: false      
34 gpt_param:    prompt_cache_all; role: // save user input and generations to prompt cache          ;  default: false      
35 gpt_param:     prompt_cache_ro; role: // open the prompt cache read-only and do not update it     ;  default: false      
36 gpt_param:           embedding; role: // get only sentence embedding                              ;  default: false      
37 gpt_param:   interactive_first; role: // wait for user input immediately                          ;  default: false      
38 gpt_param:     multiline_input; role: // reverse the usage of `\`                                 ;  default: false      
39 gpt_param:           simple_io; role: // improves compat'y with subprocs and ltd consoles         ;  default: false      
40 gpt_param:    input_prefix_bos; role: // prefix BOS to user inputs, preceding input_prefix        ;  default: false      
41 gpt_param:            instruct; role: // instruction mode (used for Alpaca models)                ;  default: false      
42 gpt_param:          logits_all; role: // return logits for all tokens in the batch                ;  default: false      
43 gpt_param:                numa; role: // attempt optimizations that help on some NUMA systems     ;  default: false      
44 gpt_param:      verbose_prompt; role: // print prompt tokens before generation                    ;  default: false      

Searching the other way round is more efficient:

 1 key:                 embedding; role: // get only sentence embedding                              ; default: false     
 2 key:          prompt_cache_all; role: // save user input and generations to prompt cache          ; default: false     
 3 key:               interactive; role: // interactive mode                                         ; default: false     
 4 key:                  instruct; role: // instruction mode (used for Alpaca models)                ; default: false     
 5 key:          presence_penalty; role: // 0.0 disabled                                             ; default: 0.00f     
 6 key:           multiline_input; role: // reverse the usage of `\`                                 ; default: false     
 7 key:                logits_all; role: // return logits for all tokens in the batch                ; default: false     
 8 key:                    prompt; role: // user-provided single prompt                              ; default: ""        
 9 key:                 cfg_scale; role: // How strong is guidance                                   ; default: 1.f       
10 key:                  mirostat; role: // 0 disabled, 1 mirostat, 2 mirostat 2.0                   ; default: 0         
11 key:         path_prompt_cache; role: // path to file for saving/loading prompt eval state        ; default: ""        
12 key:                      numa; role: // attempt optimizations that help on some NUMA systems     ; default: false     
13 key:           prompt_cache_ro; role: // open the prompt cache read-only and do not update it     ; default: false     
14 key:            rope_freq_base; role: // RoPE base frequency                                      ; default: 0.0f      
15 key:                     tfs_z; role: // 1.0 disabled                                             ; default: 1.00f     
16 key:                     top_k; role: // < 0 to use vocab size                                    ; default: 40        
17 key:            verbose_prompt; role: // print prompt tokens before generation                    ; default: false     
18 key:                 simple_io; role: // improves compat'y with subprocs and ltd consoles         ; default: false     
19 key:             repeat_last_n; role: // last n tokens to penalize (0 disable, -1 cxt size)       ; default: 64        
20 key:       cfg_negative_prompt; role: // string to help guidance                                  ; default: //        
21 key:                     n_ctx; role: // context size                                             ; default: 512       
22 key:                      seed; role: // RNG seed                                                 ; default: -1        
23 key:                   n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ; default: 512       
24 key:         interactive_first; role: // wait for user input immediately                          ; default: false     
25 key:                 typical_p; role: // 1.0 disabled                                             ; default: 1.00f     
26 key:              input_prefix; role: // string to prefix user inputs with                        ; default: ""        
27 key:                     top_p; role: // 1.0 disabled                                             ; default: 0.95f     
28 key:              input_suffix; role: // string to suffix user inputs with                        ; default: ""        
29 key:          input_prefix_bos; role: // prefix BOS to user inputs, preceding input_prefix        ; default: false     
30 key:                 use_color; role: // use color to distinguish generations and inputs          ; default: false     
31 key:            repeat_penalty; role: // 1.0 disabled                                             ; default: 1.10f     
32 key:         frequency_penalty; role: // 0.0 disabled                                             ; default: 0.00f     
33 key:              mirostat_eta; role: // learning rate                                            ; default: 0.10f     
34 key:                      temp; role: // 1.0 disabled                                             ; default: 0.80f     
35 key:                   grammar; role: // optional BNF-like grammar to constrain sampling          ; default: ""        
36 key:                    logdir; role: // directory in which to save YAML log files                ; default: ""        
37 key:              mirostat_tau; role: // target entropy                                           ; default: 5.00f     
38 key:                      help; role: // universal help parameter                                 ; default: ""        
39 key:                    n_keep; role: // number of tokens to keep from initial prompt             ; default: 0         
40 key:           rope_freq_scale; role: // RoPE frequency scaling factor                            ; default: 0.0f      
41 key:                 n_predict; role: // new tokens to predict                                    ; default: -1        
42 key:             random_prompt; role: // do not randomize prompt if none provided                 ; default: false     
43 key:                logit_bias; role: // way to alter prob of particular words                    ; default: 0         
44 key:                antiprompt; role: // string upon seeing which more user input is prompted     ; default: //        

No unresolved help-list incompatibilities with main.cpp


###################
Filename: metal.cpp
###################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  -h, --help            show this help message and exit\n"

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         

Searching the other way round is more efficient:

 1 key:                      help; role: // universal help parameter                                 ; default: ""        

No unresolved help-list incompatibilities with metal.cpp


######################
Filename: parallel.cpp
######################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  --logits-all          return logits for all tokens in the batch (default: disabled)\n"
 2 help:  -n N, --n-predict N   number of tokens to predict (default: %d, -1 = infinity, -2 = until context filled)\n", params.n-predict
 3 help:  -ns N, --sequences N  number of sequences to decode (default: %d)\n", params.n-sequences
 4 help:  -b N, --batch-size N  batch size for prompt processing (default: %d)\n", params.n-batch
 5 help:  --numa                attempt optimizations that help on some NUMA systems\n"
 6 help:  -cb, --cont-batching  enable continuous batching (a.k.a dynamic batching) (default: disabled)\n"
 7 help:  -h, --help            show this help message and exit\n"
 8 help:  -np N, --parallel N   number of parallel sequences to decode (default: %d)\n", params.n-parallel
 9 help:  -c N, --ctx-size N    size of the prompt context (default: %d, 0 = loaded from model)\n", params.n-ctx

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:           n_predict; role: // new tokens to predict                                    ;  default: -1         
 2 gpt_param:               n_ctx; role: // context size                                             ;  default: 512        
 3 gpt_param:             n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ;  default: 512        
 4 gpt_param:          n_parallel; role: // number of parallel sequences to decode                   ;  default: 1          
 5 gpt_param:         n_sequences; role: // number of sequences to decode                            ;  default: 1          
 6 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         
 7 gpt_param:       cont_batching; role: // insert new sequences for decoding on-the-fly             ;  default: false      
 8 gpt_param:          logits_all; role: // return logits for all tokens in the batch                ;  default: false      
 9 gpt_param:                numa; role: // attempt optimizations that help on some NUMA systems     ;  default: false      

Searching the other way round is more efficient:

 1 key:                     n_ctx; role: // context size                                             ; default: 512       
 2 key:                n_parallel; role: // number of parallel sequences to decode                   ; default: 1         
 3 key:             cont_batching; role: // insert new sequences for decoding on-the-fly             ; default: false     
 4 key:                      numa; role: // attempt optimizations that help on some NUMA systems     ; default: false     
 5 key:                   n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ; default: 512       
 6 key:                      help; role: // universal help parameter                                 ; default: ""        
 7 key:               n_sequences; role: // number of sequences to decode                            ; default: 1         
 8 key:                 n_predict; role: // new tokens to predict                                    ; default: -1        
 9 key:                logits_all; role: // return logits for all tokens in the batch                ; default: false     

No unresolved help-list incompatibilities with parallel.cpp


########################
Filename: perplexity.cpp
########################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  --logits-all          return logits for all tokens in the batch (default: disabled)\n"
 2 help:  --random-prompt       start with a randomized prompt.\n"
 3 help:  --hellaswag           compute HellaSwag score over random tasks from datafile supplied with -f\n"
 4 help:  -b N, --batch-size N  batch size for prompt processing (default: %d)\n", params.n-batch
 5 help:  --ppl-output-type     0 (default): ppl output as usual, 1: ppl output num-tokens, one per line\n"
 6 help:  -s SEED, --seed SEED  RNG seed (default: -1, use random seed for < 0)\n"
 7 help:  -p PROMPT, --prompt PROMPT\n"
 8 help:  --chunks N            max number of chunks to process (default: %d, -1 = all)\n", params.n-chunks
 9 help:  --ppl-stride          stride for ppl calcs. 0 (default): the pre-existing approach will be used.\n"
10 help:  --numa                attempt optimizations that help on some NUMA systems\n"
11 help:  --hellaswag-tasks N   number of tasks to use when computing the HellaSwag score (default: %zu)\n", params.hellaswag-tasks
12 help:  -ld LOGDIR, --logdir LOGDIR\n"
13 help:  -h, --help            show this help message and exit\n"
14 help:  -c N, --ctx-size N    size of the prompt context (default: %d, 0 = loaded from model)\n", params.n-ctx

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:                seed; role: // RNG seed                                                 ;  default: -1         
 2 gpt_param:               n_ctx; role: // context size                                             ;  default: 512        
 3 gpt_param:             n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ;  default: 512        
 4 gpt_param:            n_chunks; role: // max number of chunks to process (-1 unlimited)           ;  default: -1         
 5 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         
 6 gpt_param:              prompt; role: // user-provided single prompt                              ;  default: ""         
 7 gpt_param:              logdir; role: // directory in which to save YAML log files                ;  default: ""         
 8 gpt_param:          ppl_stride; role: // stride for ppl calcs. 0: the pre-existing approach will be used.;  default: 0          
 9 gpt_param:     ppl_output_type; role: // 0: ppl output as usual, 1: ppl output num_tokens, ppl, one per line;  default: 0          
10 gpt_param:           hellaswag; role: // compute HellaSwag score from datafile given in prompt    ;  default: false      
11 gpt_param:     hellaswag_tasks; role: // number of tasks to use when computing the HellaSwag score;  default: 400        
12 gpt_param:       random_prompt; role: // do not randomize prompt if none provided                 ;  default: false      
13 gpt_param:          logits_all; role: // return logits for all tokens in the batch                ;  default: false      
14 gpt_param:                numa; role: // attempt optimizations that help on some NUMA systems     ;  default: false      

Searching the other way round is more efficient:

 1 key:                    prompt; role: // user-provided single prompt                              ; default: ""        
 2 key:           hellaswag_tasks; role: // number of tasks to use when computing the HellaSwag score; default: 400       
 3 key:                     n_ctx; role: // context size                                             ; default: 512       
 4 key:           ppl_output_type; role: // 0: ppl output as usual, 1: ppl output num_tokens, ppl, one per line; default: 0         
 5 key:                ppl_stride; role: // stride for ppl calcs. 0: the pre-existing approach will be used.; default: 0         
 6 key:                      seed; role: // RNG seed                                                 ; default: -1        
 7 key:                 hellaswag; role: // compute HellaSwag score from datafile given in prompt    ; default: false     
 8 key:                    logdir; role: // directory in which to save YAML log files                ; default: ""        
 9 key:                   n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ; default: 512       
10 key:                      numa; role: // attempt optimizations that help on some NUMA systems     ; default: false     
11 key:                      help; role: // universal help parameter                                 ; default: ""        
12 key:             random_prompt; role: // do not randomize prompt if none provided                 ; default: false     
13 key:                  n_chunks; role: // max number of chunks to process (-1 unlimited)           ; default: -1        
14 key:                logits_all; role: // return logits for all tokens in the batch                ; default: false     

No unresolved help-list incompatibilities with perplexity.cpp


############################
Filename: quantize-stats.cpp
############################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  -h, --help            show this help message and exit\n"
 2 help:  -m FNAME, --model FNAME\n"   

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         
 2 gpt_param:               model; role: // model path                                               ;  default: "models/7B/ggml-model-f16.gguf" 

Searching the other way round is more efficient:

 1 key:                     model; role: // model path                                               ; default: "models/7B/ggml-model-f16.gguf"
 2 key:                      help; role: // universal help parameter                                 ; default: ""        

No unresolved help-list incompatibilities with quantize-stats.cpp


######################
Filename: quantize.cpp
######################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  -h, --help            show this help message and exit\n"

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         

Searching the other way round is more efficient:

 1 key:                      help; role: // universal help parameter                                 ; default: ""        

No unresolved help-list incompatibilities with quantize.cpp


#############################
Filename: save-load-state.cpp
#############################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  -n N, --n-predict N   number of tokens to predict (default: %d, -1 = infinity, -2 = until context filled)\n", params.n-predict
 2 help:  -s SEED, --seed SEED  RNG seed (default: -1, use random seed for < 0)\n"
 3 help:  -p PROMPT, --prompt PROMPT\n"
 4 help:  -t N, --threads N     number of threads to use during generation (default: %d)\n", params.n-threads
 5 help:  --repeat-last-n N     last n tokens to consider for penalize (default: %d, 0 = disabled, -1 = ctx-size)\n", params.repeat-last-n
 6 help:  -h, --help            show this help message and exit\n"

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:                seed; role: // RNG seed                                                 ;  default: -1         
 2 gpt_param:           n_threads; role: // user-defined or num of internal physical cores           ;  default: get_num_physical_cores() 
 3 gpt_param:           n_predict; role: // new tokens to predict                                    ;  default: -1         
 4 gpt_param:       repeat_last_n; role: // last n tokens to penalize (0 disable, -1 cxt size)       ;  default: 64         
 5 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         
 6 gpt_param:              prompt; role: // user-provided single prompt                              ;  default: ""         

Searching the other way round is more efficient:

 1 key:                    prompt; role: // user-provided single prompt                              ; default: ""        
 2 key:                 n_threads; role: // user-defined or num of internal physical cores           ; default: get_num_physical_cores()
 3 key:                      seed; role: // RNG seed                                                 ; default: -1        
 4 key:                      help; role: // universal help parameter                                 ; default: ""        
 5 key:                 n_predict; role: // new tokens to predict                                    ; default: -1        
 6 key:             repeat_last_n; role: // last n tokens to penalize (0 disable, -1 cxt size)       ; default: 64        

No unresolved help-list incompatibilities with save-load-state.cpp


####################
Filename: server.cpp
####################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  --top-p N             top-p sampling (default: %.1f, 1.0 = disabled)\n", (double)params.top-p
 2 help:  --embedding           0 (default): get only sentence embedding\n"
 3 help:  --lora-base FNAME     optional model to use as a base for the layers modified by the LoRA adapter\n"
 4 help:  -n N, --n-predict N   number of tokens to predict (default: %d, -1 = infinity, -2 = until context filled)\n", params.n-predict
 5 help:  -tb N, --threads-batch N\n"  
 6 help:  -b N, --batch-size N  batch size for prompt processing (default: %d)\n", params.n-batch
 7 help:  --alias               model alias (default: 'unknown')\n"
 8 help:  --in-suffix STRING    string to suffix after user inputs with (default: empty)\n"
 9 help:  -r PROMPT, --reverse-prompt PROMPT\n"
10 help:  --rope-freq-scale N   RoPE frequency linear scaling factor (default: loaded from model)\n"
11 help:  -l T, --logit-bias T  T = TOKEN-ID(plus/minus)BIAS\n"
12 help:  --typical N           locally typical sampling, parameter p (default: %.1f, 1.0 = disabled)\n", (double)params.typical-p
13 help:  --beams N             0 (default): if non-zero use beam search of given width N.\n"
14 help:  -m FNAME, --model FNAME\n"   
15 help:  --presence-penalty N  repeat alpha presence penalty (default: %.1f, 0.0 = disabled)\n", (double)params.presence-penalty
16 help:  --no-mmap             0 (default): if true use mmap for faster loads.\n"
17 help:  -c N, --ctx-size N    size of the prompt context (default: %d, 0 = loaded from model)\n", params.n-ctx
18 help:  --mirostat N          use Mirostat sampling.\n"
19 help:  --mlock               0 (default): if true keep model in memory.\n"
   help:                        i.e. `--logit-bias 15043+1` to increase likelihood of token ' Hello',\n"
20 help:  -t N, --threads N     number of threads to use during generation (default: %d)\n", params.n-threads
21 help:  --tfs N               tail free sampling, parameter z (default: %.1f, 1.0 = disabled)\n", (double)params.tfs-z
22 help:  --frequency-penalty N repeat alpha frequency penalty (default: %.1f, 0.0 = disabled)\n", (double)params.frequency-penalty
23 help:  -ngl N, --n-gpu-layers N\n"  
24 help:  -h, --help            show this help message and exit\n"
25 help:  --grammar GRAMMAR     BNF-like grammar to constrain generations (see samples in grammars/ dir)\n"
   help:                        or `--logit-bias 15043-1` to decrease likelihood of token ' Hello'\n"
26 help:  -mg i, --main-gpu i   the GPU to use for scratch and small tensors\n"
27 help:  --no-mmap             do not memory-map model (slower load but may reduce pageouts if not using mlock)\n"
28 help:  -s SEED, --seed SEED  RNG seed (default: -1, use random seed for < 0)\n"
29 help:  --repeat-last-n N     last n tokens to consider for penalize (default: %d, 0 = disabled, -1 = ctx-size)\n", params.repeat-last-n
30 help:  --temp N              temperature (default: %.1f)\n", (double)params.temp
31 help:  --numa                attempt optimizations that help on some NUMA systems\n"
32 help:  --top-k N             top-k sampling (default: %d, 0 = disabled)\n", params.top-k
33 help:  --mlock               force system to keep model in RAM rather than swapping or compressing\n"
34 help:  --repeat-penalty N    penalize repeat sequence of tokens (default: %.1f, 1.0 = disabled)\n", (double)params.repeat-penalty
35 help:  --nprobs N            if > 0 output the probabilities of the top N tokens\n"
36 help:  --no-penalize-nl      do not penalize newline token (default is DO penalise nl token)\n"
37 help:  --in-prefix STRING    string to prefix user inputs with (default: empty)\n"
38 help:  --rope-freq-base N    RoPE base frequency, used by NTK-aware scaling (default: loaded from model)\n"
39 help:  --mirostat-ent N      Mirostat target entropy, parameter tau (default: %.1f)\n", (double)params.mirostat-tau
40 help:  --mirostat-lr N       Mirostat learning rate, parameter eta (default: %.1f)\n", (double)params.mirostat-eta
41 help:  --keep N              number of tokens to keep from the initial prompt (default: %d, -1 = all)\n", params.n-keep

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:          logit_bias; role: // way to alter prob of particular words                    ;  default: 0          
 2 gpt_param:                seed; role: // RNG seed                                                 ;  default: -1         
 3 gpt_param:           n_threads; role: // user-defined or num of internal physical cores           ;  default: get_num_physical_cores() 
 4 gpt_param:     n_threads_batch; role: // num threads for batch proc (-1 use n_threads)            ;  default: -1         
 5 gpt_param:           n_predict; role: // new tokens to predict                                    ;  default: -1         
 6 gpt_param:               n_ctx; role: // context size                                             ;  default: 512        
 7 gpt_param:             n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ;  default: 512        
 8 gpt_param:              n_keep; role: // number of tokens to keep from initial prompt             ;  default: 0          
 9 gpt_param:        n_gpu_layers; role: // num layers stored in VRAM (-1 for default)               ;  default: -1         
10 gpt_param:            main_gpu; role: // the GPU that is used for scratch and small tensors       ;  default: 0          
11 gpt_param:             n_probs; role: // if > 0, output probabilities of top n_probs tokens.      ;  default: 0          
12 gpt_param:             n_beams; role: // if non-zero then use beam search of given width.         ;  default: 0          
13 gpt_param:      rope_freq_base; role: // RoPE base frequency                                      ;  default: 0.0f       
14 gpt_param:     rope_freq_scale; role: // RoPE frequency scaling factor                            ;  default: 0.0f       
15 gpt_param:               top_k; role: // < 0 to use vocab size                                    ;  default: 40         
16 gpt_param:               top_p; role: // 1.0 disabled                                             ;  default: 0.95f      
17 gpt_param:               tfs_z; role: // 1.0 disabled                                             ;  default: 1.00f      
18 gpt_param:           typical_p; role: // 1.0 disabled                                             ;  default: 1.00f      
19 gpt_param:                temp; role: // 1.0 disabled                                             ;  default: 0.80f      
20 gpt_param:      repeat_penalty; role: // 1.0 disabled                                             ;  default: 1.10f      
21 gpt_param:       repeat_last_n; role: // last n tokens to penalize (0 disable, -1 cxt size)       ;  default: 64         
22 gpt_param:   frequency_penalty; role: // 0.0 disabled                                             ;  default: 0.00f      
23 gpt_param:    presence_penalty; role: // 0.0 disabled                                             ;  default: 0.00f      
24 gpt_param:            mirostat; role: // 0 disabled, 1 mirostat, 2 mirostat 2.0                   ;  default: 0          
25 gpt_param:        mirostat_tau; role: // target entropy                                           ;  default: 5.00f      
26 gpt_param:        mirostat_eta; role: // learning rate                                            ;  default: 0.10f      
27 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         
28 gpt_param:               model; role: // model path                                               ;  default: "models/7B/ggml-model-f16.gguf" 
29 gpt_param:         model_alias; role: // model alias                                              ;  default: "unknown"  
30 gpt_param:        input_prefix; role: // string to prefix user inputs with                        ;  default: ""         
31 gpt_param:        input_suffix; role: // string to suffix user inputs with                        ;  default: ""         
32 gpt_param:             grammar; role: // optional BNF-like grammar to constrain sampling          ;  default: ""         
33 gpt_param:          antiprompt; role: // string upon seeing which more user input is prompted     ;  default: //         
34 gpt_param:           lora_base; role: // base model path for the lora adapter                     ;  default: ""         
35 gpt_param:           mul_mat_q; role: // if true, use mul_mat_q kernels instead of cuBLAS         ;  default: true       
36 gpt_param:          memory_f16; role: // use f16 instead of f32 for memory kv                     ;  default: true       
37 gpt_param:           embedding; role: // get only sentence embedding                              ;  default: false      
38 gpt_param:         penalize_nl; role: // consider newlines as a repeatable token                  ;  default: true       
39 gpt_param:            use_mmap; role: // use mmap for faster loads                                ;  default: true       
40 gpt_param:           use_mlock; role: // use mlock to keep model in memory                        ;  default: false      
41 gpt_param:                numa; role: // attempt optimizations that help on some NUMA systems     ;  default: false      

Searching the other way round is more efficient:

 1 key:                     model; role: // model path                                               ; default: "models/7B/ggml-model-f16.gguf"
 2 key:                 embedding; role: // get only sentence embedding                              ; default: false     
 3 key:          presence_penalty; role: // 0.0 disabled                                             ; default: 0.00f     
 4 key:              n_gpu_layers; role: // num layers stored in VRAM (-1 for default)               ; default: -1        
 5 key:                 mul_mat_q; role: // if true, use mul_mat_q kernels instead of cuBLAS         ; default: true      
 6 key:                  mirostat; role: // 0 disabled, 1 mirostat, 2 mirostat 2.0                   ; default: 0         
 7 key:                 n_threads; role: // user-defined or num of internal physical cores           ; default: get_num_physical_cores()
 8 key:                      numa; role: // attempt optimizations that help on some NUMA systems     ; default: false     
 9 key:                   n_beams; role: // if non-zero then use beam search of given width.         ; default: 0         
10 key:            rope_freq_base; role: // RoPE base frequency                                      ; default: 0.0f      
11 key:                     tfs_z; role: // 1.0 disabled                                             ; default: 1.00f     
12 key:                     top_k; role: // < 0 to use vocab size                                    ; default: 40        
13 key:             repeat_last_n; role: // last n tokens to penalize (0 disable, -1 cxt size)       ; default: 64        
14 key:                     n_ctx; role: // context size                                             ; default: 512       
15 key:                      seed; role: // RNG seed                                                 ; default: -1        
16 key:                   n_probs; role: // if > 0, output probabilities of top n_probs tokens.      ; default: 0         
17 key:               model_alias; role: // model alias                                              ; default: "unknown" 
18 key:                 use_mlock; role: // use mlock to keep model in memory                        ; default: false     
19 key:                   n_batch; role: // batch size for prompt proc (> 32 to use BLAS)            ; default: 512       
20 key:                 typical_p; role: // 1.0 disabled                                             ; default: 1.00f     
21 key:              input_prefix; role: // string to prefix user inputs with                        ; default: ""        
22 key:               penalize_nl; role: // consider newlines as a repeatable token                  ; default: true      
23 key:                     top_p; role: // 1.0 disabled                                             ; default: 0.95f     
24 key:              input_suffix; role: // string to suffix user inputs with                        ; default: ""        
25 key:                  main_gpu; role: // the GPU that is used for scratch and small tensors       ; default: 0         
26 key:            repeat_penalty; role: // 1.0 disabled                                             ; default: 1.10f     
27 key:         frequency_penalty; role: // 0.0 disabled                                             ; default: 0.00f     
28 key:              mirostat_eta; role: // learning rate                                            ; default: 0.10f     
29 key:                memory_f16; role: // use f16 instead of f32 for memory kv                     ; default: true      
30 key:                      temp; role: // 1.0 disabled                                             ; default: 0.80f     
31 key:                   grammar; role: // optional BNF-like grammar to constrain sampling          ; default: ""        
32 key:                  use_mmap; role: // use mmap for faster loads                                ; default: true      
33 key:              mirostat_tau; role: // target entropy                                           ; default: 5.00f     
34 key:                      help; role: // universal help parameter                                 ; default: ""        
35 key:                 n_predict; role: // new tokens to predict                                    ; default: -1        
36 key:                 lora_base; role: // base model path for the lora adapter                     ; default: ""        
37 key:                    n_keep; role: // number of tokens to keep from initial prompt             ; default: 0         
38 key:           rope_freq_scale; role: // RoPE frequency scaling factor                            ; default: 0.0f      
39 key:           n_threads_batch; role: // num threads for batch proc (-1 use n_threads)            ; default: -1        
40 key:                logit_bias; role: // way to alter prob of particular words                    ; default: 0         
41 key:                antiprompt; role: // string upon seeing which more user input is prompted     ; default: //        

No unresolved help-list incompatibilities with server.cpp


####################
Filename: simple.cpp
####################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  -tb N, --threads-batch N\n"  
 2 help:  -p PROMPT, --prompt PROMPT\n"
 3 help:  -t N, --threads N     number of threads to use during generation (default: %d)\n", params.n-threads
 4 help:  --numa                attempt optimizations that help on some NUMA systems\n"
 5 help:  -m FNAME, --model FNAME\n"   
 6 help:  -h, --help            show this help message and exit\n"

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:           n_threads; role: // user-defined or num of internal physical cores           ;  default: get_num_physical_cores() 
 2 gpt_param:     n_threads_batch; role: // num threads for batch proc (-1 use n_threads)            ;  default: -1         
 3 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         
 4 gpt_param:               model; role: // model path                                               ;  default: "models/7B/ggml-model-f16.gguf" 
 5 gpt_param:              prompt; role: // user-provided single prompt                              ;  default: ""         
 6 gpt_param:                numa; role: // attempt optimizations that help on some NUMA systems     ;  default: false      

Searching the other way round is more efficient:

 1 key:                     model; role: // model path                                               ; default: "models/7B/ggml-model-f16.gguf"
 2 key:                    prompt; role: // user-provided single prompt                              ; default: ""        
 3 key:                 n_threads; role: // user-defined or num of internal physical cores           ; default: get_num_physical_cores()
 4 key:                      numa; role: // attempt optimizations that help on some NUMA systems     ; default: false     
 5 key:                      help; role: // universal help parameter                                 ; default: ""        
 6 key:           n_threads_batch; role: // num threads for batch proc (-1 use n_threads)            ; default: -1        

No unresolved help-list incompatibilities with simple.cpp


#########################
Filename: speculative.cpp
#########################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  -ngld N, --n-gpu-layers-draft N\n"
 2 help:  --logits-all          return logits for all tokens in the batch (default: disabled)\n"
 3 help:  --draft N             number of tokens to draft for speculative decoding (default: %d)\n", params.n-draft
 4 help:  -n N, --n-predict N   number of tokens to predict (default: %d, -1 = infinity, -2 = until context filled)\n", params.n-predict
 5 help:  -md FNAME, --model-draft FNAME\n"
 6 help:  -p PROMPT, --prompt PROMPT\n"
 7 help:  --numa                attempt optimizations that help on some NUMA systems\n"
 8 help:  -ngl N, --n-gpu-layers N\n"  
 9 help:  -m FNAME, --model FNAME\n"   
10 help:  -h, --help            show this help message and exit\n"
11 help:  --grammar GRAMMAR     BNF-like grammar to constrain generations (see samples in grammars/ dir)\n"

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:           n_predict; role: // new tokens to predict                                    ;  default: -1         
 2 gpt_param:             n_draft; role: // number of tokens to draft during speculative decoding    ;  default: 16         
 3 gpt_param:        n_gpu_layers; role: // num layers stored in VRAM (-1 for default)               ;  default: -1         
 4 gpt_param:  n_gpu_layers_draft; role: // num layers stored in VRAM for draft mod (-1 for default) ;  default: -1         
 5 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         
 6 gpt_param:               model; role: // model path                                               ;  default: "models/7B/ggml-model-f16.gguf" 
 7 gpt_param:         model_draft; role: // draft model for speculative decoding                     ;  default: ""         
 8 gpt_param:              prompt; role: // user-provided single prompt                              ;  default: ""         
 9 gpt_param:             grammar; role: // optional BNF-like grammar to constrain sampling          ;  default: ""         
10 gpt_param:          logits_all; role: // return logits for all tokens in the batch                ;  default: false      
11 gpt_param:                numa; role: // attempt optimizations that help on some NUMA systems     ;  default: false      

Searching the other way round is more efficient:

 1 key:                     model; role: // model path                                               ; default: "models/7B/ggml-model-f16.gguf"
 2 key:                    prompt; role: // user-provided single prompt                              ; default: ""        
 3 key:               model_draft; role: // draft model for speculative decoding                     ; default: ""        
 4 key:                   grammar; role: // optional BNF-like grammar to constrain sampling          ; default: ""        
 5 key:                      numa; role: // attempt optimizations that help on some NUMA systems     ; default: false     
 6 key:        n_gpu_layers_draft; role: // num layers stored in VRAM for draft mod (-1 for default) ; default: -1        
 7 key:                      help; role: // universal help parameter                                 ; default: ""        
 8 key:                 n_predict; role: // new tokens to predict                                    ; default: -1        
 9 key:              n_gpu_layers; role: // num layers stored in VRAM (-1 for default)               ; default: -1        
10 key:                logits_all; role: // return logits for all tokens in the batch                ; default: false     
11 key:                   n_draft; role: // number of tokens to draft during speculative decoding    ; default: 16        

No unresolved help-list incompatibilities with speculative.cpp


#####################################
Filename: train-text-from-scratch.cpp
#####################################

Command-line arguments available and gpt-params functions implemented (TODO: multi-line helps NEED SOME WORK):

 1 help:  -t N, --threads N     number of threads to use during generation (default: %d)\n", params.n-threads
 2 help:  --rope-freq-scale N   RoPE frequency linear scaling factor (default: loaded from model)\n"
 3 help:  --rope-freq-base N    RoPE base frequency, used by NTK-aware scaling (default: loaded from model)\n"
 4 help:  -h, --help            show this help message and exit\n"
 5 help:  -c N, --ctx-size N    size of the prompt context (default: %d, 0 = loaded from model)\n", params.n-ctx

Now we extract the original gpt_params definition from common.h with the defaults for implemented arguments:

 1 gpt_param:           n_threads; role: // user-defined or num of internal physical cores           ;  default: get_num_physical_cores() 
 2 gpt_param:               n_ctx; role: // context size                                             ;  default: 512        
 3 gpt_param:      rope_freq_base; role: // RoPE base frequency                                      ;  default: 0.0f       
 4 gpt_param:     rope_freq_scale; role: // RoPE frequency scaling factor                            ;  default: 0.0f       
 5 gpt_param:                help; role: // universal help parameter                                 ;  default: ""         

Searching the other way round is more efficient:

 1 key:                 n_threads; role: // user-defined or num of internal physical cores           ; default: get_num_physical_cores()
 2 key:            rope_freq_base; role: // RoPE base frequency                                      ; default: 0.0f      
 3 key:                     n_ctx; role: // context size                                             ; default: 512       
 4 key:                      help; role: // universal help parameter                                 ; default: ""        
 5 key:           rope_freq_scale; role: // RoPE frequency scaling factor                            ; default: 0.0f      

No unresolved help-list incompatibilities with train-text-from-scratch.cpp

[pudepiedj]

Now opened in PR #3556.