nvcc fatal : 'f16c': expected a number #1467

sunyuhan19981208 · 2023-05-15T11:14:57Z

Prerequisites

Please answer the following questions for yourself before submitting an issue.

[NO] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
[YES] I carefully followed the README.md.
[YES] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
[YES] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior & Failure Information & Failure Logs

I have successfully compiled the code in CPU-only mode. However, when I attempt to compile it with the '-DLLAMA_CUBLAS=1' flag to enable cuBLAS support, I encounter an error. I expect the compilation to be successful with cuBLAS enabled. Below, I have included the error logs for reference:

#cmake --build . --config Release -- VERBOSE=1
/home/sunyuhan/syh/sunyuhan/cmake-3.26.3-linux-x86_64/bin/cmake -S/home/sunyuhan/syh/sunyuhan/zju/llama.cpp -B/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build --check-build-system CMakeFiles/Makefile.cmake 0
/home/sunyuhan/syh/sunyuhan/cmake-3.26.3-linux-x86_64/bin/cmake -E cmake_progress_start /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build/CMakeFiles /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build//CMakeFiles/progress.marks
/usr/bin/make  -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
/usr/bin/make  -f CMakeFiles/BUILD_INFO.dir/build.make CMakeFiles/BUILD_INFO.dir/depend
make[2]: Entering directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
cd /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build && /home/sunyuhan/syh/sunyuhan/cmake-3.26.3-linux-x86_64/bin/cmake -E cmake_depends "Unix Makefiles" /home/sunyuhan/syh/sunyuhan/zju/llama.cpp /home/sunyuhan/syh/sunyuhan/zju/llama.cpp /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build/CMakeFiles/BUILD_INFO.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
/usr/bin/make  -f CMakeFiles/BUILD_INFO.dir/build.make CMakeFiles/BUILD_INFO.dir/build
make[2]: Entering directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
make[2]: Nothing to be done for 'CMakeFiles/BUILD_INFO.dir/build'.
make[2]: Leaving directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
[  2%] Built target BUILD_INFO
/usr/bin/make  -f CMakeFiles/ggml.dir/build.make CMakeFiles/ggml.dir/depend
make[2]: Entering directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
cd /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build && /home/sunyuhan/syh/sunyuhan/cmake-3.26.3-linux-x86_64/bin/cmake -E cmake_depends "Unix Makefiles" /home/sunyuhan/syh/sunyuhan/zju/llama.cpp /home/sunyuhan/syh/sunyuhan/zju/llama.cpp /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build/CMakeFiles/ggml.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
/usr/bin/make  -f CMakeFiles/ggml.dir/build.make CMakeFiles/ggml.dir/build
make[2]: Entering directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
[  5%] Building C object CMakeFiles/ggml.dir/ggml.c.o
/usr/bin/gcc -DGGML_USE_CUBLAS -I/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/. -isystem /usr/local/cuda/include -O3 -DNDEBUG -std=gnu11 -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -mf16c -mfma -mavx -mavx2 -pthread -MD -MT CMakeFiles/ggml.dir/ggml.c.o -MF CMakeFiles/ggml.dir/ggml.c.o.d -o CMakeFiles/ggml.dir/ggml.c.o -c /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/ggml.c
[  8%] Building CUDA object CMakeFiles/ggml.dir/ggml-cuda.cu.o
/usr/local/cuda/bin/nvcc -forward-unknown-to-host-compiler -DGGML_USE_CUBLAS --options-file CMakeFiles/ggml.dir/includes_CUDA.rsp -O3 -DNDEBUG -std=c++11 -mf16c -mfma -mavx -mavx2 -Xcompiler -pthread -MD -MT CMakeFiles/ggml.dir/ggml-cuda.cu.o -MF CMakeFiles/ggml.dir/ggml-cuda.cu.o.d -x cu -c /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/ggml-cuda.cu -o CMakeFiles/ggml.dir/ggml-cuda.cu.o
nvcc fatal   : 'f16c': expected a number
CMakeFiles/ggml.dir/build.make:90: recipe for target 'CMakeFiles/ggml.dir/ggml-cuda.cu.o' failed
make[2]: *** [CMakeFiles/ggml.dir/ggml-cuda.cu.o] Error 1
make[2]: Leaving directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
CMakeFiles/Makefile2:359: recipe for target 'CMakeFiles/ggml.dir/all' failed
make[1]: *** [CMakeFiles/ggml.dir/all] Error 2
make[1]: Leaving directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
Makefile:100: recipe for target 'all' failed
make: *** [all] Error 2

Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

Physical (or virtual) hardware you are using, e.g. for Linux:

$ lscpu

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              48
On-line CPU(s) list: 0-47
Thread(s) per core:  2
Core(s) per socket:  12
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Gold 6146 CPU @ 3.20GHz
Stepping:            4
CPU MHz:             3202.505
CPU max MHz:         3200.0000
CPU min MHz:         1200.0000
BogoMIPS:            6400.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            25344K
NUMA node0 CPU(s):   0-11,24-35
NUMA node1 CPU(s):   12-23,36-47
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm arat pln pts pku ospke md_clear flush_l1d

Operating System, e.g. for Linux:

$ uname -a

Linux 7acf3cf70158 5.4.0-91-generic #102~18.04.1-Ubuntu SMP Thu Nov 11 14:46:36 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

SDK version, e.g. for Linux:

$ make --version

GNU Make 4.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

$ g++ --version

g++ (Ubuntu 8.4.0-1ubuntu1~18.04) 8.4.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$nvcc -V

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0

Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

git clone the latest code
mkdir build
cmake .. -DLLAMA_CUBLAS=1
cmake --build . --config Release -- VERBOSE=1

The text was updated successfully, but these errors were encountered:

sunyuhan19981208 · 2023-05-15T12:26:34Z

After I change my docker from CUDA11.0 to CUDA11.1, nothing happen. So the lowest version that can run llama.cpp is CUDA11.1?

kexul · 2023-05-15T13:08:27Z

I got similar error nvcc fatal : Value 'native' is not defined for option 'gpu-architecture' with cuda 11.0. But I'm not able to upgrade my cuda...

SlyEcho · 2023-05-15T14:41:05Z

What if you disable -mf16c? -DLLAMA_F16C=OFF

As for native, I was able to get around it by using all: -DCMAKE_CUDA_ARCHITECTURES=all

kexul · 2023-05-15T15:26:24Z

Hi @SlyEcho , I tried -DCMAKE_CUDA_ARCHITECTURES=all, another error occured:

CMake Error at /usr/local/share/cmake-3.20/Modules/CMakeDetermineCompilerId.cmake:676 (message):
  Compiling the CUDA compiler identification source file
  "CMakeCUDACompilerId.cu" failed.

  Compiler: /usr/local/cuda-11.0/bin/nvcc

  Build flags:

  Id flags: --keep;--keep-dir;tmp;-gencode=arch=compute_,code=sm_ -v

  

  The output was:

  1

  nvcc fatal : Unsupported gpu architecture 'compute_'

Then I tried -DCMAKE_CUDA_ARCHITECTURES=80, now I got the exactly same error as @sunyuhan19981208 :

[  2%] Building C object CMakeFiles/ggml.dir/ggml.c.o
[  5%] Building CUDA object CMakeFiles/ggml.dir/ggml-cuda.cu.o
nvcc fatal   : 'f16c': expected a number
make[2]: *** [CMakeFiles/ggml.dir/build.make:90: CMakeFiles/ggml.dir/ggml-cuda.cu.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:646: CMakeFiles/ggml.dir/all] Error 2
make: *** [Makefile:101: all] Error 2

Then I used -DLLAMA_F16C=OFF, this time I got another error:

[  2%] Building C object CMakeFiles/ggml.dir/ggml.c.o
[  5%] Building CUDA object CMakeFiles/ggml.dir/ggml-cuda.cu.o
nvcc fatal   : 'fma': expected a number
make[2]: *** [CMakeFiles/ggml.dir/build.make:90: CMakeFiles/ggml.dir/ggml-cuda.cu.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:646: CMakeFiles/ggml.dir/all] Error 2
make: *** [Makefile:101: all] Error 2

Then I add -DLLAMA_FMA=OFF, now:

[  2%] Building C object CMakeFiles/ggml.dir/ggml.c.o
In file included from /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/include/immintrin.h:97,
                 from /data/playground/ll2/llama.cpp/ggml.c:189:
/data/playground/ll2/llama.cpp/ggml.c: In function ‘ggml_vec_dot_q4_0_q8_0’:
/opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/include/fmaintrin.h:63:1: error: inlining failed in call to always_inline ‘_mm256_fmadd_ps’: target specific option mismatch
 _mm256_fmadd_ps (__m256 __A, __m256 __B, __m256 __C)
 ^~~~~~~~~~~~~~~
/data/playground/ll2/llama.cpp/ggml.c:2166:15: note: called from here
         acc = _mm256_fmadd_ps( d, q, acc );
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/include/immintrin.h:97,
                 from /data/playground/ll2/llama.cpp/ggml.c:189:
/opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/include/fmaintrin.h:63:1: error: inlining failed in call to always_inline ‘_mm256_fmadd_ps’: target specific option mismatch
 _mm256_fmadd_ps (__m256 __A, __m256 __B, __m256 __C)
 ^~~~~~~~~~~~~~~
/data/playground/ll2/llama.cpp/ggml.c:2166:15: note: called from here
         acc = _mm256_fmadd_ps( d, q, acc );
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/include/immintrin.h:97,
                 from /data/playground/ll2/llama.cpp/ggml.c:189:
/opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/include/fmaintrin.h:63:1: error: inlining failed in call to always_inline ‘_mm256_fmadd_ps’: target specific option mismatch
 _mm256_fmadd_ps (__m256 __A, __m256 __B, __m256 __C)
 ^~~~~~~~~~~~~~~
/data/playground/ll2/llama.cpp/ggml.c:2166:15: note: called from here
         acc = _mm256_fmadd_ps( d, q, acc );
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/include/immintrin.h:97,
                 from /data/playground/ll2/llama.cpp/ggml.c:189:
/opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/include/fmaintrin.h:63:1: error: inlining failed in call to always_inline ‘_mm256_fmadd_ps’: target specific option mismatch
 _mm256_fmadd_ps (__m256 __A, __m256 __B, __m256 __C)
 ^~~~~~~~~~~~~~~
/data/playground/ll2/llama.cpp/ggml.c:2166:15: note: called from here
         acc = _mm256_fmadd_ps( d, q, acc );
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [CMakeFiles/ggml.dir/build.make:76: CMakeFiles/ggml.dir/ggml.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:646: CMakeFiles/ggml.dir/all] Error 2
make: *** [Makefile:101: all] Error 2

Many thanks for your help!

SlyEcho · 2023-05-15T15:59:20Z

I think these kinds of errors come when the compiler doesn't know what CPU it is building for, if you want to use AVX and AVX2, you can try the option DLLAMA_NATIVE=ON, if that doesn't work, add the flag manually with -DCMAKE_C_FLAGS=-march=skylake -CMAKE_CXX_FLAGS=-march=skylake.

Take a look at what options your compiler has: gcc --target-help

kexul · 2023-05-15T16:36:58Z

I think these kinds of errors come when the compiler doesn't know what CPU it is building for, if you want to use AVX and AVX2, you can try the option DLLAMA_NATIVE=ON, if that doesn't work, add the flag manually with -DCMAKE_C_FLAGS=-march=skylake -CMAKE_CXX_FLAGS=-march=skylake.
Take a look at what options your compiler has: gcc --target-help

Thanks for your detailed information, but none of these worked here 😭
cmake .. -DLLAMA_CUBLAS=1 -DCMAKE_CUDA_ARCHITECTURES=80 -DLLAMA_F16C=OFF -DLLAMA_FMA=OFF -DLLAMA_NATIVE=ON produce:

[  2%] Building C object CMakeFiles/ggml.dir/ggml.c.o
[  5%] Building CUDA object CMakeFiles/ggml.dir/ggml-cuda.cu.o
nvcc fatal   : 'arch=native': expected a number
make[2]: *** [CMakeFiles/ggml.dir/build.make:90: CMakeFiles/ggml.dir/ggml-cuda.cu.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:646: CMakeFiles/ggml.dir/all] Error 2
make: *** [Makefile:101: all] Error 2

After adding -DCMAKE_C_FLAGS=-march=skylake -CMAKE_CXX_FLAGS=-march=skylake, It becomes: cmake .. -DLLAMA_CUBLAS=1 -DCMAKE_CUDA_ARCHITECTURES=80 -DLLAMA_F16C=OFF -DLLAMA_FMA=OFF -DCMAKE_C_FLAGS=-march=skylake -CMAKE_CXX_FLAGS=-march=skylake, now cmake failed.

Performing C SOURCE FILE Test CMAKE_HAVE_LIBC_PTHREAD failed with the following output:
Change Dir: /data/playground/ll2/llama.cpp/build/CMakeFiles/CMakeTmp

Run Build Command(s):/opt/rh/devtoolset-7/root/usr/bin/gmake -f Makefile cmTC_8a69a/fast && /opt/rh/devtoolset-7/root/usr/bin/gmake  -f CMakeFiles/cmTC_8a69a.dir/build.make CMakeFiles/cmTC_8a69a.dir/build
gmake[1]: Entering directory '/data/playground/ll2/llama.cpp/build/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_8a69a.dir/src.c.o
/opt/rh/devtoolset-8/root/usr/bin/cc -DCMAKE_HAVE_LIBC_PTHREAD  -std=gnu11 -o CMakeFiles/cmTC_8a69a.dir/src.c.o -c /data/playground/ll2/llama.cpp/build/CMakeFiles/CMakeTmp/src.c
Linking C executable cmTC_8a69a
/usr/local/bin/cmake -E cmake_link_script CMakeFiles/cmTC_8a69a.dir/link.txt --verbose=1
/opt/rh/devtoolset-8/root/usr/bin/cc CMakeFiles/cmTC_8a69a.dir/src.c.o -o cmTC_8a69a
CMakeFiles/cmTC_8a69a.dir/src.c.o: In function `main':
src.c:(.text+0x2d): undefined reference to `pthread_create'
src.c:(.text+0x39): undefined reference to `pthread_detach'
src.c:(.text+0x45): undefined reference to `pthread_cancel'
src.c:(.text+0x56): undefined reference to `pthread_join'
src.c:(.text+0x6a): undefined reference to `pthread_atfork'
collect2: error: ld returned 1 exit status
gmake[1]: *** [CMakeFiles/cmTC_8a69a.dir/build.make:99: cmTC_8a69a] Error 1
gmake[1]: Leaving directory '/data/playground/ll2/llama.cpp/build/CMakeFiles/CMakeTmp'
gmake: *** [Makefile:127: cmTC_8a69a/fast] Error 2

SlyEcho · 2023-05-15T17:20:39Z

Umm, -DCMAKE_EXE_LINKER_FLAGS=-lpthread?

You can also edit the CMakeCache.txt file in the build directory instead of using commands.

kexul · 2023-05-16T16:33:51Z

Umm, -DCMAKE_EXE_LINKER_FLAGS=-lpthread?

No luck, still lots of error:

Performing C SOURCE FILE Test CMAKE_HAVE_LIBC_PTHREAD failed with the following output:
Change Dir: /data/playground/ll2/llama.cpp/build/CMakeFiles/CMakeTmp

Run Build Command(s):/opt/rh/devtoolset-7/root/usr/bin/gmake -f Makefile cmTC_f7d55/fast && /opt/rh/devtoolset-7/root/usr/bin/gmake  -f CMakeFiles/cmTC_f7d55.dir/build.make CMakeFiles/cmTC_f7d55.dir/build
gmake[1]: Entering directory '/data/playground/ll2/llama.cpp/build/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_f7d55.dir/src.c.o
/opt/rh/devtoolset-8/root/usr/bin/cc -DCMAKE_HAVE_LIBC_PTHREAD  -march=skylake  -std=gnu11 -o CMakeFiles/cmTC_f7d55.dir/src.c.o -c /data/playground/ll2/llama.cpp/build/CMakeFiles/CMakeTmp/src.c
Linking C executable cmTC_f7d55
/usr/local/bin/cmake -E cmake_link_script CMakeFiles/cmTC_f7d55.dir/link.txt --verbose=1
/opt/rh/devtoolset-8/root/usr/bin/cc -march=skylake  -lpthread  CMakeFiles/cmTC_f7d55.dir/src.c.o -o cmTC_f7d55
CMakeFiles/cmTC_f7d55.dir/src.c.o: In function `main':
src.c:(.text+0x6a): undefined reference to `pthread_atfork'
collect2: error: ld returned 1 exit status
gmake[1]: *** [CMakeFiles/cmTC_f7d55.dir/build.make:99: cmTC_f7d55] Error 1
gmake[1]: Leaving directory '/data/playground/ll2/llama.cpp/build/CMakeFiles/CMakeTmp'
gmake: *** [Makefile:127: cmTC_f7d55/fast] Error 2

SlyEcho · 2023-05-16T20:03:53Z

This system seems to be very old. Could you use something newer?

Maybe add -pthread to C and CXX flags?

kexul · 2023-05-17T03:21:03Z

@SlyEcho , Thanks for your continued help! I just tried compile llama.cpp in a docker container with cuda11.8, it works now!

aditya624 · 2023-05-20T12:52:57Z

i got the same error. how to solve it. iam very confused.

kexul · 2023-05-20T12:56:53Z

After I change my docker from CUDA11.0 to CUDA11.1, nothing happen. So the lowest version that can run llama.cpp is CUDA11.1?

@aditya624 what's your version of cuda? try newer versino if it's 11.0.

aditya624 · 2023-05-20T12:59:51Z

i am using cuda 11.7. i think not because of cuda version. because i have another server with same cuda version running fine. but there is 1 server that has an error like this.

kexul · 2023-05-20T13:07:30Z

i am using cuda 11.7. i think not because of cuda version. because i have another server with same cuda version running fine. but there is 1 server that has an error like this.

Then I'm not sure what's going on... Maybe @SlyEcho got more insight 😉

SlyEcho · 2023-05-21T21:52:50Z

I really can't tell much from that info, @aditya624. I don't use CUDA much myself, but I managed to compile llama.cpp on CUDA 11.4 a couple times.

There are a lot of different things we tried here, maybe it's a start.

EagleChen · 2023-05-26T12:18:43Z

cmake .. -DLLAMA_CUBLAS=1 -DCMAKE_CUDA_ARCHITECTURES=80 -DLLAMA_F16C=OFF -DLLAMA_FMA=OFF -DCMAKE_C_FLAGS='-march=native -pthread' -DCMAKE_CXX_FLAGS='-march=native -pthread' -DLLAMA_AVX=OFF -DLLAMA_AVX2=OFF

the above command works for me.

cuda version V11.0.221

edp1096 · 2023-05-29T13:17:22Z

I'v met same issue on 1080ti / ubuntu 20.04 / nvidia driver 525 / cuda toolkit 12.0
In my case, this solved by driver and cuda toolkit upgrading. nvidia driver 530 / cuda toolkit 12.1

Tried just basic flags.

cmake .. -DLLAMA_CUBLAS=1 -DLLAMA_BUILD_EXAMPLES=1 -DLLAMA_BUILD_TESTS=0

kexul · 2023-06-05T11:55:44Z

cmake .. -DLLAMA_CUBLAS=1 -DCMAKE_CUDA_ARCHITECTURES=80 -DLLAMA_F16C=OFF -DLLAMA_FMA=OFF -DCMAKE_C_FLAGS='-march=native -pthread' -DCMAKE_CXX_FLAGS='-march=native -pthread' -DLLAMA_AVX=OFF -DLLAMA_AVX2=OFF

Awesome! that worked for me too! cuda version V11.0.194

ggerganov · 2023-07-05T15:59:25Z

For me, I had 2 nvcc compilers and it was using the "older" one by default.
I resolve the problem by specifying the path to the newer one:

cmake .. -DLLAMA_CUBLAS=1 -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.2/bin/nvcc

luav · 2023-07-11T13:10:12Z

@kexul You just missed several more flags that are required for legacy Intel CPUs:

Thanks for your detailed information, but none of these worked here sob cmake .. -DLLAMA_CUBLAS=1 -DCMAKE_CUDA_ARCHITECTURES=80 -DLLAMA_F16C=OFF -DLLAMA_FMA=OFF -DLLAMA_NATIVE=ON produce:
[  2%] Building C object CMakeFiles/ggml.dir/ggml.c.o
[  5%] Building CUDA object CMakeFiles/ggml.dir/ggml-cuda.cu.o
nvcc fatal   : 'arch=native': expected a number
make[2]: *** [CMakeFiles/ggml.dir/build.make:90: CMakeFiles/ggml.dir/ggml-cuda.cu.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:646: CMakeFiles/ggml.dir/all] Error 2
make: *** [Makefile:101: all] Error 2
After adding -DCMAKE_C_FLAGS=-march=skylake -CMAKE_CXX_FLAGS=-march=skylake, It becomes: cmake .. -DLLAMA_CUBLAS=1 -DCMAKE_CUDA_ARCHITECTURES=80 -DLLAMA_F16C=OFF -DLLAMA_FMA=OFF -DCMAKE_C_FLAGS=-march=skylake -CMAKE_CXX_FLAGS=-march=skylake, now cmake failed.

@EagleChen is right, and generalizing his build command for legacy Intel CPUs:

$ cmake -DCMAKE_CUDA_ARCHITECTURES="all" -DLLAMA_F16C=0 -DLLAMA_FMA=0 -DLLAMA_AVX=0 -DLLAMA_AVX2=0 -DCMAKE_C_FLAGS="-march=native" -DLLAMA_CUBLAS=1 ..

For modern AMD CPUs, it is sufficient to use the build command given by @ggerganov:

cmake -DCMAKE_CUDA_ARCHITECTURES="all" -DCMAKE_C_FLAGS="-march=native" -DLLAMA_CUBLAS=1 ..

However, in the latter case, the default -mf16c, -mfma, ... flags are supplied to nvcc instead of cc on some platforms:

...
[  5%] Building CUDA object CMakeFiles/ggml.dir/ggml-cuda.cu.o
nvcc fatal   : 'fma': expected a number
...

In that case, use the build command listed above:

$ cmake -DCMAKE_CUDA_ARCHITECTURES="all" -DLLAMA_F16C=0 -DLLAMA_FMA=0 -DLLAMA_AVX=0 -DLLAMA_AVX2=0 -DCMAKE_C_FLAGS="-march=native" -DLLAMA_CUBLAS=1 ..

dshwei · 2023-08-26T11:35:26Z

For me, I had 2 nvcc compilers and it was using the "older" one by default. I resolve the problem by specifying the path to the newer one:
cmake .. -DLLAMA_CUBLAS=1 -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.2/bin/nvcc

thanks ， my problem was resolved by this method.
my problem is nvcc fatal : 'f16c': expected a number when commands cmake .. -DLLAMA_CUBLAS=ON and cmake --build . --config Release were executed using cmake way

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 23 G /Xwayland N/A |
+---------------------------------------------------------------------------------------+

find /usr -name nvvc , it return result as following :
/usr/local/cuda-12.2/bin/nvcc
/usr/lib/nvidia-cuda-toolkit/bin/nvcc
/usr/bin/nvcc

xzitlou · 2023-09-14T12:05:04Z

Hello there, I'm still getting the same error in Linus:
This is the cmake version I'm already using

cmake --version
cmake version 3.27.4

I tried bellow code without success:

cd llm/llama.cpp/ggml
cmake -DLLAMA_CUBLAS=1 -DCMAKE_CUDA_COMPILER=/usr/lib/nvidia-cuda-toolkit/bin/nvcc

> nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

CUDA version

> sudo nvidia-smi -l 1
NVIDIA-SMI 525.125.06   Driver Version: 525.125.06   CUDA Version: 12.0

Any suggestion?

sunyuhan19981208 changed the title ~~[User] nvcc fatal : 'f16c': expected a number~~ nvcc fatal : 'f16c': expected a number May 15, 2023

sunyuhan19981208 closed this as completed May 17, 2023

marella mentioned this issue Jul 16, 2023

Failed to compile on linux ubuntu with gpu support marella/ctransformers#53

Closed

xzitlou mentioned this issue Sep 12, 2023

first pass at linux gpu support ollama/ollama#454

Merged

Weaxs mentioned this issue Dec 19, 2023

MacOS (x86_64) + Metal Framework error with ggml Weaxs/go-chatglm.cpp#4

Closed

Bearsaerker mentioned this issue Mar 12, 2025

Eval bug: Gemma 3 extremly slow prompt processing when using quantized kv cache. #12352

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nvcc fatal : 'f16c': expected a number #1467

nvcc fatal : 'f16c': expected a number #1467

sunyuhan19981208 commented May 15, 2023 •

edited

Loading

sunyuhan19981208 commented May 15, 2023

kexul commented May 15, 2023

SlyEcho commented May 15, 2023

kexul commented May 15, 2023

SlyEcho commented May 15, 2023

kexul commented May 15, 2023

SlyEcho commented May 15, 2023

kexul commented May 16, 2023

SlyEcho commented May 16, 2023

kexul commented May 17, 2023

aditya624 commented May 20, 2023

kexul commented May 20, 2023

aditya624 commented May 20, 2023

kexul commented May 20, 2023

SlyEcho commented May 21, 2023

EagleChen commented May 26, 2023

edp1096 commented May 29, 2023

kexul commented Jun 5, 2023

ggerganov commented Jul 5, 2023

luav commented Jul 11, 2023

dshwei commented Aug 26, 2023 •

edited

Loading

xzitlou commented Sep 14, 2023

nvcc fatal : 'f16c': expected a number #1467

nvcc fatal : 'f16c': expected a number #1467

Comments

sunyuhan19981208 commented May 15, 2023 • edited Loading

Prerequisites

Expected Behavior & Failure Information & Failure Logs

Environment and Context

Steps to Reproduce

sunyuhan19981208 commented May 15, 2023

kexul commented May 15, 2023

SlyEcho commented May 15, 2023

kexul commented May 15, 2023

SlyEcho commented May 15, 2023

kexul commented May 15, 2023

SlyEcho commented May 15, 2023

kexul commented May 16, 2023

SlyEcho commented May 16, 2023

kexul commented May 17, 2023

aditya624 commented May 20, 2023

kexul commented May 20, 2023

aditya624 commented May 20, 2023

kexul commented May 20, 2023

SlyEcho commented May 21, 2023

EagleChen commented May 26, 2023

edp1096 commented May 29, 2023

kexul commented Jun 5, 2023

ggerganov commented Jul 5, 2023

luav commented Jul 11, 2023

dshwei commented Aug 26, 2023 • edited Loading

xzitlou commented Sep 14, 2023

sunyuhan19981208 commented May 15, 2023 •

edited

Loading

dshwei commented Aug 26, 2023 •

edited

Loading