My attempt and progress so far at building local-ai natively on windows

[dionysius]

I'm no C++ dev and just liked to see how far I can get. There is no guarantee that I even use the tools correctly or configure stuff correctly. There is no need for Visual Studio and might even be better to not have it and/or the visual build tools installed as it is annoyingly interfering in some circumstances.

As a first attempt there are no gpu accelerators set, so trying to build it as minimally as possible.

---

Prereqs:

This helps to describe the base line of the environment and operation. There are definitely alternative ways. My current package manager is scoop, but this works definitely also with chocolatey packages and other package managers etc. - choose your favourite. Git is expected to be already installed.

scoop install gcc nasm make cmake sed zlib

and follow printed post-installation commands.

Open fresh git bash so we have all the new envs.

---

Preparation:

Get a clean checkout of your wished repo state and make a new branch, e.g. git checkout -b windows_build master; git reset --hard; make clean; git clean -fdx
Create the following files:

windows_env

# define your compiler explicitly
# https://github.com/grpc/grpc/issues/18657#issuecomment-567830909 we need to use MSVC?
export CMAKE_C_COMPILER=$(which gcc)
export CMAKE_CXX_COMPILER=$(which g++)
export CMAKE_ASM_NASM_COMPILER=$(which nasm)
# so far none of below worked for me to get .o files instead of .obj files, so the patch is needed
export CMAKE_C_OUTPUT_EXTENSION=.o
export CMAKE_CXX_OUTPUT_EXTENSION=.o
export CMAKE_ASM_OUTPUT_EXTENSION=.o
export CMAKE_C_OUTPUT_EXTENSION_REPLACE=1
export CMAKE_CXX_OUTPUT_EXTENSION_REPLACE=1
export CMAKE_ASM_OUTPUT_EXTENSION_REPLACE=1
export OBJECTS_SUFFIXES=.o
export OBJECTS=.o
export OBJCFLAGS="-c -o"
export OBJCXXFLAGS="-c -o"
# various settings for cmake and vcpkg
# there may be too many settings as actually needed, this list grew during trial and error
# export CMAKE_SYSTEM_NAME=LINUX
export CMAKE_GENERATOR="Unix Makefiles"
# export CMAKE_PLATFORM=x64
export PATH="${PATH}:$(realpath .)/backend/cpp/grpc/installed_packages/bin"
# https://github.com/abseil/abseil-cpp/issues/1510
# export ABSL_FORCE_WAITER_MODE=4
#export CMAKE_CXX_FLAGS="-DABSL_FORCE_WAITER_MODE=4"
# export CGO_LDFLAGS=-Wl,--no-as-needed #maybe not needed
export CC=$(which gcc)

windows_patch

diff --git a/sources/go-bert/Makefile b/sources/go-bert/Makefile
index 8e374a3f..b8512596 100644
--- a/sources/go-bert/Makefile
+++ b/sources/go-bert/Makefile
@@ -140,7 +140,7 @@ $(info )
 
 
 ggml.o: bert.o
-	cd bert.cpp/build && make VERBOSE=1 ggml && cp -rf ggml/src/CMakeFiles/ggml.dir/ggml.c.o ../../ggml.o
+	cd bert.cpp/build && make VERBOSE=1 ggml && cp -rf ggml/src/CMakeFiles/ggml.dir/ggml.c.obj ../../ggml.o
 
 bert.cpp/build:
 	cd bert.cpp && mkdir build
@@ -148,7 +148,7 @@ bert.cpp/build:
 bert.o: bert.cpp/build
 	sed "s/#include <regex>/#include <regex>\n#include <unordered_map>/" bert.cpp/bert.cpp > bert.cpp/bert.tmp && mv bert.cpp/bert.tmp bert.cpp/bert.cpp
 	cd bert.cpp/build && cmake .. -DBUILD_SHARED_LIBS=OFF -DCMAKE_BUILD_TYPE=Release && make
-	cp bert.cpp/build/CMakeFiles/bert.dir/bert.cpp.o bert.o
+	cp bert.cpp/build/CMakeFiles/bert.dir/bert.cpp.obj bert.o
 
 clean:
 	rm -f *.o
diff --git a/sources/go-ggml-transformers/Makefile b/sources/go-ggml-transformers/Makefile
index 9cffc3e6..ff061580 100644
--- a/sources/go-ggml-transformers/Makefile
+++ b/sources/go-ggml-transformers/Makefile
@@ -147,9 +147,9 @@ $(info )
 
 ggml.o:
 	mkdir build
-	cd build && cmake ../ggml.cpp $(CMAKE_ARGS) && make VERBOSE=1 ggml && cp -rf src/CMakeFiles/ggml.dir/ggml.c.o ../ggml.o
+	cd build && cmake ../ggml.cpp $(CMAKE_ARGS) && make VERBOSE=1 ggml && cp -rf src/CMakeFiles/ggml.dir/ggml.c.obj ../ggml.o
 	@if [ "$(BUILD_TYPE)" = "cublas" ]; then \
-		cp -rf build/src/CMakeFiles/ggml.dir/ggml-cuda.cu.o ggml-cuda.o ;\
+		cp -rf build/src/CMakeFiles/ggml.dir/ggml-cuda.cu.obj ggml-cuda.o ;\
 	fi
 
 generic-ggml.o:
diff --git a/sources/go-llama/Makefile b/sources/go-llama/Makefile
index 7dc8ffa7..a5543e12 100644
--- a/sources/go-llama/Makefile
+++ b/sources/go-llama/Makefile
@@ -200,32 +200,32 @@ $(info )
 # Use this if you want to set the default behavior
 
 llama.cpp/grammar-parser.o:
-	cd build && cp -rf common/CMakeFiles/common.dir/grammar-parser.cpp.o ../llama.cpp/grammar-parser.o
+	cd build && cp -rf common/CMakeFiles/common.dir/grammar-parser.cpp.obj ../llama.cpp/grammar-parser.o
 
 llama.cpp/ggml-alloc.o:
-	cd build && cp -rf CMakeFiles/ggml.dir/ggml-alloc.c.o ../llama.cpp/ggml-alloc.o
+	cd build && cp -rf CMakeFiles/ggml.dir/ggml-alloc.c.obj ../llama.cpp/ggml-alloc.o
 
 llama.cpp/ggml.o: prepare
 	mkdir -p build
-	cd build && CC="$(CC)" CXX="$(CXX)" cmake ../llama.cpp $(CMAKE_ARGS) && VERBOSE=1 cmake --build . --config Release && cp -rf CMakeFiles/ggml.dir/ggml.c.o ../llama.cpp/ggml.o
+	cd build && cmake ../llama.cpp $(CMAKE_ARGS) && VERBOSE=1 cmake --build . --config Release && cp -rf CMakeFiles/ggml.dir/ggml.c.obj ../llama.cpp/ggml.o
 
 llama.cpp/ggml-cuda.o: llama.cpp/ggml.o
 	cd build && cp -rf "$(GGML_CUDA_OBJ_PATH)" ../llama.cpp/ggml-cuda.o
 
 llama.cpp/ggml-opencl.o: llama.cpp/ggml.o
-	cd build && cp -rf CMakeFiles/ggml.dir/ggml-opencl.cpp.o ../llama.cpp/ggml-opencl.o
+	cd build && cp -rf CMakeFiles/ggml.dir/ggml-opencl.cpp.obj ../llama.cpp/ggml-opencl.o
 
 llama.cpp/ggml-metal.o: llama.cpp/ggml.o
-	cd build && cp -rf CMakeFiles/ggml.dir/ggml-metal.m.o ../llama.cpp/ggml-metal.o
+	cd build && cp -rf CMakeFiles/ggml.dir/ggml-metal.m.obj ../llama.cpp/ggml-metal.o
 
 llama.cpp/k_quants.o: llama.cpp/ggml.o
-	cd build && cp -rf CMakeFiles/ggml.dir/k_quants.c.o ../llama.cpp/k_quants.o
+	cd build && cp -rf CMakeFiles/ggml.dir/k_quants.c.obj ../llama.cpp/k_quants.o
 
 llama.cpp/llama.o:
-	cd build && cp -rf CMakeFiles/llama.dir/llama.cpp.o ../llama.cpp/llama.o
+	cd build && cp -rf CMakeFiles/llama.dir/llama.cpp.obj ../llama.cpp/llama.o
 
 llama.cpp/common.o:
-	cd build && cp -rf common/CMakeFiles/common.dir/common.cpp.o ../llama.cpp/common.o
+	cd build && cp -rf common/CMakeFiles/common.dir/common.cpp.obj ../llama.cpp/common.o
 
 binding.o: prepare llama.cpp/ggml.o llama.cpp/llama.o llama.cpp/common.o llama.cpp/grammar-parser.o llama.cpp/ggml-alloc.o
 	$(CXX) $(CXXFLAGS) -I./llama.cpp -I./llama.cpp/common binding.cpp -o binding.o -c $(LDFLAGS)

diff --git a/sources/go-llama-ggml/Makefile b/sources/go-llama-ggml/Makefile
index 1d703877..e3e7f2ae 100644
--- a/sources/go-llama-ggml/Makefile
+++ b/sources/go-llama-ggml/Makefile
@@ -176,32 +176,32 @@ $(info )
 # Use this if you want to set the default behavior

 llama.cpp/grammar-parser.o:
-       cd build && cp -rf examples/CMakeFiles/common.dir/grammar-parser.cpp.o ../llama.cpp/grammar-parser.o
+       cd build && cp -rf examples/CMakeFiles/common.dir/grammar-parser.cpp.obj ../llama.cpp/grammar-parser.o

 llama.cpp/ggml-alloc.o:
-       cd build && cp -rf CMakeFiles/ggml.dir/ggml-alloc.c.o ../llama.cpp/ggml-alloc.o
+       cd build && cp -rf CMakeFiles/ggml.dir/ggml-alloc.c.obj ../llama.cpp/ggml-alloc.o

 llama.cpp/ggml.o: prepare
        mkdir -p build
-       cd build && cmake ../llama.cpp $(CMAKE_ARGS) && VERBOSE=1 cmake --build . --config Release && cp -rf CMakeFiles/ggml.dir/ggml.c.o ../llama.cpp/ggml.o
+       cd build && cmake ../llama.cpp $(CMAKE_ARGS) && VERBOSE=1 cmake --build . --config Release && cp -rf CMakeFiles/ggml.dir/ggml.c.obj ../llama.cpp/ggml.o

 llama.cpp/ggml-cuda.o: llama.cpp/ggml.o
-       cd build && cp -rf CMakeFiles/ggml.dir/ggml-cuda.cu.o ../llama.cpp/ggml-cuda.o
+       cd build && cp -rf CMakeFiles/ggml.dir/ggml-cuda.cu.obj ../llama.cpp/ggml-cuda.o

 llama.cpp/ggml-opencl.o: llama.cpp/ggml.o
-       cd build && cp -rf CMakeFiles/ggml.dir/ggml-opencl.cpp.o ../llama.cpp/ggml-opencl.o
+       cd build && cp -rf CMakeFiles/ggml.dir/ggml-opencl.cpp.obj ../llama.cpp/ggml-opencl.o

 llama.cpp/ggml-metal.o: llama.cpp/ggml.o
-       cd build && cp -rf CMakeFiles/ggml.dir/ggml-metal.m.o ../llama.cpp/ggml-metal.o
+       cd build && cp -rf CMakeFiles/ggml.dir/ggml-metal.m.obj ../llama.cpp/ggml-metal.o

 llama.cpp/k_quants.o: llama.cpp/ggml.o
-       cd build && cp -rf CMakeFiles/ggml.dir/k_quants.c.o ../llama.cpp/k_quants.o
+       cd build && cp -rf CMakeFiles/ggml.dir/k_quants.c.obj ../llama.cpp/k_quants.o

 llama.cpp/llama.o:
-       cd build && cp -rf CMakeFiles/llama.dir/llama.cpp.o ../llama.cpp/llama.o
+       cd build && cp -rf CMakeFiles/llama.dir/llama.cpp.obj ../llama.cpp/llama.o

 llama.cpp/common.o:
-       cd build && cp -rf examples/CMakeFiles/common.dir/common.cpp.o ../llama.cpp/common.o
+       cd build && cp -rf examples/CMakeFiles/common.dir/common.cpp.obj ../llama.cpp/common.o

 binding.o: prepare llama.cpp/ggml.o llama.cpp/llama.o llama.cpp/common.o llama.cpp/grammar-parser.o llama.cpp/ggml-alloc.o
        $(CXX) $(CXXFLAGS) -I./llama.cpp -I./llama.cpp/examples binding.cpp -o binding.o -c $(LDFLAGS)
diff --git a/sources/gpt4all/gpt4all-bindings/golang/Makefile b/sources/gpt4all/gpt4all-bindings/golang/Makefile
index 9d230008..a3e88f08 100644
--- a/sources/gpt4all/gpt4all-bindings/golang/Makefile
+++ b/sources/gpt4all/gpt4all-bindings/golang/Makefile
@@ -141,8 +141,8 @@ $(info )
 llmodel.o:
        mkdir buildllm
        cd buildllm && cmake ../../../gpt4all-backend/ $(CMAKEFLAGS) && make
-       cd buildllm && cp -rf CMakeFiles/llmodel.dir/llmodel_c.cpp.o ../llmodel_c.o
-       cd buildllm && cp -rf CMakeFiles/llmodel.dir/llmodel.cpp.o ../llmodel.o
+       cd buildllm && cp -rf CMakeFiles/llmodel.dir/llmodel_c.cpp.obj ../llmodel_c.o
+       cd buildllm && cp -rf CMakeFiles/llmodel.dir/llmodel.cpp.obj ../llmodel.o

 clean:
        rm -f *.o

diff --git a/backend/cpp/grpc/script/build_grpc.sh b/backend/cpp/grpc/script/build_grpc.sh
index 98ece5b1..cb0d9274 100755
--- a/backend/cpp/grpc/script/build_grpc.sh
+++ b/backend/cpp/grpc/script/build_grpc.sh
@@ -1,5 +1,7 @@
 #!/bin/bash

+set -euo pipefail
+
 # Builds locally from sources the packages needed by the llama cpp backend.

 # Makes sure a few base packages exist.
@@ -15,10 +17,10 @@ if [ -z ${CPP_INSTALLED_PACKAGES_DIR} ]; then
     exit -1
 fi

-if [ -d "${CPP_INSTALLED_PACKAGES_DIR}" ]; then
-  echo "gRPC installation directory already exists. Nothing to do."
-  exit 0
-fi
+# if [ -d "${CPP_INSTALLED_PACKAGES_DIR}" ]; then
+#   echo "gRPC installation directory already exists. Nothing to do."
+#   exit 0
+# fi

 # The depth when cloning a git repo. 1 speeds up the clone when the repo history is not needed.
 GIT_CLONE_DEPTH=1
@@ -38,19 +40,19 @@ echo "GRPC_BUILD_DIR: ${GRPC_BUILD_DIR}"

 mkdir -pv ${GRPC_REPO_DIR}

-rm   -rf ${GRPC_BUILD_DIR}
+# rm -rf ${GRPC_BUILD_DIR}
 mkdir -pv ${GRPC_BUILD_DIR}

 mkdir -pv ${CPP_INSTALLED_PACKAGES_DIR}
-
+
 if [ -d "${SRC_DIR_LIB_GRPC}" ]; then
   echo "gRPC source already exists locally. Not cloned again."
-else
+else
   ( cd ${GRPC_REPO_DIR} && \
     git clone --depth ${GIT_CLONE_DEPTH} -b ${TAG_LIB_GRPC} ${GIT_REPO_LIB_GRPC} && \
     cd ${SRC_DIR_LIB_GRPC} && \
     git submodule update --init --recursive --depth ${GIT_CLONE_DEPTH}
-  )
+  )
 fi

 ( cd ${GRPC_BUILD_DIR} && \
@@ -70,12 +72,12 @@ fi
      -DgRPC_BUILD_GRPC_RUBY_PLUGIN=OFF \
      -Dprotobuf_WITH_ZLIB=ON \
      -DRE2_BUILD_TESTING=OFF \
+     -DOPENSSL_NO_ASM=ON \
      -DCMAKE_INSTALL_PREFIX=${CPP_INSTALLED_PACKAGES_DIR}/ \
      ${SRC_DIR_LIB_GRPC}  && \
   cmake --build .  -- -j ${NUM_BUILD_THREADS} && \
   cmake --build .  --target install -- -j ${NUM_BUILD_THREADS}
 )

-rm -rf ${GRPC_BUILD_DIR}
-rm -rf ${GRPC_REPO_DIR}
-
+# rm -rf ${GRPC_BUILD_DIR}
+# rm -rf ${GRPC_REPO_DIR}
diff --git a/backend/cpp/llama/Makefile b/backend/cpp/llama/Makefile
index a64ee1b4..bc60b6e0 100644
--- a/backend/cpp/llama/Makefile
+++ b/backend/cpp/llama/Makefile
@@ -28,9 +28,9 @@ llama.cpp:

 llama.cpp/examples/grpc-server:
        mkdir -p llama.cpp/examples/grpc-server
-       cp -r $(abspath ./)/CMakeLists.txt llama.cpp/examples/grpc-server/
-       cp -r $(abspath ./)/grpc-server.cpp llama.cpp/examples/grpc-server/
-       cp -rfv $(abspath ./)/json.hpp llama.cpp/examples/grpc-server/
+       cp -r $(realpath ./)/CMakeLists.txt llama.cpp/examples/grpc-server/
+       cp -r $(realpath ./)/grpc-server.cpp llama.cpp/examples/grpc-server/
+       cp -rfv $(realpath ./)/json.hpp llama.cpp/examples/grpc-server/
        echo "add_subdirectory(grpc-server)" >> llama.cpp/examples/CMakeLists.txt
 ## XXX: In some versions of CMake clip wasn't being built before llama.
 ## This is an hack for now, but it should be fixed in the future.
@@ -38,9 +38,9 @@ llama.cpp/examples/grpc-server:
        cp -rfv llama.cpp/examples/llava/clip.cpp llama.cpp/examples/grpc-server/clip.cpp

 rebuild:
-       cp -rfv $(abspath ./)/CMakeLists.txt llama.cpp/examples/grpc-server/
-       cp -rfv $(abspath ./)/grpc-server.cpp llama.cpp/examples/grpc-server/
-       cp -rfv $(abspath ./)/json.hpp llama.cpp/examples/grpc-server/
+       cp -rfv $(realpath ./)/CMakeLists.txt llama.cpp/examples/grpc-server/
+       cp -rfv $(realpath ./)/grpc-server.cpp llama.cpp/examples/grpc-server/
+       cp -rfv $(realpath ./)/json.hpp llama.cpp/examples/grpc-server/
        rm -rf grpc-server
        $(MAKE) grpc-server

• fixes issues with pwd in toplevel Makefile fixed since merge minor: replace shell pwd in Makefile with CURDIR for better windows compatibility #1571
• the env I used so far (file: windows_env)
• a secondary patch after submodules are checked out (file: windows_patch)

source windows_env so you have my current used env for cmake and vcpkg - There should be no error! Check with printenv that everything looks fine for CMAKE env variables and we have access to the required binaries.

---

First steps:

First we only checkout the submodules because we need to patch files there a well. We can hopefully skip this in the future.
make get-sources

Hacking the .git subfolders away so we can apply the next patch after:
rm -rf sources/*/.git sources/*/*/.git

Apply the patch for the sources (So far I couldn't figure out to tell gcc to make .o files instead of .obj)
git apply windows_patch

---

Building:

We can finally start building. We need to build grpc completely again so we are forced to use:
make BUILD_GRPC_FOR_BACKEND_LLAMA=true build

[dionysius]

@mudler I have a windows build! Not the best but at least a build.

C:\Users\Dionysius\Projects\github.com\mudler\LocalAI (windows_build)
λ local-ai.exe --version
LocalAI version v2.5.1-4-g900253d2 (900253d272a2fcb9330d68a9248fa0de4b102a55)

C:\Users\Dionysius\Projects\github.com\mudler\LocalAI (windows_build)
λ file local-ai.exe
local-ai: PE32+ executable (console) x86-64, for MS Windows, 15 sections

C:\Users\Dionysius\Projects\github.com\mudler\LocalAI (windows_build)
λ local-ai.exe          
11:00PM DBG no galleries to load
11:00PM INF Starting LocalAI using 4 threads, with models path: C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models
11:00PM INF LocalAI version: v2.5.1-4-g900253d2 (900253d272a2fcb9330d68a9248fa0de4b102a55)
11:00PM INF Preloading models from C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models

 ┌───────────────────────────────────────────────────┐ 
 │                   Fiber v2.50.0                   │
 │               http://127.0.0.1:8080               │
 │       (bound on host 0.0.0.0 and port 8080)       │
 │                                                   │
 │ Handlers ............ 73  Processes ........... 1 │
 │ Prefork ....... Disabled  PID ............. 16136 │
 └───────────────────────────────────────────────────┘ 

Now figuring out what works and what not

C:\Users\Dionysius\Projects\github.com\mudler\LocalAI (windows_build)
λ local-ai.exe phi-2
11:07PM DBG no galleries to load
11:07PM INF Starting LocalAI using 4 threads, with models path: C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models
11:07PM INF LocalAI version: v2.5.1-4-g900253d2 (900253d272a2fcb9330d68a9248fa0de4b102a55)
11:07PM INF Downloading "https://raw.githubusercontent.com/mudler/LocalAI/master/examples/configurations/phi-2.yaml"
11:07PM ERR error loading model: failed to rename temporary file C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models\1701d57f28d47552516c2b6ecc3cc719.yaml.partial -> C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models\1701d57f28d47552516c2b6ecc3cc719.yaml: rename C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models\1701d57f28d47552516c2b6ecc3cc719.yaml.partial C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models\1701d57f28d47552516c2b6ecc3cc719.yaml: The process cannot access the file because it is being used by another process.
11:07PM INF Preloading models from C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models
11:07PM INF Downloading "https://huggingface.co/TheBloke/phi-2-GGUF/resolve/main/phi-2.Q8_0.gguf"
11:07PM INF Downloading C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models\c0c3c83d0ec33ffe925657a56b06771b.partial: 313.2 MiB/2.8 GiB (11.10%) ETA: 40.037470154s
11:08PM INF Downloading C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models\c0c3c83d0ec33ffe925657a56b06771b.partial: 702.1 MiB/2.8 GiB (24.89%) ETA: 30.181442906s
11:08PM INF Downloading C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models\c0c3c83d0ec33ffe925657a56b06771b.partial: 1.1 GiB/2.8 GiB (38.38%) ETA: 24.087225277s
11:08PM INF Downloading C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models\c0c3c83d0ec33ffe925657a56b06771b.partial: 1.4 GiB/2.8 GiB (50.83%) ETA: 19.346209405s
11:08PM INF Downloading C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models\c0c3c83d0ec33ffe925657a56b06771b.partial: 1.8 GiB/2.8 GiB (64.38%) ETA: 13.833141664s
11:08PM INF Downloading C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models\c0c3c83d0ec33ffe925657a56b06771b.partial: 2.2 GiB/2.8 GiB (78.52%) ETA: 8.207628437s
11:08PM INF Downloading C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models\c0c3c83d0ec33ffe925657a56b06771b.partial: 2.5 GiB/2.8 GiB (91.98%) ETA: 3.052312911s
11:08PM ERR error downloading models: failed to rename temporary file C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models\c0c3c83d0ec33ffe925657a56b06771b.partial -> C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models\c0c3c83d0ec33ffe925657a56b06771b: rename C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models\c0c3c83d0ec33ffe925657a56b06771b.partial C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models\c0c3c83d0ec33ffe925657a56b06771b: The process cannot access the file because it is being used by another process.

fixing models manually...

C:\Users\Dionysius\Projects\github.com\mudler\LocalAI (windows_build)
λ ls -lah models\
total 2.8G
drwxr-xr-x 1 Dionysius 197121    0 Jan 10 23:17 ./
drwxr-xr-x 1 Dionysius 197121    0 Jan 10 22:58 ../
-rw-r--r-- 1 Dionysius 197121    0 Dec  4 21:30 .keep
-rw-r--r-- 1 Dionysius 197121 2.8G Jan 10 23:14 phi-2.Q4_K_M.gguf
-rw-r--r-- 1 Dionysius 197121  210 Jan 10 23:18 phi-test.yaml

C:\Users\Dionysius\Projects\github.com\mudler\LocalAI (windows_build)
λ local-ai.exe   
11:20PM DBG no galleries to load
11:20PM INF Starting LocalAI using 4 threads, with models path: C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models
11:20PM INF LocalAI version: v2.5.1-4-g900253d2 (900253d272a2fcb9330d68a9248fa0de4b102a55)
11:20PM INF Preloading models from C:\Users\Dionysius\Projects\github.com\mudler\LocalAI\models
11:20PM INF Model name: phi-test

 ┌───────────────────────────────────────────────────┐ 
 │                   Fiber v2.50.0                   │
 │               http://127.0.0.1:8080               │
 │       (bound on host 0.0.0.0 and port 8080)       │
 │                                                   │
 │ Handlers ............ 73  Processes ........... 1 │
 │ Prefork ....... Disabled  PID ............. 17064 │
 └───────────────────────────────────────────────────┘ 

11:20PM INF Loading model 'phi-2.Q4_K_M.gguf' with backend llama-cpp

$ curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
          "model": "phi-test",
          "messages": [{"role": "user", "content": "How are you doing?", "temperature": 0.1}]
      }'
{"error":{"code":500,"message":"grpc process not found: \\tmp\\localai\\backend_data\\backend-assets\\grpc\\llama-cpp. some backends(stablediffus
ion, tts) require LocalAI compiled with GO_TAGS","type":""}}

Dionysius@pc-dionysius MINGW64 ~/Projects/github.com/mudler/LocalAI (windows_build)
$ ls -lah backend/cpp/llama/
total 23M
drwxr-xr-x 1 Dionysius 197121    0 Jan 10 21:48 ./
drwxr-xr-x 1 Dionysius 197121    0 Dec  4 21:30 ../
-rw-r--r-- 1 Dionysius 197121 3.0K Jan  5 17:00 CMakeLists.txt
-rw-r--r-- 1 Dionysius 197121 2.2K Jan 10 18:24 Makefile
-rw-r--r-- 1 Dionysius 197121  97K Jan  5 17:00 grpc-server.cpp
-rwxr-xr-x 1 Dionysius 197121  22M Jan 10 22:56 grpc-server.exe*
-rw-r--r-- 1 Dionysius 197121 911K Dec  4 21:30 json.hpp
drwxr-xr-x 1 Dionysius 197121    0 Jan 10 18:32 llama.cpp/

Dionysius@pc-dionysius MINGW64 ~/Projects/github.com/mudler/LocalAI (windows_build)
$ file backend/cpp/llama/grpc-server.exe
backend/cpp/llama/grpc-server.exe: PE32+ executable (console) x86-64, for MS Windows, 19 sections

Dionysius@pc-dionysius MINGW64 ~/Projects/github.com/mudler/LocalAI (windows_build)
$ backend/cpp/llama/grpc-server.exe
Segmentation fault

Yeah, not the best result :)

[dionysius]

Last update here, from now on work on the build will result in PRs or issues. All the current changes (breaks unixoide systems currently): dionysius/LocalAI@master...windows_build

1. scoop install gcc nasm make cmake sed zlib (or similiar for your build environment)

• follow printed post-installation notes!

2. one of the following:

• cherry pick the differences dionysius/LocalAI@master...windows_build on top of your wished branch
• or just check out my windows_build branch https://github.com/dionysius/LocalAI/tree/windows_build

3. Make sure that your git and make state is really clean (make clean, git clean -fdx)
4. start fresh git bash
5. source windows_env
6. make BUILD_GRPC_FOR_BACKEND_LLAMA=true build

should build through and you'll have a local-ai.exe in the current folder.