Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restructuring, getting rid of more memory leaks, and improving readability #70

Open
wants to merge 66 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
5e59947
Adding back SqlStatement
polterguy Jun 26, 2023
6e2e09a
Removing unused code path
polterguy Jun 26, 2023
f513a1d
Making sure we use nullptr instead of 0
polterguy Jun 26, 2023
3f9d111
Update sqlite-vss.cpp
polterguy Jun 26, 2023
f83febe
Update sqlite-vss.cpp
polterguy Jun 26, 2023
98420ad
Encapsulating members
polterguy Jun 26, 2023
53f6206
Moving SqlStatement into separate header file
polterguy Jun 26, 2023
e5f412d
Better structure
polterguy Jun 26, 2023
8590e0b
Moving more stuff into separate files
polterguy Jun 26, 2023
d42db30
Restructuring by encapsulating members
polterguy Jun 26, 2023
765746a
Using nullptr + code formatting
polterguy Jun 26, 2023
600f67c
Comment
polterguy Jun 26, 2023
a33b38c
Using nullptr to be more semantic correct
polterguy Jun 26, 2023
87b058b
Better encapsulation and cohesion
polterguy Jun 26, 2023
d9d1c41
Better cohesion + encapsulation
polterguy Jun 26, 2023
89d3138
Comments
polterguy Jun 26, 2023
1eeda54
Bug fix
polterguy Jun 26, 2023
7ce8e0f
Trying to save build
polterguy Jun 26, 2023
7253c9f
Puuh, logical error fixed
polterguy Jun 26, 2023
fe77733
Optimising and trying to get rid of build bug
polterguy Jun 26, 2023
feace32
Update sqlite-vss.cpp
polterguy Jun 26, 2023
bcef4f3
Fixing bug
polterguy Jun 26, 2023
ca7763b
Revert "Fixing bug"
polterguy Jun 26, 2023
0d2b514
bug fix
polterguy Jun 26, 2023
8b1c9b0
Minor cleanups
polterguy Jun 26, 2023
3f0551e
Better encapsulation and cohesion
thomas-hansen-resolve Jun 26, 2023
1ff2619
More encapsulation
thomas-hansen-resolve Jun 26, 2023
dd6cfd1
Bumping to C++ 17 - Required to have shared_mutex
thomas-hansen-resolve Jun 26, 2023
543384c
Further encapsulating getting ready for mutex
thomas-hansen-resolve Jun 26, 2023
4b540be
Adding locks in crucial methods
thomas-hansen-resolve Jun 26, 2023
4d98e55
Creating factory constructor getting ready to cache faiss Index *
polterguy Jun 26, 2023
ebcc291
Moving more stuff around to get ready for caching of index
polterguy Jun 26, 2023
cbe056f
Oops, bug fix
polterguy Jun 26, 2023
4525660
Improving structure further to get ready for caching
polterguy Jun 26, 2023
d71031c
Re-structuring
polterguy Jun 26, 2023
1347944
Update vss-index.h
polterguy Jun 26, 2023
cbf8546
Locking access to factory methods
polterguy Jun 26, 2023
f773d25
Caching faiss indexes sharing between multiple connections
polterguy Jun 26, 2023
b6c1aa4
Resetting index on close
polterguy Jun 26, 2023
97dc2c7
Minor bugfix
polterguy Jun 26, 2023
247db3c
Formatting
polterguy Jun 26, 2023
aadc50b
Triggering build
polterguy Jun 26, 2023
1c3f9ff
Cleanups
polterguy Jun 27, 2023
1c73d9c
Started moving vector parts into separate files
polterguy Jun 27, 2023
cf0a7db
Cleanups
polterguy Jun 27, 2023
202367a
Minor changes
polterguy Jun 27, 2023
b845616
Removing caching - Doesn't work
polterguy Jun 27, 2023
8501219
Revert "Triggering build"
polterguy Jun 27, 2023
a42d4e3
Removing limit requirement
polterguy Sep 4, 2023
400d4d5
Update release.yaml
polterguy Sep 4, 2023
62e1363
Update release.yaml
polterguy Sep 4, 2023
86161da
Update sqlite-vss.cpp
polterguy Sep 4, 2023
8180c32
Update release.yaml
polterguy Mar 8, 2024
c8f5d23
Update release.yaml
polterguy Mar 8, 2024
e2aef0e
Update release.yaml
polterguy Mar 8, 2024
7d37828
Update release.yaml
polterguy Mar 8, 2024
5831d61
Update release.yaml
polterguy Mar 8, 2024
313210b
Update release.yaml
polterguy Mar 8, 2024
6eb557b
Update release.yaml
polterguy Mar 8, 2024
60482aa
Update release.yaml
polterguy Mar 8, 2024
93398a9
Update release.yaml
polterguy Mar 8, 2024
197a485
Update release.yaml
polterguy Mar 8, 2024
a787053
Update release.yaml
polterguy Mar 8, 2024
ecdc4e4
Update release.yaml
polterguy Mar 8, 2024
c867250
Update release.yaml
polterguy Mar 9, 2024
aafe8ae
Update release.yaml
polterguy Mar 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
305 changes: 35 additions & 270 deletions .github/workflows/release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,11 @@ permissions:
env:
ARTIFACT-LINUX-X86_64-EXTENSION: sqlite-vss-linux-x86_64
ARTIFACT-MACOS-X86_64-EXTENSION: sqlite-vss-macos-x86_64
ARTIFACT-MACOS-AARCH64-EXTENSION: sqlite-vss-macos-aarch64
ARTIFACT-WINDOWS-X86_64-EXTENSION: sqlite-vss-windows-x86_64
ARTIFACT-LINUX-X86_64-WHEELS: sqlite-vss-linux-x86_64-wheels
ARTIFACT-MACOS-X86_64-WHEELS: sqlite-vss-macos-x86_64-wheels
ARTIFACT-MACOS-AARCH64-WHEELS: sqlite-vss-macos-aarch64-wheels
ARTIFACT-MACOS-AARCH64-EXTENSION: sqlite-vss-macos-aarch64
jobs:
build-linux-x86_64-extension:
runs-on: ubuntu-20.04
build-macos-aarch64-extension:
runs-on: flyci-macos-large-latest-m1
steps:
- uses: actions/checkout@v3
with:
Expand All @@ -30,33 +27,26 @@ jobs:
- if: steps.cache-sqlite-build.outputs.cache-hit != 'true'
working-directory: vendor/sqlite
run: ./configure && make

# TODO how cache this?
- run: sudo apt-get install -y cmake libgomp1
- run: brew install llvm
- id: cache-cmake-build
uses: actions/cache@v3
with:
path: build
key: ${{ runner.os }}-build
- run: make patch-openmp
- run: make loadable-release static-release
env:
# `brew info libomp` gives the correct one, with .a file for static openmp builds
CC: /opt/homebrew/opt/llvm/bin/clang
CXX: /opt/homebrew/opt/llvm/bin/clang++
LDFLAGS: "-L/usr/local/opt/libomp/lib/"
CPPFLAGS: "-I/usr/local/opt/libomp/include/"
- uses: actions/upload-artifact@v3
with:
name: ${{ env.ARTIFACT-LINUX-X86_64-EXTENSION }}
name: ${{ env.ARTIFACT-MACOS-AARCH64-EXTENSION }}
path: dist/release/*
build-linux-x86_64-python:
build-linux-x86_64-extension:
runs-on: ubuntu-20.04
needs: [build-linux-x86_64-extension]
steps:
- uses: actions/checkout@v3
- uses: actions/download-artifact@v3
with:
name: ${{ env.ARTIFACT-LINUX-X86_64-EXTENSION }}
path: dist/release/
- uses: actions/setup-python@v3
- run: pip install wheel
- run: make python-release
- run: make datasette-release
- uses: actions/upload-artifact@v3
with:
name: ${{ env.ARTIFACT-LINUX-X86_64-WHEELS }}
path: dist/release/wheels/*.whl
build-macos-x86_64-extension:
runs-on: macos-latest
steps:
- uses: actions/checkout@v3
with:
Expand All @@ -71,47 +61,20 @@ jobs:
- if: steps.cache-sqlite-build.outputs.cache-hit != 'true'
working-directory: vendor/sqlite
run: ./configure && make
- run: brew install llvm
- id: cache-cmake-build
uses: actions/cache@v3
with:
path: build
key: ${{ runner.os }}-build
- run: make patch-openmp

# TODO how cache this?
- run: sudo apt-get install -y cmake libgomp1
- run: make loadable-release static-release
env:
# `brew info libomp` gives the correct one, with .a file for static openmp builds
CC: /usr/local/opt/llvm/bin/clang
CXX: /usr/local/opt/llvm/bin/clang++
LDFLAGS: "-L/usr/local/opt/libomp/lib/"
CPPFLAGS: "-I/usr/local/opt/libomp/include/"
- uses: actions/upload-artifact@v3
with:
name: ${{ env.ARTIFACT-MACOS-X86_64-EXTENSION }}
name: ${{ env.ARTIFACT-LINUX-X86_64-EXTENSION }}
path: dist/release/*
build-macos-x86_64-python:
build-macos-x86_64-extension:
runs-on: macos-latest
needs: [build-macos-x86_64-extension]
steps:
- uses: actions/checkout@v3
- uses: actions/download-artifact@v3
with:
name: ${{ env.ARTIFACT-MACOS-X86_64-EXTENSION }}
path: dist/release/
- uses: actions/setup-python@v3
- run: pip install wheel
- run: make python-release
- run: make datasette-release
- uses: actions/upload-artifact@v3
with:
name: ${{ env.ARTIFACT-MACOS-X86_64-WHEELS }}
path: dist/release/wheels/*.whl
build-macos-aarch64-extension:
runs-on: [self-hosted, mm1]
steps:
- uses: actions/checkout@v3
with:
submodules: "recursive"
submodules: recursive
- id: cache-sqlite-build
uses: actions/cache@v3
with:
Expand All @@ -122,61 +85,24 @@ jobs:
- if: steps.cache-sqlite-build.outputs.cache-hit != 'true'
working-directory: vendor/sqlite
run: ./configure && make
- run: brew install llvm
- id: cache-cmake-build
uses: actions/cache@v3
with:
path: build
key: ${{ runner.os }}-build
- run: make patch-openmp
- run: make loadable-release static-release
env:
# `brew info libomp` gives the correct one, with .a file for static openmp builds
CC: /opt/homebrew/opt/llvm/bin/clang
CXX: /opt/homebrew/opt/llvm/bin/clang++
LDFLAGS: "-L/opt/homebrew/opt/libomp/lib"
CPPFLAGS: "-I/opt/homebrew/opt/libomp/include"
CC: /usr/local/opt/llvm/bin/clang
CXX: /usr/local/opt/llvm/bin/clang++
LDFLAGS: "-L/usr/local/opt/libomp/lib/"
CPPFLAGS: "-I/usr/local/opt/libomp/include/"
- uses: actions/upload-artifact@v3
with:
name: ${{ env.ARTIFACT-MACOS-AARCH64-EXTENSION }}
name: ${{ env.ARTIFACT-MACOS-X86_64-EXTENSION }}
path: dist/release/*
build-macos-aarch64-python:
runs-on: [self-hosted, mm1]
needs: [build-macos-aarch64-extension]
steps:
- uses: actions/checkout@v3
- uses: actions/download-artifact@v3
with:
name: ${{ env.ARTIFACT-MACOS-AARCH64-EXTENSION }}
path: dist/release/
- run: pip3 install wheel
- run: make python-release IS_MACOS_ARM=1
- run: make datasette-release
- uses: actions/upload-artifact@v3
with:
name: ${{ env.ARTIFACT-MACOS-AARCH64-WHEELS }}
path: dist/release/wheels/*.whl
upload-deno:
needs:
[
build-macos-x86_64-extension,
build-macos-aarch64-extension,
build-linux-x86_64-extension,
]
permissions:
contents: write
runs-on: ubuntu-latest
outputs:
deno-checksums: ${{ steps.deno-assets.outputs.result }}
steps:
- uses: actions/checkout@v3
- uses: actions/download-artifact@v2
- id: deno-assets
uses: actions/github-script@v6
env:
ARTIFACT-LINUX-X86_64-EXTENSION: ${{ env.ARTIFACT-LINUX-X86_64-EXTENSION }}
ARTIFACT-MACOS-X86_64-EXTENSION: ${{ env.ARTIFACT-MACOS-X86_64-EXTENSION }}
ARTIFACT-MACOS-AARCH64-EXTENSION: ${{ env.ARTIFACT-MACOS-AARCH64-EXTENSION }}
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
result-encoding: string
script: |
const script = require('.github/workflows/upload-deno-assets.js')
return await script({github, context})
upload-extensions:
needs:
[
Expand All @@ -201,164 +127,3 @@ jobs:
macos-x86_64: ${{ env.ARTIFACT-MACOS-X86_64-EXTENSION }}/*
macos-aarch64: ${{ env.ARTIFACT-MACOS-AARCH64-EXTENSION }}/*
linux-x86_64: ${{ env.ARTIFACT-LINUX-X86_64-EXTENSION }}/*
upload-checksums:
needs: [upload-extensions, upload-deno]
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/github-script@v6
env:
CHECKSUMS: "${{ needs.upload-extensions.outputs.checksums }}\n${{ needs.upload-deno.outputs.deno-checksums }}"
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const { owner, repo } = context.repo;
const release = await github.rest.repos.getReleaseByTag({
owner,
repo,
tag: process.env.GITHUB_REF.replace("refs/tags/", ""),
});
const release_id = release.data.id;
github.rest.repos.uploadReleaseAsset({
owner,
repo,
release_id,
name: "checksums.txt",
data: process.env.CHECKSUMS,
});
upload-hex:
runs-on: ubuntu-latest
needs: [upload-extensions]
steps:
- uses: actions/checkout@v2
- uses: erlef/setup-beam@v1
with:
otp-version: "24"
rebar3-version: "3.16.1"
elixir-version: "1.14"
- run: ./scripts/elixir_generate_checksum.sh "${{ needs.upload-extensions.outputs.checksums }}"
- run: mix deps.get
working-directory: ./bindings/elixir
- run: mix compile --docs
working-directory: ./bindings/elixir
- run: mix hex.publish --yes
working-directory: ./bindings/elixir
env:
HEX_API_KEY: ${{ secrets.HEX_API_KEY }}
upload-npm:
needs:
[
build-macos-x86_64-extension,
build-macos-aarch64-extension,
build-linux-x86_64-extension,
]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/download-artifact@v2
- run: |
cp ${{ env.ARTIFACT-LINUX-X86_64-EXTENSION }}/*.so bindings/node/sqlite-vss-linux-x64/lib/
cp ${{ env.ARTIFACT-MACOS-X86_64-EXTENSION }}/*.dylib bindings/node/sqlite-vss-darwin-x64/lib/
cp ${{ env.ARTIFACT-MACOS-AARCH64-EXTENSION }}/*.dylib bindings/node/sqlite-vss-darwin-arm64/lib/
- uses: actions/setup-node@v3
with:
node-version: "16"
registry-url: "https://registry.npmjs.org"
- name: Publish NPM sqlite-vss-linux-x64
working-directory: bindings/node/sqlite-vss-linux-x64
run: npm publish --access public
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
- name: Publish NPM sqlite-vss-darwin-x64
working-directory: bindings/node/sqlite-vss-darwin-x64
run: npm publish --access public
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
- name: Publish NPM sqlite-vss-darwin-arm64
working-directory: bindings/node/sqlite-vss-darwin-arm64
run: npm publish --access public
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
- name: Publish NPM sqlite-vss
working-directory: bindings/node/sqlite-vss
run: npm publish --access public
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
upload-pypi:
needs:
[
build-linux-x86_64-python,
build-macos-x86_64-python,
build-macos-aarch64-python,
]
runs-on: ubuntu-latest
steps:
- uses: actions/download-artifact@v3
with:
name: ${{ env.ARTIFACT-LINUX-X86_64-WHEELS }}
path: dist
- uses: actions/download-artifact@v3
with:
name: ${{ env.ARTIFACT-MACOS-X86_64-WHEELS }}
path: dist
- uses: actions/download-artifact@v3
with:
name: ${{ env.ARTIFACT-MACOS-AARCH64-WHEELS }}
path: dist
- uses: pypa/gh-action-pypi-publish@release/v1
with:
password: ${{ secrets.PYPI_API_TOKEN }}
skip-existing: true
upload-gem:
needs:
[
build-macos-x86_64-extension,
build-macos-aarch64-extension,
build-linux-x86_64-extension,
]
permissions:
contents: write
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/download-artifact@v2
- uses: ruby/setup-ruby@v1
with:
ruby-version: 3.2
- run: |
rm bindings/ruby/lib/*.{dylib,so,dll} || true
cp ${{ env.ARTIFACT-MACOS-X86_64-EXTENSION }}/*.dylib bindings/ruby/lib
gem -C bindings/ruby build -o x86_64-darwin.gem sqlite_vss.gemspec
env:
PLATFORM: x86_64-darwin
- run: |
rm bindings/ruby/lib/*.{dylib,so,dll} || true
cp ${{ env.ARTIFACT-MACOS-AARCH64-EXTENSION }}/*.dylib bindings/ruby/lib
gem -C bindings/ruby build -o arm64-darwin.gem sqlite_vss.gemspec
env:
PLATFORM: arm64-darwin
- run: |
rm bindings/ruby/lib/*.{dylib,so,dll} || true
cp ${{ env.ARTIFACT-LINUX-X86_64-EXTENSION }}/*.so bindings/ruby/lib
gem -C bindings/ruby build -o x86_64-linux.gem sqlite_vss.gemspec
env:
PLATFORM: x86_64-linux
- run: |
gem push bindings/ruby/x86_64-darwin.gem
gem push bindings/ruby/arm64-darwin.gem
gem push bindings/ruby/x86_64-linux.gem
env:
GEM_HOST_API_KEY: ${{ secrets.GEM_HOST_API_KEY }}
upload-crate:
runs-on: ubuntu-latest
needs: [upload-extensions]
steps:
- uses: actions/checkout@v2
- uses: actions-rs/toolchain@v1
with:
toolchain: stable
- run: cargo publish --no-verify
working-directory: ./bindings/rust
env:
CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}
Loading