feat: allow setting sqlite compiler flags via extconf args

[flavorjones]

Closes #401

[fractaledmind]

I don't have my laptop available this evening, but I will test this branch manually tomorrow. The code looks very clear tho, so I'm confident this will do precisely what the docs describe.

This is such a lovely feature. After digging into the SQLite docs, I was surprised to learn how "tamely" they tuned the default configuration.

[flavorjones]

I'm going to cut a release candidate so you should be able to kick the tires more easily.

[flavorjones]

I was surprised to learn how "tamely" they tuned the default configuration

@fractaledmind If there are some obvious options that are good for general-purpose sqlite usage, please open an issue to suggest them be made the default when the gem builds sqlite!

[flavorjones]

@fractaledmind v1.6.5.rc1 has been uploaded to rubygems. Give it a try and if it works, I'll make a real release.

(For the record, it worked for me: a bundler config is able to set page size and cache size for me on both Linux and macOS.)

[fractaledmind]

It definitely tried to compile SQLite using the config flags, but on my macOS machine, I hit an error. Here is the output of my ~/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/extensions/arm64-darwin-21/3.2.0/sqlite3-1.6.5.rc1/gem_make.out file. TL;DR the key lines are:

linking shared-object sqlite3/sqlite3_native.bundle
Undefined symbols for architecture arm64:

current directory: /Users/fractaled/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/sqlite3-1.6.5.rc1/ext/sqlite3
/Users/fractaled/.asdf/installs/ruby/3.2.1/bin/ruby -I /Users/fractaled/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0 extconf.rb --with-sqlite-cflags\=-DSQLITE_DEFAULT_MEMSTATUS\=0\ -DSQLITE_DEFAULT_PAGE_SIZE\=16384\ -DSQLITE_DEFAULT_WAL_SYNCHRONOUS\=1\ -DSQLITE_DQS\=0\ -DSQLITE_ENABLE_FTS5\ -DSQLITE_LIKE_DOESNT_MATCH_BLOBS\ -DSQLITE_MAX_EXPR_DEPTH\=0\ -DSQLITE_OMIT_DECLTYPE\ -DSQLITE_OMIT_PROGRESS_CALLBACK\ -DSQLITE_OMIT_SHARED_CACHE\ -DSQLITE_THREADSAFE\=0\ -DSQLITE_USE_ALLOCA
Building sqlite3-ruby using packaged sqlite3.
Extracting sqlite-autoconf-3430000.tar.gz into tmp/arm64-apple-darwin21.6.0/ports/sqlite3/3.43.0... OK
Running 'configure' for sqlite3 3.43.0... OK
Running 'compile' for sqlite3 3.43.0... OK
Running 'install' for sqlite3 3.43.0... OK
Activating sqlite3 3.43.0 (from /Users/fractaled/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/sqlite3-1.6.5.rc1/ports/arm64-apple-darwin21.6.0/sqlite3/3.43.0)...
checking for whether -L/Users/fractaled/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/sqlite3-1.6.5.rc1/ports/arm64-apple-darwin21.6.0/sqlite3/3.43.0/lib is accepted as LDFLAGS... yes
checking for whether -lsqlite3 is accepted as LDFLAGS... yes
checking for whether -lz is accepted as LDFLAGS... yes
checking for whether -fvisibility=hidden is accepted as CFLAGS... yes
checking for sqlite3.h... yes
checking for sqlite3_libversion_number() in -lsqlite3... yes
checking for rb_proc_arity()... yes
checking for rb_integer_pack()... yes
checking for sqlite3_initialize()... yes
checking for sqlite3_backup_init()... yes
checking for sqlite3_column_database_name()... no
checking for sqlite3_enable_load_extension()... yes
checking for sqlite3_load_extension()... yes
checking for sqlite3_open_v2()... yes
checking for sqlite3_prepare_v2()... yes
checking for sqlite3_int64 in sqlite3.h... yes
checking for sqlite3_uint64 in sqlite3.h... yes
creating Makefile

current directory: /Users/fractaled/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/sqlite3-1.6.5.rc1/ext/sqlite3
make DESTDIR\= sitearchdir\=./.gem.20230909-33769-tta6re sitelibdir\=./.gem.20230909-33769-tta6re clean

current directory: /Users/fractaled/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/sqlite3-1.6.5.rc1/ext/sqlite3
make DESTDIR\= sitearchdir\=./.gem.20230909-33769-tta6re sitelibdir\=./.gem.20230909-33769-tta6re
compiling aggregator.c
compiling backup.c
compiling database.c
database.c:130:18: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32]
  return INT2NUM((long)sqlite3_total_changes(ctx->db));
         ~~~~~~~ ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
database.c:171:67: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32]
  VALUE result = rb_funcall(handle, rb_intern("call"), 1, INT2NUM((long)count));
                                                          ~~~~~~~ ^~~~~~~~~~~
database.c:416:18: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32]
  return INT2NUM((long)sqlite3_errcode(ctx->db));
         ~~~~~~~ ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3 warnings generated.
compiling exception.c
compiling sqlite3.c
compiling statement.c
statement.c:351:18: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32]
  return INT2NUM((long)sqlite3_column_count(ctx->st));
         ~~~~~~~ ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
statement.c:400:18: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32]
  return INT2NUM((long)sqlite3_bind_parameter_count(ctx->st));
         ~~~~~~~ ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2 warnings generated.
linking shared-object sqlite3/sqlite3_native.bundle
Undefined symbols for architecture arm64:
  "_sqlite3_column_decltype", referenced from:
      _column_decltype in statement.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [sqlite3_native.bundle] Error 1

make failed, exit code 2

[flavorjones]

@fractaledmind Help me reproduce this? Do you have the same problem with compiling the source gem for v1.6.4?

[flavorjones]

OK, I'm trying to reproduce with this in my bundler config:

BUNDLE_BUILD__SQLITE3: "--with-sqlite-cflags='-DSQLITE_DEFAULT_MEMSTATUS=0 -DSQLITE_DEFAULT_PAGE_SIZE=16384 -DSQLITE_DEFAULT_WAL_SYNCHRONOUS=1 -DSQLITE_DQS=0 -DSQLITE_ENABLE_FTS5 -DSQLITE_LIKE_DOESNT_MATCH_BLOBS -DSQLITE_MAX_EXPR_DEPTH=0 -DSQLITE_OMIT_DECLTYPE -DSQLITE_OMIT_PROGRESS_CALLBACK -DSQLITE_OMIT_SHARED_CACHE -DSQLITE_THREADSAFE=0 -DSQLITE_USE_ALLOCA'"

It's fine on Linux, but I can reproduce what you're seeing on my M1.

I think this is because of one of the -D defines you're passing in. With no -D flags this works OK. I'll try to bisect to tell you which flag is messing you up.

[flavorjones]

Looks like -DSQLITE_OMIT_DECLTYPE is is not compatible with the gem, because we have Statement#column_decltype which references the function sqlite3_column_decltype.

[fractaledmind]

That was fast. I was planning to bisect the flags, presuming it was one that wasn't compatible with the gem (as I know is true with the SQLITE_OMIT_DEPRECATED flag). But, you beat me to it. I will remove that flag and test right now.

[fractaledmind]

It works!!

% bin/rails c
Loading development environment (Rails 7.1.0.alpha)
 development  irb(main):001> ActiveRecord::Base.connection.execute 'PRAGMA compile_options'
   (0.2ms)  PRAGMA compile_options
=> 
[{"compile_options"=>"ATOMIC_INTRINSICS=1"},
 {"compile_options"=>"COMPILER=clang-14.0.0"},
 {"compile_options"=>"DEFAULT_AUTOVACUUM"},
 {"compile_options"=>"DEFAULT_CACHE_SIZE=-2000"},
 {"compile_options"=>"DEFAULT_FILE_FORMAT=4"},
 {"compile_options"=>"DEFAULT_JOURNAL_SIZE_LIMIT=-1"},
 {"compile_options"=>"DEFAULT_MEMSTATUS=0"},
 {"compile_options"=>"DEFAULT_MMAP_SIZE=0"},
 {"compile_options"=>"DEFAULT_PAGE_SIZE=16384"},
 {"compile_options"=>"DEFAULT_PCACHE_INITSZ=20"},
 {"compile_options"=>"DEFAULT_RECURSIVE_TRIGGERS"},
 {"compile_options"=>"DEFAULT_SECTOR_SIZE=4096"},
 {"compile_options"=>"DEFAULT_SYNCHRONOUS=2"},
 {"compile_options"=>"DEFAULT_WAL_AUTOCHECKPOINT=1000"},
 {"compile_options"=>"DEFAULT_WAL_SYNCHRONOUS=1"},
 {"compile_options"=>"DEFAULT_WORKER_THREADS=0"},
 {"compile_options"=>"DQS=0"},
 {"compile_options"=>"ENABLE_FTS3"},
 {"compile_options"=>"ENABLE_FTS4"},
 {"compile_options"=>"ENABLE_FTS5"},
 {"compile_options"=>"ENABLE_GEOPOLY"},
 {"compile_options"=>"ENABLE_MATH_FUNCTIONS"},
 {"compile_options"=>"ENABLE_RTREE"},
 {"compile_options"=>"LIKE_DOESNT_MATCH_BLOBS"},
 {"compile_options"=>"MALLOC_SOFT_LIMIT=1024"},
 {"compile_options"=>"MAX_ATTACHED=10"},
 {"compile_options"=>"MAX_COLUMN=2000"},
 {"compile_options"=>"MAX_COMPOUND_SELECT=500"},
 {"compile_options"=>"MAX_DEFAULT_PAGE_SIZE=8192"},
 {"compile_options"=>"MAX_EXPR_DEPTH=0"},
 {"compile_options"=>"MAX_FUNCTION_ARG=127"},
 {"compile_options"=>"MAX_LENGTH=1000000000"},
 {"compile_options"=>"MAX_LIKE_PATTERN_LENGTH=50000"},
 {"compile_options"=>"MAX_MMAP_SIZE=0x7fff0000"},
 {"compile_options"=>"MAX_PAGE_COUNT=1073741823"},
 {"compile_options"=>"MAX_PAGE_SIZE=65536"},
 {"compile_options"=>"MAX_SQL_LENGTH=1000000000"},
 {"compile_options"=>"MAX_TRIGGER_DEPTH=1000"},
 {"compile_options"=>"MAX_VARIABLE_NUMBER=32766"},
 {"compile_options"=>"MAX_VDBE_OP=250000000"},
 {"compile_options"=>"MAX_WORKER_THREADS=0"},
 {"compile_options"=>"MUTEX_OMIT"},
 {"compile_options"=>"OMIT_PROGRESS_CALLBACK"},
 {"compile_options"=>"OMIT_SHARED_CACHE"},
 {"compile_options"=>"SYSTEM_MALLOC"},
 {"compile_options"=>"TEMP_STORE=1"},
 {"compile_options"=>"THREADSAFE=0"},
 {"compile_options"=>"USE_ALLOCA"}]

[fractaledmind]

@flavorjones: Thank you so much for your help with this feature. Let me know when the new version is released, because I want to write about this as soon as possible. I am genuinely giddy with excitement that we found such a clean and simple way to allow developer's to fine-tune their SQLite with compile-time options for their Rails apps. I can't wait to share with folks the optimizations they can unlock, without needing Docker or any other containerization setup.

You have been a real joy to work with (tho, to be honest, I did very little of the work) and to start to get to know. It is precisely individuals like yourself that makes me so happy to have found the Ruby world oh those years ago. Thanks for being such an excellent open-source maintainer.

[flavorjones]

👍 1.6.5 has been released: Release 1.6.5 / 2023-09-08 · sparklemotion/sqlite3-ruby