2.3.2-beta

Major features

• Searchd now uses mmap(). Daemon is available immediately and spawns
  a separate thread that loads the indexes.
• Built-in suggest using CALL QSUGGEST statement for indexes with
  infixing enabled and dict=keywords
• HTTP protocol interface

New functions and options

• added RAND(),HOUR(),MINUTE(),SECOND() functions
• addedFLUSH HOSTNAMES SphinxQL statement and hostname_lookup directive
• added RELOAD INDEX SphinxQL statement
• added sphinxql_timeout directive

Changes and improvements

• changed SHOW INDEX STATUS now displays performed number of queries,
  query times and found rows of last 1,5,15 minutes and total since
  daemon start
• changed retry_count behaviour
• added wildcard support and new options for CALL KEYWORDS SphinxQL
  statement
• faster RT inserts
• faster CSV/TSV indexing
• query cache is disabled by default
• indexer --keeep-attrs support specific path
• daemon now picks changed index path when it receives HUP
• added alias support for MVA attributes
• added SphinxQL support for comparison, IN, and BETWEEN conditions over
  ANY/ALL(mva)
• added explicit JSON type conversion in WHERE clause

Bug fixes

• fixed #2503 update of attributes at index prevents binlog from clean
• fixed #2516 suggest for index with exact_word or morphology options
• fixed #2507 .NET Connector overflow exception (unsigned id support)
• Fixed initial round-robin counter
• Thread-safety checks added (backported)
• Refactored dl-staff
• added per-index statistics to 'show index status'
• fixed #2502 final calculation of expression at RT index (optimized
  calls count)
• Refactored ha-staff
• Added begin() and end() to CSphVector, CSphTightVector
• fixed error handle for API protocol net loop
• Fixed crash on exit (revealed in test 234 on Ubuntu 16.04)
• added token_filter and string list filter to API (php, python); set
  client ver to 32; fixed filter string list escape; updated
  token_filter plugin interface
• Backported behavior for pthread_mutex_timedlock, SCHED_IDLE
• Fast runaround for issue #877
• fixed #2496 profiler counts multiple sequential queries with
  thread_pool worker
• fixed #1825 added support for embedded zeroes in fields for pgsql,
  odbc data sources
• PHP sphinx api: renamed SphinxClient c-tr to __construct
• fixed #2461 crash of daemon with worker thread_pool on high load of
  fast queries
• fixed uninitialized m_bSync variable
• fixed #2461 crash of daemon with worker thread_pool on high load of
  fast queries
• fixed #2456 daemon stuck on rotating index due to high amount of
  search threads
• Fixed internal date calculation which caused different result of
  day(NUM) function in different timezones
• fixed #2400 crash of daemon on CALL KEYWORDS to RT index with disk
  chunks and regexp filter; added regression to test 194
• added #2393 feature wildcards for CALL KEYWORDS; bunch of options
  (fold_wildcards, fold_lemmas, fold_blended, expansion_limit, stats);
  added cases to test 254; fixed github #17
• fixed #2390 latency at workers thread_pool added net-loop wakeup on
  job done added send at the end of job then transfer left data to
  net-loop added spin-wait at polling wait added socket_pair emulation
  for windows version of net-loop added eventfd checks to configure
• added length() for expressions, disabled Expr_Time_c hashing, fixed
  test_253
• fixed Expr_Time and Expr_Timediff always returning empty strings
• fixed minor expression hash calc bug
• added a big test for GetHash in expressions; added Expr_Now_c; fixed
  template expression name check
• fixed several filters vs qcache issues
• check filter expression tree when caching queries
• fixed #2384 replace large index list at message to distributed index
  name; added regression to test 153
• fixed #2384 fold large index list at message from distributed index;
  added regression to test 153
  • fixed #2372 ALL(mva) filter passed from master to agent as legacy
    filter; added regressions to test 244; set master version to 13
• fixed #2371 warning on query via API with filter on MVA attribute;
  added cases to test 244
• fixed query cache vs filters with expressions
• fixed #2351 ALTER RECONFIGURE skipped for RT index with only re2 or
  rlp changes; added regression as test 252 set binlog version to 6
• fixed daemon to work with --nodetach option after previous commit
  breaks it
• fixed #2358 mmap memory to be fork-less fixed bitvec copying fixing
  false socket shutdown at net-loop added ping handling to net-loop
  instead API command added feature to distributed index to break
  kill-list of local indexes sequence
• fixed a memory leak on inserts with aot enabled
• fixed #2062 attribute name shadows field with same name; added check
  at ALTER and RT index config; added regressions to test 214
• fixed #2330 daemon shutdown stopped waiting searching threads
• fixed dlopen bug on linux while reloading udf
• fixed (searchd.cpp split issues): stats mutex leak and crash of
  dashboard at distributed index setup due to config reload; added tests
  248, 249
• fixed #2299 crash of indexer due to empty xmlpipe2 source with
  embedded schema; added regression to test 68
• fixed RLP vs non-CJK fields (missing trailing zero)
• refactored RLP to work as a field filter (preprocessor)
• fixed RLP enabled build
• fixed ubertest to pass on different linux platforms
• added SphinxQL support for comparison, IN, and BETWEEN conditions over
  ANY/ALL(mva); and added missing "ident NOT BETWEEN x AND y" syntax
• fixed #2277 network connection timeout overflow for agent with worker
  = thread_pool added test 243
• fixed mantis-2156 (COUNT(DISTINCT attr) does not work with strings)
• updated old links to code.google.com to new links to github.com
• fixed embedded zeroes in rt inserts
• fixed mantis-1825 (no support for embedded zeroes in fields)
• Removed CodeBlocks. Modified .gitignore for clion
• fixed examples version in documentation, rebuilt docs
• added #2262 new blend_mode trim_all added cases to test 192
• fixed #2261 ngram chars presence at charset_table, now it warns for
  such config added regression to tests fixed test 19
• fixed multiform handling (multiform + lemmatizer case) in CALL
  KEYWORDS
• fixed libre2.patch to be in sync with latest re2 changes
• Eliminated gcc warnings in http_parser.c. Eliminated msvc warning in
  sphinxquery.cpp.
• Windows yy.cmd synced with bash yy.sh script
• lex/bison files and rules fixed for bison >1.875
• do not create tokenizer for every document in batch insert, create it
  just once and reuse instead
• fixed bug #1766 (UPDATE does not correctly update negative values for
  bigint and float attributes)
• fixed hits duplicates at RT index on document indexing fixed aggregate
  depended expression at RT index fixed tests 162, 192, 205 to pass rt
  mode updated visual studio 2013 project file
• optimized away crazy memmove() in CSV/TSV parser, much faster CSV/TSV
  indexing (more than 10x on a synthetic test)
• field lengths are no longer required to be last in schema
• initial per-index field lengths support for RT, fixes test 217 --rt
• fixed CSphMatchVariant::ToDocid conversion to match plain index
  behavior (fixes test 047 --rt)
• fixed duplicates handling vs RT INSERT (first row wins now, not the
  last one)
• added fetched_* counters collection to rt (fixes test_209 in --rt
  mode)
• fixed keyword expansion in rt with docinfo=inline (fixes test_126 in
  --rt mode)
• unified CSphIndex::SetupQueryTokenizer and sphSetupQueryTokenizer
  implementations, fixes most (but not all) of test 165 --rt
• fixed off-by-1 in non-stemmed stopword check; fixed that lemmas got
  stemmed; fixed that wordforms could get applied twice through
  exact_dict; and rebuilt test 207 accordingly
• improved RT insert speed (%7 gain in my batch insert test case)
• indextool needs to preread checkpoints and infixes too
• fixed mlock option on caching index files
• fixed #2223 query cache last entry eviction during search cause daemon
  to hung
• Expr_Rand_c speedup and fixes, thread-safe XorShift64, updated test
  125
• fixed #2053 added RAND() function
• fixed #2230 memory corruption at daemon on inserting data into RT with
  bad HTML markup added regression to tests
• fixed span length and lcs calculation in proximity queries
• fixed performance on reading a lot of small buffers
• fixed #2223 crash at watchdog shutdown on some OSes like centos, rhel
• optimize RT inserts
• refactoring
• improving insertion speed into RT index (5% gain in my test)
• refactoring, removed unneeded code
• added RELOAD INDEX to SphinxQL
• fixed #2209 prohibited order by MVA, added error message
• fixed undefined reference to void ISphOutputBuffer::SendT in release
  version
• new qcache defaults
• lets handle 32bit weights in qcache
• fixed a couple of memory leaks
• fixed typo in vs2008 proj; added lost files to codeblocks projects
• searchd.cpp splitted
• fixed agent dashboard setup due to remove of workers
• added test_232, positions coming out of the matching engine
• fixed several bugs in qcache (bug #2191 and some more)
• use RAII on CSphMutex instead of separate initialization method, fixed
  clang warnings
• added feature #2195 memory mapping of all index files with separated
  caching thread daemon (re)start should be immediately and fix of 'old'
  ondisk* issue fixed update of attributes for indexes with ondisk*
  option got rig and prohibit 32bit to 64bit index conversion on load
  got rid of all shared memory code
• fixes in variant_match model generation (more compatible attr types,
  and better diff report)
• fixed HTML stripper handling of broken PI (processing instruction)
  tags
• added #2179 SphinxQL client timeout searchd section option
  sphinxql_timeout, default value is 900 seconds

2.2.11-release

• fixed #2499 crash of daemon at phrase node with star shift; added regressions to test 41
• Backported RE2 patch and solutions from master
• fixed #2488 performance issue with matching hitless terms
• fixed #2498 wrong profiling report (was filter instead get_hits)
• fixed #2497 windows service does not handle system shutdown
• fixed #2484 plugin ranker at distributed index, crash of daemon in case no plugin at agent and check of cache
• fixed #2495 filtering by false/true/null values without using aliases
• fixed #2484 plugin ranker and token_filter work with distributed index; set master ver to 14
• fixed #2485 client failed to parse reply with string_ptr attribute via API; added regression to test 159
• fixed #2462 added OPTION statements to SphinxQL log fixed string split buffer overrun
• fixed #2458 expression with escaped quote was cut at distributed index; added regressions to test 125
• fixed #2459 incorrect indexer exit code for indexing multiple indexes
• fixed #2452 group by aliased JSON array
• optimized sphWildcardMatch (UTF-8 vs ASCII)
• fixed #2451 UTF-8 support for extended wildcards (?, %)
• fixed #2434 multiple declaration of same attribute breaks *sv pipe data sources and index; added test 257
• fixed #2437 64-bit values comparison for ALL/ANY/INDEXOF functions
• fixed #2434 field modifier error for over-short word; added regression to test 211
• fixed #2435 added HAVING statement to sphinxql query log
• fixed #2433 crash of indexer for csv source with escaped quote inside quotation; added regression to tests
• fixed #2431 indexer crash on multiple escape chars at csv source; added regression to tests
• backported svn:r5092 git:3b5bf10bb852e992f4e02d6f379899413549b5fe
• fixed #2429 crash of service threads has no crash log
• fixed #2416 crash of daemon on hit stream with wrong qpos; fixed tests 184, 251
• fixed #2427 no warnings for short wildcards inside word; added regressions to test 173
• fixed #2420 count(*) statement vs space characters at facet; added cases to test 226
• fixed #2209 prohibited order by MVA, backported from master, added error message; fixed tests 20, 180
• fixed #2419 ATTACH RT index missed doc-id duplicates for id32; fixed test 187
• fixed #2421 daemon crash on complex query with field start and quorum operators
• fixed #2417 destination wordform from multiform was stemmed; added regressions to test 22
• fixed #2419 ATTACH RT index missed doc-id duplicates; added regression to test 184
• fixed #2418 wildcards do not work at snippets beside star; added cases to test 40
• fixed #2406 ALTER RECONFIGURE mess with wordforms due to wrong creation order; added regression to test 255
• fixed #2405 crash of daemon on quorum query with duplicates from expand_keywords; added regression to test 54
• fixed #2404 ok reply for ALTER query to missed index; added regression to test 213
• fixed #2394 multi query with profiling enabled; added statements SET, SHOW PLAN, SHOW profile to multi query; added regression to test 113; fixed ubertest to clean up multi query result sets
• fixed #2398 lcs calculation for large delta positions; added regression to tests 175, 68
• fixed #2399 crash of daemon for query with json attribute at HAVING; added regressions to test 171
• fixed #2397 inplace JSON update for several 64-bit values
• fixed #1415 (destination tokens in wordforms couldn't be stopwords)
• fixed #2315 crash of daemon on rotating config without indexes
• fixed #14 (github) RT index without any attribute; fixed tests 88, 175, 181, 192
• fixed #2389 indextool now reports on empty segment of RT index at check mode
• fixed #2383 wrong indexer exit cqode with nohup option
• fixed #2365 multiple sysvars support in select statement (for Connector/J 5.1.36+)
• fixed #2363 ping to bad ha mirror pause accept thread at daemon
• fixed #2376 json-dependent columns vs is null and order by
• fixed #2355 result set missed SNIPPET expression calc for query with offset; added regressions to test 143
• fixed JSON type conversion, fixed memory leak, updated test 206
• fixed #2375 json field type autoconversion in expressions
• fixed #2363 ping to bad ha mirror pause accept thread at daemon; fixed missed statistics lock
• fixed #2373 group by 1-character json field, updated test 206
• fixed crash log time double setup from previous commit
• fixed #2363 ping to bad ha mirror pause accept thread at daemon (affects all workers but not prefork), moved ping to a separate thread
• fixed #2364 case-sensitive fields support for distributed indexes
• fixed #2361 daemon crash with quorum operator; added warning on replacing quorum operator
• fixed #2359 agent balancing added tests on balancing fixed ping to affect only counters but not timings fixed network statistics aggregation fixed timeout exit on reading agent reply fixed mirror statistics output to be more consistent added feature log_debug_filter to log only matched messages
• fixed #2356 added error handling for large (>0x400000 bytes) JSON fields
• fixed #2348 max_matches option did not affect facet queries
• fixed #2337 crash of indexer on indexing large data passed regexp_filter; added regression to test 194
• fixed #2336 field modifiers assigned improperly for query with multi destination wordforms; added regressions to test 22
• fixed #2341 crash of daemon with memory corruption at query to multiple indexes with pool attributes; added regression to test 159
• fixed #2339 indexer crashed on lemma length overflow; added regression to test 207
• fixed #1912 indextool memory usage during check of huge index (skiplist keep on disk and docid list variants to use less memory)
• added string list filter; set master version to 12, API protocol version to 1.31 fixed #2334 daemon crash on filtering by string column via API; fixed string values IN() filter for distributed index; added regression to tests 60, 125
• fixed #2296 wrong snippet produced from wordforms with multiple destination tokens and AOT dictionary; added regressions to test 223
• fixed #1675 ZONESPAN was silently ignored with BEFORE operator, added warning
• fixed #2329 slow wildcard matching (star vs long terms might took above 100 seconds); added regressions to tests
• fixed #2323 missed field id at packedfactors with json enabled output; fixed model at test 217
• fixed test 125, enabled back regression
• fixed #2325 false positive result of indextool check mode for bad index with crc dictionary
• fixed #2324 daemon crash on handle query via api with empty select list to indexes with dissimilar structure; added regression to test 113
• fixed JSON mapping from the previous commit (consumes less memory this way)
• fixed #2320 rt index crashes on groupby() for large JSON fields
• fixed indextool --check vs nested JSON objects

2.2.10-release

Sphinx 2.2.10 is released