feat(doc): What’s new in 3.4.0?

src/docs/ext/linter.py

@@ -107,7 +107,7 @@ def silent_scream(file):
         else:
             IGNORE_ERROR = False
 
-        match = re.match("\s*.. lint: ignore errors for the next (\d+) line?", line)
+        match = re.match(r"\s*.. lint: ignore errors for the next (\d+) line?", line)
         if match:
             # +1 for empty line right after comment
             counter = int(match.group(1)) + 1

src/docs/src/experimental.rst

@@ -37,6 +37,8 @@ sent for everything in /_utils.
 
 Then restart CouchDB.
 
+.. _nouveauconfig:
+
 Nouveau Server (new Apache Lucene integration)
 ==============================================
 

src/docs/src/whatsnew/3.4.rst

@@ -0,0 +1,359 @@
+.. Licensed under the Apache License, Version 2.0 (the "License"); you may not
+.. use this file except in compliance with the License. You may obtain a copy of
+.. the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+.. WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+.. License for the specific language governing permissions and limitations under
+.. the License.
+
+.. _release/3.4.x:
+
+============
+3.4.x Branch
+============
+
+.. contents::
+    :depth: 1
+    :local:
+
+.. _release/3.4.0:
+
+Version 3.4.0
+=============
+
+Breaking Changes
+----------------
+
+* :ghissue:`5046`: JWT: require valid ``exp`` claim by default
+
+  Users of JWT rightly expect tokens to be considered invalid once they expire. It
+  is a surprise to some that this requires a change to the default
+  configuration. In the interest of security we will now require a valid ``exp``
+  claim in tokens. Administrators can disable the check by changing
+  ``required_claims`` back to the empty string.
+
+  We recommend adding ``nbf`` as a required claim if you know your tokens will
+  include it.
+
+Highlights
+----------
+
+.. code-block:: text
+
+      .-.
+          /  |
+         /\  | .-._.)  (  )   .-..-.  .-.  )  (
+        /  \ |(   )(    )(   / ./.-'_(  | (    )
+   .-' /    \| `-'  `--': \_/  (__.'  `-'-'`--':
+  (__.'      `.
+
+* :ghissue:`4291`: Introducing Nouveau (beta) a modern, from-the-ground-up
+  implementation of Lucene-based full-text search for CouchDB. Please test
+  this thoroughly and report back any issues you might find.
+
+  - :ref:`Setup instructions <nouveauconfig>`
+  - :ref:`Usage <ddoc/nouveau>`
+  - `Report a bug`_.
+
+.. _Report a bug: https://github.com/apache/couchdb/issues/new/choose
+
+* :ghissue:`4627`: Add `QuickJS`_ as a JavaScript engine option.
+
+.. _QuickJS: https://bellard.org/quickjs/`
+
+  Advantages over SpiderMonkey:
+
+  - significantly smaller and easier to integrate codebase. We're using ~6
+    C files vs 700+ SM91 C++ files.
+  - Built with Apache CouchDB as opposed having to maintain a separate
+    SpiderMonkey package for OSs that don’t support it
+    (\*cough*RedHat9\*cough\*).
+  - Better sandboxing support.
+  - Preliminary test results show multiple performance improvements.
+
+      - 4x faster than SpiderMonkey 1.8.5.
+      - 5x faster than SpiderMonkey 91.
+      - 6x reduced memory usage per ``couchjs`` process (5MB vs 30MB).
+
+  - Allows compiling JavaScript bytecode ahead of time.
+  - QuickJS can be built alongside SpiderMonkey and toggled on/off at runtime::
+
+          ./configure --dev --js-engine=quickjs
+
+  - This makes it the default engine. But SpiderMonkey can still be set in the
+    config option::
+
+          [couchdb]
+          js_engine = spidermonkey | quickjs
+
+* :ghissue:`4570`, :ghissue:`4578`, :ghissue:`4576`: Adopt xxHash in favour
+  of md5 for ``couch_file`` checksums and ETag calculation. 30% performance
+  increase for large (128K) docs. No difference for smaller docs.
+
+* :ghissue:`4814`: Introduce PBKDF2-SHA256 for password hashing. The existing
+  PBKDF2-SHA1 variant is now deprecated. Increases the default iteration count
+  to ``500000``. Also introduce a password hash in-memory cache with a low
+  iteration number, to keep interactive requests fast for a fixed time.
+
+  Entries in the password hash cache are time-limited, unused entries are
+  automatically deleted, and there is a capacity bound.
+
+* :ghissue:`4512`: Mango: add keys-only covering indexes. Improves query
+  response times for certain queries up to 10x at p(95).
+
+* :ghissue:`4681`: Introduce optional countermeasures as we run out of
+  disk space.
+
+* :ghissue:`4847`: Require auth for ``_replicate`` endpoint. This continues
+  the 3.x closed-by-default design goal.
+
+* Many small small performance improvements, see :ref:`the Performance
+  section <performance430>`.
+
+Features and Enhancements
+-------------------------
+
+* :ghissue:`5069`: Update Fauxton to v1.3.1.
+* :ghissue:`5067`: Support Erlang/OTP 27.
+* :ghissue:`5053`: Use the built-in ``crypto:pbkdf2_hmac`` function.
+* :ghissue:`5036`: Remove ``replication_job_supervisor``.
+* :ghissue:`5035`: Modernise ``couch_replicator_supervisor``.
+* :ghissue:`5019`: Remove unused build files.
+* :ghissue:`5017`: Remove unused ``boot_dev_cluster.sh``.
+* :ghissue:`5014`: Add Couch Scanner module.
+* :ghissue:`5013`: Improve dist diagnostics.
+* :ghissue:`4990`: Add ``dbname`` to mango exec stats.
+* :ghissue:`4987`: Replace ``khash`` with maps in ``ddoc_cache_lru``.
+* :ghissue:`4984`: Fabric: switch to maps for view rows.
+* :ghissue:`4979`: Git ignore top level ``clouseau`` directory.
+* :ghissue:`4977`: Replace ``khash`` with maps in ``couch_event_server``.
+* :ghissue:`4976`: Add metrics for fast vs slow password hashing.
+* :ghissue:`4965`: Handle multiple response copies for ``_purged_infos`` API.
+* :ghissue:`4878`: Add an option to scrub some sensitive headers from external
+  json.
+* :ghissue:`4834`: Wait for newly set admin creds to be hashed in setup.
+* :ghissue:`4821`: Do not fail compactions if the last step is delayed
+  by ``ioq``.
+* :ghissue:`4810`: Mango: add ``$beginsWith`` operator .
+* :ghissue:`4769`: Improve replicator error handling.
+* :ghissue:`4766`: Add new HTTP endpoint ``/_node/_local/_smoosh/status``.
+* :ghissue:`4736`: Stop client process and clean up if client disconnects.
+* :ghissue:`4703`: Add ``_purged_infos`` endpoint.
+* :ghissue:`4685`: Add ``"CouchDB-Replicator/..."`` user agent to
+  replicator ``/_session`` requests
+* :ghissue:`4680`: Shard splitting: allow resumption of failed jobs and
+  make timeout configurable.
+* :ghissue:`4677`: Crash replication jobs on unexpected ``4xx`` errors.
+* :ghissue:`4670`: Allow setting of additional ``ibrowse`` options
+  like ``prefer_ipv6``.
+* :ghissue:`4662`: Mango: extend ``_explain`` with candidate indexes and
+  selector hints.
+* :ghissue:`4625`: Add optional logging of security issues when
+  replicating.
+* :ghissue:`4623`: Better upgrade handling of ``instance_start_time``
+  in replicator.
+* :ghissue:`4613`: Add option to suppress version info via HTTP header.
+* :ghissue:`4601`: Add simple ``fabric`` benchmark.
+* :ghissue:`4581`: Support Erlang/OTP 26.
+* :ghissue:`4575`: Add ``{verify, verify_peer}`` for TLS validation.
+* :ghissue:`4569`: Mango: add ``keys_examined`` for ``execution_stats``.
+* :ghissue:`4558`: Make Erlang/OTP 24 the minimum supported Erlang version.
+* :ghissue:`4503`: Make timeouts for ``_view`` and ``_search`` configurable.
+* :ghissue:`4483`: Add RFC5424 compliant report logging.
+* :ghissue:`4475`: Add type and descriptions to prometheus output.
+* :ghissue:`4443`: Automatically enable FIPS mode at runtime.
+* :ghissue:`4438`: Upgrade hash algorithm for proxy auth.
+* :ghissue:`4432`: Hide shard-sync and purge documents from ``_local_docs``.
+* :ghissue:`4431`: Allow definition of JWT roles claim as comma-separated
+  list.
+* :ghissue:`4404`: Respond with 503 immediately if search not available.
+* :ghissue:`4347`: Remove failed ``couch_plugins`` experiment.
+* :ghissue:`5046`: JWT: require valid `exp` claim by default
+* :ghissue:`5065`: Update Fauxton UI to version v1.3.1
+
+.. _performance430:
+
+Performance
+-----------
+
+* :ghissue:`5037`: Optimise ``fabric:all_dbs()``.
+* :ghissue:`4911`: Optimise and clean up ``couch_multidb_changes``.
+* :ghissue:`4852`: Optimise ``_active_tasks``.
+* :ghissue:`4786`, :ghissue:`4789`: Add extra timing stats for ``couch_js``
+  engine commands.
+* :ghissue:`4679`: Fix multipart parse ``attachment longer than expected``
+  error.
+* :ghissue:`4672`: Remove ``folsom`` and reimplement required functionality
+  with new Erlang/OTP primitives resulting in up to 19x faster histogram
+  operations.
+* :ghissue:`4617`: Use a faster sets implementation available since OTP 24.
+* :ghissue:`4608`: Add metrics for fsync calls and query engine operations.
+* :ghissue:`4604`: 6x speedup for common ``mem3:dbname/1`` function.
+* :ghissue:`4603`: Update ``vm.args`` settings, increased Erlang distribution
+  buffer size to 32MB.
+* :ghissue:`4598`: Speed up internal replicator.
+* :ghissue:`4507`, :ghissue:`4525`: Add more prometheus metrics.
+* :ghissue:`4505`: Treat JavaScript internal errors as fatal.
+* :ghissue:`4494`: Treat single-element keys as key.
+* :ghissue:`4473`: Avoid re-compiling filter view functions.
+* :ghissue:`4401`: Enforce doc ids ``_changes`` filter optimisation limit and
+  raise it from ``100`` to ``1000``.
+* :ghissue:`4394`: Mango: push ``fields`` selection down to data nodes.
+
+Bugfixes
+--------
+
+* :ghissue:`5066`: Handle multiple ``Set-Cookie`` headers in replicator session
+  plugin.
+* :ghissue:`5060`: Cleanup a few clauses in ``fabric_view_changes``.
+* :ghissue:`5030`: Always commit if we upgrade 2.x view files. Fixes
+  misleading ``wrong signature`` error.
+* :ghissue:`5025`: Fix ``seedlist`` to not return duplicate json keys.
+* :ghissue:`5008`: Fix case clause error in replicator ``_scheduler/docs``
+  response.
+* :ghissue:`5000`: Remove repetitive word in source commends (5000!).
+* :ghissue:`4962`: Make multidb changes shard map aware.
+* :ghissue:`4958`: Mango: use rolling execution statistics.
+* :ghissue:`4921`: Make sure to reply to ``couch_index_server`` clients.
+* :ghissue:`4910`: ``couch_passwords:verify`` should always return false for
+  bad inputs.
+* :ghissue:`4908`: Mango: communicate rows read for global stats collection.
+* :ghissue:`4906`: Flush ``chttpd_db`` monitor refs on demonitor.
+* :ghissue:`4904`: Git ignore all .hypothesis directories.
+* :ghissue:`4887`: Look up search node name in config for weatherreport.
+* :ghissue:`4837`: Fix update bug in ``ets_lru``.
+* :ghissue:`4811`: Prevent delayed opener error from crashing index servers.
+* :ghissue:`4794`: Fix incorrect raising of ``database_does_not_exist`` error.
+* :ghissue:`4784`: Fix parsing of node name from ``ERL_FLAGS1``
+  in ``remsh``.
+* :ghissue:`4782`, :ghissue:`4891`: Mango: prevent occasional
+  duplication of paginated text results.
+* :ghissue:`4761`: Fix badrecord error when replicator is logging HTTP usage.
+* :ghissue:`4759`: TLS: use HTTP rules for hostname verification.
+* :ghissue:`4758`: Remove sensitive headers from the ``mochiweb`` request
+  in ``pdict``.
+* :ghissue:`4751`: Mango: correct behaviour of ``fields`` on ``_explain``.
+* :ghissue:`4722`: Fix badmatch error when purge requests time out.
+* :ghissue:`4716`: Fix pending count for reverse changes feed.
+* :ghissue:`4709`: Mango: improve handling of invalid fields.
+* :ghissue:`4704`, :ghissue:`4707`: Fix empty facet search results.
+* :ghissue:`4682`: ``_design_doc/queries`` with ``keys`` should only return
+  design docs.
+* :ghissue:`4669`: Allow for more than two replicator socket options.
+* :ghissue:`4666`: Improve error handling in config API.
+* :ghissue:`4659`: Mango: remove duplicates from ``indexable_fields/1``
+  results.
+* :ghissue:`4658`: Fix undefined range in ``mem3_rep`` purge replication
+  logic.
+* :ghissue:`4653`: Fix ability to use ``;`` inside of config values.
+* :ghissue:`4629`: Fix prometheus to survive ``mem3_sync`` termination.
+* :ghissue:`4626`: Fix purge infos replicating to the wrong shards
+  during shard splitting.
+* :ghissue:`4602`: Fix error handling for the ``_index`` endpoint and
+  document ``_index/_bulk_delete``.
+* :ghissue:`4555`: Fix race condition when creating indexes.
+* :ghissue:`4524`: Querying ``_all_docs`` with non-string key should
+  return an empty list.
+* :ghissue:`4514`: GET invalid path under ``_index`` should not cause
+  500 response.
+* :ghissue:`4509`: Make ``remsh`` work with quoted cookie.
+* :ghissue:`4503`: Add ``error_info`` clause for ``410 Gone``.
+* :ghissue:`4491`: Fix ``couch_index`` to avoid crashes under certain
+  conditions.
+* :ghissue:`4485`: Catch and log any error from ``mem3:local_shards`` in
+  ``index_server``.
+* :ghissue:`4473`: Fix prometheus counter metric naming.
+* :ghissue:`4458`: Mango: Fix text index selection for queries with ``$regex``.
+* :ghissue:`4416`: Allow ``_local`` doc writes to the replicator dbs.
+* :ghissue:`4370`: Ensure design docs are uploaded individually when
+  replicating with ``bulk_get``.
+* :ghissue:`4363`: Fix replication ``_scheduler/docs`` ``total_rows``.
+* :ghissue:`4360`: Fix handling forbidden exceptions from workers in
+  ``fabric_doc_update``.
+* :ghissue:`4353`: Fix replication ``job_link``.
+* :ghissue:`4348`: Fix undefined function warning in weatherreport.
+* :ghissue:`4343`: Fix undef when parsing replication doc body.
+
+Tests
+-----
+
+* :ghissue:`5068`: Fix flakiness in ``fabric_bench``.
+* :ghissue:`5054`: Update a few deps and improve CI.
+* :ghissue:`5050`: Update CI OSes.
+* :ghissue:`5048`: Update CI Erlang versions.
+* :ghissue:`5040`: Fix invalid call to ``exit/2`` in ``couch_server``.
+* :ghissue:`5039`: Improve fabric ``all_dbs`` test.
+* :ghissue:`5024`: Fix flaky ``_changes`` async test.
+* :ghissue:`4982`: Fix flaky password hashing test.
+* :ghissue:`4980`: Fix password test timeout.
+* :ghissue:`4973`: Handling node number configuration in ``dev/run``.
+* :ghissue:`4959`: Enable Clouseau for more platforms.
+* :ghissue:`4953`: Improve retries in dev/run cluster setup.
+* :ghissue:`4947`: Add tests for ``_changes`` endpoint.
+* :ghissue:`4938`: Add tests for ``_changes`` with different parameters.
+* :ghissue:`4903`: Add extra rev tree changes tests.
+* :ghissue:`4902`: Fix flaky tests by increasing timeout.
+* :ghissue:`4900`: More flaky fixes for cluster setup.
+* :ghissue:`4899`: Reduce EUnit log noise.
+* :ghissue:`4898`: Simplify ``couch_changes_tests.erl`` using macro
+  ``?TDEF_FE``.
+* :ghissue:`4893`: Relax restriction on ``[admins]`` in dev ``local.ini``.
+* :ghissue:`4889`: Do not use admin party for integration tests.
+* :ghissue:`4873`: Fix test for `text` index creation.
+* :ghissue:`4863`: Fix flaky ``users_db_security`` test.
+* :ghissue:`4808`: Fix flaky ``couch_stream`` test.
+* :ghissue:`4806`: Mango: do not skip json tests when Clouseau installed.
+* :ghissue:`4803`: Fix flaky ``ddoc_cache`` test some more.
+* :ghissue:`4765`: Fix flaky mem3 reshard test.
+* :ghissue:`4763`: Plug hole in unit test coverage of view curser functions.
+* :ghissue:`4726`: Support Elixir 1.15.
+* :ghissue:`4691`: ``make elixir`` should match what we run in CI.
+* :ghissue:`4632`: Fix test database recreation logic.
+* :ghissue:`4630`: Add extra assert in flaky couch_file test.
+* :ghissue:`4620`: Add Erlang/OTP 26 to Pull Request CI matrix.
+* :ghissue:`4552`, :ghissue:`4553`: Fix flaky couchjs error test.
+* :ghissue:`4453`: Fix flaky LRU test that the new super fast macOS CI worker
+  noticed.
+* :ghissue:`4422`: Clean up JSON index selection and add unit tests.
+* :ghissue:`4345`: Add test coverage for replicator ``user_ctx`` parser.
+
+Docs
+----
+
+* :ghissue:`5063`: Fix duplicate keys in find query.
+* :ghissue:`5045`: Create Python ``virtualenv`` on Windows for docs.
+* :ghissue:`5038`: Fix small detail about conflicts in Overview section.
+* :ghissue:`4999`: Change server instance to cluster for UUID docs.
+* :ghissue:`4955`: Revamp the installation instructions for FreeBSD.
+* :ghissue:`4951`: Add extension for copying code blocks with just one click.
+* :ghissue:`4950`: Improve changes feed API documentation.
+* :ghissue:`4948`: Update Sphinx package version to 7.2.6.
+* :ghissue:`4946`: Update Sphinx/RTD dependencies.
+* :ghissue:`4942`: Fix invalid JSON in ``_db_updates`` example.
+* :ghissue:`4940`: Re-wrote snap installation guide lines for 3.3.
+* :ghissue:`4933`: Set docs version numbers dynamically from file.
+* :ghissue:`4928`: Add missing installation OSes for convenience binaries.
+* :ghissue:`4925`: Break long lines for better readability within tables.
+* :ghissue:`4774`: Amend description of ``use_index`` on ``/{db}/_find``.
+* :ghissue:`4743`: Ban the laster monster.
+* :ghissue:`4684`: Add ``_design_docs/queries`` and
+  ``_local_docs/queries``.
+* :ghissue:`4645`: Add authentication data to examples.
+* :ghissue:`4636`: Clarify default quorum calculation.
+* :ghissue:`4561`: Clarify encoding length in performance section.
+* :ghissue:`4402`: Fix example code in partitioned databases.
+
+Builds
+------
+* :ghissue:`4840`: Add Debian 12 (bookworm) to CI and binary packages.
+
+.. _release/3.4.x/breakingchanges:
+
+Other
+-----
+
+What’s new, Scooby-Doo?

src/docs/src/whatsnew/index.rst

@@ -20,6 +20,7 @@ Release Notes
     :glob:
     :maxdepth: 2
 
+    3.4
     3.3
     3.2
     3.1