[new release] raven (11 packages) (1.0.0~alpha2)

[tmattio]

Meta package for the Raven ML ecosystem

• Project page: https://github.com/raven-ml/raven
• Documentation: https://raven-ml.dev/docs/

CHANGES:

We're excited to announce the release of Raven 1.0.0~alpha2! Less than a month after alpha1, this release notably includes contributions from Outreachy applicants in preparation for the upcoming two internships.

Some highlights from this release include:

• NumPy-compatible text I/O with Nx_io.{save,load}_text
• Lots of new functions in Nx/Rune, including neural-net ones dropout, log_softmax, batch_norm, layer_norm, and activation functions like celu and celu, and generic ones like conjugate, index_put, and more.
• Addition of .top libraries for nx, rune, and hugin that auto-install pretty-printers in the OCaml toplevel. You can run e.g. #require "nx.top".
• Addition of a visualization API in Fehu via the new fehu.visualize library, supporting video recording.
• Redesign of Kaun core datastructure and checkpointing subsystem for complete snapshotting.
• Many, many bug fixes and correctness improvements.

We've also made numerous performance improvements across the board:

• Nx elementwise ops: 5–50× faster (e.g., Add 50×50 f32 88.81 µs → 1.83 µs, 48×; Mul 100×100 f32 78.51 µs → 2.41 µs, 33×).
• Nx conv2d: 4–5× faster on common shapes; up to 115× on heavy f64 batched cases (e.g., B16 C64→128 16×16 K3 f64 1.61 s → 13.96 ms).
• Rune autodiff: 1.2–3.7× faster on core grads (e.g., MatMulGrad Medium 34.04 ms → 11.91 ms, 2.86×; Large 190.19 ms → 50.97 ms, 3.73×).
• Talon dataframes: big wins in joins and group-bys (Join 805.35 ms → 26.10 ms, 31×; Group-by 170.80 ms → 19.03 ms, 9×; Filter 9.93 ms → 3.39 ms, 3×).
• Saga tokenizers: realistic workloads 4–17% faster (e.g., WordPiece encode single 136.05 µs → 115.92 µs, 1.17×; BPE batch_32 24.52 ms → 22.27 ms, 1.10×)

We're closing 8 user-reported issues or feature requests and are totalling 30 community contributions from 8 unique contributors.

Nx

• Add Nx_io.Cache_dir module with consolidated cache directory utilities respecting RAVEN_CACHE_ROOT, XDG_CACHE_HOME, and HOME fallback, replacing project-specific cache logic across the whole raven ecosystem (Consolidate cache directory utilities into Nx.Cache module raven-ml/raven#134, @Arsalaan-Alam)
• Add Nx_io.save_txt / Nx_io.load_txt with NumPy-compatible formatting, comments, and dtype support (nx_io: add save_txt/load_txt for 1D/2D tensors raven-ml/raven#120, @six-shot)
• Optimize multi_dot for matrix chains, reducing intermediate allocations and improving performance (@tmattio)
• Add public index_put function for indexed updates (@tmattio)
• Clarify reshape documentation to match its view-only semantics (@tmattio)
• Provide nx.top, rune.top, and hugin.top libraries that auto-install pretty printers in the OCaml toplevel and update Quill to load them (@tmattio)
• Add ifill for explicit in-place fills and make fill return a copied tensor (@tmattio)
• Speed up contiguous elementwise ops via vectorized loops (@tmattio)
• Fast-path contiguous single-axis reductions to avoid iterator fallback (@tmattio)
• Speed up float reductions with contiguous multi-axis fast paths (@tmattio)
• Fast-path padding-free unfold to lower conv2d overhead (@tmattio)
• Move neural-network operations (softmax, log_softmax, relu, gelu, silu, sigmoid, tanh) from Kaun to Nx (@tmattio)
• Add public conjugate function for complex number conjugation (fix: make conjugate publicly available raven-ml/raven#125, @Arsalaan-Alam)
• Fix complex vdot to conjugate first tensor before multiplication, ensuring correct mathematical behavior (fix: complex numbers in Nx.vdot raven-ml/raven#123, @Arsalaan-Alam)
• Update comparison and conditional operations to use boolean tensors (Replace uint8 Masks with Bool Tensors raven-ml/raven#115, @nirnayroy)
• Add support for rcond parameter and underdetermined systems to lstsq (lstsq ~rcond and underdetermined systems implemented raven-ml/raven#102, @Shocker444)
• Fix matrix_rank/pinv Hermitian fast paths to use eigen-decomposition and match NumPy for complex inputs (Implement hermitian optimization for matrix_rank and pinv raven-ml/raven#96, @six-shot, @tmattio)
• Optimize matmul BLAS dispatch for strided tensors, improving matrix multiplication performance (@tmattio)
• Fix slow builds reported since alpha1 (Fix slow builds raven-ml/raven#88, @tmattio)
• Fix macOS ARM crash when loading extended bigarray kinds (@tmattio)
• Add float16 and bfloat16 support to safetensors I/O, including precise conversions that preserve denormals/NaNs (Fix F16/BF16 loading in safetensors raven-ml/raven#84, @six-shot, @tmattio)
• Refined View internals for leaner contiguity checks and stride handling, cutting redundant materialization on hot paths (@tmattio)
• Merge Lazy_view into the core View API so movement ops operate on a single composed view (@tmattio)
• Documented the reworked View interface (@tmattio)
• Documented the Symbolic_shape interface (@tmattio)
• Added Accelerate framework flag when compiling on macOS, fixing issues in some environments (fix: add framework Accelerate when compiling on Macosx raven-ml/raven#129, @nirnayroy)

Hugin

• Let Hugin.show windows close cleanly via the window button or Esc/q, avoiding frozen macOS REPL sessions (@tmattio)

Rune

• Add Rune.no_grad and Rune.detach to mirror JAX stop-gradient semantics (@tmattio)
• Improve gradient performance slightly by replace the reverse-mode tape's linear PhysicalTbl with an identity hash table (@tmattio)
• Fix Rune.Rng.shuffle flattening outputs for multi-dimensional tensors; the
  shuffle now gathers along axis 0 and keeps shapes intact (@tmattio)
• Replace Rune.Rng.truncated_normal clipping with rejection sampling so
  samples stay inside the requested interval without boundary spikes (@tmattio)
• Add support for categorical sampling with Rune.Rng.categorical (chore/implement categorical sampling in rune raven-ml/raven#89, @nirnayroy)
• Allow plain llvm-config in discovery, fixing build in some platforms (fix(rune): allow plain llvm-config in discovery raven-ml/raven#71, @stepbrobd)

Kaun

• Added Similarity and Polysemy analysis to the BERT example (Added Similarity and Polysemy analysis in BERT example raven-ml/raven#137, @nirnayroy)
• Support attention masks via the new Kaun.Attention module (@tmattio)
• Support loading sharded Hugging Face safetensors (@tmattio)
• Fix BERT and GPT‑2 model loading (@tmattio)
• API simplification: removed type parameters from public types; Ptree now supports mixed‑dtype trees via packed tensors with typed getters. (@tmattio)
• Checkpointing overhaul: versioned Train_state with schema tagging, explicit Checkpoint.{Snapshot,Artifact,Manifest,Repository} (retention, tags, metadata), and simple save/load helpers for snapshots and params. (@tmattio)
• Overhaul dataset combinators: derive tensor specs from Rune dtype, fix sampling/window bugs, validate weighted sampling, and respect drop_remainder (@tmattio)
• Make dataset prefetch truly asynchronous with background domains and allow reusing an external Domainslib pool via parallel_map ~pool (@tmattio)
• Use Dataset.iter for epoch batches to reduce overhead (@tmattio)
• Update BERT and GPT-2 tokenizer cache to use Nx.Cache for consistent cache directory resolution (Feature wish: an implementation for op_qr in Rune raven-ml/raven#133, @Arsalaan-Alam)
• Honor text dataset encodings via incremental Uutf decoding (Update text encoding in kaun raven-ml/raven#122, @Satarupa22-SD).
• Preserve empty sequential modules when unflattening so indices stay aligned for checkpoint round-tripping (@tmattio)
• Prevent Training.fit/evaluate from consuming entire datasets eagerly and fail fast when a dataset yields no batches, avoiding hangs and division-by-zero crashes (@tmattio)
• Allow metric history to tolerate metrics that appear or disappear between epochs so dynamic metric sets no longer raise during training (@tmattio)
• Make Optimizer.clip_by_global_norm robust to zero gradients and empty parameter trees to avoid NaNs during training (@tmattio)
• Split CSV loader into from_csv and from_csv_with_labels to retain labels when requested (Update Kaun-CSV loader raven-ml/raven#114, @Satarupa22-SD)
• Implement AUC-ROC and AUC-PR in Kaun metrics and simplify their signatures (Implement AUC-ROC in Kaun metrics raven-ml/raven#124, Implement AUC-PR in Kaun metrics raven-ml/raven#131, @Shocker444)
• Add mean absolute percentage error, explained variance, R² (with optional adjustment), KL-divergence, and top-k accuracy to Kaun metrics (@tmattio)
• Add NDCG, MAP, and MRR ranking metrics to Kaun metrics (@tmattio)
• Add BLEU, ROUGE, and METEOR metrics to Kaun for pre-tokenized sequences, removing tokenizer dependencies (@tmattio)
• Add SSIM, IoU, and Dice metrics for vision workloads in Kaun (@tmattio)

Talon

• Remove automatic sentinel-based null detection for numeric columns; explicit masks (via [_opt] constructors) now define missing data semantics (@tmattio)
• Replace join nested loops with hashed join indices, cutting lookup from O(n·m) to near O(n) (@tmattio)
• Reuse a shared Nx-based column reindexer so filter/sample paths avoid repeated array copies (@tmattio)
• Fix fillna to honor column null masks and replacements, restoring expected nullable semantics (@tmattio)
• Preserve null masks when reindexing during joins so sentinel values remain valid data (@tmattio)
• Handle numeric index columns in pivot, preventing distinct keys from collapsing into a single bucket (@tmattio)
• Respect null masks when serializing numeric columns to JSON, emitting JSON null instead of sentinel values (@tmattio)
• Detect big integers as int64 in Talon CSV loader (fix: Detect big integers as int64 in Talon CSV loader raven-ml/raven#121, @Arsalaan-Alam)
• Allow forcing column types in Talon JSON loader (Allow forcing column types in Talon JSON loader raven-ml/raven#104, @nirnayroy)

Saga

• Remove legacy Normalizers.nmt and Normalizers.precompiled constructors (and their JSON serializers) so the public surface only advertises supported normalizers (@tmattio)
• Tighten template processor JSON parsing: require integer type ids, drop the legacy special-token list format, and ensure multi-id special tokens round-trip with the new record fields (@tmattio)
• Make tokenizer JSON loading tolerant of HuggingFace quirks (missing model.type, string-encoded merges), restoring compatibility with upstream tokenizer.json files (@tmattio)
• Cache byte-level encode/decode lookup tables to avoid rebuilding them during tokenization, trimming avoidable allocations (@tmattio)
• Skip BPE dropout sampling when dropout is disabled, removing redundant RNG work on common hot paths (@tmattio)
• Fix Unigram tokenization so longest matches are emitted without aborting the sequence when a vocab hit occurs (@tmattio)
• Recompute pad token ids when the pad special string changes, preventing padding with stale ids (@tmattio)
• Fix Unigram token_to_id/id_to_token vocabulary lookups (Fix Unigram tokenizer vocabulary lookup (token_to_id and id_to_token) raven-ml/raven#117, @RidwanAdebosin)
• Optimize Pre_tokenizers.whitespace to reduce allocations and improve tokenization performance (@tmattio)
• Simplify tokenizers interface (@tmattio)

Sowilo

• Add resize (nearest & bilinear) that works for 2D, batched, and NHWC tensors (@tmattio)
• Update grayscale conversion and RGB/BGR channel swaps to run entirely on Rune ops, keeping batched inputs compatible with JIT backends (@tmattio)
• Make median_blur compute the true median so salt-and-pepper noise is removed as expected (@tmattio)
• Fix erode/dilate so custom structuring elements (e.g. cross vs. square) and batched tensors produce the correct morphology result (@tmattio)

Fehu

• Added snapshot-based save/load for DQN and REINFORCE agents (Implement save/load for Fehu's DQN agent and add regression tests raven-ml/raven#127, @RidwanAdebosin, @tmattio)
• Added typed Render payloads with enforced render_mode selection in Env.create, auto human-mode rendering, and vectorized Env.render accessors so environments consistently expose frames for downstream tooling (@tmattio)
• Introduced the Fehu_visualize library with ffmpeg/gif/W&B sinks, overlay combinators, rollout/evaluation recorders, and video wrappers for single and vectorized environments, providing a cohesive visualization stack for Fehu (@tmattio)
• Added a Fehu.Policy helper module (random/deterministic/greedy) and sink with_* guards so visualization sinks handle directory creation and cleanup automatically (@tmattio)
• Added Buffer.Replay.sample_tensors to streamline batched training loops and exploration handling (@tmattio)
• Reworked Fehu_algorithms.Dqn around init/step/train primitives with functional state, warmup control, and snapshotting helpers (@tmattio)
• Rebuilt Fehu_algorithms.Reinforce on the same init/step/train interface with optional baselines, tensor-based rollouts, snapshot save/load, and updated tests/examples/docs using the new workflow (@tmattio)
• Upgraded the GridWorld environment to return ANSI and RGB-array frames using the new render types, and updated the DQN example to optionally record pre- and post-training rollouts via FEHU_DQN_RECORD_DIR using Fehu_visualize sinks (@tmattio)
• Reworked space sampling to return (value, next_rng) and split keys internally, fixing correlated draws in Box/Multi-discrete/Tuple/Dict/Sequence/Text samplers while adding Space.boundary_values for deterministic compatibility checks (@tmattio)
• Extended vectorized environments to reuse space boundary probes and now store structured final_observation payloads in Info, improving downstream consumption (@tmattio)
• Added Buffer.Replay.add_many and Buffer.Replay.sample_arrays, preserved backing storage on clear, and exposed struct-of-arrays batches for vectorised learners (@tmattio)
• Tightened Env.create diagnostics with contextual error messages and an optional ~validate_transition hook for custom invariants (@tmattio)
• Enriched Wrapper utilities with map_info, Box clip_action/clip_observation, and time-limit info reporting elapsed steps (@tmattio)
• Upgraded Info values to carry int/float/bool arrays with stable JSON round-tripping (handling NaN/∞) and sorted metadata serialization for deterministic diffs (@tmattio)
• Improved training helpers: Welford-based normalization with optional unbiased variance, documented done = terminated || truncated, and returned nan when explained variance is undefined (@tmattio)
• Treat time-limit truncations as terminals when computing rollout advantages and expose the truncated flag in buffer steps (@tmattio)
• Require callers of Training.compute_gae to pass final bootstrapping values and ensure Training.evaluate feeds the current observation to policies (@tmattio)
• Allow Space.Sequence.create to omit max_length, keeping sequences unbounded above while preserving validation and sampling semantics (@tmattio)
• Validate vectorized environments by round-tripping sample actions/observations across every instance, preventing incompatible spaces from slipping through (@tmattio)
• Finish clipped value loss support in Fehu.Training (Finish clipped value loss support in Fehu.Training raven-ml/raven#119, @nirnayroy)

Nx-datasets

• Migrate to Nx.Cache for cache directory resolution, enabling consistent behavior. (Feature wish: an implementation for op_qr in Rune raven-ml/raven#133, @Arsalaan-Alam)
• Fix cache directory resolution to respect RAVEN_CACHE_ROOT (or fall back to XDG_CACHE_HOME/HOME), allowing custom cache locations. (fix: Let nx-datasets respect XDG cache locations raven-ml/raven#128, @Arsalaan-Alam)
• Switch CIFAR-10 loader to the binary archive so parsing succeeds again (@tmattio)
• Add a CIFAR-10 example (@tmattio)
• Standardize dataset examples on Logs (@tmattio)
• Use Logs for dataset loader logging (Update dataset loaders raven-ml/raven#95, @Satarupa22-SD)

[stepbrobd]

i'll update the pr in nixpkgs (NixOS/nixpkgs#445206) to track alpha 2 by the end of this week

[jmid]

There's also a Base64 lower bound issue in hugin:

#=== ERROR while compiling hugin.1.0.0~alpha2 =================================#
# context              2.5.0~alpha1 | linux/x86_64 | ocaml-base-compiler.5.3.0 | pinned(https://github.com/raven-ml/raven/releases/download/1.0.0_alpha2/raven-1.0.0.alpha2.tbz)
# path                 ~/.opam/5.3/.opam-switch/build/hugin.1.0.0~alpha2
# command              ~/.opam/opam-init/hooks/sandbox.sh build dune build -p hugin -j 71 --promote-install-files=false @install
# exit-code            1
# env-file             ~/.opam/log/hugin-7-a9468b.env
# output-file          ~/.opam/log/hugin-7-a9468b.out
### output ###
# (cd _build/default && /home/opam/.opam/5.3/bin/ocamlc.opt -w -40 -g -bin-annot -bin-annot-occurrences -I hugin/top/.hugin_top.objs/byte -I /home/opam/.opam/5.3/lib/base64 -I /home/opam/.opam/5.3/lib/ocaml/compiler-libs -I hugin/lib/.hugin.objs/byte -H /home/opam/.opam/5.3/lib/bytes -H /home/opam/.opam/5.3/lib/cairo2 -H /home/opam/.opam/5.3/lib/nx -H /home/opam/.opam/5.3/lib/nx/bigarray_ext -H /home/opam/.opam/5.3/lib/nx/c -H /home/opam/.opam/5.3/lib/nx/core -H /home/opam/.opam/5.3/lib/nx/io -H /home/opam/.opam/5.3/lib/nx/io/npy -H /home/opam/.opam/5.3/lib/nx/io/stb_image -H /home/opam/.opam/5.3/lib/nx/io/stb_image_write -H /home/opam/.opam/5.3/lib/nx/pocketfft -H /home/opam/.opam/5.3/lib/nx/safetensors -H /home/opam/.opam/5.3/lib/nx/xdg -H /home/opam/.opam/5.3/lib/nx/zip -H /home/opam/.opam/5.3/lib/ocaml/compiler-libs -H /home/opam/.opam/5.3/lib/ocaml/str -H /home/opam/.opam/5.3/lib/ocaml/unix -H /home/opam/.opam/5.3/lib/stdlib-shims -H hugin/usdl/.usdl.objs/byte -no-alias-deps -o hugin/top/.hugin_top.objs/byte/hugin_top.cmo -c -impl hugin/top/hugin_top.ml)
# File "hugin/top/hugin_top.ml", line 12, characters 20-40:
# 12 |   let base64_data = Base64.encode_string image_data in
#                          ^^^^^^^^^^^^^^^^^^^^
# Error: Unbound module "Base64"

[tmattio]

Thanks @jmid, these should be fixed now.