Update Kaun-CSV loader #114
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@tmattio please let me know if any changes are needed. |
|
Thanks @Satarupa22-SD! I've pushed a change to split the function into two variants: |
|
@tmattio Thank you ! |
tmattio
added a commit
to tmattio/opam-repository
that referenced
this pull request
Nov 3, 2025
CHANGES:
We're excited to announce the release of Raven 1.0.0~alpha2! Less than a month after alpha1, this release notably includes contributions from Outreachy applicants in preparation for the upcoming _two_ internships.
Some highlights from this release include:
- NumPy-compatible text I/O with `Nx_io.{save,load}_text`
- Lots of new functions in Nx/Rune, including neural-net ones `dropout`, `log_softmax`, `batch_norm`, `layer_norm`, and activation functions like `celu` and `celu`, and generic ones like `conjugate`, `index_put`, and more.
- Addition of `.top` libraries for `nx`, `rune`, and `hugin` that auto-install pretty-printers in the OCaml toplevel. You can run e.g. `#require "nx.top"`.
- Addition of a visualization API in Fehu via the new `fehu.visualize` library, supporting video recording.
- Redesign of Kaun core datastructure and checkpointing subsystem for complete snapshotting.
- Many, many bug fixes and correctness improvements.
We've also made numerous performance improvements across the board:
- Nx elementwise ops: 5–50× faster (e.g., Add 50×50 f32 88.81 µs → 1.83 µs, **48×**; Mul 100×100 f32 78.51 µs → 2.41 µs, **33×**).
- Nx conv2d: **4–5×** faster on common shapes; up to **115×** on heavy f64 batched cases (e.g., B16 C64→128 16×16 K3 f64 1.61 s → 13.96 ms).
- Rune autodiff: **1.2–3.7×** faster on core grads (e.g., MatMulGrad Medium 34.04 ms → 11.91 ms, **2.86×**; Large 190.19 ms → 50.97 ms, **3.73×**).
- Talon dataframes: big wins in joins and group-bys (Join 805.35 ms → 26.10 ms, **31×**; Group-by 170.80 ms → 19.03 ms, **9×**; Filter 9.93 ms → 3.39 ms, **3×**).
- Saga tokenizers: realistic workloads **4–17%** faster (e.g., WordPiece encode single 136.05 µs → 115.92 µs, **1.17×**; BPE batch_32 24.52 ms → 22.27 ms, **1.10×**)
We're closing 8 user-reported issues or feature requests and are totalling 30 contributions from 15 unique contributors.
### Nx
- Add `Nx_io.Cache_dir` module with consolidated cache directory utilities respecting `RAVEN_CACHE_ROOT`, `XDG_CACHE_HOME`, and `HOME` fallback, replacing project-specific cache logic across the whole raven ecosystem (raven-ml/raven#134, @Arsalaan-Alam)
- Add `Nx_io.save_txt` / `Nx_io.load_txt` with NumPy-compatible formatting, comments, and dtype support (raven-ml/raven#120, @six-shot)
- Optimize `multi_dot` for matrix chains, reducing intermediate allocations and improving performance (@tmattio)
- Add public `index_put` function for indexed updates (@tmattio)
- Clarify `reshape` documentation to match its view-only semantics (@tmattio)
- Provide `nx.top`, `rune.top`, and `hugin.top` libraries that auto-install pretty printers in the OCaml toplevel and update Quill to load them (@tmattio)
- Add `ifill` for explicit in-place fills and make `fill` return a copied tensor (@tmattio)
- Speed up contiguous elementwise ops via vectorized loops (@tmattio)
- Fast-path contiguous single-axis reductions to avoid iterator fallback (@tmattio)
- Speed up float reductions with contiguous multi-axis fast paths (@tmattio)
- Fast-path padding-free `unfold` to lower conv2d overhead (@tmattio)
- Move neural-network operations (softmax, log_softmax, relu, gelu, silu, sigmoid, tanh) from Kaun to Nx (@tmattio)
- Add public `conjugate` function for complex number conjugation (raven-ml/raven#125, @Arsalaan-Alam)
- Fix complex vdot to conjugate first tensor before multiplication, ensuring correct mathematical behavior (raven-ml/raven#123, @Arsalaan-Alam)
- Update comparison and conditional operations to use boolean tensors (raven-ml/raven#115, @nirnayroy)
- Add support for rcond parameter and underdetermined systems to `lstsq` (raven-ml/raven#102, @Shocker444)
- Fix `matrix_rank`/`pinv` Hermitian fast paths to use eigen-decomposition and match NumPy for complex inputs (raven-ml/raven#96, @six-shot, @tmattio)
- Optimize matmul BLAS dispatch for strided tensors, improving matrix multiplication performance (@tmattio)
- Fix slow builds reported since alpha1 (raven-ml/raven#88, @tmattio)
- Fix macOS ARM crash when loading extended bigarray kinds (@tmattio)
- Add float16 and bfloat16 support to safetensors I/O, including precise conversions that preserve denormals/NaNs (raven-ml/raven#84, @six-shot, @tmattio)
- Refined `View` internals for leaner contiguity checks and stride handling, cutting redundant materialization on hot paths (@tmattio)
- Merge `Lazy_view` into the core `View` API so movement ops operate on a single composed view (@tmattio)
- Documented the reworked `View` interface (@tmattio)
- Documented the `Symbolic_shape` interface (@tmattio)
- Added Accelerate framework flag when compiling on macOS, fixing issues in some environments (raven-ml/raven#129, @nirnayroy)
### Hugin
- Let `Hugin.show` windows close cleanly via the window button or `Esc`/`q`, avoiding frozen macOS REPL sessions (@tmattio)
### Rune
- Add `Rune.no_grad` and `Rune.detach` to mirror JAX stop-gradient semantics (@tmattio)
- Improve gradient performance slightly by replace the reverse-mode tape's linear PhysicalTbl with an identity hash table (@tmattio)
- Fix `Rune.Rng.shuffle` flattening outputs for multi-dimensional tensors; the
shuffle now gathers along axis 0 and keeps shapes intact (@tmattio)
- Replace `Rune.Rng.truncated_normal` clipping with rejection sampling so
samples stay inside the requested interval without boundary spikes (@tmattio)
- Add support for categorical sampling with `Rune.Rng.categorical` (raven-ml/raven#89, @nirnayroy)
- Allow plain `llvm-config` in discovery, fixing build in some platforms (raven-ml/raven#71, @stepbrobd)
### Kaun
- Added Similarity and Polysemy analysis to the BERT example (raven-ml/raven#137, @nirnayroy)
- Support attention masks via the new `Kaun.Attention` module (@tmattio)
- Support loading sharded Hugging Face safetensors (@tmattio)
- Fix BERT and GPT‑2 model loading (@tmattio)
- API simplification: removed type parameters from public types; `Ptree` now supports mixed‑dtype trees via packed tensors with typed getters. (@tmattio)
- Checkpointing overhaul: versioned `Train_state` with schema tagging, explicit `Checkpoint.{Snapshot,Artifact,Manifest,Repository}` (retention, tags, metadata), and simple save/load helpers for snapshots and params. (@tmattio)
- Overhaul dataset combinators: derive tensor specs from Rune dtype, fix sampling/window bugs, validate weighted sampling, and respect `drop_remainder` (@tmattio)
- Make dataset `prefetch` truly asynchronous with background domains and allow reusing an external Domainslib pool via `parallel_map ~pool` (@tmattio)
- Use `Dataset.iter` for epoch batches to reduce overhead (@tmattio)
- Update BERT and GPT-2 tokenizer cache to use `Nx.Cache` for consistent cache directory resolution (raven-ml/raven#133, @Arsalaan-Alam)
- Honor text dataset encodings via incremental Uutf decoding (raven-ml/raven#122, @Satarupa22-SD).
- Preserve empty sequential modules when unflattening so indices stay aligned for checkpoint round-tripping (@tmattio)
- Prevent `Training.fit`/`evaluate` from consuming entire datasets eagerly and fail fast when a dataset yields no batches, avoiding hangs and division-by-zero crashes (@tmattio)
- Allow metric history to tolerate metrics that appear or disappear between epochs so dynamic metric sets no longer raise during training (@tmattio)
- Make `Optimizer.clip_by_global_norm` robust to zero gradients and empty parameter trees to avoid NaNs during training (@tmattio)
- Split CSV loader into `from_csv` and `from_csv_with_labels` to retain labels when requested (raven-ml/raven#114, @Satarupa22-SD)
- Implement AUC-ROC and AUC-PR in Kaun metrics and simplify their signatures (raven-ml/raven#124, raven-ml/raven#131, @Shocker444)
- Add mean absolute percentage error, explained variance, R² (with optional adjustment), KL-divergence, and top-k accuracy to Kaun metrics (@tmattio)
- Add NDCG, MAP, and MRR ranking metrics to Kaun metrics (@tmattio)
- Add BLEU, ROUGE, and METEOR metrics to Kaun for pre-tokenized sequences, removing tokenizer dependencies (@tmattio)
- Add SSIM, IoU, and Dice metrics for vision workloads in Kaun (@tmattio)
### Talon
- Remove automatic sentinel-based null detection for numeric columns; explicit masks (via [_opt] constructors) now define missing data semantics (@tmattio)
- Replace join nested loops with hashed join indices, cutting lookup from O(n·m) to near O(n) (@tmattio)
- Reuse a shared Nx-based column reindexer so filter/sample paths avoid repeated array copies (@tmattio)
- Fix `fillna` to honor column null masks and replacements, restoring expected nullable semantics (@tmattio)
- Preserve null masks when reindexing during joins so sentinel values remain valid data (@tmattio)
- Handle numeric index columns in `pivot`, preventing distinct keys from collapsing into a single bucket (@tmattio)
- Respect null masks when serializing numeric columns to JSON, emitting JSON `null` instead of sentinel values (@tmattio)
- Detect big integers as int64 in Talon CSV loader (raven-ml/raven#121, @Arsalaan-Alam)
- Allow forcing column types in Talon JSON loader (raven-ml/raven#104, @nirnayroy)
### Saga
- Remove legacy `Normalizers.nmt` and `Normalizers.precompiled` constructors (and their JSON serializers) so the public surface only advertises supported normalizers (@tmattio)
- Tighten template processor JSON parsing: require integer type ids, drop the legacy special-token list format, and ensure multi-id special tokens round-trip with the new record fields (@tmattio)
- Make tokenizer JSON loading tolerant of HuggingFace quirks (missing `model.type`, string-encoded merges), restoring compatibility with upstream `tokenizer.json` files (@tmattio)
- Cache byte-level encode/decode lookup tables to avoid rebuilding them during tokenization, trimming avoidable allocations (@tmattio)
- Skip BPE dropout sampling when dropout is disabled, removing redundant RNG work on common hot paths (@tmattio)
- Fix Unigram tokenization so longest matches are emitted without aborting the sequence when a vocab hit occurs (@tmattio)
- Recompute pad token ids when the pad special string changes, preventing padding with stale ids (@tmattio)
- Fix Unigram `token_to_id`/`id_to_token` vocabulary lookups (raven-ml/raven#117, @RidwanAdebosin)
- Optimize `Pre_tokenizers.whitespace` to reduce allocations and improve tokenization performance (@tmattio)
- Simplify tokenizers interface (@tmattio)
### Sowilo
- Add `resize` (nearest & bilinear) that works for 2D, batched, and NHWC tensors (@tmattio)
- Update grayscale conversion and RGB/BGR channel swaps to run entirely on Rune ops, keeping batched inputs compatible with JIT backends (@tmattio)
- Make `median_blur` compute the true median so salt-and-pepper noise is removed as expected (@tmattio)
- Fix `erode`/`dilate` so custom structuring elements (e.g. cross vs. square) and batched tensors produce the correct morphology result (@tmattio)
### Fehu
- Added snapshot-based save/load for DQN and REINFORCE agents (raven-ml/raven#127, @RidwanAdebosin, @tmattio)
- Added typed `Render` payloads with enforced `render_mode` selection in `Env.create`, auto human-mode rendering, and vectorized `Env.render` accessors so environments consistently expose frames for downstream tooling (@tmattio)
- Introduced the `Fehu_visualize` library with ffmpeg/gif/W&B sinks, overlay combinators, rollout/evaluation recorders, and video wrappers for single and vectorized environments, providing a cohesive visualization stack for Fehu (@tmattio)
- Added a `Fehu.Policy` helper module (random/deterministic/greedy) and sink `with_*` guards so visualization sinks handle directory creation and cleanup automatically (@tmattio)
- Added `Buffer.Replay.sample_tensors` to streamline batched training loops and exploration handling (@tmattio)
- Reworked `Fehu_algorithms.Dqn` around `init`/`step`/`train` primitives with functional state, warmup control, and snapshotting helpers (@tmattio)
- Rebuilt `Fehu_algorithms.Reinforce` on the same `init`/`step`/`train` interface with optional baselines, tensor-based rollouts, snapshot save/load, and updated tests/examples/docs using the new workflow (@tmattio)
- Upgraded the GridWorld environment to return ANSI and RGB-array frames using the new render types, and updated the DQN example to optionally record pre- and post-training rollouts via `FEHU_DQN_RECORD_DIR` using `Fehu_visualize` sinks (@tmattio)
- Reworked space sampling to return `(value, next_rng)` and split keys internally, fixing correlated draws in Box/Multi-discrete/Tuple/Dict/Sequence/Text samplers while adding `Space.boundary_values` for deterministic compatibility checks (@tmattio)
- Extended vectorized environments to reuse space boundary probes and now store structured `final_observation` payloads in `Info`, improving downstream consumption (@tmattio)
- Added `Buffer.Replay.add_many` and `Buffer.Replay.sample_arrays`, preserved backing storage on `clear`, and exposed struct-of-arrays batches for vectorised learners (@tmattio)
- Tightened `Env.create` diagnostics with contextual error messages and an optional `~validate_transition` hook for custom invariants (@tmattio)
- Enriched `Wrapper` utilities with `map_info`, Box `clip_action`/`clip_observation`, and time-limit info reporting elapsed steps (@tmattio)
- Upgraded `Info` values to carry int/float/bool arrays with stable JSON round-tripping (handling NaN/∞) and sorted metadata serialization for deterministic diffs (@tmattio)
- Improved training helpers: Welford-based normalization with optional unbiased variance, documented `done = terminated || truncated`, and returned `nan` when explained variance is undefined (@tmattio)
- Treat time-limit truncations as terminals when computing rollout advantages and expose the `truncated` flag in buffer steps (@tmattio)
- Require callers of `Training.compute_gae` to pass final bootstrapping values and ensure `Training.evaluate` feeds the current observation to policies (@tmattio)
- Allow `Space.Sequence.create` to omit `max_length`, keeping sequences unbounded above while preserving validation and sampling semantics (@tmattio)
- Validate vectorized environments by round-tripping sample actions/observations across every instance, preventing incompatible spaces from slipping through (@tmattio)
- Finish clipped value loss support in Fehu.Training (raven-ml/raven#119, @nirnayroy)
### Nx-datasets
- Migrate to `Nx.Cache` for cache directory resolution, enabling consistent behavior. (raven-ml/raven#133, @Arsalaan-Alam)
- Fix cache directory resolution to respect `RAVEN_CACHE_ROOT` (or fall back to `XDG_CACHE_HOME`/`HOME`), allowing custom cache locations. (raven-ml/raven#128, @Arsalaan-Alam)
- Switch CIFAR-10 loader to the binary archive so parsing succeeds again (@tmattio)
- Add a CIFAR-10 example (@tmattio)
- Standardize dataset examples on `Logs` (@tmattio)
- Use `Logs` for dataset loader logging (raven-ml/raven#95, @Satarupa22-SD)
tmattio
added a commit
to tmattio/opam-repository
that referenced
this pull request
Nov 3, 2025
CHANGES:
We're excited to announce the release of Raven 1.0.0~alpha2! Less than a month after alpha1, this release notably includes contributions from Outreachy applicants in preparation for the upcoming _two_ internships.
Some highlights from this release include:
- NumPy-compatible text I/O with `Nx_io.{save,load}_text`
- Lots of new functions in Nx/Rune, including neural-net ones `dropout`, `log_softmax`, `batch_norm`, `layer_norm`, and activation functions like `celu` and `celu`, and generic ones like `conjugate`, `index_put`, and more.
- Addition of `.top` libraries for `nx`, `rune`, and `hugin` that auto-install pretty-printers in the OCaml toplevel. You can run e.g. `#require "nx.top"`.
- Addition of a visualization API in Fehu via the new `fehu.visualize` library, supporting video recording.
- Redesign of Kaun core datastructure and checkpointing subsystem for complete snapshotting.
- Many, many bug fixes and correctness improvements.
We've also made numerous performance improvements across the board:
- Nx elementwise ops: 5–50× faster (e.g., Add 50×50 f32 88.81 µs → 1.83 µs, **48×**; Mul 100×100 f32 78.51 µs → 2.41 µs, **33×**).
- Nx conv2d: **4–5×** faster on common shapes; up to **115×** on heavy f64 batched cases (e.g., B16 C64→128 16×16 K3 f64 1.61 s → 13.96 ms).
- Rune autodiff: **1.2–3.7×** faster on core grads (e.g., MatMulGrad Medium 34.04 ms → 11.91 ms, **2.86×**; Large 190.19 ms → 50.97 ms, **3.73×**).
- Talon dataframes: big wins in joins and group-bys (Join 805.35 ms → 26.10 ms, **31×**; Group-by 170.80 ms → 19.03 ms, **9×**; Filter 9.93 ms → 3.39 ms, **3×**).
- Saga tokenizers: realistic workloads **4–17%** faster (e.g., WordPiece encode single 136.05 µs → 115.92 µs, **1.17×**; BPE batch_32 24.52 ms → 22.27 ms, **1.10×**)
We're closing 8 user-reported issues or feature requests and are totalling 30 community contributions from 8 unique contributors.
### Nx
- Add `Nx_io.Cache_dir` module with consolidated cache directory utilities respecting `RAVEN_CACHE_ROOT`, `XDG_CACHE_HOME`, and `HOME` fallback, replacing project-specific cache logic across the whole raven ecosystem (raven-ml/raven#134, @Arsalaan-Alam)
- Add `Nx_io.save_txt` / `Nx_io.load_txt` with NumPy-compatible formatting, comments, and dtype support (raven-ml/raven#120, @six-shot)
- Optimize `multi_dot` for matrix chains, reducing intermediate allocations and improving performance (@tmattio)
- Add public `index_put` function for indexed updates (@tmattio)
- Clarify `reshape` documentation to match its view-only semantics (@tmattio)
- Provide `nx.top`, `rune.top`, and `hugin.top` libraries that auto-install pretty printers in the OCaml toplevel and update Quill to load them (@tmattio)
- Add `ifill` for explicit in-place fills and make `fill` return a copied tensor (@tmattio)
- Speed up contiguous elementwise ops via vectorized loops (@tmattio)
- Fast-path contiguous single-axis reductions to avoid iterator fallback (@tmattio)
- Speed up float reductions with contiguous multi-axis fast paths (@tmattio)
- Fast-path padding-free `unfold` to lower conv2d overhead (@tmattio)
- Move neural-network operations (softmax, log_softmax, relu, gelu, silu, sigmoid, tanh) from Kaun to Nx (@tmattio)
- Add public `conjugate` function for complex number conjugation (raven-ml/raven#125, @Arsalaan-Alam)
- Fix complex vdot to conjugate first tensor before multiplication, ensuring correct mathematical behavior (raven-ml/raven#123, @Arsalaan-Alam)
- Update comparison and conditional operations to use boolean tensors (raven-ml/raven#115, @nirnayroy)
- Add support for rcond parameter and underdetermined systems to `lstsq` (raven-ml/raven#102, @Shocker444)
- Fix `matrix_rank`/`pinv` Hermitian fast paths to use eigen-decomposition and match NumPy for complex inputs (raven-ml/raven#96, @six-shot, @tmattio)
- Optimize matmul BLAS dispatch for strided tensors, improving matrix multiplication performance (@tmattio)
- Fix slow builds reported since alpha1 (raven-ml/raven#88, @tmattio)
- Fix macOS ARM crash when loading extended bigarray kinds (@tmattio)
- Add float16 and bfloat16 support to safetensors I/O, including precise conversions that preserve denormals/NaNs (raven-ml/raven#84, @six-shot, @tmattio)
- Refined `View` internals for leaner contiguity checks and stride handling, cutting redundant materialization on hot paths (@tmattio)
- Merge `Lazy_view` into the core `View` API so movement ops operate on a single composed view (@tmattio)
- Documented the reworked `View` interface (@tmattio)
- Documented the `Symbolic_shape` interface (@tmattio)
- Added Accelerate framework flag when compiling on macOS, fixing issues in some environments (raven-ml/raven#129, @nirnayroy)
### Hugin
- Let `Hugin.show` windows close cleanly via the window button or `Esc`/`q`, avoiding frozen macOS REPL sessions (@tmattio)
### Rune
- Add `Rune.no_grad` and `Rune.detach` to mirror JAX stop-gradient semantics (@tmattio)
- Improve gradient performance slightly by replace the reverse-mode tape's linear PhysicalTbl with an identity hash table (@tmattio)
- Fix `Rune.Rng.shuffle` flattening outputs for multi-dimensional tensors; the
shuffle now gathers along axis 0 and keeps shapes intact (@tmattio)
- Replace `Rune.Rng.truncated_normal` clipping with rejection sampling so
samples stay inside the requested interval without boundary spikes (@tmattio)
- Add support for categorical sampling with `Rune.Rng.categorical` (raven-ml/raven#89, @nirnayroy)
- Allow plain `llvm-config` in discovery, fixing build in some platforms (raven-ml/raven#71, @stepbrobd)
### Kaun
- Added Similarity and Polysemy analysis to the BERT example (raven-ml/raven#137, @nirnayroy)
- Support attention masks via the new `Kaun.Attention` module (@tmattio)
- Support loading sharded Hugging Face safetensors (@tmattio)
- Fix BERT and GPT‑2 model loading (@tmattio)
- API simplification: removed type parameters from public types; `Ptree` now supports mixed‑dtype trees via packed tensors with typed getters. (@tmattio)
- Checkpointing overhaul: versioned `Train_state` with schema tagging, explicit `Checkpoint.{Snapshot,Artifact,Manifest,Repository}` (retention, tags, metadata), and simple save/load helpers for snapshots and params. (@tmattio)
- Overhaul dataset combinators: derive tensor specs from Rune dtype, fix sampling/window bugs, validate weighted sampling, and respect `drop_remainder` (@tmattio)
- Make dataset `prefetch` truly asynchronous with background domains and allow reusing an external Domainslib pool via `parallel_map ~pool` (@tmattio)
- Use `Dataset.iter` for epoch batches to reduce overhead (@tmattio)
- Update BERT and GPT-2 tokenizer cache to use `Nx.Cache` for consistent cache directory resolution (raven-ml/raven#134, @Arsalaan-Alam)
- Honor text dataset encodings via incremental Uutf decoding (raven-ml/raven#122, @Satarupa22-SD).
- Preserve empty sequential modules when unflattening so indices stay aligned for checkpoint round-tripping (@tmattio)
- Prevent `Training.fit`/`evaluate` from consuming entire datasets eagerly and fail fast when a dataset yields no batches, avoiding hangs and division-by-zero crashes (@tmattio)
- Allow metric history to tolerate metrics that appear or disappear between epochs so dynamic metric sets no longer raise during training (@tmattio)
- Make `Optimizer.clip_by_global_norm` robust to zero gradients and empty parameter trees to avoid NaNs during training (@tmattio)
- Split CSV loader into `from_csv` and `from_csv_with_labels` to retain labels when requested (raven-ml/raven#114, @Satarupa22-SD)
- Implement AUC-ROC and AUC-PR in Kaun metrics and simplify their signatures (raven-ml/raven#124, raven-ml/raven#131, @Shocker444)
- Add mean absolute percentage error, explained variance, R² (with optional adjustment), KL-divergence, and top-k accuracy to Kaun metrics (@tmattio)
- Add NDCG, MAP, and MRR ranking metrics to Kaun metrics (@tmattio)
- Add BLEU, ROUGE, and METEOR metrics to Kaun for pre-tokenized sequences, removing tokenizer dependencies (@tmattio)
- Add SSIM, IoU, and Dice metrics for vision workloads in Kaun (@tmattio)
### Talon
- Remove automatic sentinel-based null detection for numeric columns; explicit masks (via [_opt] constructors) now define missing data semantics (@tmattio)
- Replace join nested loops with hashed join indices, cutting lookup from O(n·m) to near O(n) (@tmattio)
- Reuse a shared Nx-based column reindexer so filter/sample paths avoid repeated array copies (@tmattio)
- Fix `fillna` to honor column null masks and replacements, restoring expected nullable semantics (@tmattio)
- Preserve null masks when reindexing during joins so sentinel values remain valid data (@tmattio)
- Handle numeric index columns in `pivot`, preventing distinct keys from collapsing into a single bucket (@tmattio)
- Respect null masks when serializing numeric columns to JSON, emitting JSON `null` instead of sentinel values (@tmattio)
- Detect big integers as int64 in Talon CSV loader (raven-ml/raven#121, @Arsalaan-Alam)
- Allow forcing column types in Talon JSON loader (raven-ml/raven#104, @nirnayroy)
### Saga
- Remove legacy `Normalizers.nmt` and `Normalizers.precompiled` constructors (and their JSON serializers) so the public surface only advertises supported normalizers (@tmattio)
- Tighten template processor JSON parsing: require integer type ids, drop the legacy special-token list format, and ensure multi-id special tokens round-trip with the new record fields (@tmattio)
- Make tokenizer JSON loading tolerant of HuggingFace quirks (missing `model.type`, string-encoded merges), restoring compatibility with upstream `tokenizer.json` files (@tmattio)
- Cache byte-level encode/decode lookup tables to avoid rebuilding them during tokenization, trimming avoidable allocations (@tmattio)
- Skip BPE dropout sampling when dropout is disabled, removing redundant RNG work on common hot paths (@tmattio)
- Fix Unigram tokenization so longest matches are emitted without aborting the sequence when a vocab hit occurs (@tmattio)
- Recompute pad token ids when the pad special string changes, preventing padding with stale ids (@tmattio)
- Fix Unigram `token_to_id`/`id_to_token` vocabulary lookups (raven-ml/raven#117, @RidwanAdebosin)
- Optimize `Pre_tokenizers.whitespace` to reduce allocations and improve tokenization performance (@tmattio)
- Simplify tokenizers interface (@tmattio)
### Sowilo
- Add `resize` (nearest & bilinear) that works for 2D, batched, and NHWC tensors (@tmattio)
- Update grayscale conversion and RGB/BGR channel swaps to run entirely on Rune ops, keeping batched inputs compatible with JIT backends (@tmattio)
- Make `median_blur` compute the true median so salt-and-pepper noise is removed as expected (@tmattio)
- Fix `erode`/`dilate` so custom structuring elements (e.g. cross vs. square) and batched tensors produce the correct morphology result (@tmattio)
### Fehu
- Added snapshot-based save/load for DQN and REINFORCE agents (raven-ml/raven#127, @RidwanAdebosin, @tmattio)
- Added typed `Render` payloads with enforced `render_mode` selection in `Env.create`, auto human-mode rendering, and vectorized `Env.render` accessors so environments consistently expose frames for downstream tooling (@tmattio)
- Introduced the `Fehu_visualize` library with ffmpeg/gif/W&B sinks, overlay combinators, rollout/evaluation recorders, and video wrappers for single and vectorized environments, providing a cohesive visualization stack for Fehu (@tmattio)
- Added a `Fehu.Policy` helper module (random/deterministic/greedy) and sink `with_*` guards so visualization sinks handle directory creation and cleanup automatically (@tmattio)
- Added `Buffer.Replay.sample_tensors` to streamline batched training loops and exploration handling (@tmattio)
- Reworked `Fehu_algorithms.Dqn` around `init`/`step`/`train` primitives with functional state, warmup control, and snapshotting helpers (@tmattio)
- Rebuilt `Fehu_algorithms.Reinforce` on the same `init`/`step`/`train` interface with optional baselines, tensor-based rollouts, snapshot save/load, and updated tests/examples/docs using the new workflow (@tmattio)
- Upgraded the GridWorld environment to return ANSI and RGB-array frames using the new render types, and updated the DQN example to optionally record pre- and post-training rollouts via `FEHU_DQN_RECORD_DIR` using `Fehu_visualize` sinks (@tmattio)
- Reworked space sampling to return `(value, next_rng)` and split keys internally, fixing correlated draws in Box/Multi-discrete/Tuple/Dict/Sequence/Text samplers while adding `Space.boundary_values` for deterministic compatibility checks (@tmattio)
- Extended vectorized environments to reuse space boundary probes and now store structured `final_observation` payloads in `Info`, improving downstream consumption (@tmattio)
- Added `Buffer.Replay.add_many` and `Buffer.Replay.sample_arrays`, preserved backing storage on `clear`, and exposed struct-of-arrays batches for vectorised learners (@tmattio)
- Tightened `Env.create` diagnostics with contextual error messages and an optional `~validate_transition` hook for custom invariants (@tmattio)
- Enriched `Wrapper` utilities with `map_info`, Box `clip_action`/`clip_observation`, and time-limit info reporting elapsed steps (@tmattio)
- Upgraded `Info` values to carry int/float/bool arrays with stable JSON round-tripping (handling NaN/∞) and sorted metadata serialization for deterministic diffs (@tmattio)
- Improved training helpers: Welford-based normalization with optional unbiased variance, documented `done = terminated || truncated`, and returned `nan` when explained variance is undefined (@tmattio)
- Treat time-limit truncations as terminals when computing rollout advantages and expose the `truncated` flag in buffer steps (@tmattio)
- Require callers of `Training.compute_gae` to pass final bootstrapping values and ensure `Training.evaluate` feeds the current observation to policies (@tmattio)
- Allow `Space.Sequence.create` to omit `max_length`, keeping sequences unbounded above while preserving validation and sampling semantics (@tmattio)
- Validate vectorized environments by round-tripping sample actions/observations across every instance, preventing incompatible spaces from slipping through (@tmattio)
- Finish clipped value loss support in Fehu.Training (raven-ml/raven#119, @nirnayroy)
### Nx-datasets
- Migrate to `Nx.Cache` for cache directory resolution, enabling consistent behavior. (raven-ml/raven#133, @Arsalaan-Alam)
- Fix cache directory resolution to respect `RAVEN_CACHE_ROOT` (or fall back to `XDG_CACHE_HOME`/`HOME`), allowing custom cache locations. (raven-ml/raven#128, @Arsalaan-Alam)
- Switch CIFAR-10 loader to the binary archive so parsing succeeds again (@tmattio)
- Add a CIFAR-10 example (@tmattio)
- Standardize dataset examples on `Logs` (@tmattio)
- Use `Logs` for dataset loader logging (raven-ml/raven#95, @Satarupa22-SD)
tmattio
added a commit
to tmattio/opam-repository
that referenced
this pull request
Nov 4, 2025
CHANGES:
We're excited to announce the release of Raven 1.0.0~alpha2! Less than a month after alpha1, this release notably includes contributions from Outreachy applicants in preparation for the upcoming _two_ internships.
Some highlights from this release include:
- NumPy-compatible text I/O with `Nx_io.{save,load}_text`
- Lots of new functions in Nx/Rune, including neural-net ones `dropout`, `log_softmax`, `batch_norm`, `layer_norm`, and activation functions like `celu` and `celu`, and generic ones like `conjugate`, `index_put`, and more.
- Addition of `.top` libraries for `nx`, `rune`, and `hugin` that auto-install pretty-printers in the OCaml toplevel. You can run e.g. `#require "nx.top"`.
- Addition of a visualization API in Fehu via the new `fehu.visualize` library, supporting video recording.
- Redesign of Kaun core datastructure and checkpointing subsystem for complete snapshotting.
- Many, many bug fixes and correctness improvements.
We've also made numerous performance improvements across the board:
- Nx elementwise ops: 5–50× faster (e.g., Add 50×50 f32 88.81 µs → 1.83 µs, **48×**; Mul 100×100 f32 78.51 µs → 2.41 µs, **33×**).
- Nx conv2d: **4–5×** faster on common shapes; up to **115×** on heavy f64 batched cases (e.g., B16 C64→128 16×16 K3 f64 1.61 s → 13.96 ms).
- Rune autodiff: **1.2–3.7×** faster on core grads (e.g., MatMulGrad Medium 34.04 ms → 11.91 ms, **2.86×**; Large 190.19 ms → 50.97 ms, **3.73×**).
- Talon dataframes: big wins in joins and group-bys (Join 805.35 ms → 26.10 ms, **31×**; Group-by 170.80 ms → 19.03 ms, **9×**; Filter 9.93 ms → 3.39 ms, **3×**).
- Saga tokenizers: realistic workloads **4–17%** faster (e.g., WordPiece encode single 136.05 µs → 115.92 µs, **1.17×**; BPE batch_32 24.52 ms → 22.27 ms, **1.10×**)
We're closing 8 user-reported issues or feature requests and are totalling 30 community contributions from 8 unique contributors.
### Nx
- Fix einsum output axis ordering for free axes (e.g., `i,jk->jki`, `ij,klj->kli`) by correcting final transpose permutation and intermediate left-axis reordering. (@tmattio)
- Add `Nx_io.Cache_dir` module with consolidated cache directory utilities respecting `RAVEN_CACHE_ROOT`, `XDG_CACHE_HOME`, and `HOME` fallback, replacing project-specific cache logic across the whole raven ecosystem (raven-ml/raven#134, @Arsalaan-Alam)
- Add `Nx_io.save_txt` / `Nx_io.load_txt` with NumPy-compatible formatting, comments, and dtype support (raven-ml/raven#120, @six-shot)
- Optimize `multi_dot` for matrix chains, reducing intermediate allocations and improving performance (@tmattio)
- Add public `index_put` function for indexed updates (@tmattio)
- Clarify `reshape` documentation to match its view-only semantics (@tmattio)
- Provide `nx.top`, `rune.top`, and `hugin.top` libraries that auto-install pretty printers in the OCaml toplevel and update Quill to load them (@tmattio)
- Add `ifill` for explicit in-place fills and make `fill` return a copied tensor (@tmattio)
- Speed up contiguous elementwise ops via vectorized loops (@tmattio)
- Fast-path contiguous single-axis reductions to avoid iterator fallback (@tmattio)
- Speed up float reductions with contiguous multi-axis fast paths (@tmattio)
- Fast-path padding-free `unfold` to lower conv2d overhead (@tmattio)
- Move neural-network operations (softmax, log_softmax, relu, gelu, silu, sigmoid, tanh) from Kaun to Nx (@tmattio)
- Add public `conjugate` function for complex number conjugation (raven-ml/raven#125, @Arsalaan-Alam)
- Fix complex vdot to conjugate first tensor before multiplication, ensuring correct mathematical behavior (raven-ml/raven#123, @Arsalaan-Alam)
- Update comparison and conditional operations to use boolean tensors (raven-ml/raven#115, @nirnayroy)
- Add support for rcond parameter and underdetermined systems to `lstsq` (raven-ml/raven#102, @Shocker444)
- Fix `matrix_rank`/`pinv` Hermitian fast paths to use eigen-decomposition and match NumPy for complex inputs (raven-ml/raven#96, @six-shot, @tmattio)
- Optimize matmul BLAS dispatch for strided tensors, improving matrix multiplication performance (@tmattio)
- Fix slow builds reported since alpha1 (raven-ml/raven#88, @tmattio)
- Fix macOS ARM crash when loading extended bigarray kinds (@tmattio)
- Add float16 and bfloat16 support to safetensors I/O, including precise conversions that preserve denormals/NaNs (raven-ml/raven#84, @six-shot, @tmattio)
- Refined `View` internals for leaner contiguity checks and stride handling, cutting redundant materialization on hot paths (@tmattio)
- Merge `Lazy_view` into the core `View` API so movement ops operate on a single composed view (@tmattio)
- Documented the reworked `View` interface (@tmattio)
- Documented the `Symbolic_shape` interface (@tmattio)
- Added Accelerate framework flag when compiling on macOS, fixing issues in some environments (raven-ml/raven#129, @nirnayroy)
### Hugin
- Let `Hugin.show` windows close cleanly via the window button or `Esc`/`q`, avoiding frozen macOS REPL sessions (@tmattio)
### Rune
- Add `Rune.no_grad` and `Rune.detach` to mirror JAX stop-gradient semantics (@tmattio)
- Improve gradient performance slightly by replace the reverse-mode tape's linear PhysicalTbl with an identity hash table (@tmattio)
- Fix `Rune.Rng.shuffle` flattening outputs for multi-dimensional tensors; the
shuffle now gathers along axis 0 and keeps shapes intact (@tmattio)
- Replace `Rune.Rng.truncated_normal` clipping with rejection sampling so
samples stay inside the requested interval without boundary spikes (@tmattio)
- Add support for categorical sampling with `Rune.Rng.categorical` (raven-ml/raven#89, @nirnayroy)
- Allow plain `llvm-config` in discovery, fixing build in some platforms (raven-ml/raven#71, @stepbrobd)
### Kaun
- Added Similarity and Polysemy analysis to the BERT example (raven-ml/raven#137, @nirnayroy)
- Support attention masks via the new `Kaun.Attention` module (@tmattio)
- Support loading sharded Hugging Face safetensors (@tmattio)
- Fix BERT and GPT‑2 model loading (@tmattio)
- API simplification: removed type parameters from public types; `Ptree` now supports mixed‑dtype trees via packed tensors with typed getters. (@tmattio)
- Checkpointing overhaul: versioned `Train_state` with schema tagging, explicit `Checkpoint.{Snapshot,Artifact,Manifest,Repository}` (retention, tags, metadata), and simple save/load helpers for snapshots and params. (@tmattio)
- Overhaul dataset combinators: derive tensor specs from Rune dtype, fix sampling/window bugs, validate weighted sampling, and respect `drop_remainder` (@tmattio)
- Make dataset `prefetch` truly asynchronous with background domains and allow reusing an external Domainslib pool via `parallel_map ~pool` (@tmattio)
- Use `Dataset.iter` for epoch batches to reduce overhead (@tmattio)
- Update BERT and GPT-2 tokenizer cache to use `Nx.Cache` for consistent cache directory resolution (raven-ml/raven#134, @Arsalaan-Alam)
- Honor text dataset encodings via incremental Uutf decoding (raven-ml/raven#122, @Satarupa22-SD).
- Preserve empty sequential modules when unflattening so indices stay aligned for checkpoint round-tripping (@tmattio)
- Prevent `Training.fit`/`evaluate` from consuming entire datasets eagerly and fail fast when a dataset yields no batches, avoiding hangs and division-by-zero crashes (@tmattio)
- Allow metric history to tolerate metrics that appear or disappear between epochs so dynamic metric sets no longer raise during training (@tmattio)
- Make `Optimizer.clip_by_global_norm` robust to zero gradients and empty parameter trees to avoid NaNs during training (@tmattio)
- Split CSV loader into `from_csv` and `from_csv_with_labels` to retain labels when requested (raven-ml/raven#114, @Satarupa22-SD)
- Implement AUC-ROC and AUC-PR in Kaun metrics and simplify their signatures (raven-ml/raven#124, raven-ml/raven#131, @Shocker444)
- Add mean absolute percentage error, explained variance, R² (with optional adjustment), KL-divergence, and top-k accuracy to Kaun metrics (@tmattio)
- Add NDCG, MAP, and MRR ranking metrics to Kaun metrics (@tmattio)
- Add BLEU, ROUGE, and METEOR metrics to Kaun for pre-tokenized sequences, removing tokenizer dependencies (@tmattio)
- Add SSIM, IoU, and Dice metrics for vision workloads in Kaun (@tmattio)
### Talon
- Remove automatic sentinel-based null detection for numeric columns; explicit masks (via [_opt] constructors) now define missing data semantics (@tmattio)
- Replace join nested loops with hashed join indices, cutting lookup from O(n·m) to near O(n) (@tmattio)
- Reuse a shared Nx-based column reindexer so filter/sample paths avoid repeated array copies (@tmattio)
- Fix `fillna` to honor column null masks and replacements, restoring expected nullable semantics (@tmattio)
- Preserve null masks when reindexing during joins so sentinel values remain valid data (@tmattio)
- Handle numeric index columns in `pivot`, preventing distinct keys from collapsing into a single bucket (@tmattio)
- Respect null masks when serializing numeric columns to JSON, emitting JSON `null` instead of sentinel values (@tmattio)
- Detect big integers as int64 in Talon CSV loader (raven-ml/raven#121, @Arsalaan-Alam)
- Allow forcing column types in Talon JSON loader (raven-ml/raven#104, @nirnayroy)
### Saga
- Remove legacy `Normalizers.nmt` and `Normalizers.precompiled` constructors (and their JSON serializers) so the public surface only advertises supported normalizers (@tmattio)
- Tighten template processor JSON parsing: require integer type ids, drop the legacy special-token list format, and ensure multi-id special tokens round-trip with the new record fields (@tmattio)
- Make tokenizer JSON loading tolerant of HuggingFace quirks (missing `model.type`, string-encoded merges), restoring compatibility with upstream `tokenizer.json` files (@tmattio)
- Cache byte-level encode/decode lookup tables to avoid rebuilding them during tokenization, trimming avoidable allocations (@tmattio)
- Skip BPE dropout sampling when dropout is disabled, removing redundant RNG work on common hot paths (@tmattio)
- Fix Unigram tokenization so longest matches are emitted without aborting the sequence when a vocab hit occurs (@tmattio)
- Recompute pad token ids when the pad special string changes, preventing padding with stale ids (@tmattio)
- Fix Unigram `token_to_id`/`id_to_token` vocabulary lookups (raven-ml/raven#117, @RidwanAdebosin)
- Optimize `Pre_tokenizers.whitespace` to reduce allocations and improve tokenization performance (@tmattio)
- Simplify tokenizers interface (@tmattio)
### Sowilo
- Add `resize` (nearest & bilinear) that works for 2D, batched, and NHWC tensors (@tmattio)
- Update grayscale conversion and RGB/BGR channel swaps to run entirely on Rune ops, keeping batched inputs compatible with JIT backends (@tmattio)
- Make `median_blur` compute the true median so salt-and-pepper noise is removed as expected (@tmattio)
- Fix `erode`/`dilate` so custom structuring elements (e.g. cross vs. square) and batched tensors produce the correct morphology result (@tmattio)
### Fehu
- Added snapshot-based save/load for DQN and REINFORCE agents (raven-ml/raven#127, @RidwanAdebosin, @tmattio)
- Added typed `Render` payloads with enforced `render_mode` selection in `Env.create`, auto human-mode rendering, and vectorized `Env.render` accessors so environments consistently expose frames for downstream tooling (@tmattio)
- Introduced the `Fehu_visualize` library with ffmpeg/gif/W&B sinks, overlay combinators, rollout/evaluation recorders, and video wrappers for single and vectorized environments, providing a cohesive visualization stack for Fehu (@tmattio)
- Added a `Fehu.Policy` helper module (random/deterministic/greedy) and sink `with_*` guards so visualization sinks handle directory creation and cleanup automatically (@tmattio)
- Added `Buffer.Replay.sample_tensors` to streamline batched training loops and exploration handling (@tmattio)
- Reworked `Fehu_algorithms.Dqn` around `init`/`step`/`train` primitives with functional state, warmup control, and snapshotting helpers (@tmattio)
- Rebuilt `Fehu_algorithms.Reinforce` on the same `init`/`step`/`train` interface with optional baselines, tensor-based rollouts, snapshot save/load, and updated tests/examples/docs using the new workflow (@tmattio)
- Upgraded the GridWorld environment to return ANSI and RGB-array frames using the new render types, and updated the DQN example to optionally record pre- and post-training rollouts via `FEHU_DQN_RECORD_DIR` using `Fehu_visualize` sinks (@tmattio)
- Reworked space sampling to return `(value, next_rng)` and split keys internally, fixing correlated draws in Box/Multi-discrete/Tuple/Dict/Sequence/Text samplers while adding `Space.boundary_values` for deterministic compatibility checks (@tmattio)
- Extended vectorized environments to reuse space boundary probes and now store structured `final_observation` payloads in `Info`, improving downstream consumption (@tmattio)
- Added `Buffer.Replay.add_many` and `Buffer.Replay.sample_arrays`, preserved backing storage on `clear`, and exposed struct-of-arrays batches for vectorised learners (@tmattio)
- Tightened `Env.create` diagnostics with contextual error messages and an optional `~validate_transition` hook for custom invariants (@tmattio)
- Enriched `Wrapper` utilities with `map_info`, Box `clip_action`/`clip_observation`, and time-limit info reporting elapsed steps (@tmattio)
- Upgraded `Info` values to carry int/float/bool arrays with stable JSON round-tripping (handling NaN/∞) and sorted metadata serialization for deterministic diffs (@tmattio)
- Improved training helpers: Welford-based normalization with optional unbiased variance, documented `done = terminated || truncated`, and returned `nan` when explained variance is undefined (@tmattio)
- Treat time-limit truncations as terminals when computing rollout advantages and expose the `truncated` flag in buffer steps (@tmattio)
- Require callers of `Training.compute_gae` to pass final bootstrapping values and ensure `Training.evaluate` feeds the current observation to policies (@tmattio)
- Allow `Space.Sequence.create` to omit `max_length`, keeping sequences unbounded above while preserving validation and sampling semantics (@tmattio)
- Validate vectorized environments by round-tripping sample actions/observations across every instance, preventing incompatible spaces from slipping through (@tmattio)
- Finish clipped value loss support in Fehu.Training (raven-ml/raven#119, @nirnayroy)
### Nx-datasets
- Migrate to `Nx.Cache` for cache directory resolution, enabling consistent behavior. (raven-ml/raven#133, @Arsalaan-Alam)
- Fix cache directory resolution to respect `RAVEN_CACHE_ROOT` (or fall back to `XDG_CACHE_HOME`/`HOME`), allowing custom cache locations. (raven-ml/raven#128, @Arsalaan-Alam)
- Switch CIFAR-10 loader to the binary archive so parsing succeeds again (@tmattio)
- Add a CIFAR-10 example (@tmattio)
- Standardize dataset examples on `Logs` (@tmattio)
- Use `Logs` for dataset loader logging (raven-ml/raven#95, @Satarupa22-SD)
tmattio
added a commit
to tmattio/opam-repository
that referenced
this pull request
Nov 4, 2025
CHANGES:
We're excited to announce the release of Raven 1.0.0~alpha2! Less than a month after alpha1, this release notably includes contributions from Outreachy applicants in preparation for the upcoming _two_ internships.
Some highlights from this release include:
- NumPy-compatible text I/O with `Nx_io.{save,load}_text`
- Lots of new functions in Nx/Rune, including neural-net ones `dropout`, `log_softmax`, `batch_norm`, `layer_norm`, and activation functions like `celu` and `celu`, and generic ones like `conjugate`, `index_put`, and more.
- Addition of `.top` libraries for `nx`, `rune`, and `hugin` that auto-install pretty-printers in the OCaml toplevel. You can run e.g. `#require "nx.top"`.
- Addition of a visualization API in Fehu via the new `fehu.visualize` library, supporting video recording.
- Redesign of Kaun core datastructure and checkpointing subsystem for complete snapshotting.
- Many, many bug fixes and correctness improvements.
We've also made numerous performance improvements across the board:
- Nx elementwise ops: 5–50× faster (e.g., Add 50×50 f32 88.81 µs → 1.83 µs, **48×**; Mul 100×100 f32 78.51 µs → 2.41 µs, **33×**).
- Nx conv2d: **4–5×** faster on common shapes; up to **115×** on heavy f64 batched cases (e.g., B16 C64→128 16×16 K3 f64 1.61 s → 13.96 ms).
- Rune autodiff: **1.2–3.7×** faster on core grads (e.g., MatMulGrad Medium 34.04 ms → 11.91 ms, **2.86×**; Large 190.19 ms → 50.97 ms, **3.73×**).
- Talon dataframes: big wins in joins and group-bys (Join 805.35 ms → 26.10 ms, **31×**; Group-by 170.80 ms → 19.03 ms, **9×**; Filter 9.93 ms → 3.39 ms, **3×**).
- Saga tokenizers: realistic workloads **4–17%** faster (e.g., WordPiece encode single 136.05 µs → 115.92 µs, **1.17×**; BPE batch_32 24.52 ms → 22.27 ms, **1.10×**)
We're closing 8 user-reported issues or feature requests and are totalling 30 community contributions from 8 unique contributors.
### Nx
- Fix einsum output axis ordering for free axes (e.g., `i,jk->jki`, `ij,klj->kli`) by correcting final transpose permutation and intermediate left-axis reordering. (@tmattio)
- Add `Nx_io.Cache_dir` module with consolidated cache directory utilities respecting `RAVEN_CACHE_ROOT`, `XDG_CACHE_HOME`, and `HOME` fallback, replacing project-specific cache logic across the whole raven ecosystem (raven-ml/raven#134, @Arsalaan-Alam)
- Add `Nx_io.save_txt` / `Nx_io.load_txt` with NumPy-compatible formatting, comments, and dtype support (raven-ml/raven#120, @six-shot)
- Optimize `multi_dot` for matrix chains, reducing intermediate allocations and improving performance (@tmattio)
- Add public `index_put` function for indexed updates (@tmattio)
- Clarify `reshape` documentation to match its view-only semantics (@tmattio)
- Provide `nx.top`, `rune.top`, and `hugin.top` libraries that auto-install pretty printers in the OCaml toplevel and update Quill to load them (@tmattio)
- Add `ifill` for explicit in-place fills and make `fill` return a copied tensor (@tmattio)
- Speed up contiguous elementwise ops via vectorized loops (@tmattio)
- Fast-path contiguous single-axis reductions to avoid iterator fallback (@tmattio)
- Speed up float reductions with contiguous multi-axis fast paths (@tmattio)
- Fast-path padding-free `unfold` to lower conv2d overhead (@tmattio)
- Move neural-network operations (softmax, log_softmax, relu, gelu, silu, sigmoid, tanh) from Kaun to Nx (@tmattio)
- Add public `conjugate` function for complex number conjugation (raven-ml/raven#125, @Arsalaan-Alam)
- Fix complex vdot to conjugate first tensor before multiplication, ensuring correct mathematical behavior (raven-ml/raven#123, @Arsalaan-Alam)
- Update comparison and conditional operations to use boolean tensors (raven-ml/raven#115, @nirnayroy)
- Add support for rcond parameter and underdetermined systems to `lstsq` (raven-ml/raven#102, @Shocker444)
- Fix `matrix_rank`/`pinv` Hermitian fast paths to use eigen-decomposition and match NumPy for complex inputs (raven-ml/raven#96, @six-shot, @tmattio)
- Optimize matmul BLAS dispatch for strided tensors, improving matrix multiplication performance (@tmattio)
- Fix slow builds reported since alpha1 (raven-ml/raven#88, @tmattio)
- Fix macOS ARM crash when loading extended bigarray kinds (@tmattio)
- Add float16 and bfloat16 support to safetensors I/O, including precise conversions that preserve denormals/NaNs (raven-ml/raven#84, @six-shot, @tmattio)
- Refined `View` internals for leaner contiguity checks and stride handling, cutting redundant materialization on hot paths (@tmattio)
- Merge `Lazy_view` into the core `View` API so movement ops operate on a single composed view (@tmattio)
- Documented the reworked `View` interface (@tmattio)
- Documented the `Symbolic_shape` interface (@tmattio)
- Added Accelerate framework flag when compiling on macOS, fixing issues in some environments (raven-ml/raven#129, @nirnayroy)
### Hugin
- Let `Hugin.show` windows close cleanly via the window button or `Esc`/`q`, avoiding frozen macOS REPL sessions (@tmattio)
### Rune
- Add `Rune.no_grad` and `Rune.detach` to mirror JAX stop-gradient semantics (@tmattio)
- Improve gradient performance slightly by replace the reverse-mode tape's linear PhysicalTbl with an identity hash table (@tmattio)
- Fix `Rune.Rng.shuffle` flattening outputs for multi-dimensional tensors; the
shuffle now gathers along axis 0 and keeps shapes intact (@tmattio)
- Replace `Rune.Rng.truncated_normal` clipping with rejection sampling so
samples stay inside the requested interval without boundary spikes (@tmattio)
- Add support for categorical sampling with `Rune.Rng.categorical` (raven-ml/raven#89, @nirnayroy)
- Allow plain `llvm-config` in discovery, fixing build in some platforms (raven-ml/raven#71, @stepbrobd)
### Kaun
- Added Similarity and Polysemy analysis to the BERT example (raven-ml/raven#137, @nirnayroy)
- Support attention masks via the new `Kaun.Attention` module (@tmattio)
- Support loading sharded Hugging Face safetensors (@tmattio)
- Fix BERT and GPT‑2 model loading (@tmattio)
- API simplification: removed type parameters from public types; `Ptree` now supports mixed‑dtype trees via packed tensors with typed getters. (@tmattio)
- Checkpointing overhaul: versioned `Train_state` with schema tagging, explicit `Checkpoint.{Snapshot,Artifact,Manifest,Repository}` (retention, tags, metadata), and simple save/load helpers for snapshots and params. (@tmattio)
- Overhaul dataset combinators: derive tensor specs from Rune dtype, fix sampling/window bugs, validate weighted sampling, and respect `drop_remainder` (@tmattio)
- Make dataset `prefetch` truly asynchronous with background domains and allow reusing an external Domainslib pool via `parallel_map ~pool` (@tmattio)
- Use `Dataset.iter` for epoch batches to reduce overhead (@tmattio)
- Update BERT and GPT-2 tokenizer cache to use `Nx.Cache` for consistent cache directory resolution (raven-ml/raven#134, @Arsalaan-Alam)
- Honor text dataset encodings via incremental Uutf decoding (raven-ml/raven#122, @Satarupa22-SD).
- Preserve empty sequential modules when unflattening so indices stay aligned for checkpoint round-tripping (@tmattio)
- Prevent `Training.fit`/`evaluate` from consuming entire datasets eagerly and fail fast when a dataset yields no batches, avoiding hangs and division-by-zero crashes (@tmattio)
- Allow metric history to tolerate metrics that appear or disappear between epochs so dynamic metric sets no longer raise during training (@tmattio)
- Make `Optimizer.clip_by_global_norm` robust to zero gradients and empty parameter trees to avoid NaNs during training (@tmattio)
- Split CSV loader into `from_csv` and `from_csv_with_labels` to retain labels when requested (raven-ml/raven#114, @Satarupa22-SD)
- Implement AUC-ROC and AUC-PR in Kaun metrics and simplify their signatures (raven-ml/raven#124, raven-ml/raven#131, @Shocker444)
- Add mean absolute percentage error, explained variance, R² (with optional adjustment), KL-divergence, and top-k accuracy to Kaun metrics (@tmattio)
- Add NDCG, MAP, and MRR ranking metrics to Kaun metrics (@tmattio)
- Add BLEU, ROUGE, and METEOR metrics to Kaun for pre-tokenized sequences, removing tokenizer dependencies (@tmattio)
- Add SSIM, IoU, and Dice metrics for vision workloads in Kaun (@tmattio)
### Talon
- Remove automatic sentinel-based null detection for numeric columns; explicit masks (via [_opt] constructors) now define missing data semantics (@tmattio)
- Replace join nested loops with hashed join indices, cutting lookup from O(n·m) to near O(n) (@tmattio)
- Reuse a shared Nx-based column reindexer so filter/sample paths avoid repeated array copies (@tmattio)
- Fix `fillna` to honor column null masks and replacements, restoring expected nullable semantics (@tmattio)
- Preserve null masks when reindexing during joins so sentinel values remain valid data (@tmattio)
- Handle numeric index columns in `pivot`, preventing distinct keys from collapsing into a single bucket (@tmattio)
- Respect null masks when serializing numeric columns to JSON, emitting JSON `null` instead of sentinel values (@tmattio)
- Detect big integers as int64 in Talon CSV loader (raven-ml/raven#121, @Arsalaan-Alam)
- Allow forcing column types in Talon JSON loader (raven-ml/raven#104, @nirnayroy)
### Saga
- Remove legacy `Normalizers.nmt` and `Normalizers.precompiled` constructors (and their JSON serializers) so the public surface only advertises supported normalizers (@tmattio)
- Tighten template processor JSON parsing: require integer type ids, drop the legacy special-token list format, and ensure multi-id special tokens round-trip with the new record fields (@tmattio)
- Make tokenizer JSON loading tolerant of HuggingFace quirks (missing `model.type`, string-encoded merges), restoring compatibility with upstream `tokenizer.json` files (@tmattio)
- Cache byte-level encode/decode lookup tables to avoid rebuilding them during tokenization, trimming avoidable allocations (@tmattio)
- Skip BPE dropout sampling when dropout is disabled, removing redundant RNG work on common hot paths (@tmattio)
- Fix Unigram tokenization so longest matches are emitted without aborting the sequence when a vocab hit occurs (@tmattio)
- Recompute pad token ids when the pad special string changes, preventing padding with stale ids (@tmattio)
- Fix Unigram `token_to_id`/`id_to_token` vocabulary lookups (raven-ml/raven#117, @RidwanAdebosin)
- Optimize `Pre_tokenizers.whitespace` to reduce allocations and improve tokenization performance (@tmattio)
- Simplify tokenizers interface (@tmattio)
### Sowilo
- Add `resize` (nearest & bilinear) that works for 2D, batched, and NHWC tensors (@tmattio)
- Update grayscale conversion and RGB/BGR channel swaps to run entirely on Rune ops, keeping batched inputs compatible with JIT backends (@tmattio)
- Make `median_blur` compute the true median so salt-and-pepper noise is removed as expected (@tmattio)
- Fix `erode`/`dilate` so custom structuring elements (e.g. cross vs. square) and batched tensors produce the correct morphology result (@tmattio)
### Fehu
- Added snapshot-based save/load for DQN and REINFORCE agents (raven-ml/raven#127, @RidwanAdebosin, @tmattio)
- Added typed `Render` payloads with enforced `render_mode` selection in `Env.create`, auto human-mode rendering, and vectorized `Env.render` accessors so environments consistently expose frames for downstream tooling (@tmattio)
- Introduced the `Fehu_visualize` library with ffmpeg/gif/W&B sinks, overlay combinators, rollout/evaluation recorders, and video wrappers for single and vectorized environments, providing a cohesive visualization stack for Fehu (@tmattio)
- Added a `Fehu.Policy` helper module (random/deterministic/greedy) and sink `with_*` guards so visualization sinks handle directory creation and cleanup automatically (@tmattio)
- Added `Buffer.Replay.sample_tensors` to streamline batched training loops and exploration handling (@tmattio)
- Reworked `Fehu_algorithms.Dqn` around `init`/`step`/`train` primitives with functional state, warmup control, and snapshotting helpers (@tmattio)
- Rebuilt `Fehu_algorithms.Reinforce` on the same `init`/`step`/`train` interface with optional baselines, tensor-based rollouts, snapshot save/load, and updated tests/examples/docs using the new workflow (@tmattio)
- Upgraded the GridWorld environment to return ANSI and RGB-array frames using the new render types, and updated the DQN example to optionally record pre- and post-training rollouts via `FEHU_DQN_RECORD_DIR` using `Fehu_visualize` sinks (@tmattio)
- Reworked space sampling to return `(value, next_rng)` and split keys internally, fixing correlated draws in Box/Multi-discrete/Tuple/Dict/Sequence/Text samplers while adding `Space.boundary_values` for deterministic compatibility checks (@tmattio)
- Extended vectorized environments to reuse space boundary probes and now store structured `final_observation` payloads in `Info`, improving downstream consumption (@tmattio)
- Added `Buffer.Replay.add_many` and `Buffer.Replay.sample_arrays`, preserved backing storage on `clear`, and exposed struct-of-arrays batches for vectorised learners (@tmattio)
- Tightened `Env.create` diagnostics with contextual error messages and an optional `~validate_transition` hook for custom invariants (@tmattio)
- Enriched `Wrapper` utilities with `map_info`, Box `clip_action`/`clip_observation`, and time-limit info reporting elapsed steps (@tmattio)
- Upgraded `Info` values to carry int/float/bool arrays with stable JSON round-tripping (handling NaN/∞) and sorted metadata serialization for deterministic diffs (@tmattio)
- Improved training helpers: Welford-based normalization with optional unbiased variance, documented `done = terminated || truncated`, and returned `nan` when explained variance is undefined (@tmattio)
- Treat time-limit truncations as terminals when computing rollout advantages and expose the `truncated` flag in buffer steps (@tmattio)
- Require callers of `Training.compute_gae` to pass final bootstrapping values and ensure `Training.evaluate` feeds the current observation to policies (@tmattio)
- Allow `Space.Sequence.create` to omit `max_length`, keeping sequences unbounded above while preserving validation and sampling semantics (@tmattio)
- Validate vectorized environments by round-tripping sample actions/observations across every instance, preventing incompatible spaces from slipping through (@tmattio)
- Finish clipped value loss support in Fehu.Training (raven-ml/raven#119, @nirnayroy)
### Nx-datasets
- Migrate to `Nx.Cache` for cache directory resolution, enabling consistent behavior. (raven-ml/raven#133, @Arsalaan-Alam)
- Fix cache directory resolution to respect `RAVEN_CACHE_ROOT` (or fall back to `XDG_CACHE_HOME`/`HOME`), allowing custom cache locations. (raven-ml/raven#128, @Arsalaan-Alam)
- Switch CIFAR-10 loader to the binary archive so parsing succeeds again (@tmattio)
- Add a CIFAR-10 example (@tmattio)
- Standardize dataset examples on `Logs` (@tmattio)
- Use `Logs` for dataset loader logging (raven-ml/raven#95, @Satarupa22-SD)
tmattio
added a commit
to tmattio/opam-repository
that referenced
this pull request
Nov 4, 2025
CHANGES:
We're excited to announce the release of Raven 1.0.0~alpha2! Less than a month after alpha1, this release notably includes contributions from Outreachy applicants in preparation for the upcoming _two_ internships.
Some highlights from this release include:
- NumPy-compatible text I/O with `Nx_io.{save,load}_text`
- Lots of new functions in Nx/Rune, including neural-net ones `dropout`, `log_softmax`, `batch_norm`, `layer_norm`, and activation functions like `celu` and `celu`, and generic ones like `conjugate`, `index_put`, and more.
- Addition of `.top` libraries for `nx`, `rune`, and `hugin` that auto-install pretty-printers in the OCaml toplevel. You can run e.g. `#require "nx.top"`.
- Addition of a visualization API in Fehu via the new `fehu.visualize` library, supporting video recording.
- Redesign of Kaun core datastructure and checkpointing subsystem for complete snapshotting.
- Many, many bug fixes and correctness improvements.
We've also made numerous performance improvements across the board:
- Nx elementwise ops: 5–50× faster (e.g., Add 50×50 f32 88.81 µs → 1.83 µs, **48×**; Mul 100×100 f32 78.51 µs → 2.41 µs, **33×**).
- Nx conv2d: **4–5×** faster on common shapes; up to **115×** on heavy f64 batched cases (e.g., B16 C64→128 16×16 K3 f64 1.61 s → 13.96 ms).
- Rune autodiff: **1.2–3.7×** faster on core grads (e.g., MatMulGrad Medium 34.04 ms → 11.91 ms, **2.86×**; Large 190.19 ms → 50.97 ms, **3.73×**).
- Talon dataframes: big wins in joins and group-bys (Join 805.35 ms → 26.10 ms, **31×**; Group-by 170.80 ms → 19.03 ms, **9×**; Filter 9.93 ms → 3.39 ms, **3×**).
- Saga tokenizers: realistic workloads **4–17%** faster (e.g., WordPiece encode single 136.05 µs → 115.92 µs, **1.17×**; BPE batch_32 24.52 ms → 22.27 ms, **1.10×**)
We're closing 8 user-reported issues or feature requests and are totalling 30 community contributions from 8 unique contributors.
### Nx
- Fix einsum output axis ordering for free axes (e.g., `i,jk->jki`, `ij,klj->kli`) by correcting final transpose permutation and intermediate left-axis reordering. (@tmattio)
- Add `Nx_io.Cache_dir` module with consolidated cache directory utilities respecting `RAVEN_CACHE_ROOT`, `XDG_CACHE_HOME`, and `HOME` fallback, replacing project-specific cache logic across the whole raven ecosystem (raven-ml/raven#134, @Arsalaan-Alam)
- Add `Nx_io.save_txt` / `Nx_io.load_txt` with NumPy-compatible formatting, comments, and dtype support (raven-ml/raven#120, @six-shot)
- Optimize `multi_dot` for matrix chains, reducing intermediate allocations and improving performance (@tmattio)
- Add public `index_put` function for indexed updates (@tmattio)
- Clarify `reshape` documentation to match its view-only semantics (@tmattio)
- Provide `nx.top`, `rune.top`, and `hugin.top` libraries that auto-install pretty printers in the OCaml toplevel and update Quill to load them (@tmattio)
- Add `ifill` for explicit in-place fills and make `fill` return a copied tensor (@tmattio)
- Speed up contiguous elementwise ops via vectorized loops (@tmattio)
- Fast-path contiguous single-axis reductions to avoid iterator fallback (@tmattio)
- Speed up float reductions with contiguous multi-axis fast paths (@tmattio)
- Fast-path padding-free `unfold` to lower conv2d overhead (@tmattio)
- Move neural-network operations (softmax, log_softmax, relu, gelu, silu, sigmoid, tanh) from Kaun to Nx (@tmattio)
- Add public `conjugate` function for complex number conjugation (raven-ml/raven#125, @Arsalaan-Alam)
- Fix complex vdot to conjugate first tensor before multiplication, ensuring correct mathematical behavior (raven-ml/raven#123, @Arsalaan-Alam)
- Update comparison and conditional operations to use boolean tensors (raven-ml/raven#115, @nirnayroy)
- Add support for rcond parameter and underdetermined systems to `lstsq` (raven-ml/raven#102, @Shocker444)
- Fix `matrix_rank`/`pinv` Hermitian fast paths to use eigen-decomposition and match NumPy for complex inputs (raven-ml/raven#96, @six-shot, @tmattio)
- Optimize matmul BLAS dispatch for strided tensors, improving matrix multiplication performance (@tmattio)
- Fix slow builds reported since alpha1 (raven-ml/raven#88, @tmattio)
- Fix macOS ARM crash when loading extended bigarray kinds (@tmattio)
- Add float16 and bfloat16 support to safetensors I/O, including precise conversions that preserve denormals/NaNs (raven-ml/raven#84, @six-shot, @tmattio)
- Refined `View` internals for leaner contiguity checks and stride handling, cutting redundant materialization on hot paths (@tmattio)
- Merge `Lazy_view` into the core `View` API so movement ops operate on a single composed view (@tmattio)
- Documented the reworked `View` interface (@tmattio)
- Documented the `Symbolic_shape` interface (@tmattio)
- Added Accelerate framework flag when compiling on macOS, fixing issues in some environments (raven-ml/raven#129, @nirnayroy)
### Hugin
- Let `Hugin.show` windows close cleanly via the window button or `Esc`/`q`, avoiding frozen macOS REPL sessions (@tmattio)
### Rune
- Add `Rune.no_grad` and `Rune.detach` to mirror JAX stop-gradient semantics (@tmattio)
- Improve gradient performance slightly by replace the reverse-mode tape's linear PhysicalTbl with an identity hash table (@tmattio)
- Fix `Rune.Rng.shuffle` flattening outputs for multi-dimensional tensors; the
shuffle now gathers along axis 0 and keeps shapes intact (@tmattio)
- Replace `Rune.Rng.truncated_normal` clipping with rejection sampling so
samples stay inside the requested interval without boundary spikes (@tmattio)
- Add support for categorical sampling with `Rune.Rng.categorical` (raven-ml/raven#89, @nirnayroy)
- Allow plain `llvm-config` in discovery, fixing build in some platforms (raven-ml/raven#71, @stepbrobd)
### Kaun
- Added Similarity and Polysemy analysis to the BERT example (raven-ml/raven#137, @nirnayroy)
- Support attention masks via the new `Kaun.Attention` module (@tmattio)
- Support loading sharded Hugging Face safetensors (@tmattio)
- Fix BERT and GPT‑2 model loading (@tmattio)
- API simplification: removed type parameters from public types; `Ptree` now supports mixed‑dtype trees via packed tensors with typed getters. (@tmattio)
- Checkpointing overhaul: versioned `Train_state` with schema tagging, explicit `Checkpoint.{Snapshot,Artifact,Manifest,Repository}` (retention, tags, metadata), and simple save/load helpers for snapshots and params. (@tmattio)
- Overhaul dataset combinators: derive tensor specs from Rune dtype, fix sampling/window bugs, validate weighted sampling, and respect `drop_remainder` (@tmattio)
- Make dataset `prefetch` truly asynchronous with background domains and allow reusing an external Domainslib pool via `parallel_map ~pool` (@tmattio)
- Use `Dataset.iter` for epoch batches to reduce overhead (@tmattio)
- Update BERT and GPT-2 tokenizer cache to use `Nx.Cache` for consistent cache directory resolution (raven-ml/raven#134, @Arsalaan-Alam)
- Honor text dataset encodings via incremental Uutf decoding (raven-ml/raven#122, @Satarupa22-SD).
- Preserve empty sequential modules when unflattening so indices stay aligned for checkpoint round-tripping (@tmattio)
- Prevent `Training.fit`/`evaluate` from consuming entire datasets eagerly and fail fast when a dataset yields no batches, avoiding hangs and division-by-zero crashes (@tmattio)
- Allow metric history to tolerate metrics that appear or disappear between epochs so dynamic metric sets no longer raise during training (@tmattio)
- Make `Optimizer.clip_by_global_norm` robust to zero gradients and empty parameter trees to avoid NaNs during training (@tmattio)
- Split CSV loader into `from_csv` and `from_csv_with_labels` to retain labels when requested (raven-ml/raven#114, @Satarupa22-SD)
- Implement AUC-ROC and AUC-PR in Kaun metrics and simplify their signatures (raven-ml/raven#124, raven-ml/raven#131, @Shocker444)
- Add mean absolute percentage error, explained variance, R² (with optional adjustment), KL-divergence, and top-k accuracy to Kaun metrics (@tmattio)
- Add NDCG, MAP, and MRR ranking metrics to Kaun metrics (@tmattio)
- Add BLEU, ROUGE, and METEOR metrics to Kaun for pre-tokenized sequences, removing tokenizer dependencies (@tmattio)
- Add SSIM, IoU, and Dice metrics for vision workloads in Kaun (@tmattio)
### Talon
- Remove automatic sentinel-based null detection for numeric columns; explicit masks (via [_opt] constructors) now define missing data semantics (@tmattio)
- Replace join nested loops with hashed join indices, cutting lookup from O(n·m) to near O(n) (@tmattio)
- Reuse a shared Nx-based column reindexer so filter/sample paths avoid repeated array copies (@tmattio)
- Fix `fillna` to honor column null masks and replacements, restoring expected nullable semantics (@tmattio)
- Preserve null masks when reindexing during joins so sentinel values remain valid data (@tmattio)
- Handle numeric index columns in `pivot`, preventing distinct keys from collapsing into a single bucket (@tmattio)
- Respect null masks when serializing numeric columns to JSON, emitting JSON `null` instead of sentinel values (@tmattio)
- Detect big integers as int64 in Talon CSV loader (raven-ml/raven#121, @Arsalaan-Alam)
- Allow forcing column types in Talon JSON loader (raven-ml/raven#104, @nirnayroy)
### Saga
- Remove legacy `Normalizers.nmt` and `Normalizers.precompiled` constructors (and their JSON serializers) so the public surface only advertises supported normalizers (@tmattio)
- Tighten template processor JSON parsing: require integer type ids, drop the legacy special-token list format, and ensure multi-id special tokens round-trip with the new record fields (@tmattio)
- Make tokenizer JSON loading tolerant of HuggingFace quirks (missing `model.type`, string-encoded merges), restoring compatibility with upstream `tokenizer.json` files (@tmattio)
- Cache byte-level encode/decode lookup tables to avoid rebuilding them during tokenization, trimming avoidable allocations (@tmattio)
- Skip BPE dropout sampling when dropout is disabled, removing redundant RNG work on common hot paths (@tmattio)
- Fix Unigram tokenization so longest matches are emitted without aborting the sequence when a vocab hit occurs (@tmattio)
- Recompute pad token ids when the pad special string changes, preventing padding with stale ids (@tmattio)
- Fix Unigram `token_to_id`/`id_to_token` vocabulary lookups (raven-ml/raven#117, @RidwanAdebosin)
- Optimize `Pre_tokenizers.whitespace` to reduce allocations and improve tokenization performance (@tmattio)
- Simplify tokenizers interface (@tmattio)
### Sowilo
- Add `resize` (nearest & bilinear) that works for 2D, batched, and NHWC tensors (@tmattio)
- Update grayscale conversion and RGB/BGR channel swaps to run entirely on Rune ops, keeping batched inputs compatible with JIT backends (@tmattio)
- Make `median_blur` compute the true median so salt-and-pepper noise is removed as expected (@tmattio)
- Fix `erode`/`dilate` so custom structuring elements (e.g. cross vs. square) and batched tensors produce the correct morphology result (@tmattio)
### Fehu
- Added snapshot-based save/load for DQN and REINFORCE agents (raven-ml/raven#127, @RidwanAdebosin, @tmattio)
- Added typed `Render` payloads with enforced `render_mode` selection in `Env.create`, auto human-mode rendering, and vectorized `Env.render` accessors so environments consistently expose frames for downstream tooling (@tmattio)
- Introduced the `Fehu_visualize` library with ffmpeg/gif/W&B sinks, overlay combinators, rollout/evaluation recorders, and video wrappers for single and vectorized environments, providing a cohesive visualization stack for Fehu (@tmattio)
- Added a `Fehu.Policy` helper module (random/deterministic/greedy) and sink `with_*` guards so visualization sinks handle directory creation and cleanup automatically (@tmattio)
- Added `Buffer.Replay.sample_tensors` to streamline batched training loops and exploration handling (@tmattio)
- Reworked `Fehu_algorithms.Dqn` around `init`/`step`/`train` primitives with functional state, warmup control, and snapshotting helpers (@tmattio)
- Rebuilt `Fehu_algorithms.Reinforce` on the same `init`/`step`/`train` interface with optional baselines, tensor-based rollouts, snapshot save/load, and updated tests/examples/docs using the new workflow (@tmattio)
- Upgraded the GridWorld environment to return ANSI and RGB-array frames using the new render types, and updated the DQN example to optionally record pre- and post-training rollouts via `FEHU_DQN_RECORD_DIR` using `Fehu_visualize` sinks (@tmattio)
- Reworked space sampling to return `(value, next_rng)` and split keys internally, fixing correlated draws in Box/Multi-discrete/Tuple/Dict/Sequence/Text samplers while adding `Space.boundary_values` for deterministic compatibility checks (@tmattio)
- Extended vectorized environments to reuse space boundary probes and now store structured `final_observation` payloads in `Info`, improving downstream consumption (@tmattio)
- Added `Buffer.Replay.add_many` and `Buffer.Replay.sample_arrays`, preserved backing storage on `clear`, and exposed struct-of-arrays batches for vectorised learners (@tmattio)
- Tightened `Env.create` diagnostics with contextual error messages and an optional `~validate_transition` hook for custom invariants (@tmattio)
- Enriched `Wrapper` utilities with `map_info`, Box `clip_action`/`clip_observation`, and time-limit info reporting elapsed steps (@tmattio)
- Upgraded `Info` values to carry int/float/bool arrays with stable JSON round-tripping (handling NaN/∞) and sorted metadata serialization for deterministic diffs (@tmattio)
- Improved training helpers: Welford-based normalization with optional unbiased variance, documented `done = terminated || truncated`, and returned `nan` when explained variance is undefined (@tmattio)
- Treat time-limit truncations as terminals when computing rollout advantages and expose the `truncated` flag in buffer steps (@tmattio)
- Require callers of `Training.compute_gae` to pass final bootstrapping values and ensure `Training.evaluate` feeds the current observation to policies (@tmattio)
- Allow `Space.Sequence.create` to omit `max_length`, keeping sequences unbounded above while preserving validation and sampling semantics (@tmattio)
- Validate vectorized environments by round-tripping sample actions/observations across every instance, preventing incompatible spaces from slipping through (@tmattio)
- Finish clipped value loss support in Fehu.Training (raven-ml/raven#119, @nirnayroy)
### Nx-datasets
- Migrate to `Nx.Cache` for cache directory resolution, enabling consistent behavior. (raven-ml/raven#133, @Arsalaan-Alam)
- Fix cache directory resolution to respect `RAVEN_CACHE_ROOT` (or fall back to `XDG_CACHE_HOME`/`HOME`), allowing custom cache locations. (raven-ml/raven#128, @Arsalaan-Alam)
- Switch CIFAR-10 loader to the binary archive so parsing succeeds again (@tmattio)
- Add a CIFAR-10 example (@tmattio)
- Standardize dataset examples on `Logs` (@tmattio)
- Use `Logs` for dataset loader logging (raven-ml/raven#95, @Satarupa22-SD)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
updated the
from_csvfunction to support label_columns. Added testcases and updated the metadata to support the dataset file.#94