Version 1.9.0

[jpivarski]

These are differences from 1.8.0 to 1.9.0. Most only affect Awkward Array version 2, in the awkward._v2 submodule of 1.9.0. The exceptions are called out with "v1" in the subject line.

Features

• feat: C++ refactoring: ak.to_pandas by @ioanaif in C++ refactoring: ak.to_pandas #1369
• feat: C++ refactoring: ak.nan_to_num by @ioanaif in C++ refactoring: ak.nan_to_num #1352
• feat: C++ refactoring: ak.run_lengths by @ioanaif in C++ refactoring: ak.run_lengths #1347
• feat: add is_tuple describe operation by @agoose77 in Feat: add is_tuple describe operation #1351
• feat: C++ refactoring: ak.unzip by @ioanaif in C++ refactoring: ak.unzip #1354
• feat: pure Cling demo and improvements to C++ JIT infrastructure by @jpivarski in Pure Cling demo and improvements to C++ JIT infrastructure #1359
• feat: C++ refactoring: ak.broadcast_arrays by @ioanaif in C++ refactoring: ak.broadcast_arrays #1368
• feat: irst version of ak._v2.from_parquet by @jpivarski in First version of ak._v2.from_parquet #1338
• feat: C++ refactoring: ak.copy by @ioanaif in C++ refactoring: ak.copy #1367
• feat: C++ refactoring: ak.unflatten by @ioanaif in C++ refactoring: ak.unflatten #1360
• feat: add depth_limit to ak.broadcast_arrays by @agoose77 in Feat: add depth_limit to ak.broadcast_arrays #1373
• feat: Allow NumPy arrays in CppStatements; fix row_groups in single-file from_parquet. by @jpivarski in Allow NumPy arrays in CppStatements; fix row_groups in single-file from_parquet. #1376
• feat: This PR sets up the architecture to call CuPy Raw Kernels from Awkward. by @swishdiff in This PR sets up the architecture to call CuPy Raw Kernels from Awkward. #1355
• feat: Reducers with axis=None and typetracers. by @jpivarski in Reducers with axis=None and typetracers. #1380
• feat: This PR adds the generated kernels and simplifies the template specialization generation process. by @swishdiff in This PR adds the generated kernels and simplifies the template specialization generation process. #1381
• feat: High-level ak._v2.Array clean-ups. by @jpivarski in High-level ak._v2.Array clean-ups. #1392
• feat: This PR adds all the remaining kernels in the studies directory by @swishdiff in This PR adds all the remaining kernels in the studies directory #1390
• feat: This PR adds JAX as a new nplike by @swishdiff in This PR adds JAX as a new nplike #1399
• feat: Passing behaviour in ak._v2 functions by @ioanaif in Passing behaviour in ak._v2 functions #1415
• feat: Enable broadcasting of string equality. by @jpivarski in Enable broadcasting of string equality. #1427
• feat: Enabled string/categorical behavior by @ioanaif in Enabled string/categorical behavior #1421
• feat: Implements ak.nan_to_none and all of the ak.nan* functions to override NumPy's. by @jpivarski in Implements ak.nan_to_none and all of the ak.nan* functions to override NumPy's. #1428
• feat: Pretty-printing types by @jpivarski in Pretty-printing types #1430
• feat: Register both v1 and v2 Arrays in Numba entry_points. by @jpivarski in Register both v1 and v2 Arrays in Numba entry_points. #1432
• feat: Add Array and Record.delitem. And fix show(type=True). by @jpivarski in Add Array and Record.__delitem__. And fix show(type=True). #1433
• feat: Enable categorical behavior - testing by @ioanaif in Enable categorical behavior - testing #1434
• feat: Enable mixins behavior by @ioanaif in Enable mixins behavior #1437
• feat: Implementing ak._v2.to_parquet. by @jpivarski in Implementing ak._v2.to_parquet. #1440
• feat: awkward to rdataframe by @ianna in awkward to rdataframe #1374
• feat: Enable ak.firsts by @ioanaif in Enable ak.firsts  #1443
• feat: Enable ak.singletons by @ioanaif in Enable ak.singletons #1444
• feat: Revamping the to_json/from_json interface. by @jpivarski in Revamping the to_json/from_json interface. #1449
• feat: This PR attempts to add autodifferentiation support for Awkward Arrays using JAX pytrees. by @swishdiff in This PR attempts to add autodifferentiation support for Awkward Arrays using JAX pytrees. #1447
• feat: Adding a Forth Based Avro Reader by @aryan26roy in Adding a Forth Based Avro Reader #1491
• feat: Adding repr overriden behavior by @ioanaif in Adding repr overriden behavior #1487
• feat: rdataframe to awkward by @ianna in rdataframe to awkward #1474
• feat: add missing _like methods to TypeTracer by @agoose77 in Feat: add missing _like methods to TypeTracer #1505
• feat: from rdataframe for awkward arrays by @ianna in from rdataframe for awkward arrays #1508
• feat: Add typeparser to v2. by @jpivarski in Add typeparser to v2. #1513
• feat: add C++ headers-only distribution configuration by @ianna in add C++ headers-only distribution configuration #1523
• feat: Growable Buffer header by @ManasviGoyal in Growable Buffer header #1535
• feat: parquet redux by @martindurant in parquet redux #1476
• feat: Templated LayoutBuilder by @ManasviGoyal in Templated LayoutBuilder #1494
• feat: raise AttributeError for public Array attributes by @agoose77 in Feat: raise AttributeError for public Array attributes #1573
• feat: Public interface for layout.recursively_apply and broadcast_and_apply. by @jpivarski in Public interface for layout.recursively_apply and broadcast_and_apply. #1610
• feat: prevent reducers like ak.sum on records (v2) by @ioanaif in feat: prevent reducers like ak.sum on records (v2) #1607
• feat: Drop ak.behavior['.', 'Name'] = cls, which isn't working/isn't tested. by @jpivarski in feat: Drop ak.behavior['.', 'Name'] = cls, which isn't working/isn't tested. #1651
• feat: retrieve multiple columns from RDataFrame in a single event loop by @ianna in feat: retrieve multiple columns from RDataFrame in a single event loop #1625

Bug-fixes

• fix: Fixed ak.num with axis=0 in typetracer. by @jpivarski in Fixed ak.num with axis=0 in typetracer. #1329
• fix: ak.flatten and ak.ravel should test for nplike.ndarray, not np.ndarray. by @jpivarski in ak.flatten and ak.ravel should test for nplike.ndarray, not np.ndarray. #1340
• fix: Straighten out error handling via a thread-local (but otherwise global) context by @jpivarski in Straighten out error handling via a thread-local (but otherwise global) context #1327
• fix: Fix PR Differing behavior depending on placement of ak.with_field #788: avoid materializing VirtualArrays in ak.with_name. by @jpivarski in Fix PR #788: avoid materializing VirtualArrays in ak.with_name. #1346
• fix: fix docstring line in is_tuple (v1) by @agoose77 in Fix: fix docstring line in is_tuple (v1) #1356
• fix: fixes ak.zip does not accept generator expressions for arrays #1363 by ensuring that arguments documented as 'iterable of X' aren't used in 'len(X)'. by @jpivarski in Fixes #1363 by ensuring that arguments documented as 'iterable of X' aren't used in 'len(X)'. #1364
• fix: Allow NumPy arrays in CppStatements; fix row_groups in single-file from_parquet. by @jpivarski in Allow NumPy arrays in CppStatements; fix row_groups in single-file from_parquet. #1372
• fix: Fixes nonlocal reducers in which the first list is empty. by @jpivarski in Fixes nonlocal reducers in which the first list is empty. #1378
• fix: Fix _prettyprint after 'for i in range' changed to 'for i, val in enumerate'. by @jpivarski in Fix _prettyprint after 'for i in range' changed to 'for i, val in enumerate'. #1384
• fix: bump black to 22.3.0 due to click 8.1 release by @henryiii in fix: bump black to 22.3.0 due to click 8.1 release #1385
• fix: No zero-length shortcuts for ak.argsort (v1 & v2). by @jpivarski in No zero-length shortcuts for ak.argsort (v1 & v2). #1387
• fix: ErrorContexts should only contain strings. by @jpivarski in ErrorContexts should only contain strings. #1393
• fix: ROOT doesn't recognize for-each iterators without operator== by @jpivarski in ROOT doesn't recognize for-each iterators without operator== #1398
• fix: Implement recursively_apply for Record by @agoose77 in Implement recursively_apply for Record #1401
• fix: from_numpy references ListArray64 by @agoose77 in Fix: from_numpy references ListArray64 #1404
• fix: Windows builds stopped working; be looser about directory name. by @jpivarski in Windows builds stopped working; be looser about directory name. #1407
• fix: pypy 3.9 by @henryiii in fix: pypy 3.9 #1412
• fix: replace llvmlite.ir instead of llvmlite.llvmpy.core by @Ahmad-AlSubaie in replace llvmlite.ir instead of llvmlite.llvmpy.core #1413
• fix: Fix performance issue in v2 tolist. by @jpivarski in Fix performance issue in v2 tolist. #1418
• fix: Fix iteration over NumpyArray type. by @jpivarski in Fix iteration over NumpyArray type. #1419
• fix: Removed bytemask() in favour of mask_as_bool() by @ioanaif in Removed bytemask() in favour of mask_as_bool() #1410
• fix: Fix ak._v2.to_arrow for sliced ListOffsetArray. by @jpivarski in Fix ak._v2.to_arrow for sliced ListOffsetArray. #1425
• fix: ListOffsetArray._reduce_next is not implemented for 32-bit. by @jpivarski in ListOffsetArray._reduce_next is not implemented for 32-bit. #1426
• fix: Fixed miscellaneous optiontype-in-Parquet bugs by @jpivarski in Fixed miscellaneous optiontype-in-Parquet bugs #1431
• fix: Remove duplicated import of to/from-parquet by @douglasdavis in Remove duplicated import of to/from-parquet #1435
• fix: is_unique() for IndexedArray by @ioanaif in Fix: is_unique() for IndexedArray #1429
• fix: Fixes to_layout with allow_records=False and allows single-record writing to Arrow and Parquet by @jpivarski in Fixes to_layout with allow_records=False and allows single-record writing to Arrow and Parquet #1456
• fix: Fix RDataFrame GetColumnNames order in test. by @jpivarski in Fix RDataFrame GetColumnNames order in test. #1457
• fix: Streamline recursively_apply for small slices of big arrays. by @jpivarski in Streamline recursively_apply for small slices of big arrays. #1458
• fix: getitem* functions must consistently set the slicer in handle_error. by @jpivarski in _getitem_* functions must consistently set the slicer in handle_error. #1469
• fix: to_rdataframe extensive tests and bug fixes by @ianna in to_rdataframe extensive tests and bug fixes #1478
• fix: Fix selecting columns from Parquet. by @jpivarski in Fix selecting columns from Parquet. #1486
• fix: Fix categorical equality handling (bad copy-paste from v1). by @jpivarski in Fix categorical equality handling (bad copy-paste from v1). #1492
• fix: _to_numpy method should return a numpy array by @swishdiff in _to_numpy method should return a numpy array #1496
• fix: Fix/Fixed slicing shape for array of booleans by @ioanaif in Fix/Fixed slicing shape for array of booleans #1497
• fix: Fixed typo in unmaskedarray.py. by @jpivarski in Fixed typo in unmaskedarray.py. #1498
• fix: Fix slicing for UnmaskedArrays (which come from Arrow). by @jpivarski in Fix slicing for UnmaskedArrays (which come from Arrow). #1499
• fix: Fix for issue 1406 by @ioanaif in Fix for issue 1406 #1502
• fix: Update type-parser for v2 by @jpivarski in Update type-parser for v2 #1514
• fix: numba pre-commit issues by @ioanaif in Fix: numba pre-commit issues #1533
• fix: Initialize values behind the mask in ak.to_numpy by @ioanaif in Fix: Initialize values behind the mask in ak.to_numpy #1531
• fix: Fixed RecordArray.repr (last vestige of 'override' misunderstanding). by @jpivarski in Fixed RecordArray.__repr__ (last vestige of 'override' misunderstanding). #1537
• fix: Error when using ak.copy in v2 by @ioanaif in Fix: Error when using ak.copy in v2 #1532
• fix: ak._v2.is_none check for axis value by @ioanaif in Fix: ak._v2.is_none check for axis value  #1539
• fix: Solving the endian bug on MacOS by @aryan26roy in Solving the endian bug on MacOS #1538
• fix: fix ak2 convert class name msg by @Moelf in fix ak2 convert class name msg #1544
• fix: GrowableBuffer.h's missing import broke localbuild.py; fixing it. by @jpivarski in GrowableBuffer.h's missing <cstring> import broke localbuild.py; fixing it. #1555
• fix: Remove std::cout from ArrayBuilder code. by @jpivarski in Remove std::cout from ArrayBuilder code. #1561
• fix: Missing axis_wrap_if_negative Record method in both v1 and v2 by @ioanaif in Fix: Missing axis_wrap_if_negative Record method in both v1 and v2 #1565
• fix: ufuncs on records should not be allowed unless overridden by @ioanaif in Fix: ufuncs on records should not be allowed unless overridden #1559
• fix: add int64_t definition for Windows by @ianna in add int64_t definition for Windows #1572
• fix: allow empty RecordArrays in ak.to_layout by @agoose77 in Fix: allow empty RecordArrays in ak.to_layout #1581
• fix: tolist() bug from Uproot (longlong type for Index) by @ioanaif in Fix: tolist() bug from Uproot (longlong type for Index)  #1567
• fix: Lengths of empty regular slices by @ioanaif in Fix: Lengths of empty regular slices #1568
• fix: RegularArray: maybe_toNumpy() by @ioanaif in Fix: RegularArray: maybe_toNumpy() #1589
• fix: Raises an exception on mixed Awkward/NumPy slices. by @jpivarski in Raises an exception on mixed Awkward/NumPy slices. #1598
• fix: ak.concatenate (mergemany) should preserve regular-type. by @jpivarski in ak.concatenate (mergemany) should preserve regular-type. #1604
• fix: ak.concatenate of identical Forms preserves the Form by @jpivarski in ak.concatenate of identical Forms preserves the Form #1605
• fix: ak.concatenate should preserve regular-type for axis>0, too. by @jpivarski in ak.concatenate should preserve regular-type for axis>0, too. #1609
• fix: improve ak.from_iter performance for long (axis=0) arrays by @agoose77 in fix: improve ak.from_iter performance for long (axis=0) arrays #1614
• fix: generator tolayout in cling by @ianna in fix: generator tolayout in cling #1613
• fix: Use dtype=np.int64, not int, for platform independence (Windows). by @jpivarski in fix: Use dtype=np.int64, not int, for platform independence (Windows). #1636
• fix: support root empty field in Parquet file by @agoose77 in fix: support root empty field in Parquet file #1619
• fix: empty slice lists of record arrays (RecordArray loses its field in a slice that doesn't select fields (v2) #1593) by @agoose77 in fix: empty slice lists of record arrays (#1593) #1597
• fix: ak.from_iter should interpret top-level tuples as ak.Array (v1 and v2). by @jpivarski in fix: ak.from_iter should interpret top-level tuples as ak.Array (v1 and v2). #1642
• fix: Ensure that ak._v2.to_json raises errors when appropriate. by @jpivarski in fix: Ensure that ak._v2.to_json raises errors when appropriate. #1649
• fix: pass a copy of RecordArray's internal fields in HL API by @Saransh-cpp in fix: pass a copy of RecordArray's internal fields in HL API #1650
• fix: include self._length in RegularArray.mergemany by @agoose77 in fix: include self._length in RegularArray.mergemany #1645
• fix: jagged slicing for ListArray by @agoose77 in fix: jagged slicing for ListArray #1408
• fix: to_list must follow getitem implementations, even in Records by @jpivarski in fix: to_list must follow __getitem__ implementations, even in Records #1652

Other

• perf: ak.concatenate should flatten for first axis, single-array by @agoose77 in perf: ak.concatenate should flatten for first axis, single-array #1641
• refactor: cleanup reducer by @agoose77 in Refactor: cleanup reducer #1365
• refactor: split up functions by @martindurant in split up functions #1397
• refactor: Flatten directory structure under src/awkward/_v2/operations. by @jpivarski in Flatten directory structure under src/awkward/_v2/operations. #1467
• refactor: Rename low-level methods to match high-level function names. by @jpivarski in Rename low-level methods to match high-level function names. #1468
• refactor: Rename fillna -> fill_none. by @jpivarski in Rename fillna -> fill_none. #1470
• refactor: Refactoring to include index_nplike and reducers by @swishdiff in Refactoring to include index_nplike and reducers #1490
• refactor: LayoutBuilder migration to v2 by @ianna in LayoutBuilder migration to v2 #1484
• refactor: restructure cpp headers by @ianna in restructure cpp headers #1524
• refactor: migrate ArrayBuilder to new GrowableBuffer by @ianna in migrate ArrayBuilder to new GrowableBuffer #1542
• refactor: builder options and layout builder updates by @ianna in builder options and layout builder updates #1560
• refactor: Renamed ak.to_pandas -> ak.to_dataframe. by @jpivarski in Renamed ak.to_pandas -> ak.to_dataframe. #1603
• refactor: replace 3 ak.from_json* functions with one by @jpivarski in refactor: replace 3 ak.from_json* functions with one #1617
• refactor: from_rdataframe to use LayoutBuilder by @ianna in refactor: from_rdataframe to use LayoutBuilder #1620
• docs: add Ahmad-AlSubaie as a contributor for code by @allcontributors in docs: add Ahmad-AlSubaie as a contributor for code #1416
• docs: Update AwkwardForth documentation and move it from the wiki to the standard docs. by @jpivarski in Update AwkwardForth documentation and move it from the wiki to the standard docs. #1422
• docs: update README: from awkward-1.0 to awkward by @ianna in update README: from awkward-1.0 to awkward #1475
• docs: Add the beginning of an example of templated LayoutBuilder. by @jpivarski in Add the beginning of an example of templated LayoutBuilder. #1485
• docs: add ManasviGoyal as a contributor for code by @allcontributors in docs: add ManasviGoyal as a contributor for code #1540
• docs: add aryan26roy as a contributor for code by @allcontributors in docs: add aryan26roy as a contributor for code #1541
• docs: Doxygen Documentation of GrowableBuffer and LayoutBuilder. by @ManasviGoyal in Doxygen Documentation of GrowableBuffer and LayoutBuilder. #1579
• docs: Documented ak.from_json and completed it. by @jpivarski in docs: Documented ak.from_json and completed it. #1621
• docs: add Saransh-cpp as a contributor for code by @allcontributors in docs: add Saransh-cpp as a contributor for code #1653
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in [pre-commit.ci] pre-commit autoupdate #1370
• ci: Protect test 1300 from ROOT without C++17 (or, at least, without std::optional). by @jpivarski in Protect test 1300 from ROOT without C++17 (or, at least, without std::optional). #1383
• ci: Build(deps): bump pypa/cibuildwheel from 2.3.1 to 2.4.0 by @dependabot in Build(deps): bump pypa/cibuildwheel from 2.3.1 to 2.4.0 #1394
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in [pre-commit.ci] pre-commit autoupdate #1395
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in [pre-commit.ci] pre-commit autoupdate #1409
• ci: Pass on skipped v2 tests by @ioanaif in Pass on skipped v2 tests #1445
• ci: Build(deps): bump pypa/cibuildwheel from 2.4.0 to 2.5.0 by @dependabot in Build(deps): bump pypa/cibuildwheel from 2.4.0 to 2.5.0 #1455
• ci: awkward rdataframe source tests by @ianna in awkward rdataframe source tests #1446
• ci: Ignore a NumPy 1.22 warning in Numba and fix the flake8-print T001 --> T201 change. by @jpivarski in Ignore a NumPy 1.22 warning in Numba and fix the flake8-print T001 --> T201 change. #1464
• ci: Build(deps): bump docker/setup-qemu-action from 1.2.0 to 2.0.0 by @dependabot in Build(deps): bump docker/setup-qemu-action from 1.2.0 to 2.0.0 #1462
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in [pre-commit.ci] pre-commit autoupdate #1465
• ci: Build(deps): bump pypa/cibuildwheel from 2.5.0 to 2.6.1 by @dependabot in Build(deps): bump pypa/cibuildwheel from 2.5.0 to 2.6.1 #1493
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in [pre-commit.ci] pre-commit autoupdate #1503
• ci: Build(deps): bump pypa/cibuildwheel from 2.6.1 to 2.7.0 by @dependabot in Build(deps): bump pypa/cibuildwheel from 2.6.1 to 2.7.0 #1510
• ci: Swap 'merged dtype same as NumPy' test of v1 for test of v2. by @jpivarski in Swap 'merged dtype same as NumPy' test of v1 for test of v2. #1515
• ci: Build(deps): bump actions/upload-artifact from 2 to 3 by @dependabot in Build(deps): bump actions/upload-artifact from 2 to 3 #1519
• ci: Build(deps): bump actions/checkout from 2 to 3 by @dependabot in Build(deps): bump actions/checkout from 2 to 3 #1520
• ci: Build(deps): bump actions/download-artifact from 2 to 3 by @dependabot in Build(deps): bump actions/download-artifact from 2 to 3 #1521
• ci: Build(deps): bump pypa/cibuildwheel from 2.7.0 to 2.8.0 by @dependabot in Build(deps): bump pypa/cibuildwheel from 2.7.0 to 2.8.0 #1536
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in [pre-commit.ci] pre-commit autoupdate #1527
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in [pre-commit.ci] pre-commit autoupdate #1543
• ci: Build(deps): bump pypa/cibuildwheel from 2.8.0 to 2.8.1 by @dependabot in Build(deps): bump pypa/cibuildwheel from 2.8.0 to 2.8.1 #1556
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in [pre-commit.ci] pre-commit autoupdate #1566
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in [pre-commit.ci] pre-commit autoupdate #1576
• ci: Build(deps): bump pypa/gh-action-pypi-publish from 1.5.0 to 1.5.1 by @dependabot in Build(deps): bump pypa/gh-action-pypi-publish from 1.5.0 to 1.5.1 #1564
• ci: use GHA instead of Azure 🔨 by @agoose77 in ci: use GHA instead of Azure 🔨  #1550
• ci: Upgrade Linux 18.04 -> 20.04 and DOS line endings. by @jpivarski in ci: Upgrade Linux 18.04 -> 20.04 and DOS line endings. #1616
• ci: lint PR titles according to conventional commits by @agoose77 in ci: lint PR titles according to conventional commits #1615
• ci: aarch64 should also use cibuildwheel v2.9.0 by @jpivarski in ci: aarch64 should also use cibuildwheel v2.9.0 #1627
• ci: run semantic-pr-title in concurrency group by @agoose77 in ci: run semantic-pr-title in concurrency group #1632
• ci: add Linux ROOT build to run ROOT-based tests by @ianna in ci: add Linux ROOT build to run ROOT-based tests #1629
• package: fix MANIFEST to include cpp-headers by @agoose77 in Package: fix MANIFEST to include cpp-headers #1516
• chore: remove unneeded lines by @henryiii in chore: remove unneeded lines #1518
• chore: include Python 3.11 by @henryiii in chore: include Python 3.11 #1602
• chore: update pre-commit hooks by @pre-commit-ci in chore: update pre-commit hooks #1618

New Contributors

• @Ahmad-AlSubaie made their first contribution in replace llvmlite.ir instead of llvmlite.llvmpy.core #1413
• @martindurant made their first contribution in split up functions #1397
• @aryan26roy made their first contribution in Adding a Forth Based Avro Reader #1491
• @ManasviGoyal made their first contribution in Growable Buffer header #1535
• @Moelf made their first contribution in fix ak2 convert class name msg #1544

Full Changelog: 1.8.0...v1.9.0

---

This discussion was created from the release Version 1.9.0.