v23.01.00

This release adds initial support for using the UCX Realm networking backend (for more efficient multi-node communication) and using Legion's new "collective views" feature (for improved scheduling of reduction operations). Both of these features are currently in preview mode, and not enabled by default. They are planned to become the default by next release, following further verification and tuning.

This release improves the build experience for developers, with fixes to corner cases in the cmake configuration, a rewrite of the build documentation, and a script for generating complete conda environments for development, covering all supported platforms.

This release also introduces improvements in user interface (improved jupyter support, more CLI options for debugging and profiling), memory usage (through better instance management in the mapper) and the Legate programming model (allowing libraries to add custom profiler annotations, and use arbitrary communicator libraries in their tasks).

Conda packages for this release are available at https://anaconda.org/legate/legate-core.

What's Changed

🐛 Bug Fixes

• Legion bug WAR: don't instantiate futures on framebuffer by @manopapad in #413
• Handle conflicts for library-level args by @bryevdv in #416
• Fix Transform class hierarchy by @manopapad in #427
• Handle scalar outputs correctly in manual tasks by @magnatelee in #432
• Explicitly build Legion if legion_dir or legion_src_dir is not provided by @trxcllnt in #411
• Fix GPU shard computation by @bryevdv in #433
• Only set default CMake generator if Ninja is available: Issue #374 by @jjwilke in #379
• Fix an issue with editable installs by @bryevdv in #434
• Allow only one of --legion-dir and --legion-src-dir by @jjwilke in #387
• legate/util: fix a mypy error on MacOS by @rohany in #438
• Improvements to legate.jupyter by @bryevdv in #425
• Fix for cunumeric#668 by @manopapad in #453
• Only keep traceback reprs, to avoid cycles by @bryevdv in #447
• Fix returned legion paths for editable install with separate legion b… by @jjwilke in #442
• Make install.py reconfigure editable installs when build type changes by @trxcllnt in #455
• fix for -ll:networks none, we will init MPI if it has not been initialized by @eddy16112 in #465
• Construct region-backed 0D stores in a correct way by @magnatelee in #450
• Pass a sufficiently high default value for gasnet's ibv-max-hcas by @manopapad in #477
• Make overlap check tight by @manopapad in #479
• Conda env script fixes by @manopapad in #481
• Fix some typos by @manopapad in #485
• fix several reference cycle / leak related bugs by @rohany in #488
• legate/core: fix FutureMap leak in communicator shutdown by @rohany in #495
• src/core/mapping: adjust indirect copy mapping for GPUs by @rohany in #499
• Don't access stream pools unless we're on GPUs by @magnatelee in #503
• Update env gen script so OS type works for mac by @m3vaz in #523
• Don't check for collective behavior when we have WRITE privilege by @manopapad in #526
• All NCCL ranks on the same node must get the same NCCL_IB_HCA by @manopapad in #528
• legate/core/_legion: add default new argument to dep part functions by @rohany in #527
• Don't turn on Legate debug checks on debug-rel builds by @manopapad in #533
• src/core: guard against missing projection functors in collective check by @rohany in #534
• Erase cached reduction instances that cannot be acquired by @magnatelee in #536

🚀 New Features

• Support for library specific annotations by @magnatelee in #464
• Cycle detection check by @manopapad in #361
• Implementing logic for reuse of reduction instances by @ipdemes in #511
• Use collective views when mapping by @ipdemes in #466

🛠️ Improvements

• Driver verbose only for rank 0 or "none" launcher by @bryevdv in #403
• Consolidate driver and test driver codebases by @bryevdv in #397
• On mapping failure retry after tightening non-RO reqs by @manopapad in #423
• More changes for provenance by @magnatelee in #417
• Allow launcher_extra to split quoted values by @bryevdv in #444
• Add script to generate conda envs by @bryevdv in #367
• Mapper improvements by @magnatelee in #452
• Support for concurrent launches by @magnatelee in #459
• legate/core/types: add missing to_pandas_type on Complex types by @rohany in #467
• Add --cprofile driver option by @bryevdv in #475
• Optimize scalar extraction by @magnatelee in #472
• Refactor CPU collective communicator by @eddy16112 in #468
• Refactoring changes by @magnatelee in #478
• Regenerate install_info.py on every build by @trxcllnt in #486
• Update create_buffer to use socket memories whenever available by @magnatelee in #487
• Check for cycles involving Futures after runtime shutdown by @manopapad in #496
• Fix for 509 by @magnatelee in #510
• Improve build documentation by @manopapad in #517
• Pass CMAKE_GENERATOR to scikit-build by @trxcllnt in #529
• Change the default CPU architecture to haswell. by @marcinz in #538
• Build rust legion_prof by @trxcllnt in #535
• adding logic for collective instances to the legate_select_sources by @ipdemes in #532
• Add support for building Legion with the UCX backend by @SeyedMir in #516

New Contributors

• @robinw0928 made their first contribution in #431
• @SeyedMir made their first contribution in #516

Full Changelog: v22.10.00...v23.01.00