0.9.0 Release

[grondo]

As we taper work on the splash app use-case, we should generate a new 0.9.0 release of flux-core (and corresponding flux-sched release)

For now we can tie remaining issues we'd like to wrap up before 0.9.0 tat to the release 0.9.0 milestone. This is just a placeholder to coordinate on the release (and I'll try to paste a candidate NEWS.md here for review)

[dongahn]

Yup this is a good idea. There are a few remaining sched items I'd loke to fold into this release as well.

[grondo]

Initial NEWS entry for 0.9.0. This needs an editing pass and of course to collect the last few PRs we merge before the tag:

This summarizes the 175(!) PRs we've merged since 0.9.0

flux-core version 0.9.0 - 2018-05-10

Fixes

• numerous memory leak fixes (libjsc: fix leaks detected by valgrind #1494, Fix memleak: free the job object returned. #1322)
• support for C++ code (kvs.h add extern "C" wrapper #1225, Improvements for cplusplus #1223, Configuration changes to support C++ code #1204)
• build fixes (build: fix newer compiler warnings about enum passed to va_start #1184
• fixes for flux-wreck purge (Fix flux-wreck purge #1357)
• scalability fixes for flux-wreck subcommands (flux-wreck: address scalability issue in 'ls' and 'timing' subcommands #1372)
• massive scalability improvement for libkz readers (libkz: avoid blocking RPCs in reactor callbacks #1411, libkz: defer kvs_watch until necessary #1424)
• increase job submission throughput (wreck: rework job.submit, job.create, wrexecd.run handlers with continuations #1472, jsc: support for augmented wreck.state.submitted event #1389)
• reduce amount of information collected in resource-hwloc to
  enhance large instance startup (Colllection of small fixes for splash project (part 2) #1457)
• i686 portability fixes (portabilty fixes for gcc 7.2 and Ubuntu 17.10 i686  #1296)
• fixes for flux-kvs dir and ls usage (cmd/flux-kvs: Update dir and ls usage #1444, cmd/flux-kvs: Fix dir -R corner case #1452)
• fix for clock_gettime workaround in Lua bindings (bindings/lua: fix clock_gettime workaround #1371)
• update minimum libhwloc to 1.11.1 to avoid assertion failure (update minimum hwloc version to 1.11.1 #1478)
• fix incorrect output from option parsing when invalid short
  option is grouped with valid options in many commands (optparse: fix incorrect error message when invalid short option is grouped with valid short options #1183)
• fix thread cancellation in sqlite module (Issue#1194: delay thread cancellation during sqlite. #1196)
• fix segfault on 32bit systems in cron module (modules/cron: fix segfault on 32 bit systems #1178)
• log errors from event redistribution (Colllection of small fixes for splash project (part 2) #1457)
• increase number of open files in wrexecd (Collection of small fixes for splash project #1450)
• fix job hangs during final task IO output flush (Collection of small fixes for splash project #1450)
• general reduction in log messages at INFO level (Collection of small fixes for splash project #1450)
• improve valgrind.h detection (build: switch to AX_VALGRIND_H for valgrind.h detection #1502)
• fix pkg-config pc name for liboptparse (pkgconfig: fix pkg-config pc names #1506)

New Features

• support config file boot method for broker (broker: add config file boot method #1320)
• new flux-kvs ls command (kvs: add flux kvs ls subcommand #1172, cmd/flux-kvs: Update dir and ls usage #1444)
• new kvs transaction API (kvs: support ability to "revert" / "back out" merged commits #1346, kvs: Rename internal "commit" API to "kvstxn" API #1348, libkvs: Require txn argument in flux_kvs_commit() and flux_kvs_fence() #1351)
• support for KVS namespaces (kvs: Refactor in preparation for private namespaces #1286, kvs: support removal of a KVS namespace #1299, modules/kvs: Support namespace timeout #1316, kvs: support non-instance owners of namespaces #1323, broker: add config file boot method #1320, kvs: Change "namespace" to "name_space" in external header #1327,
  kvs: Support ability to list namespaces #1336, kvs: Refactor prep work for namespace prefix support #1390, kvs: Support namespace prefix in key passed in by user #1423, kvs: Support namespace prefixes in symlinks #1432, kvs: don't check namespace if user passes in at reference #1436)
• support for node inclusion,exclusion via flux-wreck command (wreck: Add support for node exclusion and inclusion #1418)
• initial parser for jobspec (libjobspec: Initial flux jobspec parser #1201, jobspec: replace minus operator to multiplication #1293, jobspec: Use "yaml-cpp" instead of "Yaml-cpp" #1306)
• store child instance URI in enclosing instance (store rank 0 instance URIs in enclosing instance KVS #1429)
• new flux-wreck uri command to fetch child instance URIs (store rank 0 instance URIs in enclosing instance KVS #1429)
• additional states from kvs module (kvs: add more kvs stats #1310)
• append support for KVS values (kvs: support append operation #1265)
• support multiple blobrefs per valref in kvs (KVS: Support reading valref objects with multiple blobrefs #1227, KVS: Support valref pointing to zero length blob objects #1237)
• add flux_kvs_lookup_get_raw(3) (kvs: add flux_kvs_lookup_get_raw() #1218)
• add flux_kvs_lookup_get_key(3) (kvs: add flux_kvs_lookup_get_key() #1414)
• add flux_event_publish(3) to libflux API (libflux: add flux_event_publish() interface #1512)
• support for composite futures in libflux (libflux/future: minor interface changes to support composite futures #1188)
• add flux_future_reset(3) to support multi-response RPCs (libflux: add flux_future_reset() #1503)
• new libflux-idset library (libidset: public lib for numerically sorted, non-negative integer sets #1498)
• support raw payloads in flux-event (flux-event: support raw payloads, pub --loopback, and man page #1488)
• add raw encode/decode to flux_event API (libflux: add flux_event_encode_raw(), flux_event_decode_raw() #1486)
• introduce R_lite format for job allocation description (wreck incremental improvements #1399, Extend JSC to add R_lite support #1485)
• new flux-hostlist command for listing hostnames for jobs (Add flux-hostlist command and initialize resource.hosts in rc1 #1499)
• new flux-wreck environment manipulation commands (wreck: better integration of global lwj.environ environment for wreck jobs #1405)
• flux-wreck ls returns active jobs first (wreck: track active jobs #1481)
• flux-wreck tools allow filtering on active,inactive jobs (wreck: track active jobs #1481)
• flux-wreckrun will now block until job is scheduled by default, use the
  new --immediate flag to get old behavior (wreck incremental improvements #1399)
• add flux-wreck cancel command to cancel pending job (wreck: add flux-wreck cancel command #1365, flux-wreck cancel should send jobid as JSON number #1367, flux-wreck cancel: fall back to kill -9 if job is not pending #1385)
• add flux-wreck dumplog command to dump error log for jobs (Collection of small fixes for splash project #1450)
• add new KZ_FLAGS_NOFOLLOW flag to avoid blocking when no data in a kz
  file (Collection of small fixes for splash project #1450)
• add -n, --no-follow option to flux-wreck attach (Collection of small fixes for splash project #1450)
• propagate gpu and cores information for flux-wreckrun and submit
  (wreck incremental improvements #1399, submit/job/jsc: propagate gpu request information #1480)
• use cmb.exec service to launch wrexecd, not direct exec (wreck: use cmb.exec instead of direct fork/exec for wrexecd execution in job module #1508)
• support output to stderr with flux_log functions (libflux/flog: Support output to stderr #1192)

Cleanup

• kvs: major cleanup (kvs: rework internal metadata format to conform to RFC 11 design #1154, modules/kvs: handle root dir object caching more precisely #1177, modules/kvs: normalize keys before use #1182, modules/kvs: Log to flux not stderr #1190, kvs: store data pointed by valref raw and unencoded in content store #1214, Misc KVS cleanup & minor fixes #1213, libkvs: cleanup, deprecate old functions, update users #1233,
  KVS: Misc cleanup, add test coverage, minor bug fix #1235, modules/kvs: Replace EPERM w/ ENOTRECOVERABLE #1242, KVS modify commit protocol in preparation for append #1243, kvs: Misc cleanup & bug fixes #1244, kvs: refactor kvs cache to handle raw data as "primary" data #1246, kvs: minor cleanup & refactoring #1248, KVS: clean up lookup and txn API's #1253, kvs: handle non-JSON values in flux kvs get, put, dir #1257, kvs: string/JSON values should not be stored with NULL termination #1262, kvs: do not store NUL byte on json encoded strings in kvs server #1263,
  kvs: refactor server cache functions #1268, kvs: return const from cache_entry_get_json() #1269, kvs: additional cleanup and refactoring #1273, kvs: [cleanup] improve isolation in internal cache #1274, kvs: cleanup / re-organize sharness tests #1276, kvs: [cleanup] fix blobref_t typedef and unfortunate variable names #1279, modules/kvs: Refactor lookup_create() #1300, doc/man1/flux-kvs: Update language #1301, modules/kvs: Fix potential to exceed nprocs #1304, kvs: minor cleanup #1308,
  kvs: Return error that occurs in kvs.relayfence #1309, doc/man1/flux-kvs: Update language #1301, kvs: general refactoring and cleanup #1314, kvs: Make getroot() asynchronous #1315, kvs: refactor with internal kvsroot file #1321, kvs: check that fence parameters are identical #1329, kvs: cleanup callback names #1339, kvs: minor fixes/cleanup #1342, kvs: refactor commit & fence #1343, kvs: Refactor commit & fence - part 2 #1347,
  kvs: have two request handlers for commits & fences #1349, kvs: minor fixes/cleanups #1353, kvs: misc cleanup & refactoring #1383, kvs: minor cleanup & refactoring #1402, kvs: Fix corner case in which symlink points to root dir #1404, kvs: Remove unnecessary/superfluous log messages #1440, kvs: do not generate uuid for commit transactions, use fixed algorithm for 'name' #1458, kvs: couple of minor fixes #1466, modules/kvs: Enhance debug error messages #1477)
• kvs: improved test coverage (kvs: Add stats.clear test coverage #1291)
• Add const to message payload accessor functions (API: add const to message payload accessor functions #1212)
• rename flux_mrpcf, flux_mrpc_getf to flux_mrpc_pack,unpack (libflux/rpc: Rename flux_mrpcf() and flux_mrpc_getf() #1338)
• cleanup bulk message hanglers in libflux (libflux: clean up bulk message handler registration functions #1277)
• minor flux_msg_handler cleanup (common/libflux: minor flux_msg_handler cleanup/fixes #1171)
• broker: cleanup to prepare for dynamic service registration (broker/service: cleanup and prepare for dynamic service registration #1189)
• broker: general cleanup (Broker cleanup 1 #1230, Broker cleanup 2 #1234, Broker cleanup 3 #1241)
• Change key lwj to jobid in all jsc/wreck messages (jsc,job,wreck: change key from lwj to jobid for consistency #1409)
• libjsc cleanup (jsc: Make jsc_job_state2num public #1374, 1395, Drop jsc rdl_alloc JSC keys #1509)
• testsuite updates (testsuite: revert travis-ci to vm builds and fix many broken &&-chains in tests #1167, sharness: test_under_flux: add fatal error if modules loaded during test aren't unloaded at test completion #1175, kvs test cleanup #1313, fixes for broken tests, addition to gitignore #1464, generalize valgrind suppression for getaddrinfo #1266)
• Internal libutil and libflux cleanup (libutil/read_all: fix minor bugs, add test coverage #1319, [cleanup] use a better variable name for flux_msg_handler_t internally #1283, Reactor cleanup #1229, libcompat: cleanup, drop dead and little-used code #1231, libflux: Add input checks to future functions #1166)
• build system cleanup (build: Exit on error in autogen.sh plus other cleanup #1163, anonymous enums kill bindings #1354, build: fix newer compiler warnings about enum passed to va_start #1184, flux.service: Fix EXTRA_DIST and .gitignore #1200, Update to the latest ax_lua.m4 #1275, build: only set warning flags for GNU and Clang #1252)
• disable pylint by default (build: actually disable pylint by default #1255, build: drop man page stub that cannot be generated #1258)
• partial migration from json-c to jansson ([cleanup] partial migration from json-c to jansson #1501, 1508)
• drop unused ev_zlist composite watcher (libutil/ev_zlist: drop unused composite watcher #1493)

[garlick]

OK, I did some cursory testing under slurm on ipa and then went ahead and tagged 0.9.0.

And then...by magic...tarballs and release notes pop out! Nice!

[grondo]

Milestone achievement unlocked!