Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release 2.30 #1101

Closed
29 tasks done
rok-cesnovar opened this issue Jun 12, 2022 · 33 comments
Closed
29 tasks done

Release 2.30 #1101

rok-cesnovar opened this issue Jun 12, 2022 · 33 comments

Comments

@rok-cesnovar
Copy link
Member

rok-cesnovar commented Jun 12, 2022 •
edited
Loading

Feature freeze (21st of June):

  • Ensure all expiring language deprecations have been removed or given new removal dates.
  • Create and merge version updating pull requests in Math/Stan/Cmdstan. These should be the last PRs accepted before the freeze.
  • Create Math/Stan RC releases.
  • Create Stanc3 RC binary.
  • Create a release candidate tarball for x86. Make sure RC tarballs include stanc3 binaries.
  • Check external links in docs (i.e. TBB docs link) - use https://github.com/tcort/markdown-link-check
  • Create a release candidate feature/bugfix list (major features/bugfixes that need testing, link to new docs in Github)
  • Run CmdStanR tests with the RC tarball.
  • Run CmdStanPy tests with the RC tarball.
  • Make a Discourse RC post.
  • Post a tweet with a link to the Discourse RC post.

Release (10 days after the RC):

  • Create the Math Release notes.
  • Create the Stan Release notes.
  • Create the Cmdstan Release notes.
  • Create the Stanc3 Release notes, include new deprecations and removals.
  • Rebuild and publish docs for the new version.
  • Check that docs for the previous release links correctly to the newest docs.
  • Create the Math release.
  • Create the Stan release.
  • Create the Stanc3 release.
  • Create x86 CmdStan tarballs (check version, check that the extracted folder is in the cmdstan-version format).
  • Create non-x86 CmdStan tarballs.
  • Create conda CmdStan tarballs.
  • Run CmdStanR tests with the release tarball.
  • Make a Stan blog release announcement post (thank the sponsors and all contributors, mention new devs).
  • Link to the blog post in a Discourse thread.
  • Make a Twitter announcement.
  • Update https://github.com/stan-dev/stan/wiki/Coding-Style-and-Idioms#supported-cpp-versions-and-compilers

Things to highlight in the release notes:

  • The Windows RTools35 toolchain will no longer be supported
@rok-cesnovar
Copy link
Member Author

The RC is currently set to this coming Thursday, but if it helps anyone we can move it to Monday, which is our typical "release" day. Let me know.

@WardBrian
Copy link
Member

It would be great to get any of stan-dev/math#2699 addressed before release.

I think we will need to add a message to the release notes about “Please let us know if functions you expect to work with complex matrices are not yet supported so we can prioritize what to work on” or something like that anyway, but those should be fairly obvious ones

@WardBrian
Copy link
Member

I also think we should look at #1057 - @rok-cesnovar how difficult would it be to improve the error messages for this?

@rok-cesnovar
Copy link
Member Author

On it! Been promising that one since January :)

@WardBrian
Copy link
Member

I think that one is as simple as

else {
        throw std::invalid_argument(
            "num_chains can currently only be used for HMC with adaptation engaged");
      }

following the if (is_hmc && adapt_engaged) {

@rok-cesnovar
Copy link
Member Author

lol yeah

@rok-cesnovar
Copy link
Member Author

Barring anything unexpected, the RC will be released on Tuesday. The only thing left open currently is the std_normal_qf I believe.

@rok-cesnovar
Copy link
Member Author

@serban-nicusor-toptal the current develop (master for stanc3) is what we want to release for the 2.30 RC. Whenever you are free tomorrow, feel free to start with the process. No rush.

I will make an RC announcement with some highlights until sometime afternoon CET when the US folks get up.

@rok-cesnovar
Copy link
Member Author

The draft of the RC announcement is here: https://gist.github.com/rok-cesnovar/3a1419b665f4b09c26f2e56995938c81#file-stan-2-30-rc-announcement

Would appreciate any comments.

@serban-nicusor-toptal
Copy link
Contributor

Hey @rok-cesnovar I've created the PRs to update the versions:

Would you mind taking a look and approve? Should we also skip waiting for CI and merge straight away so I can do the RC ?

@rok-cesnovar
Copy link
Member Author

Thanks! Go ahead and merge them.

@serban-nicusor-toptal
Copy link
Contributor

I am now starting with the RC, most of it shouldn't take long but we will need to wait a bit for stanc3 binaries to be built. Will reply here when all done.

@rok-cesnovar
Copy link
Member Author

We are not under any time pressure, so whenever they are built its fine.

@serban-nicusor-toptal
Copy link
Contributor

Release candidates are up now!

@rok-cesnovar
Copy link
Member Author

rok-cesnovar commented Jun 21, 2022 •
edited
Loading

Thanks. I am seeing a cmdstan-2.30.0-rc1.tgz and .tar.gz file (we typically only have the tar.gz one). And it seems as if they dont have the Stan sources inside?

@WardBrian
Copy link
Member

The RC was not tagged as a pre-release so things like cmdstanpy which look for the latest version picked it up (and, due to the issue @rok-cesnovar notes, immediately failed)

@serban-nicusor-toptal
Copy link
Contributor

serban-nicusor-toptal commented Jun 21, 2022 •
edited
Loading

Unintended mistake, I forgot to checkout submodules in cmdstan.
Everything should be fine now, updated release candidates as pre-release, removed the .tgz and re-uploaded the .tar.gz

@rok-cesnovar
Copy link
Member Author

Thanks, this seems to work. The RC announcement was posted on Discourse and Twitter. I am also running CmdStanR tests. I will check for broken links in the next few days.

@WardBrian
Copy link
Member

Ran cmdstanpy tests here: https://github.com/stan-dev/cmdstanpy/actions/runs/2557546817

@rok-cesnovar
Copy link
Member Author

Thanks! I also checked the links. All of them work, some just require an unnecessary redirect, that is addressed here: stan-dev/docs#552

@rok-cesnovar
Copy link
Member Author

rok-cesnovar commented Jun 30, 2022 •
edited
Loading

@serban-nicusor-toptal you can start the release whenever you have a moment. There were a couple of fixes merged, so please release the current develop (Math/Stan/CmdStan) and master (stanc3) branches as the official release.

Release notes:

-----------
Math
-----------

Complex:
 - Added fast Fourier transform (FFT) implementations of 1D and 2D discrete Fourier transform and their inverses. (#2686, #2750)
 - Added type traits for detecting whether a type has a value type that is complex. (#2694)
 - Added vectorized versions of get_real and get_imag. (#2700)
 - Vectorized complex `abs()`, `to_complex`. (#2737, #2749)

New functions:
 - Added the incomplete beta function inverse. (#2637)
 - Added the `wishart_cholesky_lpdf`, which is the Cholesky parameterization of the Wishart distribution of both the input matrix and the scale matrix. (#2710)
 - Added the  inverse Wishart Cholesky parameterization `lpdf\lupdf` is available along with `inv_wishart_cholesky_rng`. (#2713)
 - Added the Cholesky parameterization of the multivariate Student's T distribution. (#2715)
 - Added the quantile function of the standard normal density function with log input. (#2744)
 
Misc:
 - Upgraded Sundials to version 6.1.1. (#2692)
 - Updated Boost to 1.78.0. (#2725)
 - Added namespace qualifiers to `size` and `apply` calls for C++17 compatibility. (#2693)
 - Enabled address sanitizer and fix it when running inside Docker in Flatiron CI. (#2702)
 - Made `arena_allocator` usable for all std containers. (#2708)
 - Added gradient calculation for the a2 input in stan-math for the `grad_2F1` function and fixes incorrect return values when `z` is between -1 and 0. (#2721)
 - Updated the signatures of `eigenvalues` and `eigenvectors` to use our pseudo-concept require templates.(#2728)
 - `norm1` and `norm2` were extended with the `apply_vector_unary` to accept general std::vectors as well as the Eigen vectors previously supported. (#2742)
 - Cleaned up compiler warnings for unused variables and non-consistent use of class and struct for ops_partials_edge. (#2757)
 - Fixed the check for `y` in `gamma_lpdf` so that it now errors if y is not a positive finite value. (#2758)
 - Improved numerical stability of gamma_lcdf gradients (#2767)

-----------
Stan
-----------

- Added `const noexcept` modifiers to the three accessors related to step-size (nominal, current, and jitter). (3124)
- Removed unused variable `num_params`. (#3126)
- Code cleanup - consistent set of includes for the set of services in stan/services/sample. (#3111)

-----------
CmdStan
-----------

- Fixed makefile for use of USER_HEADER with the bernoulli example model. (#1104)
- Added an error if `num_chains` is supplied for unsupported algorithms and methods. (#1102)
- Fixed handling of size-zero JSON arrays as input. (#1093)

-----------
stanc3
-----------

Complex:
 - Added complex containers: `complex_matrix`, `complex_vector`, and `complex_row_vector` with basic arithmetic support. (#1133)
 - Fixed indexing of a complex container to return the correct type. (#1158)
 - Added many common linear algebra functions and container operations for complex types. (#1166)
 - Fixed an issue reading in `complex_matrix` in the data block. (#1168)
 - Added vectorized signatures for `get_real` and `get_imag`, e.g. `get_real(complex_matrix) => matrix`. (#1169)
 - Added `fft`, `inv_fft`, `fft2`, and `inv_fft2` to the language. (#1170)
 - `pow` (and subsequently the `^` operator) now have overloads for complex containers. (#1186)
 - `to_complex` is now vectorized. This means signatures like `to_complex(matrix, matrix) => complex_matrix` are now available. (#1208)
 - Added signatures for addition, subtraction and division which mix scalars and complex vector types, broadcasting the scalar as appropriate. (#1212)
 - Added `svd_U`, `svd_V`, `singular_values`, `eigenvalues_sym`, and `eigenvectors_sym` for complex inputs. (#1212)

New functions:
 - Added `L1` and `L2` norms. (#1140)
 - Added `log_determinant_spd` which is faster when the input matrix is known to be symmetric and positive definite. (#1178)
 - Added `wishart_cholesky_lpdf` and `wishart_cholesky_rng` distribution functions. (#1188)
 - Added `multi_student_t_cholesky_lpdf` and `multi_student_t_cholesky_rng` distribution functions. (#1197)
 - Added `inv_wishart_cholesky_lpdf` and `inv_wishart_cholesky_rng` distribution functions. (#1202)
 - Exposes inv_inc_beta. (#1211)

Misc: 
 - Re-enabled function inlining optimizations at O1. (#1097)
 - Modified the optimizer to turn statements like `3 + 4i` into `to_complex(3,4)` directly. (#1112)
 - Added `cumulative_sum(Vector/RowVector)` for `var<Matrix>` types. (#1152)
 - Improved compile times by accumulating all template requirements into a single `require_all_t` on user defined functions. (#1157)
 - The `--info` and `--debug-generate-data` commands now use the Yojson library for it's output. (#1161, #1175)
 - Bugfixed for output values in `write_array` that should be `NaN`. (#1165)
 - `--dump-stan-math-signatures` now prints an entire signature on one line, no matter how long it is. (#1167)
 - Fix --debug-generate-data for ordered vector data types. (#1173)
 - Exposed an overload for `cumulative_sum` which takes and returns an array of integers. (#1174)
 - The javascript interface (stancjs) now generates errors which look more like stanc3's by pointing out where in the supplied code the error occurred. (#1177)
 - Cleaned up the internal structuring of modules in the compiler. (#1179)
 - Cleaned up code for pretty printing the Middle Intermediate Representation (MIR). (#1180)
 - Added `--debug-mem-patterns` flag, which will print out the memory type of variables (either Array of Structs or the more efficient Struct of Arrays pattern) when optimization is enabled. (#1185)
 - Allowed the adding of new distributions without requiring them to support deprecated suffixes. (#1191)
 - Added `eigenvectors` and `eigenvalues` functions which return the complex-valued eigenvectors or eigenvalues of a real (asymmetrix) matrix. (#1192)
 - `abs` is now vectorized for container types. `fabs` is now deprecated, as all usages can be replaced with `abs` thanks to array promotion rules. (#1195)
 - Made installing the developer tools for the Stanc repository an interactive script with error handling. (#1200)
 - Internally clean up how variable declarations are represented. (#1203)
 - Simplified internal representations in the compiler. (#1204)
 - The pretty-printer now preserves multiple declarations in one line like `int a, b;` rather than splitting them into two declarations. (#1207)
 - Declarations are now allowed in places they previously would not parse, such as the only statement in the body of an if statement. (#1207)
 - `./`, `/` `.*`, and `*` now have consistent behavior with scalar arguments. (#1209)

@WardBrian
Copy link
Member

This title should probably be something more like "Re-enabled function inlining optimizations at O1", and I think the abs/fabs change should be higher up. Otherwise the notes look good to me!

@rok-cesnovar
Copy link
Member Author

Thanks. Updated my post, also added the line for the Math bugfix of gamma_lcdf.

@serban-nicusor-toptal
Copy link
Contributor

Hey @rok-cesnovar I'll try to do it later today or early tomorrow

@rok-cesnovar
Copy link
Member Author

Ok, cool.

@serban-nicusor-toptal
Copy link
Contributor

Hey @rok-cesnovar everything should be ready except docs:

2.30 documentation is currently 404 until the following PRs are approved & merged:

Please let me know if there's anything else I can do to help!

@rok-cesnovar
Copy link
Member Author

rok-cesnovar commented Jul 1, 2022 •
edited
Loading

Thanks. The tarball looks good on first look, locally everything seems to work fine. Running CmdStanR tests now.

@WardBrian
Copy link
Member

Cmdstanpy tests running here: https://github.com/stan-dev/cmdstanpy/runs/7152748217?check_suite_focus=true

@WardBrian
Copy link
Member

Docs look good to me

@rok-cesnovar
Copy link
Member Author

There is some formatting issue here: https://mc-stan.org/docs/functions-reference/array-reductions.html#norms
I am not sure exactly what is up. It's a minor issue though. Not something we need to delay anything for.

@WardBrian
Copy link
Member

Looks alright to me? What am I missing?

@rok-cesnovar
Copy link
Member Author

On refresh it did show the equations, seems just a temporary browser issue. Sorry for the false alarm.

@rok-cesnovar
Copy link
Member Author

All tasks are done so I am going to close this. Thanks all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rok-cesnovar @WardBrian @serban-nicusor-toptal