Release 2.27 checklist #1005

rok-cesnovar · 2021-05-05T15:49:53Z

rok-cesnovar · 2021-05-05T15:50:21Z

Larger projects currently in the last phases:

varmat
adjoint ODE

wds15 · 2021-05-16T08:20:29Z

Are we on track with things? Adjoint ODE looks as if it just makes the deadline... should there be any additional hickup it would be worthwhile to finish last cleanups for adjoint ODE (should there be any need).

rok-cesnovar · 2021-05-16T09:29:48Z

Its starting to look like varmat might be pushed to the next release again. Other things are on track for the freeze.

I have two minor PRs that I would like to get in (#1009, #1010), but these are not worth delaying anything for.

mitzimorris · 2021-05-16T12:46:03Z

I've got a branch for #1011 - if I got a PR in today, would that go in?

wds15 · 2021-05-16T12:51:59Z

the freeze is on the 17th end of day anywhere.. so it's the 18th...

So if the PR can be reviewed by tomorrow then it can be in.

rok-cesnovar · 2021-05-16T13:18:53Z

Its end of day on the 18th, thats how we usually do it.

wds15 · 2021-05-16T13:20:32Z

Even better

wds15 · 2021-05-16T18:48:22Z

What's the matter with varmat?

When you post the RC can you point people to the design doc of the adjoint ODE thing? This is to make it possible for people to call the new function for which I am writing the doc this week... or do you have other suggestions?

Thanks!

rok-cesnovar · 2021-05-16T18:53:15Z

What's the matter with varmat?

Missing stanc3 implementation, mostly ran out of time.

When you post the RC can you point people to the design doc of the adjoint ODE thing?

That was my plan yes!

mitzimorris · 2021-05-17T01:14:16Z

if anyone has time and feels qualified to review CmdStan pr #1017, that would be great.

rok-cesnovar · 2021-05-19T07:44:36Z

@serban-nicusor-toptal let me know when you have time to proceed with the release candidate. We only need to wait for this master stanc3 run to finish: https://jenkins.mc-stan.org/blue/organizations/jenkins/stanc3/detail/master/840/pipeline which should be soon.

serban-nicusor-toptal · 2021-05-19T10:38:08Z

Hey @rok-cesnovar I can start today around 5 PM CEST time.

rok-cesnovar · 2021-05-19T10:43:28Z

Great, thanks! If you can handle the first three points above it would be great!

Create a release candidate tarball. Make sure RC has the new CMDSTAN_VERSION and that RC tarballs have binaries.
Create Math/Stan RC releases. Make sure versions are updated.
Create Stanc3 RC binary. Make sure versions are updated.

I have the rest of the stuff ready and will check the tarballs/releases once they are ready.

serban-nicusor-toptal · 2021-05-19T17:54:35Z

Hey @rok-cesnovar here you can find the rcs:

It's taking a bit more because I am trying to figure out why this jenkins job is failing in the Build & test static Windows binary step.

+ dune build -x windows

Error: ocamlfind toolchain windows isn't defined in

/usr/local/opam/4.07.0/lib/findlib.conf.d (context: default.windows)

script returned exit code 1

rok-cesnovar · 2021-05-19T18:01:21Z

That happened a few times today yeah. Just restarted a few times and it passed then.

serban-nicusor-toptal · 2021-05-19T18:02:41Z

I see, I will restart one more and see what happens. In the meanwhile I'll try to debug locally, maybe they changed something that affects us as we didn't change anything code wise afaik.

serban-nicusor-toptal · 2021-05-19T19:39:18Z

Sadly I'm getting the same error locally

rok-cesnovar · 2021-05-19T19:48:30Z

Weird, not sure what changed. Will take a look in the morning.

serban-nicusor-toptal · 2021-05-19T19:57:20Z

I've found that here we're missing a -y which messes up the dependencies, I will update master and re-create the release, it worked fine locally so I expect it to work in Jenkins too.
I will create the tarball and tag you when done so you can review when you're available, no rush.

rok-cesnovar · 2021-05-19T20:03:29Z

Great. Fingers crossed. If you have time to build the cmdstan tarball today I can test and make an annoucement in the morning.

serban-nicusor-toptal · 2021-05-19T21:15:38Z

Everything seems to be alright now, weird how that typo got into the pipeline and actually worked until now.
Added tarball to cmdstan release: https://github.com/stan-dev/cmdstan/releases/tag/v2.27.0-rc1

rok-cesnovar · 2021-05-20T07:37:42Z

Thanks Nic! The release candidates are ready and all announcements have been posted. If all goes well the release will be next Thursday.

rok-cesnovar · 2021-05-25T08:00:24Z

Both Mac machines that are located at Columbia have gone offline yesterday. These Mac machines have been the most stable for the last few years and both of them going offline in the same day is very strange. Not much tests have run since the freeze so its very very unlikey this would be related to overload or anything like that.

We need these machines to build binaries for Mac and if we are unable to bring them online until Thursday, we will move the release to at least Monday (31st May). They will probably require physical intervention so not sure on the timeline to resolve this.

We could use the AWS Mac instances temporarily but that would mean that the minimum Mac version for CmdStan would be macOS Mojave (the oldest version we can get on AWS - a binary built on MacOS version X only work on version X and higher). This is more likely than not a non-problem, but would still like to avoid adding this restriction.

rok-cesnovar · 2021-05-25T08:01:36Z

We also need to resolve stan-dev/math#2493 prior to that. But I think that should be resolved prior to Thursday.

mitzimorris · 2021-05-25T11:03:43Z

I have access to Columbia, so I could go up there today and try to figure this out.

rok-cesnovar · 2021-05-25T11:05:26Z

Great! @serban-nicusor-toptal what would need to be done with the machine? Just reboot I am guessing?

serban-nicusor-toptal · 2021-05-25T11:07:49Z

Awesome!
Just try to reboot, make sure they're online and if possible get their IPv4 so I can try to connect to them.
If the linux/win machines are there too, it's worth checking them out too.

Thanks !!

rok-cesnovar · 2021-05-25T11:12:02Z

@bgoodri mentioned that the Mac machines are in Andrew's office right? The Linux/Windows are in Ben's office I believe so thats probably not accessible?

The Macs are definitely a bigger priority. The machines are named gelman-group-mac and old-imac, not sure if that helps identifying them.

mitzimorris · 2021-05-25T11:16:22Z

that sounds about right. one Mac machine is in my office (gelman-group-mac?) and the other is in Andrew's.
I might be able to get access to Ben's office, which is in a different building, depending on whether or not any of the admin staff in that building are back to working on campus.

mitzimorris · 2021-05-25T11:21:10Z

will be up there after 1pm EDT

rok-cesnovar · 2021-05-27T08:19:54Z

Due to the issues with the Mac computer and the performance regression we still haven't gotten to the bottom of, I propose moving the release to Monday. We are unlikely to figure the regression today and I dont want to release on a Friday.

rok-cesnovar · 2021-05-29T09:05:31Z

@serban-nicusor-toptal I think we will need to set up that AWS Mac instance for the release. The performance regression was identified and we will need the Mac instance for the bugfix tests as well as the release.

serban-nicusor-toptal · 2021-05-29T15:23:49Z

Hey @rok-cesnovar I can work on it tomorrow hoping to be ready until Monday, but just FYI Mac instances on AWS can't be bought using Spot Requests and also we can't have it on-demand as it's treated as a bare metal so we'll need to run it 24/7 with a 24-hour minimum allocation period. source
@mitzimorris hey, are you still planning in revisiting Columbia these days ? No rush in doing so just checking before I start on the AWS stuff.

mitzimorris · 2021-05-29T16:03:19Z

hi @serban-nicusor-toptal, I can only get access to Andrew's office Tues and Thurs. I couldn't get up the this past Thursday, but will go up on Tuesday and will get access, one way or another.

serban-nicusor-toptal · 2021-05-29T16:04:57Z

Ah okay then we'll need to go for aws for sure until then. Thanks!

serban-nicusor-toptal · 2021-05-31T13:13:02Z

Hey @rok-cesnovar the new AWS Mac is up and running, I hope to get all the dependencies right.
Also FYI I couldn't find llvm@6 so I've installed and using llvm@7 if that's okay, if it's not we'll need to install the older version somehow ( it's not in brew ).
Also there are some jobs that will get stuck in the queue as we don't have a GPU attached to the AWS Mac, what should we do about these ? ( gelman-mac has gpu and gpu-no-async labels )

Please let me know if you see any job failure so I can sort it out, thanks!

rok-cesnovar · 2021-05-31T13:23:48Z

We currently dont use the Mac+gpu combination, that label is not used in our tests anymore.

lets first wait for all tests to pass and merge the bugfix PRs. Its a holiday in the US today so no rush on releasing today.

serban-nicusor-toptal · 2021-05-31T14:28:35Z

Do we need this pr for the current release ? Also, is there any other PR that we should wait for ?

rok-cesnovar · 2021-05-31T14:32:13Z

Yes, that is the last one we need to merge for the release.

serban-nicusor-toptal · 2021-05-31T14:34:43Z

Huh okay that's good to know, I see it has some test failures so I'll probably have to wait until that's fixed. Please let me know if there's anything else I can do for now.

rok-cesnovar · 2021-05-31T15:58:41Z

The PR was merged so we just need to wait for the develop tests to pass so it propagates to cmdstan. That should take 4-5 hours so I guess we are probably going to do it tomorrow?

To recap:

stanc3 master is what we will release as 2.27.0 (no PRs were merged after RC)
math develop is what we will release (1 Pr was merged so we need to regenerate for the release)
stan develop is what we will release (Math submodule was updated + 1 PR was just merged)
cmdstan (there was a PR merged + we need to wait for the submodule to be updated)

serban-nicusor-toptal · 2021-05-31T16:06:52Z

Nice, we can do it today too as I'll most probably work until late or leave it for tomorrow before US wakes up both work just fine for me. Thanks for the recap!

mitzimorris · 2021-06-01T16:46:46Z

@serban Nicusor ***@***.***> I am at Columbia now and will be here until 3:30 pm New York time. do you want to co-ordinate w/r/t restarting gelman-group-mac?

…

On Mon, May 31, 2021 at 12:07 PM Nicusor Serban ***@***.***> wrote: Nice, we can do it today too as I'll most probably work until late or leave it for tomorrow before US wakes up both work just fine for me. Thanks for the recap! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1005 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJWT5HDFCJGXZ6UT3I6DHDTQOX2VANCNFSM44FILQSQ> .

serban-nicusor-toptal · 2021-06-01T16:51:59Z

Hey, sure let's go for it I'm available.

mitzimorris · 2021-06-01T16:57:18Z

OK, I just rebooted the machine. if you want to do a hangout, send invite to my email.

…

On Tue, Jun 1, 2021 at 12:52 PM Nicusor Serban ***@***.***> wrote: Hey, sure let's go for it I'm available. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1005 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJWT5D35QNWAPAUVPQE5MTTQUF35ANCNFSM44FILQSQ> .

serban-nicusor-toptal · 2021-06-01T17:10:29Z

I'm remotely connected to the machine, will try to see if I can change any setting for it.

mitzimorris · 2021-06-01T17:15:49Z

awesome. it seems to be plugged into one of the wall ethernet connections. took a photo and can reach out to CU IT if there's some kind of network issue.

serban-nicusor-toptal · 2021-06-01T17:21:26Z

I did change pmset settings to:

System-wide power settings:
 SleepDisabled          1
Active Profiles:
AC Power                -1*
Currently in use:
 standby              0
 Sleep On Power Button 1
 womp                 1
 autorestart          0
 hibernatefile        /var/vm/sleepimage
 powernap             0
 gpuswitch            2
 networkoversleep     0
 disksleep            0
 sleep                0
 autopoweroffdelay    14400
 hibernatemode        0
 autopoweroff         0
 ttyskeepawake        1
 displaysleep         10 (display sleep prevented by Bluetooth Setup Assistant)
 standbydelay         10800

To disable most of the sleep/turnoff so let's hope this fixed it.

Thanks a lot @mitzimorris !

serban-nicusor-toptal · 2021-06-01T19:07:21Z

Looks like we lost the mac machine again @mitzimorris. Nothing was running on it so we can rule out overheating also I don't think it was a standby/shutdown either as we changed those.

mitzimorris · 2021-06-01T19:30:56Z

OK, I will reach out to CU IT.
cheers,
Mitzi

wds15 · 2021-06-02T17:06:07Z

Do we have to sort out the Mac thing? Or can we proceed with different means?

serban-nicusor-toptal · 2021-06-02T17:07:00Z

Hey, I've already spin up a new Mac Instance on AWS that we can use for now.
I will start with the release in just a few moments!

serban-nicusor-toptal · 2021-06-02T23:13:49Z

Release is done but there's one thing left, to generate new docs and update release notes with the new urls.
I've tried to build docs on Mac Mojave with R 3.3.3 and Ubuntu 20.04 with R 4.0.1 but it fails with the below error on both:

python build.py 2 27

Try to find the following text in _main.Rmd:
  \subitem {\tt (data row_ 
You may need to add $ $ around a certain inline R expression `r ` in _main.Rmd (see the above hint). See https://github.com/rstudio/rmarkdown/issues/385 for more info.
Error: LaTeX failed to compile _main.tex. See https://yihui.org/tinytex/r/#debugging for debugging tips. See _main.log for more info.
In addition: Warning message:
In normalizePath(dirname(f)) : path[1]="NA": No such file or directory
Execution halted
Command Rscript -e "bookdown::render_book('index.Rmd', output_format='bookdown::pdf_book')" failed
Traceback (most recent call last):
  File "build.py", line 113, in <module>
    main()
  File "build.py", line 109, in main
    make_docs(docspath, stan_version, doc, formats)
  File "build.py", line 47, in make_docs
    shexec(command)
  File "build.py", line 35, in shexec
    raise Exception(returncode)
Exception: 1

Not sure if I've messed something up in a dependency version or if it's a code error.

mitzimorris · 2021-06-03T13:44:43Z

tried to build the docs from current develop - failed on my mac. I don't think it's your scripts that are the problem here.
investigating.

rok-cesnovar · 2021-06-03T18:46:58Z

I have managed to find a solution, one of the tables failed to render (produces latex that doesnt compile).

Setting full_width to TRUE here fixes the issue: https://github.com/stan-dev/docs/blob/master/src/stan-users-guide/sparse-ragged.Rmd#L70

serban-nicusor-toptal · 2021-06-03T20:31:11Z

Nice! Good job for finding out the error.
~~Since this is a code change I've created a PR with your suggestion. Can you please review @mitzimorris and if alright merge? After that I'll generate the docs and post here.~~

serban-nicusor-toptal · 2021-06-03T20:59:47Z

Hey PR for docs can be found here stan-dev/docs#371.
https://stan-dev.github.io is also updated.
@rok-cesnovar I've also fixed the pipeline issues it had. It fails in make manual in cmdstan, there's not that function anymore.
I will shortly update release notes in github after docs are merged.

serban-nicusor-toptal · 2021-06-03T21:24:02Z

Done! Eveything seems to be in place now.
Math v4.1.0
Stan v2.27.0
CmdStan v2.27.0
Stanc3 v2.27.0

rok-cesnovar · 2021-06-04T10:40:43Z

Thanks Nic!

All of the announcements on various channels have now been posted so I am closing this issue. Thanks to all!

Next release is planned for the 27th of September (RC on the 20th).

rok-cesnovar closed this as completed Jun 4, 2021

Release 2.27 checklist #1005

Release 2.27 checklist #1005

Comments

rok-cesnovar commented May 5, 2021 • edited Loading

rok-cesnovar commented May 5, 2021

wds15 commented May 16, 2021

rok-cesnovar commented May 16, 2021

mitzimorris commented May 16, 2021

wds15 commented May 16, 2021

rok-cesnovar commented May 16, 2021

wds15 commented May 16, 2021

wds15 commented May 16, 2021

rok-cesnovar commented May 16, 2021

mitzimorris commented May 17, 2021

rok-cesnovar commented May 19, 2021 • edited Loading

serban-nicusor-toptal commented May 19, 2021

rok-cesnovar commented May 19, 2021

serban-nicusor-toptal commented May 19, 2021 • edited Loading

rok-cesnovar commented May 19, 2021

serban-nicusor-toptal commented May 19, 2021

serban-nicusor-toptal commented May 19, 2021

rok-cesnovar commented May 19, 2021

serban-nicusor-toptal commented May 19, 2021

rok-cesnovar commented May 19, 2021

serban-nicusor-toptal commented May 19, 2021 • edited Loading

rok-cesnovar commented May 20, 2021

rok-cesnovar commented May 25, 2021

rok-cesnovar commented May 25, 2021

mitzimorris commented May 25, 2021

rok-cesnovar commented May 25, 2021

serban-nicusor-toptal commented May 25, 2021

rok-cesnovar commented May 25, 2021

mitzimorris commented May 25, 2021

mitzimorris commented May 25, 2021

rok-cesnovar commented May 27, 2021

rok-cesnovar commented May 29, 2021

serban-nicusor-toptal commented May 29, 2021

mitzimorris commented May 29, 2021

serban-nicusor-toptal commented May 29, 2021

serban-nicusor-toptal commented May 31, 2021 • edited Loading

rok-cesnovar commented May 31, 2021

serban-nicusor-toptal commented May 31, 2021

rok-cesnovar commented May 31, 2021

serban-nicusor-toptal commented May 31, 2021

rok-cesnovar commented May 31, 2021

serban-nicusor-toptal commented May 31, 2021

mitzimorris commented Jun 1, 2021 via email

serban-nicusor-toptal commented Jun 1, 2021

mitzimorris commented Jun 1, 2021 via email

serban-nicusor-toptal commented Jun 1, 2021

mitzimorris commented Jun 1, 2021

serban-nicusor-toptal commented Jun 1, 2021

serban-nicusor-toptal commented Jun 1, 2021 • edited Loading

mitzimorris commented Jun 1, 2021

wds15 commented Jun 2, 2021

serban-nicusor-toptal commented Jun 2, 2021

serban-nicusor-toptal commented Jun 2, 2021

mitzimorris commented Jun 3, 2021

rok-cesnovar commented Jun 3, 2021

serban-nicusor-toptal commented Jun 3, 2021 • edited Loading

serban-nicusor-toptal commented Jun 3, 2021 • edited Loading

serban-nicusor-toptal commented Jun 3, 2021

rok-cesnovar commented Jun 4, 2021

rok-cesnovar commented May 5, 2021 •

edited

Loading

rok-cesnovar commented May 19, 2021 •

edited

Loading

serban-nicusor-toptal commented May 19, 2021 •

edited

Loading

serban-nicusor-toptal commented May 19, 2021 •

edited

Loading

serban-nicusor-toptal commented May 31, 2021 •

edited

Loading

serban-nicusor-toptal commented Jun 1, 2021 •

edited

Loading

serban-nicusor-toptal commented Jun 3, 2021 •

edited

Loading

serban-nicusor-toptal commented Jun 3, 2021 •

edited

Loading