LSST DESC Notes documenting Twinkles 1 results

[drphilmarshall]

@cwwalter is writing the requirements doc for DC1. He is focusing on the biggie, DC1 PhoSim Deep, as the main DC1 dataset, but wants to cite Twinkles 1. This means we need to provide some citeable objects for him: talking to @wmwv about this, we think we should write up some short DESC research notes, of the kind referred to in the proposed Publication Policy. I asked around a bit for guidance on how we might do this (thanks @nregnault @alexabate @kponder :-) ) and am thinking we might knock up a template at the Hack Day, @heather999 ? Doing this now should be easy (because we just gave a bunch of talks on our results) and should help a lot with specifying exactly what Run 3 (the main data generation run) will involve. What do you think?

Meanwhile, our job in the Twinkles TF is to blat our results into simple markdown files or jupyter notebooks, and check them in to the appropriate doc folders, using pull requests from the first authors to enable Task Force review. I guess this review will be led by our TF leaders, @richardxdubois and @wmwv.

Here's the list of notes from our March 1 weekly meeting, complete with potential first authors (it's editable, please contribute changes here with explanations in comments below!).

• Twinkles 1 Overview @drphilmarshall @wmwv, rst/tex (with links to other notes) LSST DESC Note: Twinkles Design #299
• The Twinkles 1 Simulation Pipeline @danielsf, with @rbiswas4 @jbkalmbach rst (OpSim, CatSim, sprinkler concept, link to sims cookbook recipe)
• The Twinkles 1 Supernova Population @rbiswas4, ipynb (incl science motivation)
• The Twinkles 1 Strong Lens Population @jbkalmbach @drphilmarshall, ipynb, (incl science motivation, ipynb) LSST DESC Note: The Twinkles 1 Strong Lens Population #450
• Twinkles 1 PhoSim Pipeline and Results @TomGlanzman, rst LSST DESC Note: The Twinkles1 PhoSim Pipeline and Results #451
• Machine-learning Prediction of PhoSim CPU Time @sethdigel, with Humna & Tom, ipynb/rst/tex LSST DESC Note: Machine Learning Prediction of PhoSim CPU Time #452
• Twinkles 1 Emulated DM Pipeline and Results @SimonKrughoff, with @tony-johnson rst (with links to cookbook recipes, and including workflow details. Results on NERSC performance in Run 2, 3 cf SLAC in Run 1.1, as well as astronomy measurements.) LSST DESC Note: The Twinkles Emulated DM Level 2 Pipeline and Results #453
• Pserv @jchiang87, with @jbkalmbach @heather999 ipynb LSST DESC Note on Pserv  pserv#16
• The Monitor @jbkalmbach, with @rbiswas4 ipynb including examples of SNe and SL, with v brief analysis?
• Twinkles 1 SN Error Model @rbiswas4 @jbkalmbach with @drphilmarshall @wmwv v, ipynb

Other notes that could be written (add ideas here):

• The DESC Software Package Template @jchiang87
• Twinkles 1 Preliminary Results: Photometric Validation of the Run 1.1 Data @wmwv
• Twinkles 1 Preliminary Results: Modeling the Run 1.1 LightCurves of Isolated Bright Supernovae @jchiang87

We'll all need to help out with each others notes as required, but there's a lot to be said for keeping the author list as small as possible - single author could be ideal for maximum credit visibility but 2-3 authors per note is often going to be needed, I guess.

To-do list:

• Set up folders with template Notes Remake Notes folders with start_paper #413. This was done, but later reverted so we can use branches and PRs for reviews.
• Engage @jonathansick as LSST docs consultant by mentioning him on this thread
• Issue individual notes, to provide space for their discussion and to help authors manage their work. This is being done by the relevant first authors.

[drphilmarshall]

Alright, no objections to this documentation push so far! I'll go ahead and set up some files for us to work in, and ask @jonathansick from the LSST docs dept to follow along with this effort. He's helping the CI group start setting up a more general DESC Notes system, and, predictably, we are the guinea pigs :-) If you're interested in reading more, check out the epic at LSSTDESC/ComputingInfrastructure#24 and some further discussion in LSSTDESC/ComputingInfrastructure#28 . We'll also chat about the content of this set of DESC Notes at our meeting today.

[cwwalter]

Hi Phil,

I would say as a courtesy to the reader, and in future cases like this, reviewers, those should be chapters or sections in a single document describing Twinkles project. After thinking about things you might still want more than one (especially describing the analyses of results) but I don't want to have to download 11 documents to get an overview. I think you can still be sensitive to explicitly putting people's names on sections and you might imagine having multiple papers coming out of it, but I really only want to download, print, and read one thing on the airplane :)

Just some feedback to consider.

[drphilmarshall]

Thanks Chris! I agree "I want to read a lot on an airplane that doesn't have wifi" is a very important use case for the notes system. However, monoliths that are repeatedly cited as, eg, "LSST DESC (2016)" are a great way to fully disguise the individual contributions of the collaborations' members - and so I think it's worth us at least experimenting with distributing the Twinkles write-up over many separate small documents and then exploring ways to aggregate them. (It might also be that the overview you want is provided by the first Note in my proposed series.)

@jonathansick - has this problem come up in the LSST technotes yet? Has anyone asked for the ability to make a compendium of notes X, Y and A through B, for example?

[jonathansick]

From a citation perspective, I do want to promote citation of individual technotes for all the reasons you can imagine (give people credit, provide transparency and accuracy in the reference, ...).

I haven't received a request yet for a compendium of technotes but I have gotten vocal calls for better PDFs and EPUBs. That's in my queue. I could imagine concatenating PDFs together from a technote series for offline reading. I could probably make that available from the planned documentation hub.

[drphilmarshall]

One additional wrinkle: two of our proposed Twinkles Notes really belong in separate repos, I think. The LSST DESC Notes describing the Monitor and pserv packages should, I think, live in those packages' repos. @jbkalmbach @jchiang87 We can start them here in Twinkles repo if you like, but I suggest that at some point we mv those two notes' folders into the doc/LSST_DESC_Notes folders of the Monitor and pserv repos. (We don't need to change the numbering - in the LSST technotes model, numbers are assigned on creation, by hand, and across the whole collaboration.) These two notes in particular will probably work well as jupyter notebooks - indeed, you may already have demo notebooks that get you most of the way towards an acceptable Note. You'll still need to develop those Notes in your personal forks, and then your WG/TF reviewers should probably by the CI WG leads for pserv, and the SN/SL WG leads for Monitor.

This issue is about to become an Epic - I'll be editing the top comment to make it so, and then issuing the individual Notes to help us work on them. In our meeting we agreed an end-of-month deadline, which I will implement as a milestone for us. Hope this sounds OK!

[cwwalter]

I wanted to follow up a bit on my previous comment about having too many tech notes for a single topic. I think the issue is more complicated/problematic than just "I want to read a lot on an airplane that doesn't have wifi". I'm coming at this from the perspective of someone who actually ran the review (with many people) of a high profile complicated analysis that had 4 or 5 notes to document the work.

In those cases the work was really quite separate it really made sense to have them be in different documents. But, even in that case, having the multiple documents really complicated things. It is just so much easier (even if I am online) to have everything combined in one document. There is a single table of contents that allows you to flip around, you can refer to other sections, and you don't have to juggle 11 files around. I really believe there is a big advantage to having the minimum number of documents possible. In this Twinkles case, a lot of these should clearly really be chapters in the same document.

I think a lot of this is being driven by the admirable desire to give credit to individual members especially for things that are being made available publicly so they can be cited etc. But, maybe we can think a little more creatively about how to handle that problem? There may be better ways than just splitting every document up into chunks associated with each author.

Internally, I think this can be handled by listing all the authors on each tech note and clearly putting people's names on each section. Publicly, I think we need to think about it and maybe this is something the PB will address. You could imagine that public notes are different than internal notes and will be extracted from the main internal document. These would be different documents with more background each one clearly labeled with their authors for citation, or there may be the sort of technical solution that Jonathan proposed above etc. Anyway, I hope we can both address this important credit issue and not unnecessarily complicate our internal documentation simultaneously.

Thanks!

[drphilmarshall]

"I hope we can both address this important credit issue and not
unnecessarily complicate our internal documentation simultaneously."

Agreed! :-)

If web pages are well designed (or are hosted within a well-designed
system), I think they can emulate being part of a single document. If
single PDF documents are not well-designed, they can be very hard to
navigate. I think our primary logistical aspiration for our documents is
for them to be "easy to navigate" rather than "self-contained" - right?

In the science roadmap latex PDF we attempted to solve the navigation
problem with a lot of hyperlinking, and I think it worked pretty well -
except that all of those links point only to addresses that are within that
document. You can't hyperlink to section 3.2 of the science roadmap from
anywhere outside the science roadmap: the only citeable object is the whole
science roadmap.

1. To get round this problem, in some cases maybe what we need is something
   like a "book of proceedings", where the individual proceedings are citeable
   objects with small numbers of authors, that accurately present what those
   authors did, but that are all packaged together in an easily navigated
   book. A technical question would then be: how do we make a long latex
   research note whose chapters or sections are each citeable objects?

2. The converse of this could be: how do we make a compendium of short
   research notes that can be downloaded as a PDF whose links are local not
   global? Presumably this has been solved by the API docs people: can you
   "download the SciPy documentation" as a single, easily navigable PDF, for
   example? Can you do the same with just SciPy stats?

In the short term, and since this is an experiment, how about we press on
with generating content in our separate notes and give ourselves an
instance of problem 2. The reason why this is a good idea is that because
if we can enable a selection of notes to be downloaded as an easily
navigable compendium, that would make all of Chris's future airplane
flights happy ones! ;-) It also would not be very difficult to refactor
into a single document by hand. Right now, the immediate problem is getting
the Twinkles Task Force members to write up their work at all ;-)

[drphilmarshall]

@TomGlanzman I added a Markdown template to your folder (and everyone else's) for you to edit. If anyone wants latex, please let me know - or submit a pull request with an update to https://github.com/DarkEnergyScienceCollaboration/ComputingInfrastructure/tree/master/doc/LSST_DESC_Notes ...

[drphilmarshall]

A Note to Note authors: as you get started writing your Notes, please note that for this experiment I have protected the Twinkles master branch, so that (I think) your Notes have to be added by pull request. This will ensure that we do actually do some sort of internal review via each PR, which should be interesting from a pub board perspective. Good luck!

[sethdigel]

This will ensure that we do actually do some sort of internal review via each PR, which should be interesting from a pub board perspective.

For what it is worth, I would not expect the Pub Board to be interested in following the internal review of DESC Notes, or journal papers. The draft Publication Policy assigns the Pub Board no role in reviewing research notes. "Research notes may be posted to a public web site maintained by the DESC (and possibly the arXiv as well) after they are reviewed for that purpose within the working group." Even for journal papers, the Pub Board will want to understand that a paper has been through internal review, but I suspect will be agnostic regarding the process. A pull request system (with the Review Committee members in charge?) might or might not be how the authors choose to draft papers.

[drphilmarshall]

@sethdigel I like the idea of using pull requests to enable task-force/working group review of Notes. I'll rework our folders into start_paper folders and place each one in a separate branch, and see how that goes...

[drphilmarshall]

PS. The checklist needs updating, based on our weekly meeting notes.

[drphilmarshall]

OK, our note list is now up to date, in the top comment of issue 284. Feel free to edit, especially the preferred format if you have had a change of heart.