Sustainability Strategy for lessons

[acrymble]

Coming from #498, we need to develop a strategy for how authors can write in a way that minimises the need for changes in future, and for readers to understand what to expect.

[acrymble]

#499 draft blogpost had relevant text on this. But given Anandi's expertise, perhaps we should wait and do it right.

[alsalin]

I like the info in #499, but there are some things we might want to consider given the conversation during today's meeting:

Making the following explicit in the metadata at the top of each article. In my random survey, articles have this information at different spots in their introductions, which vary greatly.

• sw versions
• dependencies
• data sources
• basically anything needed to get the tutorial up and running
  Doing so would hopefully help to tailor the expectations of the user, highlighting the specific environment that the article was published for.

Sustainability ideas/questions

• I love the fact that this is addressed in Blog post on scoping lessons #499 - instead of walkthroughs, more general guidance is best. 💯
• Screenshots are the worst. They are a horror for localization and for sustainability. But we need them. I have no answer to this but just to say that it might be good to discuss screenshots in this conversation.
• We may want to keep from calling out sw add-ons that are pretty old or themselves deprecated. e.g. I saw in the Audacity tutorial that Soundflower gets called out - but it's pretty old and not being regularly updated itself.
• I like the idea of deprecating articles as they lag behind major sw releases. The issue then - with cross over into Create policy regarding ongoing revisions to lessons #498 (I think that was the one) is that if the sw changes drastically, are the lessons still derivative? For example, if we have someone make a killer ArcMap-based article for us, that has a shelf life of a few years until ESRI drops its by fancy complete design. Will a tutorial in the new ArcMap replacement that accomplishes the same end goal as the old ArcMap w/u still be derivative?

That's all I have for now!

[mdlincoln]

One note that I think @JMParr raised in the meeting today by skype message was that a reviewer for an R lesson submission said they'd require that the lesson be resubmitted using currently-maintained R packages. I think this is a great example of peer review working as intended, where a specialist in that particular language/software brings subject-specific sustainability issues to the attention of the author & editor.

Could we add a bullet point to the reviewer guidelines asking them (if they have the knowledge of the particular software environment) to evaluate how up-to-date the methodology and tooling is? We already ask them to consider the sustainability of external datasets.

[JMParr]

This sounds like a good bullet point to the reviewer guidelines. Certainly, we want to encourage the peer reviewers to point out when there are newer versions of software packages.

…

On Wed, Jul 26, 2017 at 8:16 PM, Matthew Lincoln ***@***.***> wrote: One note that I think @JMParr <https://github.com/jmparr> raised in the meeting today by skype message was that a reviewer for an R lesson submission said they'd require that the lesson be resubmitted using currently-maintained R packages. I think this is a great example of peer review *working as intended*, where a specialist in that particular language/software brings subject-specific issues to the attention of the author & editor. Could we add a bullet point to the reviewer guidelines asking them (if they have the knowledge of the particular software environment) to evaluate how up-to-date the methodology and tooling is? We already ask them to consider the sustainability of external datasets. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#534 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AVruWzTJCJhNuL9_LzsUs7oxtljPMFP8ks5sR9bhgaJpZM4OkMBO> .

[acrymble]

One of our ongoing sustainability strategies has been to encourage Wikipedia links for technical terminology and jargon. I think that's worked really well.

[mdlincoln]

From 2017-08-24 meeting: @alsalin will compile ideas from this thread and #498 and create a list of changes/additions for author, editor, and reviewer guidelines

[walshbr]

Following on the conversation in #596 - @alsalin I'm wondering if there are any other particular things that you (or anyone else) see that should be done for the backlog of lessons? Largely was thinking about this in the context of making sure that we're hosting all of our own assets, which @mdlincoln and others just did for several older lessons. @alsalin were you suggesting a more thorough sustainability review, or just working towards more proactive sustainability processes for the future. I'm all for either one, but a review of past lessons seems like a much more substantial investment of time.

I also think it would be helpful to frame some sustainability guidelines, for all of those communities but particularly reviewers, in terms of concrete things that they can be checked. Things like: what sort of sites does the lesson link to, how many external pieces of software are discussed, etc. Perhaps obvious, but I'm thinking of people like me, who recognize that sustainability is important in the abstract but am less clear about hard and fast rules I should watch out for.

[alsalin]

@walshbr I've been following the self-hosting of assets and it's a fabulous idea. I hadn't really considered an audit of previous lessons because, as you mentioned, it would be a substantial investment. That's not to say if someone sees an issue in an older article that we shouldn't address it. I was framing my suggestions for future lessons and was considering language for author, editor, and reviewer guidelines. There would be kind of a suggested checklist but then also some general guidelines to follow. Mostly my notes need to be in complete sentences, so I'll be posting some ideas soon.

[mdlincoln]

Some specific problems that show up in links:

• Do you link to the webpage for corporate commercial software like Apple or Microsoft? These are commonly NOT canonical links, and so break with high incidence. Linking to google search results for software like this can even be more effective!
• When linking to articles, try to use a DOI if possible. Even these sometimes fail, but then at least there is a DOI consortium that gets angry at the publishers for you.

[mdlincoln]

@alsalin I think it'll be important to cross-reference the lesson retirement policy with these sustainability guidelines - so I'd like them to go under the same PR #612 (not to mention, it'll help unify all the changes, thus making the Spanish team's job easier.) That means we'll need to get your suggested edits before closing out the lesson retirement policy changes. Do you have a realistic timeline for submitting these for everyone to look at?

[alsalin]

@mdlincoln I hope to have some drafts up in the next day or so. apologies for the delay, but I've been a bit under the weather the last week or so.

[mdlincoln]

No rush @alsalin - just need a realistic timeline, that's all.

[alsalin]

Hi all, here are some suggestions on verbage for sustainability. These are merely suggestions for placement and content. I've tried to summarize suggestions from a few different places, but please review for accuracy and appropriateness. Rather than repeat everything in every section, I tried to word the guidelines for each audience as best as I could. Question: would it be helpful in any of these sections to call out the retirement policy?

Editor Guidelines

Under Open Peer Review
"Before soliciting external reviews, the editor should read and try the tutorial and use their experience with the Programming Historian to help the author make initial improvements (if required)." Add The editor should complete an initial sustainability overview of the submission to ensure that software versions and dependencies are clearly marked, specificities of software like screenshots are limited to those required to complete the lesson, and that the lesson makes use of existing software documentation whenever available and appropriate. Editors should also ensure that lessons try, as much as possible, to avoid software specific directions, such as "Right-click on the • icon to access the • menu," instead favoring general methodological overviews. The Editorial Checklist contains more details about sustainability practices for PH.

Under Technical Processes of Review
Potentially insert after B:
Sustainability Review
To increase the lifespan of our articles, PH editors should complete a sustainability review as a part of their final checklist. Every submission is different and some of these areas may not be applicable to all submissions. Editors should use these areas as guidelines to ensure that lessons are as sustainable as possible from the date of publication.

• all software versions and dependencies are described in the introduction to the lesson
• sources of data for lessons are clearly noted and should be hosted at PH whenever possible
• lessons make use of existing software documentation whenever possible
• lessons link to Wikipedia for technical terminology
• screenshots of software GUIs are limited to those that are required to understand the lesson
• external links (e.g. software or data sources) are current and live though authors should consider directing users to documentation generally rather than providing links to specific documentation pages
• links to articles use DOIs if available

Reviewer Guidelines

Under What to Comment On
Potentially insert at the end: Sustainability
To increase the lifespan of our articles, PH reviewers should keep in mind the following questions about sustainability. Every submission is different and some of these areas may not be applicable to all submissions. Reviewers should use these questions as guidelines to ensure that lessons are as sustainable as possible from the date of publication.

• Are all software versions and dependencies listed in the submission? Are these assets the most recent versions? If the lesson uses older software versions, does the author note why?
• If you have expertise in the specific methodology or tool(s) for the lesson, is the methodology generally up-to-date?
• What are the data sources for the submission? Are they included in a way that does not heavily on third-party hosting?
• What kinds of other external links does the submission use? Are these current or are there other, more recent or appropriate, resources that could be linked to?

Author Guidelines

Under Proposal Guidelines (taken from #499)
To aid in the sustainability of our lessons, authors should seek to submit lessons that are not overly dependent upon specific software or user interfaces. These lessons inevitably break or need substantial revision when a new version comes out. Teaching concepts rather than 'click on the X button' helps make for sustainable tutorials.

Under Writing a New Lesson
Write Sustainably
PH strives to publish lessons that will be of use to our readers for the foreseeable future. To aid in creating sustainable lessons, we ask that you keep certain writing guidelines in mind as you create your lesson:

• Instead of focusing on software specifics, keep your lesson more geared towards methodologies and tool generalities.
• If your lesson can leverage existing software documentation, consider directing your readers to this documentation rather than repeating it in the lesson. Instead of linking directly to a software company's documentation pages (which often change), you can provide general guidance on how to find the documentation.
• Limit the use of software version-specific images, unless required to follow the lesson.
• Check any external links to ensure they are live and up-to-date.
• Consider hosting your lesson's data sources at PH.

[acrymble]

This is great. I thin we can add in new advice as and when we come across problems.

[ianmilligan1]

This is really good, thanks @alsalin!

My only hesitation is the verbiage around:

Editors should also ensure that lessons try, as much as possible, to avoid software specific directions, such as "Right-click on the • icon to access the • menu," instead favoring general methodological overviews".

I've always thought that this one of the Programming Historian's strengths: being able to take overall methodological concepts (which are important), but then assuming a low-level of technical knowledge and really breaking things down for users. Reminding them to double-click an archive file to open it, or to change the directory into the one that they just downloaded, or that x or y is hidden behind a menu item in the Tools menu. It does require updates as things change, but it also makes it usable to a person who's coming in and might not have used their system for much beyond Word or Excel.

As tutorials, especially beginner-level ones, I think we should be OK for people giving very detailed instructions.

[walshbr]

Looks really great to me. And just to underscore what @ianmilligan1 is saying - I had the same thoughts. In general, when I've been involved, those sort of additions have been based on extended back and forth with reviewers/authors who can't figure out how to do something. Ie - "It says click this to download, but it's not downloading. It's still not downloading." etc. And if something requires that sort of discussion behind the scenes for people who we're helping through the process, it seems worth documenting for a general readership. To some extent that's a question of basic web/digital literacy, but I'd rather err on the side of readability in that case. Maybe Ian is right in that it's a question of determining the difficulty level that the tutorial is pitched at. Personally I'm inclined to err towards detailed instructions in most cases, but I recognize that isn't necessarily always called for given the sustainability discussion and the difficulty rating system we've been developing.

[alsalin]

@ianmilligan1 @walshbr excellent points - I think we can certainly add language that the reviewers and editors should be aware of the level that the lesson is geared towards when making these determinations. also, it may be possible, if we implement something like hypothes.is in the future, for some of these details to be annotated and then updated by our readership. in the meantime, I think it will have to be up to the editors/reviewers to determine how much specificity is too much given the audience for the lesson.

[acrymble]

Good points all. Giving editors something to think about is different from prohibiting screenshots/instructions, etc. So if we're thinking about 'guidelines' then I think it's worth reminding editors and reviewers that certain decisions may have a future sustainability impact.

[mdlincoln]

🥇 these are really great @alsalin - as for where to reference the retirement policy, I'd link to it in the author guidelines at the end of the Write Sustainably section, e.g.

Authors should also consult our lesson retirement policy, which describes how the Programming Historian editorial team manages lessons that have become out-of-date.

Please strengthen one statement in the author guidelines:

• Consider hosting your lesson's data sources at PH.
• Data sources for lessons should be hosted at PH.

This aligns the author guidelines with the editor guidelines, which correctly state that data should almost always be hosted on our site.

[mdlincoln]

@alsalin can you please go ahead and push these edits to the archiving-policy branch of this repository? That way they can become part of the #612 PR and we can preview everything in one place.

[alsalin]

@mdlincoln do we want to wait until #617 discussion to push the edits? i'm fine with either, but i wasn't sure since it seemed like sustainability was on the meeting agenda.

[mdlincoln]

We're not merging the changes yet, just adding them to the PR so that people can preview them as they would appear on the published site - something I think will help anchor conversation. Plus, if they're committed as changes in a PR, then we can do inline commenting, so it's much easier to have concrete discussions.

[alsalin]

@mdlincoln great! i will make my attempts probably tomorrow and hopefully not destroy everything :)

[alsalin]

@mdlincoln I gave it the ol' college try. please verify that I didn't ruin PR #612 ;)

[mdlincoln]

You didn't.

All further conversation about this issue should now take place on #612

[mdlincoln]

Closed by #612