Planning for Community Discussion on Programming Historian Goals

[walshbr]

During the August call - #553 - we discussed #552 and decided that the conversation would be a good opportunity to engage the community again in the same mode as #152. The conversation internally about the goals and mission of PH is one thing, but would be useful to hear from the larger community about how they see the project, what they get out of it, what would be useful, etc. We agreed that a ticket for public consumption could be a good approach, so this ticket is meant to serve as planning for that community conversation:

What kinds of perspectives are we interested in? What questions and conversations do we want to cultivate? What is the goal of the conversation? Is GitHub ticket the best format?

After some brainstorming we can move to engaging the community more directly.

[acrymble]

I think survey monkey will be a good idea

[acrymble]

Sorry I missed the meeting. Who is the target of this consultation? As I've said a number of times, what authors got or thought they'd get out of it might be starkly different from what readers do.

Can we focus this? And what do we intend to do with it? If our readers all suggest that we take a completely different approach than we want to, do we all step down?

[walshbr]

Would you mind weighing in and reframing your thoughts on this @alsalin? I believe the idea was yours, and would want people to respond to it as you articulate it rather than as it gets inflected through my memory of the conversation.

[alsalin]

hi all, actually the initial survey idea came from @mariajoafana, I believe. the sense of it that I recall was that it might be time to check in with our readers/users to get a sense of how they use the materials they find through this publication. the survey might end up being part content survey and UX survey - how do you find our content? what do you do with it? do you use it as a reference? do you read through it in its entirety (which I'd be shocked to hear, but cool if so)? etc. I think a survey monkey or google form would be fine and we can use this space to cook up some appropriate questions. @mariajoafana - did I get the genera overview right? what did I miss?

[drjwbaker]

A general survey with elements that help us reframe our goals would be sensible. I'd be keen to ensure that we know who those completing the survey are (crudely, historian or not historian) and where they are from (geographically). I'd also be keen to ensure that we see this as a substantial piece of work, involving things like:

• keeping the survey open for a long period (6 months?)
• clear articulation of why we asking readers for their time
• in person/conference promotion of the survey
• an international strategy so we don't just get our friends completing the survey
• banners on all the lessons (start and finish?) asking readers to complete the survey (and why)
• a short co-authored report/paper based on the survey

[drjwbaker]

(I should add, sorry if a) any of this was covered in the meeting I couldn't make b) this is totally off piste and not what you want to do :) )

[acrymble]

We'd need a separate initiative for Spanish, I would think.

Should we start drafting some questions somewhere?

[walshbr]

Like all of those points a lot @drjwbaker.
@acrymble - I had intended this ticket to serve as a space for drafting of questions and the conversations around it, so go ahead. But if you think that deserves its own particular ticket feel free to move the conversation to another issue. Don't really see a problem with having that conversation happen in the open, but we could move to the google group if people desire.

[acrymble]

Of course, @walshbr.

Our earlier survey on gender had these questions, which might be repurposable:

1. Which of the following ways have you engaged with the Programming Historian? (reader, reviewer, editor, author, educator, none of the above).
2. On a scale of 1 (novice) to 6 (advanced), how would you rate your confidence with technology?

[walshbr]

I like those. Maybe something like these too -

1. In what ways does the Programming Historian enable your work? What works well for you?
2. What would help the Programming Historian be more useful for you? What could we do better?

But those might be a bit too pointed in the direction of utility, and imagine that we might want more specific questions as well. Those are a bit vague and general, but could be general grab bag, write what you will opportunities for capturing general public feeling. Those also might point a little too much towards the pedagogical aims of the project than the research publication side.

[mariajoafana]

Yes, @alsalin that sounds right. I agree with @acrymble on the need for a separate survey for Spanish.
For the survey we did in the Spanish team last year these questions might be useful here:

1. Choose your disciplinary field(s) (information science, history, art history, linguistics, literature, archeology, etc...)
2. Which digital methodologies, tools and/or workflows do you use often in your research and teaching?

[drjwbaker]

Quick holding note. I've been reading a great pedagogy book and it has an excellent section on structuring feedback. Takehomes: don't ask about things you don't want to change; structure questions in a particular less-open ended way that means you can easily analyse the results. Anyway, I don't have the book in front of me right now (hence the holding note). More tomorrow!

[acrymble]

Good points @drjwbaker. What do we want to open for change here? Not everything obviously. Presumably we don't want to consult on the lessons page having just spent a year building it. What do we want to know?

We can learn a lot about user behaviour from our Google Analytics, so it should be something we can't already find out. Knowing what proportion of our readers are humanities scholars might be helpful (though we won't be able to capture that easily). And finding out about research made possible from the lessons would also be great. But neither would be about changing the project.

I'd be more interested in consulting with our authors, because I see those as the group with whom we have the closest relationship and greatest obligations towards. The readers seem to come no matter what, but we have to be good to our authors.

[drjwbaker]

Okay. The module feedback/evaluation thing I found uses the following format.

--

Part 1 is split into two columns. Column 1 is headed 'I like the way the XXX:' and Column 2 'I would like the XXX to:'. It then has 15-20 statements in each columns that the respondent can check if they agree with it.

So, we could do something like:

Column A

I like the way the Programming Historian:
_is written by historians.
_has enabled me to do better research.
_has supported my teaching.
_is peer reviewed.
_is open access.
_has a Spanish version.

Column B

I would like the Programming Historian to:
_summarize the main points more often.
_drop peer review.
_enable me to edit lessons more easily.
_seek people willing to translate its lessons in my native language.

Then in Part 2 of the survey it has one or two more open-ended questions which people can write free text responses to.

--

The up side of using this is that it give us lots of quantifiable and easy to manage feedback, helps us steer the survey towards things we are willing to change, and encourages respondents to only write in free text things the things they really care about or which they think are not on the survey. My hunch - if we go with this model - is that we use Part A to mostly find out if what we think PH is is shared by our users (I'm not too bothered if we use leading questions, this isn't contributing to proper social research after all) + then sprinkle in a few more controversial topics that we are willing to shift on (eg, wikis, user editing, depreciation).

Oh and the book this is from is Teaching What You Don't Know. Which is super awesome and one of the many things that @gvwilson has put me onto over the years.

[drjwbaker]

If we go ahead with a survey, note idea to tie to mission statement #552 (comment)

[walshbr]

Is someone willing to take on this initiative and/or #579? My sense (that I articulated on #644) is that, as conversation has died down, we're either ready to table these for now or take action. My suspicion is that this might be an initiative that we're interested in but lacking the bandwidth for at the moment given the current people on hiatus (was thinking especially that this might be something to save for when @alsalin has more time to devote to this).

[drjwbaker]

@walshbr Revisit in the new year? PH has changed a lot this year. It would be useful to see if we are on track with our users.

[walshbr]

Discussion at #678 was that it makes sense to think in the direction of a survey. Assigning @alsalin to continue the discussion working off of what has been shared here so far.

[alsalin]

Hi folks!

To summarize, here are the suggested questions so far, though of course the way we end up phrasing them may be different (e.g. 'add specifics to guide reader as suggested by @drjwbaker and always have an 'other' field):

• Which of the following ways have you engaged with the Programming Historian? (reader, reviewer, editor, author, educator, none of the above).
• On a scale of 1 (novice) to 6 (advanced), how would you rate your confidence with technology?
• In what ways does the Programming Historian enable your work? What works well for you? (add specifics to guide reader)
• What would help the Programming Historian be more useful for you? What could we do better? (add specifics to guide reader)
• Choose your disciplinary field(s) (information science, history, art history, linguistics, literature, archeology, etc...)
• Which digital methodologies, tools and/or workflows do you use often in your research and teaching? (add specifics to guide reader)

Some more ideas?

• Where do you primarily do your DH work? (some kind of geography question)
• How do you primarily find PH lessons? (social media, google search, peer suggestion? - though we have analytics for this, this question is tied directly to the other given answers here)
• As a user of PH, would it be helpful to interact with other users of PH to discuss methods and issues with implementation of lessons? (e.g. I'm learning python and getting stuck with lesson x, who else is too?) If so, what would you like to see?
• On the PH website, are lessons easy to find?
• Do you prefer to read through an entire lesson or skim to find relevant information? (alternatively, we could ask a question about learning styles).

Other ideas? For editors, readers, authors? In UX and engagement? In content?

[acrymble]

What would we do with the information we gained from this survey? What process does this feed into?

[drjwbaker]

On which, a good way of writing a question is to think about what we might do with the answers.

For example - as @alsalin notes - it seems clear we can say we would use responses to 'How do you primarily find PH lessons?' in two ways: 1) compare/contrast with data we have on how people find PH; 2) revise comms strategy.

On 'What would help the Programming Historian be more useful for you? What could we do better?' - however - we'd have to ensure we only include things we are willing to action.

Great starting point @alsalin. I vote for keeping the survey simple and not adding more (it should take no more than 2-3 minutes to complete), and focusing - as @acrymble suggests - on what we want to get out of responses to each question.

[alsalin]

@acrymble excellent question. As I was summarizing the questions we had already, I found that topics fit into certain categories: knowing our audience (and thus reaching out to them more effectively, choosing content to publish that matches this audience and/or expanding audience), website experience (a good thing from a QA perspective after a redesign anyway), content audit of sorts (is what we have what our audience needs/expects). Not that we have to go with this set of questions at all. We could choose categories to investigate and work from there. Either way, I think we need to ensure that we are willing to engage with the survey results to change things at some level, e.g. better audience engagement or different audience engagement, more lessons on x, etc. Now what process does this feed into? That I'm not sure :) Not sure if there are existing pathways for this kind of thing, though I know surveys have been done before at PH - how were those interpreted and what came out of them?

@drjwbaker I agree - let's keep it short, preferably to a page of questions - two, my goodness, at tops.

[drjwbaker]

@alsalin Sure. To clarify, do you mean that you need help with the technical side of hosting a blog on PH or with writing? (I'm guessing more the former?)

[alsalin]

@drjwbaker yep! also, would it be best to post a draft for comment in this forum or elsewhere?

[drjwbaker]

@alsalin Not sure what the protocol is here. I've seen some blogs posted here first for comment. @mdlincoln: can we stage a blog for preview via Jekyll like we do lessons?

[mdlincoln]

Yep, every PR gets a generated preview so blog posts will show up on netlify just like lessons will.

[drjwbaker]

Thanks for confirming @mdlincoln

So @alsalin lets draft branching off https://github.com/programminghistorian/jekyll/tree/gh-pages/_posts (using a recent lesson as a template for headers et al). Are happy to make a start?

[alsalin]

@drjwbaker @mdlincoln - I've posted the draft here: #803 - Let me know if that all looks ok from a technical perspective and then beyond that, I'd love edits, suggestions, and co-writers for anyone who wants to add content!

[drjwbaker]

@alsalin I'll look tomorrow morning.

[drjwbaker]

@walshbr Once @alsalin gets #803 posted, can we push it through the twitter bot until the close of the survey? (15 July)

[walshbr]

@drjwbaker - the bot isn't super intelligent as of right now. It's only set up to tweet at random from the English and Spanish tabs from the master spreadsheet, once per day M-R with a different task depending on the day. So part of the issue would be that we would just spend weeks waiting for it to get to those publicity-oriented tweets. What kind of behavior would you want for something like this? I can imagine having a third spreadsheet tab containing promotional tweets that we have it work through, once per week on a day we pick. And we can turn that on or off and modify the contents of it based on whatever is going on presently. Someone would still have to write the tweets, but they could just be a handful of variations on the same idea. Would that work? Could also throw some tweets in about the workshop so that @acrymble and @mariajoafana get some help promoting, if they'd like.

[acrymble]

Could just schedule tweets with tweetdeck if needed

…

On Wednesday, April 18, 2018, Brandon Walsh ***@***.***> wrote: @drjwbaker <https://github.com/drjwbaker> - the bot isn't super intelligent as of right now. It's only set up to tweet at random from the English and Spanish tabs from the master spreadsheet, once per day M-R with a different task depending on the day. So part of the issue would be that we would just spend weeks waiting for it to get to those publicity-oriented tweets. What kind of behavior would you want for something like this? I can imagine having a third spreadsheet tab containing promotional tweets that we have it work through, once per week on a day we pick. And we can turn that on or off and modify the contents of it based on whatever is going on presently. Someone would still have to write the tweets, but they could just be a handful of variations on the same idea. Would that work? Could also throw some tweets in about the workshop so that @acrymble <https://github.com/acrymble> and @mariajoafana <https://github.com/mariajoafana> get some help promoting, if they'd like. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#576 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAnd90KdEgbuoYIbVYLl-kV_Z7-WKguMks5tpzeVgaJpZM4PBre1> .

[drjwbaker]

@walshbr Could a hack me to add lots of duplicates of posts we want to promote to the spreadsheet?

[drjwbaker]

Blog up https://programminghistorian.org/posts/programming-historian-community-survey Great job @alsalin. Seen as you can see the responses, any chance you could report on response rate at our upcoming team calls?

[drjwbaker]

@alsalin See https://twitter.com/anterotesis/status/986901226292998145 Should it read..

Which digital methodologies, tools and/or workflows do you primarily use often in your research and teaching?

..or be multiple choice?

[alsalin]

@drjwbaker - a multiple choice snuck in there. Everything has been changed now to check boxes so you can select multiple items.

[walshbr]

@drjwbaker - We could theoretically add a bunch of duplicate tweets to the sheet to weight the randomizer towards promotional tweets, but I'm inclined to keep the pieces of the script directed towards single tasks. I.e. - the script has two main functions at the moment that tweet about a lesson in English or Spanish. Rather than mixing that functionality with promotional stuff, I'd want to add promotional functions. Keeping the things separate would make them easier to troubleshoot and maintain and reduce the chances of me or us gumming up the works. Particularly since it only tweets out once per week effectively on each topic, odds would still be high that weeks would go by without promoting the thing you want.

I'm happy to add the functionality you're talking about to the bot if it seems useful. But I do agree with @acrymble that tweaking the bot to accommodate things like this sounds a bit like recreating Tweetdeck's functionality. Only seems useful if we'll be wanting to tweet out more than say…ten versions or iterations of the same tweet? We'll still have to write the tweets, and the bot just keeps us from having to click through Tweetdeck. Probably worth opening a separate ticket to continue the conversation - let me know what you think and feel free to assign it to me if you decide it's worth having.

[acrymble]

If you email me the tweets verbatim with a list of when you want them sent out, I will schedule it in Tweetdeck.

…

On Thu, Apr 19, 2018 at 2:12 PM, Brandon Walsh ***@***.***> wrote: @drjwbaker <https://github.com/drjwbaker> - We could theoretically add a bunch of duplicate tweets to the sheet to weight the randomizer towards promotional tweets, but I'm inclined to keep the pieces of the script directed towards single tasks. I.e. - the script has two main functions at the moment that tweet about a lesson in English or Spanish. Rather than mixing that functionality with promotional stuff, I'd want to add promotional functions. Keeping the things separate would make them easier to troubleshoot and maintain and reduce the chances of me or us gumming up the works. Particularly since it only tweets out once per week effectively on each topic, odds would still be high that weeks would go by without promoting the thing you want. I'm happy to add the functionality you're talking about to the bot if it seems useful. But I do agree with @acrymble <https://github.com/acrymble> that tweaking the bot to accommodate things like this sounds a bit like recreating Tweetdeck's functionality. Only seems useful if we'll be wanting to tweet out more than say…ten versions or iterations of the same tweet? We'll still have to write the tweets, and the bot just keeps us from having to click through Tweetdeck. Probably worth opening a separate ticket to continue the conversation - let me know what you think and feel free to assign it to me if you decide it's worth having. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#576 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAnd9yRd0qXzooZsFez58n_T0rL5iyu4ks5tqI0sgaJpZM4PBre1> .

[alsalin]

@drjwbaker re: survey responses - happy to update at any time! Maybe we can add a regular to-do to our meeting agendas to go over the responses for the month up until the survey closes. Already some very interesting responses! I also appreciate that people are pointing out where the survey and its categories fail. I think that information is crucial!

[alsalin]

May Update!

• 39 submissions so far
• All submissions so far consider themselves to be more advanced than novice, >4 (1-6)
• 2 non-academic positions
• primary discipline: History
• Text analysis reigns supreme 26/39, data cleaning 25/39, GIS 1, data vis, 21, network analysis 11/39, web development 17/39, statistical analysis 1/39, VR 2/39
• People are finding PH through primarily social media and/or going directly to website (but is this because we are only marketing the survey on Twitter and the blog?)
• People generally us PH for reference and as a learning tool for the first time
• Improvement suggestions
  -- making tutorials for global audience
  -- more variations on a theme, ex python both structured and unstructured text
  -- more traditional journal-like listing of lessons - can be difficult to tell how old a lesson is from browsing page
  -- more advanced lessons x 2in python, more APIs to other databases, advanced AI for algorithmic historiography
  -- variety of technical levels for lessons x3, updating lessons
  -- better search options x6
  -- FR language lessons x2 :)
  -- guides on what i need to learn to do X task more efficiently
  -- lessons in German x2 (also someone volunteered their services)
  -- combine lessons into learning paths
  -- combo of written and video tutorials
• for the most part lessons are easy to understand and get up and running
• instructions for authoring and reviewing for the most part clear
• things we could do better
  -- open call for topics/techniques

[arojascastro]

Thank you @alsalin for the report. Very interesting findings! I'll share on my social networks and list in the following days.

[drjwbaker]

Details of PH gmail account here https://github.com/programminghistorian/jekyll/wiki/Service-Integrations

[acrymble]

@alsalin I believe the plan was that you were going to open new tickets for any findings that came out of the survey which we need to discuss or act upon. If possible, could you do that so we can close this issue?

[alsalin]

@acrymble yep! It's gotten pushed back but I plan to have this completed before our monthly meeting. Once that's done I'll update this issue and close it out.

[alsalin]

Moving discussion to individual issues now!