Cycle 5: Astropy UX User Interviews #504
Conversation
|
I would be interested in know more about the proposed contractor, Jenn Kotler. Maybe link or include their resume? |
|
Please react to this comment to vote on this proposal (👍, 👎, or no reaction for +0). |
|
Since I rarely downvote, an explanation: I just cannot see how interviewing 10 "astronomers" is going to help... I just taught a coding class for first-year graduate students, and, no surprise, astropy is seen like numpy and python itself, a tool that exists of which one knows little, but just googles for examples. Obviously, happy to be proven wrong, but it seems just too unlikely one will get anything useful and actionable. Sorry! |
|
Just to offer a contrasting opinion to @mhvk : As part of an NSF award, the pypeit team was required to participate in the I-Corps for POSE training, which requires each team to conduct 60 "ecosystem discovery" interviews over a period of 4 weeks (@eteq was one of the people we interviewed!). Personally, I found these interviews really useful, but they were a ton of work. There can be value in these interviews, and I think it helped us set development priorities. But here are some things to consider: (1) I would argue you need more than 10 interviews. I think 60 was too many for us, but I think 10 would have been too few. (2) This is discussed broadly in the proposal, but who you interview will be very important, particularly if you only do 10. Instead of a broad target (astronomers with a "reasonable geographic and demographic spread"), I'd recommend identifying specific groups you think might have been under-represented in the user survey. (3) I found it easy to fall into the trap of listening for what I wanted or expected to hear during the interviews. I don't have any specific advice on how to avoid this, but I would find it useful to keep this in mind when developing the interview script. Having said all that, pypeit and Astropy are rather different projects, so what's true for one won't necessarily be true for the other... |
|
@kbwestfall - thanks, that's interesting to hear. I do think the projects are very different, astropy being really more like numpy, scipy and matplotlib. So much so in fact that it is very hard to get people to think of it as something they could contribute to rather than just use it. I do share your question about how the 10 would be selected... |
|
Thanks for the info, @kbwestfall ! Is the pypeit report public and available somewhere? |
Hi @pllim . Sorry, no, there was no formal report written up, but I'd be happy to share a couple of slides that gives the breakdown demographics of the people we interviewed and some of the take-home messages. I can share them in Slack, if that's helpful. |
|
PS - @tepickering was part of the pypeit team that participated in the I-Corps training. So he likely has some useful insights into this, as well. |
|
@kbwestfall , yes I would be interested to see. This is because I understand that pypeit has an visual component, so if the takeaways are all about visualization, I suspect it wouldn't overlap much with "Astropy UX". I also am not very sure where we're going with this for Astropy. Chances are the users would want everything, and some things would conflict with each other, or ask for things we can never provide with the resources available. If you want to proceed, maybe be more concrete on what exact problems you are trying to solve with these interviews. |
|
i agree with @kbwestfall and found the I-Corps interviews a lot more useful than i expected, but also a lot of work. if you get people talking, information will come out that they might not necessarily take the time to code into a survey response. being able to interactively follow up on responses can really help dig into where pre/mis-conceptions may lie and what we might be able to do about them. i also agree that 60 was an extreme number, but 10 is probably too few unless really done right. however, i think of this as a pilot project with the script and protocol as a deliverable that could be used going forward. |
UX does not necessarily mean GUI. it's overall "user experience", how they interact with the system as a whole. |
|
We did studies on users, maintainers, etc before, but did we do anything with those results? I would rather we go wrap up previous study results than to generate even more new results that we might not act upon. |
Just posted a couple of slides in Slack. |
|
Here are some past surveys of various focus that I can think of. I think we all agreed that something should be done as follow-up but no one took the lead AFAIK.
|
This is part of Jenn Kotler's expertise. |
|
The Cycle 5 funding request process has been hugely successful! On the downside, that means our funds are severely oversubscribed. Even after the Finance Committee and SPOC have taken into consideration community feedback/voting and alignment with the roadmap, there are still more funding requests than we can afford in 2026. We would like to stretch the budget as far as possible, and to fund as many activities as possible, while making sure the Project remains volunteer-driven. Hence, we would like to know if this project will still meet its deliverables if your minimum budget is reduced by 25%, 50%, or 100%. Or if there’s some other minimum, feel free to specify that instead. As a reminder, there will be more funding for 2027 and we expect the Cycle 6 call for 2027 funding requests to begin in the Fall of 2026. Thank you for your engagement and understanding as we continue to optimize our funding and budgeting processes and the balance of volunteer vs funded work! (@eteq ) |
|
@kelle - I think given the discussion above, particularly @mhvk and @kbwestfall's points that 10 is not enough people, I stick with $3600 as the minimum. What I think I learned from the post-vote discussion above is that this is probably too low, and that if this is not approaved we may go back and try for more but with a better focus along the lines of what @kbwestfall suggested. But since I don't think an increase is an option right now we'll stick with $3600 as the minimum to get the work done (note that the work definitely will not happen without the funding, though, as Jenn's time, at least, needs to be paid for for her to be allowed to work on this). |
|
One question for @mhvk - when you said
I agree with that, but that was part of the motivation for this proposal: since people don't necessarily engage with it as much as they used to, the thought was that doing interviews by someone who's professional experience includes how to do this kind of technical interview would extract this information from people who otherwise wouldn't give it. Are you saying you don't think the information will be any good (because the people won't think of their contributions even in an interview setting), that we need more than 10 people (I take your point that a lot more thought might be needed if its just 10), or something else? |
|
My fear is that people just have not thought about this at all, and that if asked to think about it, they come up with things that in practice they would not actually do or not find important. I'm just trying to imagine what possible use one would get out of me if being interviewed about python or C or fortran. That, combined with @pllim's comment that it is hardly like we acted much on previous surveys (plus my own sense that those did not provide much that was actionable) makes me sceptical of the use. At some level, we do know what is missing (e.g., nice spectroscopy tools) and are trying to address that. |
|
I for one would love to get feedback from people in our "target" population (astronomers using software and programming on a daily basis) that know about astropy but choose not to use it for anything serious. Someone mentioned on the recent "drop support for numpy 1" email thread (I think it was @hamogu ?) that some people at NASA see astropy as unreliable and don't want to build on top of the foundation we're trying to create. I would like to understand where that stand is coming from and what, if anything, we've been doing wrong or what we could be doing differently to mitigate it. |
|
Edit to clarify: This is an answer to @neutrinoceros comment, and is not directly related to the level of funding for the project discussed in this thread. You can, however, read this as my report of one example for a point of feedback that we didn't get through GH issues, but through listening to people in the community who do not use astropy. The comment I heard was "Astropy is constantly breaking backwards compatibility, thus we should not build on top of it". This was in a zoom meeting with about a dozen people on a committee tasked to shape the future of NASA's high-energy software and data archives at HEASARC. Since this was not an astropy meeting, I did not deem is appropriate to interject and say "Can you please specify that in detail and give examples of things that you relied on that broke?", so I don't really have much more information. All I can say is that several people on the call nodded approvingly to that comment. It's not an official recommendation to "not use astropy". To the contrary, the official recommendations will likely be something along the lines of "intensify collaboration and interoperability with other open source projects in the astronomy community". However, I think a comment like that does show a sentiment that is shared by a significant (though hopefully minority) fraction of the community. Because I don't have more details, I can't say what that specific person had in mind. However, I can give some general thoughts based on what I know about the people who were part of that group: In < 15 years, astropy went from version 0.4 to 7.2 and if you have a astropy 0.4 (or even 1.0 or 2.0) script laying around, it's almost impossible that that still works - back then astropy was on python 2.x. Even Python code that relies on astropy that's only 10 years old, is unlikely to work today - sure, most individual functions are still similar, but if you write something more complex, e.g. a data reduction pipeline for some specific paper you are likely to have hundreds or thousands of LOC and somewhere in this code, there will be some something that doesn't work any longer, sometimes silently. For example, we changed the default cosmology to pre-publication Plank 2018 in 4.0, and back than, most people probably didn't explicitly set or log their cosmology, but just used the default. In 4.0 (just to stick with one example), we also changed the galactocentric frame, changed the entire modeling API, and tables don't upgrade to masked tables any longer just because one column has a mask. None of that alone makes old astropy-based code unusable, but together it means that some trip-up is very likely in code that's In contrast, HEASARC's ftools (library and command line tools to manipulate fits files, often used as building block for data reduction pipelines for X-ray missions such as NICER or IXPE), is essentially unchanged compared to 1990 or so (with some functions added), XSPEC (the default X-ray fitting program) only went from major version 11 to major version 12 in the last 15 years - compare that to astropy! So what we in Astropy think of improvements and "long" depreciation times are just the blink of an eye compared to the lifetime of large missions. Chandra, XMM, HST, all have embraced Python to some degree, but even HST, probably the most Pythonic of the large observatories, took about two decades to do it (helping to invent numpy, matplotlib, and astropy in the process). So, if a manager who today is planning for a mid-size observatory (i.e. one that will produce data for the public and fly for a few years) with a launch date coming up in 2030 or so would look for something today that they would consider "stable", they would probably want a software that doesn't change noticeably between now and mission end in 2035. In other words, they would not care if we do an LTS for 6 months or 12 months, they would want an LTS for 10 years. We just can't offer that in Python. Python itself is changing much faster than that. Maybe the first usable Python 3 was 3.3 from 2012, just over a decade ago. Python 2.7 reigned longer than that. And how many packages today are backwards compatible to Python 3.3? The Python ecosystem develops much faster than small and mid-size observatories can keep up with. |
11 participants
This is 1 business day late for the draft deadline. I had some technical and logistical issues on Friday that prevented me from submitting it then, but anyone should feel free to weight this late-submission appropriately to their own thinking when considering whether they support the proposal.