DOC: Discussion on new getting started page (Numfocus Grant)

[stijnvanhoey]

As currently prepared together with @jorisvandenbossche, the following document is a proposal on how to adjust the getting started page and section of the documentation: https://docs.google.com/document/d/1Rc_eql5KLrdf0c582KyWfs2ADVNxbJy4jfosnqdrVak/edit?usp=sharing

The general idea is to split the current 10 minutes into 10 x 1' topics of Pandas. Input is very welcome on:

• the topics to pick and how to introduce them
• the content of the introduction tutorial linked to each of the topics

[jorisvandenbossche]

cc @pandas-dev/pandas-core

[datapythonista]

I don't like the idea much. I'd prefer to have few conventional end to end tutorials (solving a real world problem), targeting different use cases and users types.

If I' had never heard about pandas, I think an "introductory tutorial" on how to open data (covering many options and formats) is not what I'm looking for. And it's not giving me relevant information on whether pandas is the tool I need or not. I think the user guide already addresses that quite well, and if we want tl;dr versions of the user guide pages (may be a good idea), I think that could be added at the beginning of each of them.

What I'd personally appreciate is proper tutorials, for example one could be "Forecasting the stock market", where it's shown:

• How to open data (common formats for finance, I guess csv mainly)
• Setting a datetime index
• Reindexing
• Imputing missing values (ffill and the common methods for time series/stock data)
• Windows functions
• Plotting
• ...

Of course that would only be useful for part of users, but I guess with 3 or 4 tutorials that cover the main pandas use cases, or introductory topics we can find. Finding them is surely not easy, but things that come to my mind:

• Data exploratory (stats, groupby, visualization,...)
• Data engineering (cleaning data, reshaping, joining,...)
• Time series (the above mentioned)

I'd also add a more theoretical one, to teach the concepts (dtypes, data structures,...)

[stijnvanhoey]

Good point, but I do not completely agree. I do not think that, to make the documentation more welcoming to absolute beginners, advanced case studies ('proper tutorials') will help as the complexity will easily overwhelm new users.

What we want to cover with the new getting started page is to provide different elements in line with http://third-bit.com/2019/04/16/what-docs-when.html:

• An overview by providing the 10 advertisement statements to introduce some key Pandas concepts. Purpose: makes people aware something exists
• For each of the topics a short tutorial is the way to help people build a mental model of Pandas elements and acquire a few basic skills and redirect them to sections in the user guide.
• Keeping the link to other languages in there as well provides the translation for users

Note: case studies are mentioned to help users from a competent level to expert level.

Still, your point:

if we want tl;dr versions of the user guide pages (may be a good idea), I think that could be added at the beginning of each of them.

is true. Having tutorials in the getting started section on topics of the user guide is making the documentation more scattered. But this is not the purpose. To be clear, we are actually just talking about a new version of the current 10 minutes and nothing more elaborate. So, the aim of the proposed introduction tutorials is not to cover many options and formats, but very short introductions applied on small data sets (e.g. titanic) and real questions (e.g. I 'm interested in the titanic passengers older than 18 years). When the document says 'content to cover', this is NOT to explain these functions in detail, but to create a short storyline using these functions to answer the question. By doing so, we align with the how to section described in http://third-bit.com/2019/04/16/what-docs-when.html

[jorisvandenbossche]

Thanks for the feedback!

And it's not giving me relevant information on whether pandas is the tool I need or not. I think the user guide already addresses that quite well,

I personally think the user guide is failing hugely on that aspect. IMO no beginner should read one of our pages of the user guide; they are way too overwhelming and unstructured. But that is maybe more saying something about those pages needing some love ... :)

I always had the long term idea to split all the user guide pages in multiple subpages, with one introductory page per topic, and then several pages going in more detail in one of the subtopics (eg Indexing page: one introduction page explaining the different concepts, and then pages with more detail on advanced indexing, multi-indexing, ..).
In such a setting, it is true that the introduction tutorials of the above proposal could also fit as those introductory pages of the different user guide sections. And also then we could still keep the concept of the getting started overview but directly linking to the user guide introductory pages instead to introduction tutorials of each page.

That's certainly an option to consider (and the effort Stijn is putting in those pages could be easily redirected to user guide intro's). We don't necessarily need to restrict the work of this grant strictly to the getting started section.

---

Now, then to the question: what does a beginner need? (which is of course difficult to answer, as it is subjective and you have many different "kinds" of beginners). I personally think such a full "case study" can be easily overwhelming for newcomers, and that it can be useful to have smaller, more focused tutorials.
That is not to say that I wouldn't find such case studies useful to have in our docs as well, but I see them more for people that are already have some basic knowledge.

Terminology is always difficult :-) I would call your "proper tutorial" rather a case study or showcase, but yeah, the definitions of and boundaries between tutorials, how-to's, case studies, user guides, ... is not always clear. We used the "A tutorial is a planned lesson that helps people build a mental model of a domain and acquire a few basic skills" of Greg Wilson when speaking about tutorials here.

[datapythonista]

I think I'm misunderstanding your document. What you say about covering very short introductions applied on small data sets (e.g. titanic) and real questions (e.g. I 'm interested in the titanic passengers older than 18 years) is exactly what I'm proposing. Except that I wouldn't call them very short, I think they'll have a decent size unless you really want to answer a single question.

[datapythonista]

@jorisvandenbossche I agree on what you say (splitting the user guides...). You misunderstood (I phrased poorly) the text you quote. What I think the user guide addresses correctly is providing information about a specific topic (a "tutorial" about IO), which is what I understand from the google doc. Not whether pandas is the right tool for a new user. That's exactly the opposite of what I was trying to say.

I think we need to discuss this more in practice. Write some drafts, and try to get people not familiar with pandas involved. And get their feedback as we progress.

[jorisvandenbossche]

What you say about covering very short introductions applied on small data sets (e.g. titanic) and real questions (e.g. I 'm interested in the titanic passengers older than 18 years) is exactly what I'm proposing.

@datapythonista OK, then it would be good if you (and also others of course) could look at the specific objectives/content to cover of the different tutorials proposed in the google doc. That would be very useful.
It is always a difficult exercise to pick those items out of the full pandas API that you want to show and teach in a beginner tutorial. This also depends on the domain you are coming from, your own pandas coding habits that you have grown, etc. For example, out of your example tutorial content above (#26831 (comment)), I would not include reindexing or windowing functions in a beginner tutorial.

[jorisvandenbossche]

I would not include reindexing or windowing functions in a beginner tutorial.

Although about the windowing functions, those are maybe quite common in the financial world I suppose? A reason to include after all, those kind of things would be good to get feedback on.

[jreback]

why are we trying to recreate a branch new tutorial, rather than simply copying from all of the tutorials out there & consolidate that?

[datapythonista]

My view is that the whole point of the tutorials is to let people know about what kind of things they can do with pandas. I don't expect newcomers to pandas to be experts in reindexing, window functions or whatever after reading a tutorial. But if they are not in the tutorial, how do we expect users to know they exist and use them? I'd say people go to the user guide or the API when they know what they are looking for, I'd expect the tutorial to give a quick overview of most features.

[stijnvanhoey]

@jreback that is what we are doing, e.g. the links provided by the twitter responds.

[WillAyd]

I personally like this idea and the reference to the Django docs. From a practical use case, workflows more often that look something like:

1. Ingest data from a datsource
2. Clean / reshape / merge
3. Perform calculations
4. Visualize

I would think this is particular true for new users coming through say a Jupyter notebook, which is where I think is very friendly to newcomers.

The ten points more or less cover these and can link to deeper discussions from there.

I'd probably move plotting to be the last point out of these as kind of an end goal. Specific to point 9 I'd also make some emphasis around data types preceding it (like converting from a string to a date, why that's important) and relatedly we might want to touch lightly on why dtypes are important (not from a technical perspective, but maybe show object vs numeric somehow)

[datapythonista]

@stijnvanhoey is there any work on this that can be already seen? I need to start preparing the pandas tutorial for EuroSciPy, and I'll try to reuse material from what we're building for the official docs if it makes sense, so I can get feedback from the audience.

[stijnvanhoey]

@datapythonista I missed yourquestion on the update of material. Based on the proposed plan with the 10 section getting started, I'm compiling now a first draft compilation of these 10 sections as notebooks in https://github.com/stijnvanhoey/pandas-getting-started-tutorials. Currently working on section 9 and 10.

The idea is to have a returning pattern in the way these are setup: illustrative figure (just as intro), question/task-oriented (currently organized as quote- > sections), short explanations and redirections to relevant sections in the user guide (to user guide). At the end of the notebook a to remember section with redirection to the main section in the user guide. I used either the titanic or air quality data for the exercises.

After a revision of the material, I'll further adjust/extend the tutorials and start on the way these will be represented in the documentation (from sphinx-integration to binder-support and proper styling of these elements together with other elements in the getting started page itself.

It is a bit a struggle to find the balance between short/concise and enough informative, so any comments are certainly welcome. I do not know if the material is also fit for a workshop (as it focuses on self-learning), but it would be very interesting to test the material and the flow/logic of the content. Unfortunately I can't make it myself to EuroSciPy, but if I can do anything in the preparation, just ask!

At the same time, I'm trying to get together a set of (more) consistent schemas to explain specific concepts, see https://github.com/stijnvanhoey/pandas-getting-started-tutorials/tree/master/schemas. These still need further revision, but feel free to use and try them out already.

[datapythonista]

Great, thanks for sharing the repo.

I think it'll be quite difficult to me (and probably many others) to give feedback on whether the tutorials are good and clear enough for people starting in pandas. So, I'll see if I can use them in conferences, and gather feedback from the kind of people we're targeting with them. I'll keep you updated, thanks!

[stijnvanhoey]

Good point, certainly because we aim here the real beginners. I'll pitch/spread the material in the carpentry community the coming weeks to gather some feedback as well and check around for local workshops/contacts to test the material.

[datapythonista]

Just got the feedback that in the tutorials I'm giving at conferences I should include a tips and tricks section. Also, I think @justmarkham is having a lot of success with this way of teaching pandas. I don't have an exact proposal on how we could include those, but I think it's worth considering. One idea could be to have some info boxes with a "Did you know...?" tip during the tutorial.

[justmarkham]

Thanks for letting me know about this discussion, @datapythonista! As for pandas tips and tricks, I'm not sure the best way to integrate them into teaching materials, though I do like the idea of a "did you know" info box.

Zooming out: I haven't had a chance to digest the discussion and links above, but I'm happy to take a look and provide feedback, if it's helpful? I've been teaching pandas online for 5 years, and the biggest segment of my audience is beginner users, so I can speak somewhat knowledgeably about the beginner's perspective.

[datapythonista]

If you have time to have a look at what's been discussed and give us feedback, that would be extremely valuable. The goal here is to make people new to pandas understand what it does and get started as fast and easily as possible for them. I doubt there are many people that can add as much value as you. :)

[datapythonista]

@stijnvanhoey I was checking the notebooks, and as I said when we first discussed, I think we should have tutorials in the documentation, understanding by tutorial a notebook that shows end to end how to solve a real-life problem.

Not opposed at all to have what you are building, shorter and more focused on the features. But I think some users will appreciate to have what I propse. To me personally, when I started with Django it was very useful to follow the tutorial on how to build a poll system end to end, and I wish I had such a thing when I started with pandas.

I added to this page (https://datapythonista.github.io/pandas-web/try.html) couple of ideas of tutorials I'd like to have. Based on use cases similar to the ones that some users may have (data analysis, preprocessing to pass data to scikit-learn, and time series analysis). I'll probably build the first two for EuroSciPy, so we can discuss over them once I'm done.

[stijnvanhoey]

I do think this is complementary to the getting started notebooks, aka 10x1' material, serving a different purpose. The end-to-end provide an idea about the capabilities (what sort of tasks can I do with the tools provided by Pandas), whereas he 10x1' to pandas provides an introduction/gateway to the main features (which tools are provided by Pandas).

@jorisvandenbossche and myself developed some of these end-to-end tutorials as case studies, oriented on environmental/life sciences:

• case1_bike_count.ipynb
• case2_biodiversity_analysis.ipynb
• case2_biodiversity_processing.ipynb
• case3_bacterial_resistance_lab_experiment.ipynb
• case4_air_quality_analysis.ipynb
• case4_air_quality_processing.ipynb

Try it out on binder.

I could certainly rework some of these as well to add to the end-to-end examples. Check them out and let me know. As we're aiming for starters/beginners, work is required to keep it concise and _understandable. The challenge will also be to provide a good variety in the topics/themes, maybe the input of the recent questionnaire provides useful info for this?

[WillAyd]

Per Joris' email today I gave the notebooks a quick look. Here are just some of my thoughts:

1. Generally I didn't understand what the Content To Cover bullets were supposed to represent and I think these should be removed and/or rolled into the objectives
2. I think the first few pages make too many references to dtypes, and having worked with a lot of people coming from Excel / Access and to a certain extent SQL backgrounds I think that makes an assumption about data types that not every analyst would have. I'd prefer if we removed anything exposing dtype info (like calls to .dtypes, .info(), etc... or at least deferred it to a discussion later in the getting started
3. Particularly in the first chapter I think we should introduce a Series before a DataFrame, as the latter logically builds upon the former

Have to give it a few more passes but there's some quick thoughts.

Generally are we tied to including these in the documentation as Notebooks? I find the standard Notebook rendering pretty ugly and I don't see interactivity of these really offering that much more to a user than copy / pasting into their own notebook or terminal session. I'm also not sure what kind of maintenance hosting those notebooks directly would incur (if any)

W.r.t the website I think design is moving in the right direction. The code rendering is a little large but sounds like you are already on that. Looking forward to the next round of updates!

[stijnvanhoey]

Thanks for the feedback, certainly valid input. Some quick responds:

• Content To Cover is purely for the preparation, not part of the content. This was the list of elements to incorporate as defined in the preparation/discussion with Joris.
• Good point with respect to the dtypes, we could shift this to later sections. I agree.
• With respect to Series versus DataFrame, the idea is that people are typically first exposed to DataFrames (e.g. reading from a csv) than Series, hence putting that first.

We could do it in notebooks and integrate into sphinx (e.g. using nbsphinx), but we planned to convert the tutorials content (when more polished) into the documentation pages itself (i.e. rst). With the update of the website, we have the bootstrap classes available to customize the looks and we can provide some custom css (e.g. proper styling of the questions, now in the notebook as a quote) to provide integration with a new getting started page.

[jorisvandenbossche]

W.r.t the website I think design is moving in the right direction. The code rendering is a little large but sounds like you are already on that. Looking forward to the next round of updates!

Yes, that big code rendering is a left-over of some of the css of the old website (which is already being removed in a PR, that should fix that issue). More feedback on usability is very welcome on the theme as well, but let's keep that in #15556

[WillAyd]

@stijnvanhoey cool all sounds good - thanks for the response

[jorisvandenbossche]

Small note here that we made PRs for each notebook (still on the separate repo) which can be reviewed with the ReviewNB app (which makes it easier to review and comment on notebooks): https://github.com/stijnvanhoey/pandas-getting-started-tutorials/pulls

[datapythonista]

Read most of the tutorials, and my feedback is the same as before. I wouldn't personally continue after the first tutorial, since it's too theoretical, and don't really feel engaged or curious about what's being explained.

Working with Pandas always requires import Pandas as pd: is false, I work with pandas all the time and never use import pandas as pd. :)

I read half of the tutorials, and the module name pandas/pd is used once in every file. I still think that we should avoid this unpythonic and not really useful practice in our documentation.

[jreback]

Working with Pandas always requires import Pandas as pd: is false, I work with pandas all the time and never use import pandas as pd. :)

this a convention (so not required) and is the most common import pattern and so appropriate to include

[stijnvanhoey]

@datapythonista, I understand your point; This is indeed not required and just a community 'convention'. If we actually include it or not, is your (core dev) call. I'll adjust the text asap.

With respect to the first tutorial, I'll try to convert to a more engaging version. As mentioned earlier, the 'dtype' details will be removed to decrease theoretical level of it. Feel free to add more comments in the repo itself. Thanks for reviewing the material!

[jorisvandenbossche]

Working with Pandas always requires import Pandas as pd: is false, I work with pandas all the time and never use import pandas as pd. :)

I think I have said this before as well, but: you are welcome to start a specific discussion about this. But as long as we didn't have that, the standard we use in our docs is the status quo which is pd.
(the wording of course should reflect that this is not required, only a standard practice. This is already the case in the text itself IMO, only in the "remember summary" it was a bit stated strongly)

[jorisvandenbossche]

Read most of the tutorials, and my feedback is the same as before. I wouldn't personally continue after the first tutorial, since it's too theoretical, and don't really feel engaged or curious about what's being explained.

I agree on the first notebook that we should try to make it a bit more attractive. I also commented about that on the review PR: stijnvanhoey/pandas-getting-started-tutorials#2.

For us, the goal of those getting started pages is to teach the readers some basic concepts about pandas, to give them a better understanding of how to think about pandas.
It was explicitly not our goal of those pages to be kind of an advertisement to show fancy, powerful things that you can achieve with pandas, or to show full case studies (the idea is that this interest is sparked before, and that people who then want to learn more end up at those pages). And that is not to say that something like that wouldn't be useful to have on the website, not at all. But it was not what we were focussing on here. And that is also not to say that those getting started pages can be boring. We should try our best to make them as interesting as possible while keeping the goal of clearly explaining some concepts.

@datapythonista So if you have some more concrete feedback on what you would do differently, or what you would otherwise cover in the first notebooks, that would be very welcome.

If we still want to change the concept more fundamentally (which I have the feeling you are hinting on, Marc? Although it is not fully clear what your alternative is), we should decide that now, to decide what we think is best for Stijn to spend the last 3 days on (over the coming weeks). So for that it would also be welcome if some other people can chime in on this with their thoughts.
(as now it feels a bit like we are just ignoring Marc's comments (while, Marc, it is great that you are engaging here and that is very much appreciated!), but as long as there are not more people sharing this concern and there is not a clear alternative, it is difficult to change course).

[datapythonista]

Sorry for the noise with the alias. I agree that's a topic for a separate issue, sorry about that. I think we've already got too many open discussions with the web, so I'll create that issue at a later stage.

I don't have more specific feedback. I don't think restarting from zero with a different approach is an option now, with just 3 days of work left. Just wanted to express that for my case personally, and the way I learn, I don't think the notebooks would be useful for me. I don't have any other feedback on that more than what we already discussed in the past. I agree having feedback from several people would be ideal, even more if they are beginners, the target of those documents. Sorry I can't be of much help in this.

[justmarkham]

Hello all! I mentioned previously that I'd be happy to review the draft notebooks. Although I'm not a pandas beginner, I've been teaching pandas to beginners for many years and so I am familiar with the beginner perspective.

Sorry that it has taken so long, but I've now taken a thorough look through the draft notebooks and will post my specific comments via ReviewNB.

However, I also have some high-level comments:

1. Regarding the format of these lessons, I think that it will appeal to some beginners and not others. Some beginners will like the chosen format, which I would describe as "each page teaches you a few core concepts". Other beginners might have preferred the "case study format" that @datapythonista has mentioned, because they would get a chance to see a longer workflow and more advanced functionality, which may be more inspirational and give them a better sense of the available functionality, but would likely teach them fewer skills (since the emphasis is on demonstration rather than skill-building). That is all to say that no format will please everyone, and the chosen format is definitely a valid choice! This is truly a tough task and I commend everyone who has worked on this.

2. The included graphics (which @stijnvanhoey referred to as "schemas" above) are excellent. I think they add a lot of value to these lessons. If you've only looked at the notebooks on GitHub or via ReviewNB, you are missing out. Instead, you can see the graphics via nbviewer.

3. I like the "remember" section at the bottom of each lesson. Even though these lessons aren't that long, it's easy to lose track of the key points as you read through the document, so the recap is helpful.

4. I would propose that the lesson titles be changed from statements to questions, such as "How do I select the data I need?" rather than "Select the data you need", and "How do I visualize my data?" rather than "Plotting comes out of the box". My opinion is that the question format is more engaging for the reader, and makes it more clear to the reader what they are going to learn in each lesson. (I use this question format when titling my pandas videos and I think it has worked well.)

5. Two concepts that I think need greater highlighting in the notebooks are axis and index. Both are critically important topics that beginners are frequently confused by. I think axis is only mentioned once in the notebooks and only briefly. I think there are two times when the notebooks briefly mention that "index" is equivalent to "row labels", but otherwise it's not really discussed (as far as I can remember). Anyway, I'm mentioning this here because I don't know exactly where in the notebooks it belongs, but I do think that the notebooks should answer the questions of "why does this thing exist" and "how do I use it" (for both the axis and the index).

Hope that is helpful! I'll try to get all of my specific comments into ReviewNB today.

[jorisvandenbossche]

@justmarkham thanks a lot for looking at it and this feedback! That is really valuable and very much appreciated.

I created the PRs for the two remaining notebooks.

[jorisvandenbossche]

One other aspect to discuss: in what kind of format do we want to include this?

The plan has always been to, at some point (i.e. now), convert the notebooks in the separate repo to rst, clean them up (convert to use ipython directive, etc), and include then in that way in our sphinx docs.

But, there are actually more options, also in light of @TomAugspurger's recent exploration of putting our docs on binder (#27514, which converts out rst files to notebooks using sphinxcontrib-jupyter).
We already include the styling page as a notebook (https://dev.pandas.io/docs/user_guide/style.html) using the nbsphinx extension. So we could also use the same approach for the getting started tutorials. I think the main advantage is that it would give better notebooks to expose on eg binder (not automatically generated ones). It might also be easier to edit the source as notebooks, although reviewing PRs that edited them will be harder (but saving as markdown with jupytext might make this better). Also a bunch of rst features such as auto-linking to the API docs cannot be used in notebooks.
But as long as we don't plan to switch all our docs to notebook, there is maybe no good reason to do it for those pages while having the rest in rst, as that just creates inconsistency (for the styling page there is a good reason: we need the html output). On the other hand, this could be a case to experiment with it.

For now, we will go with rst as planned, but mentioning it here, in case we want to discuss this (although a new issue might be better suited in case we want a general discussion about this).

[stijnvanhoey]

With respect to the conversion of the tutorials towards the documentation (getting started) pages, we will css-style the recurring elements (links to user guide, remember sections, ref to raw data,...) according to the following example html: https://stijnvanhoey.github.io/pandas-getting-started-tutorials/. All adjustments are bootstrap (already loaded) classes with some CSS.

In practice: the .rst files will get raw html and/or custom directive to translate the content to the proper html. Feedback is off course welcome.

[datapythonista]

What happened with this? Was this finally discontinued?

[stijnvanhoey]

Setup is tested on the new sphinx theme, currently preparing PR for pandas itself. Can be expected coming week. Op vr 10 jan. 2020 18:01 schreef Marc Garcia <notifications@github.com>:

…

What happened with this? Was this finally discontinued? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#26831?email_source=notifications&email_token=AAFYJLRNIIOAMJOF23HMZ63Q5CSVRA5CNFSM4HXYZSXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIURT3Y#issuecomment-573118959>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFYJLWBPIBO3HGMU6Q5XWDQ5CSVRANCNFSM4HXYZSXA> .