[epic] non-API docs need to handle different versions of Qiskit & Runtime

[Eric-Arellano]

See Abby's comment below for an explanation of the overall problems and some solutions. That discussion resulted in this tasklist:

Tasks

Preview Give feedback

1. Add a way to dynamically render the version of Qiskit, Runtime, and Aer to notebooks #922
   infra 🏗️ +1
2. Automate updating requirements for notebook testing #920
   infra 🏗️ +1
3. Set up cron job to test notebooks that submit jobs #917
   infra 🏗️ +1
   

[javabster]

Ok so this issue may be larger than we originally thought, I suggest we convert this to an epic and create issues for smaller pieces. there are a few different things to consider:

• A user should be told somehow when they land on a page what versions of each package are being used in that page
• A user should reasonably be able to trust that a code example they see in a non-api ref page is up to date and runnable without errors
• when a new minor version of qiskit comes out (every 3 months between majors) it could throw deprecation warnings, we should aim to have NO deprecation warnings in live docs
• when a new minor version of qiskit comes out (every 3 months between majors) it could introduce a new feature, or new recommended way of doing something
• In general we should be encouraging users to use the latest version of a given package. BUT we also have some responsibility to continue documenting versions that are still actively maintained. For example, say once qiskit v2.0 comes out, qiskit 1.0 will still continue to be maintained for a further 6 months according to the semantic version strategy that the qiskit team follows. During this period v1.0 is still considered a valid version for people to use, hence we have some responsibility to keep 1.0 documentation alive until 1.0 itself reaches end of life.
• we use multiple different packages. qiskit is the only package currently with a robust, predictable versioning strategy. Other packages like runtime are used often in pages but does not have a predictable versioning timeline. so whatever strategy we choose will likely not perfectly align with all packages

The proposed solutions to the above constraints (some of these still need more in depth discussion):

Code block testing

Code cells in non-api reference pages should be tested in our CI, and they should test against Qiskit and Qiskit Runtime. For Qiskit code, they should be tested with any currently maintained major versions, and the latest minor version. E.g. if the latest version is v1.3 it should be tested with v1.0 and v1.3. E.g. If the latest version is v2.1 it should be tested with v1.0, v2.0 and v2.1. For non-qiskit packages that have not yet reached major versioning, I recommend we test with at least the last 6months of minor releases, or maybe even the last year?

We should update our contributing or bot processes to remind people to write/update tests for their code blocks when they submit a new page or update an existing one

For discussion: should we also test with the latest dev version of a package? So we can pre-emptively adjust content if anything breaks with an upcoming version?

For discussion: should we test with other ecosystem packages beyond qiskit and runtime? e.g. Aer and quantum serverless are important packages in a handful of pages

Version tables

On every non-api reference page a user should be able to see a list of the (IBM Quantum) packages used and what version. If we are testing all the code blocks in that page with the latest version, perhaps there is a way to dynamically update a version table with the latest version?

for discussion: how should this version table be shown on the page? Just a simple version table like we used to use in Qiskit tutorials? do we need some design input here?

Changes to code blocks

This one is tricky. Consider the hypothetical: Qiskit 1.2 is released and it introduces a new feature or a new "recommended" way of doing something that differs from v1.0. This requires updating a certain page in the docs. How should we approach this update? We always want to show people the most up-to-date recommended way of doing something, however we also want to keep documentation alive for people using the latest major version, who have not yet updated to the latest minor, and possibly won't update until 2.0. This problem persists even when 2.0 comes out because 1.0 will continue to be maintained 6 months after the release of 2.0.

There are a few options to tackle this, but they all have their drawbacks:

Option A:
We update the page to use the new feature, and add an admonition underneath the new code block showing how to do this with the old way. This option is fine when it is only small changes that are required, however if there is a large feature it may require many updates to one page, or even a complete rewrite, in which case the page would be covered in admonitions which isn't a great user experience.

Option B:
We look into how to display different versions of non-api reference pages with a version switcher, similar to how we do things in the API reference. I anticipate this will be a bit of a nightmare to support, as we don't just need to consider Qiskit versions, but also runtime and any other packages used in the page. With API reference a version switcher is simpler to implement because all the packages have their content separated. Perhaps there is a way to do this with custom components? Maybe some kind of code block that includes tabbing between versions (kind of like this)? I'm open to creative solutions here if anyone has a bright idea 💡

Option C:
If code blocks differ between versions, we include a single admonition (either at the start or end of the page) that mentions there has been a change since the last major version, and if a user wishes to see the older version provide them with a link to the old notebook in our GitHub repo history. The main drawbacks here are that a user will have to scan through the older version carefully to spot the differences themselves and it makes the last major version feel less legitimate than it actually is (e.g. it is perfectly ok for a user to still be using qiskit 1.0 when 1.3 comes out)

[Eric-Arellano]

and if a user wishes to see the older version provide them with a link to the old notebook in our GitHub repo history.

For your Option C, I was envisioning it still being a normal rendered page in the documentation. I agree with you it's valuable for the page to still look legit. We also need to render it in the site and not simply show the notebook for our custom components to work properly, like admonitions.

I'm not sure if that 'legacy' page would show up in the left sidebar or not, i.e. whether it would be an orphan page.

[javabster]

some user feedback expressing desire for clearer versioning info: https://fosstodon.org/@rdviii@famichiki.jp/111610533262044382

[arnaucasau]

In my opinion, option B is the best for the user experience. I think we can have some sort of nested tabbing to show the versions of different packages (qiskit and runtime).

For Qiskit, I think we can show the first minor version of the 2 supported major versions (in the first 6 months). In addition, if a new release introduces a new feature or a new "recommended" way of doing something we could show the latest minor version as well. With these we'll have a standard way of doing things in the major that will always work thanks to 1.0, and the most recent/recommended. The standard method could be denoted by the major version dot x (e,g, 1.x or 2.x).

For runtime, we could show, for example, the last two released versions to encourage users to use the latest version.

UX proposed

I have created a mock-up using Figma to depict an example of what I'm proposing (I'm sure the design could be improved but it helps to show the idea). Nested tabbing can be confusing, but we could have some "tag system" where we allow the user to select which versions they want and also provide an easy way of showing each version.

Each code block would have a collapsable header for each package to show the version of the code block:

Once we click on the header of a package, we'll see all the versions available for that package, allowing the user to click and change it as needed. Once a new version is selected, we could collapse the header again.

In case we want to show more than 2 versions for runtime, we could group together versions with the same code with a tag like this to avoid duplicating entries and having too many options:

The tag system, or any option with separate version selectors for each package, could have incompatibilities of available combinations. One option could be to disable some tags and show a message when hovering explaining why the option is not available. The disabled option would change depending on what version of Qiskit we have selected.

How to write docs

In the documentation, we would have as many code blocks as combinations of versions we have with some id to be able to prune incompatible combinations and to help decide when to show each code. This means that the author would have to write multiple code blocks. Limiting the number of versions we show per package would be useful here to avoid having a combinatorial explosion.

This "problem" is common for different UX options and can be treated independently from the UX proposed in the previous section.

[Eric-Arellano]

We decided that with the current resources this is not feasible, and it's also less pressing because of the nature of Runtime server updates meaning that you either way need to be using relatively new versions of the APIs. However, we do think it's important to show what version you're on: #922

This can be reopened in the future if we decide it becomes a priority, such as if Runtime has better stability guarantees or we have more resources.