Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade Checkbox Tutorial #613

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 117 additions & 0 deletions docs/tutorial/base_tutorial.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
.. _tutorials:

=============
Base Tutorial
=============

This uses a new tutorial TP, its a small addition but could make a
huge difference, we can custom tailor a new user experience with it.
It will be referenced as tutorial-plan

..

This uses explain and introduce interchangeably, by explain I don’t
mean a 300 pages novella about the feature but a dry simple
explanation of its purpose and how to use it # Introduction Here
mention what checkbox is

Installing checkbox
-------------------

Here we introduce snaps and debs for checkbox, introduce here the
concept of ``core`` and ``frontend``.

Outcome: checkbox.checkbox-cli or checkbox-cli starting

Running First Test Plan
-----------------------

Here we can explain a little bit the different kind of jobs that we have
while we encounter them in the execution, explain comments and outcomes.

Running a job will also ask if you want to submit it, it may be worth
explaining what that means! Also: outcome is saved, point to where and
how

Outcome: tutorial-plan results

Using basic launcher
--------------------

~ ``checkbox-cli launcher.conf`` Outcome: Auto-starting tutorial-plan
test, Filtered test list

Using basic configuration
-------------------------

Explain how configs are used, **where to put them** Outcome:
Auto-starting tutorial-plan test via conf, filtered test list, also:
``check-config`` usage for config validation/see what is going on

Machine Manifest
----------------

Explain how the machine manifest works/is created and where it is.
Create one with the create manifest job, now re launch and edit it. Edit
it via text editor.

Using them all
--------------

Here explain how conf/MM/launcher are prioritized between them (launcher
wins) and make the user do a couple of examples like setting start.

Recall here ``check-config``, and show how it tracks the origin of
values

Outcomes: Auto starting tutorial-plan via conf and launcher,
``check-config`` source explained

Remote testing
==============

Here say a couple of words on the motivation of remote testing, say that
if run with no args, checkbox-cli is local (and deprecated? useful to
check local configs? idk)

Basic
-----

Explicitly start two ends, one per terminal, to play around with them.

Outcome: Two terminals, one running agent, one running controller,
tutorial-plan tests run “remotely”

Configuration
-------------

Explicitly explain how configuration interact with agent/controller,
with examples go through who reads what how, recall ``check-config`` and
show it works remotely

Outcome: A couple of combination of configurations with examples,
running tests remotely with filters etc.

Test Output
===========

Here guide the user through an output, use the tutorial plan output.

Outcome: Understanding of the main, most common fields of the output

Once here, the reader of the tutorial should have a good knowledge of
the basics of checkbox, he should be able to run tests, comprehend
the output and see/modify/debug the configuration

Advanced Configs
================

Here explain every feature of config files, recall ``check-config``
somewhere so that if someone skims through the guide it is known.

This page is a pretty good beginning:
https://canonical-checkbox.readthedocs-hosted.com/en/stable/reference/launcher.html

The only thing to change is to prune uncommon config values and
explanation and have an actionable config, per section, that the user
can load and “experience”.
141 changes: 141 additions & 0 deletions docs/tutorial/extended_tutorial.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
.. _adv_tutorials:

=================
Extended Tutorial
=================

Hacking basics
==============

Link here to the QA tutorial for “normal users” and recall that we offer
snaps and debs. The
`CONTRIBUTING.md <https://github.com/canonical/checkbox/blob/main/CONTRIBUTING.md>`__
guide is a good basic as it explains how to install, sideload and launch
both remotely and locally, how to create new providers etc..

Basic -> Only use the most common options, think of ``name`` not
``category-overrides``

Basic Test Plan
===============

Create a simple test plan that includes a few test cases from tutorial
or smoke, for example, to get the hang of it.

Recall here side-loading.

Outcome: Run the created test plan, show output

Basic Test Case
===============

Now that we know how to create a custom test plan, lets create a custom
test case.

Auto Test Case
--------------

Introduce here the basic concepts of what a test case and write a simple
one. As a command do not use anything complicated (``true`` may be too
simple, but something easy like that, do not overwhelm the reader, here
he has to learn test cases, not linux).

Link back to previous to say how to include this in a test plan.

Outcome: Run the new test plan, show outcome, also create a failing one
and check that it does fail

Manual Test Case
----------------

Introduce here the basics of the purpose/idea of a manual test case.

Outcome: Run the test plan, show how the new test case is presented and
check that pass/fail is reported correctly

--------------

From here onward, we introduce features and use them in a test
case/plan, so that after each chapter it is clear what they do, how
to use them and how to use what they produce in practice

Category
========

Outcome: Create a category, assign it to a test case

Resources
=========

Here mention what a resource is and name spacing, redirect to reference
for more.

Basic usage
-----------

Here introduce how resources are formatted and write a simple resource
job. This can be: is a file there?

Now write a test that uses this resource (command true?), show that it
passes, remove the file, show that the job is skipped, introduce here
``fail-on-resource``. Show that it fails.

Outcome: Test job that uses resources, understands how to create simple
ones and use them

Combining resources
-------------------

Here create a second resource job, this can be is another file present?

Now write a test that uses one, one that uses the other and one that
uses both (``and``/``or``). Show how they are skipped or run for the
given resource evaluation results.

Consider pointing to the complexity note in the reference here

Ourcome: Two resource jobs, three tests, knowledge on how resource
expressions work.

Manifest
========

Introduce what a manifest entity is and recall where the disk cache is
stored.

Here we should only explain how to define a manifest and introduce the
basic kws (point to reference for more), then create a test plan that
uses one of the manifest entities that we just defined and run it,
filling the manifest as prompted.

Mention here the job to show all manifest entities

Outcome: Test that uses a manifest entity, knowledge on where the
manifest is store/how to read it

Templates
=========

Here introduce templates, point to reference for more.

Basic Template
--------------

The basic example
`here <https://canonical-checkbox.readthedocs-hosted.com/en/stable/reference/units/template.html#basic-example>`__
is pretty ok, I would simplify it a bit by removing one resource (3
echos) and the filtering (``requires``) as it is explained above. Also,
less text, more doing. Command should be just a plain ``echo`` so that
the user can familiarize with the template substitution

Outcome: Template that the user can try out and see the jobs that are
generated/run them and see a comprehensible output

Jinja Templates
---------------

Explain here how to use jinja templates (``template-engine: jinja2``),
and provide a basic usage. Create a template that generates a job using
at least one jinja feature, let the user run the generated test plan.

Outcome: Template that generates a test plan via jinja
13 changes: 9 additions & 4 deletions docs/tutorial/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,19 +20,24 @@ problems, we want to help you, so please let us know - `get support <#>`_.
Core tutorial
-------------

Follow the core tutorial steps in sequence; they take you on a learning
Follow the base tutorial steps in sequence; they take you on a learning
journey through the product.

.. toctree::
:maxdepth: 1

tutorial
base_tutorial

Extended tutorial
-----------------

Once you have completed the core tutorial, the extended optional tutorial
Once you have completed the base tutorial, the extended optional tutorial
sections can be followed in any order - they don’t depend on each other.

Follow the core tutorial steps in sequence; they take you on a learning journey
Follow the base tutorial steps in sequence; they take you on a learning journey
through the product.

.. toctree::
:maxdepth: 1

extended_tutorial
Loading
Loading