docs: Initial set of documentation tests

[kevgo]

Proposed changes

Checks technical parts of the documentation using a documentation testing framework (Text-Runner) and fixes a number of issues found.

Checklist

• I have read the contributing guidelines.
• I have read the security policy.
• I confirm that this pull request does not address a security
  vulnerability. If this pull request addresses a security. vulnerability, I
  confirm that I got green light (please contact
  security@ory.sh) from the maintainers to push
  the changes.
• I have added tests that prove my fix is effective or that my feature
  works.
• I have added or changed the documentation.

Further comments

Keeping it simple for now to provide an idea of the capabilities of the tool. In later stages we can automate larger tutorials as well.

[kevgo]

This is a documentation bug found by Text-Runner. make resetdb doesn't exist. Is my fix to make test-resetdb correct?

[kevgo]

@tacurran @aeneasr you can see more details when running text-run --offline --format detailed README.md. Please hit me up if you want me to walk you through this.

[aeneasr]

Nice! This will help a lot in keeping the docs tidy!

One question I have is regarding the runenr files - it looks like these are pretty generic (e.g. "is file executable?"). Instead of copying them per repository (and subsequently maintaining them in all repos), would it be possible to load them from a stdlib or a remote or something?

[kevgo]

You hit the nail on the head: sharing actions is the last big missing feature for Text-Runner. There is a stdlib of Text-Runner actions here. But many actions are too specific to be in a global stdlib. In kevgo/text-runner#896 I'm thinking about a way to load them from NPM modules. Do you have an idea how Text-Runner could discover which NPM packages listed in package.json contain Text-Runner actions?

[aeneasr]

I would add the stdlibs to the core package. For additional things, how about

<span data-textrunner="@textrunner/mypkg"/>

which translates to

const runner = require('@textrunner/mypkg")

?

[kevgo]

Your idea to make the module name a part of the action name is brilliant! This is not only straightforward to implement, it also makes clear where a particular action comes from when debugging documentation. Thanks!!

Btw stdlib is already built into Text-Runner.

[aeneasr]

Awesome! Should we wait for upstream #896 then before merge or merge right away? If we merge right away, I think it would make sense to put this into ory/meta and build automation around synching this for all the big 4 (kratos, oathkeeper, keto, hydra). What you think?

[kevgo]

It will be pretty straightforward to finish #896. I just have to find time to get to it. The latest this weekend. Let's not invest any energy into a meta setup that will be obsolete next week. Probably better to just wait.

[aeneasr]

Nice :)

[kevgo]

Please have another look. This is a small set of initial documentation tests, with a lot more possible later. I have left some comments outlining how it works.

Thanks again for your very helpful suggestions. I have finished implementing them. Since they required breaking changes on Text-Runner, I had to implement a backlog of other planned changes in order to release Text-Runner 5.0. It ended up being quite a large release and a large-scale refactor that kept me busy until now: https://github.com/kevgo/text-runner/blob/master/CHANGELOG.md#500

[kevgo]

These targets were broken. I didn't find any targets for backers and sponsors on opencollective.com, so I link to that page in both cases instead. Please let me know if there are better URLs.

[zepatrik]

Looks like this moved to #who-is-using-it

[kevgo]

Text-Runner found this orphaned tag.

[kevgo]

Here you describe shell commands to get command-line help. I tag them as shell commands. Text-Runner executes shell commands in a subshell and verifies exit code 0.

If somebody (hypothetically) changes the Kratos source so that the CLI argument is kratos --help, Text-Runner will enforce that this documentation gets updated as well!

These shell commands only succeed if somebody runs make install first to create the Kratos binary. This could be done by other tests (end-to-end tests?) that run before these documentation tests. Text-Runner can of course also run make install. It is currently documented below and takes a while, so I'm not doing it.

[kevgo]

Here I describe that this code block contains a make command. Text-Runner looks inside the Makefile and makes sure the documented targets exist there.

[kevgo]

I tag this block as containing an executable that exists in this repository. Text-Runner verifies that this file exists and is executable. It doesn't run it. I assume you have separate tests for it.

We could run it here, then we would have to tag it to be of type shell/command.

[kevgo]

This block of type npm/installed-executable contains an executable installed by an external npm package. Text-Runner verifies it by looking for this file inside node_modules/.bin.

[kevgo]

These npm packages are a form of "standard library" for Text-Runner and contain all the code block types used in this document. Everything is reusable elsewhere.

[kevgo]

The biggest change is that tags in the documentation are no longer Text-Runner specific but describe the semantics of the document content. This makes Text-Runner a lot less invasive. Knowing what type of content the different document parts contain, Text-Runner can verify that they are correct.

[zepatrik]

Nice, do you have any guide on writing these tests? We could add a reference to that in our PR template: https://github.com/ory/meta/blob/master/templates/repository/server/.github/pull_request_template.md

[kevgo]

@zepatrik Text-Runner is documented here. Not sure Text-Runner needs to be mentioned in the PR template itself but once we adopt it more broadly it might make sense to add it to the documentation guidelines.

[zepatrik]

Right, I just thought if we add it right away people will add the textrunner annotations right away.

[aeneasr]

Hey @kevgo I wanted to check in and see what the status is for this PR! Is it ready to be merged and reviewed or does it need a bit more work? Let me know!

[kevgo]

Hey @aeneasr sorry for the miscommunication, this PR was ready for 👀 since October. Should I tag you next time?

This PR makes Text-Runner work (run via make test-docs) and fixes various identified documentation errors. Not sure if the documentation fixes are in the correct place or if the README.md file is auto-generated.

The only thing left is running the doc tests on CircleCI. They need the Kratos binary and a Node runtime. The way I typically do this is (step 1) compile the binary in a Go container and save the binary via persist_to_workspace, then (step 2) copy that binary into various containers for unit, end-to-end, and documentation tests (via attach_workspace) and run those in parallel. Please guide me if and how you want to do it here. Thanks!

[aeneasr]

Addressed your questions regarding CircleCI. For the CircleCI task itself, you could probably add this to the test pipeline over here like so:

      -
        run: make test-docs

[aeneasr]

If this is instead npm run text-run and text-run is a package.json run script, then this should already work in the CI!

[kevgo]

Thanks for the suggestions! I implemented all of them but it doesn't work on CI.

• When I run this inside the test job (as suggested by you), it doesn't find npm. I assume this job runs inside a Go container.
• When I run this inside the validate job, it doesn't have the Kratos binary.

I think the validate job seems a good place to run this. Do you have any pointers how to get the Kratos binary into that container in a way that is idiomatic with your setup?

[aeneasr]

Ah ok, I should have seen that when writing my comment. The default go image does not have NodeJS, but there is one which does:

    docker:
      - image: cimg/go:1.15-node

Replacing https://github.com/ory/kratos/pull/567/files#diff-78a8a19706dbd2a4425dd72bdab0502ed7a2cef16365ab7030a5a0588927bf47R16 with that image should then make this work!

[kevgo]

Thanks for the pointer! For future reference, the cimg/go:1.15-node image has a folder structure that is incompatible with the folder structure CircleCI expects. This causes the code checkout step to fail. The circleci/golang:1.15-node image doesn't have this problem and seems to work well.

[kevgo]

@aeneasr alright I figured it out. I'm using the circleci/golang:1.15-node Docker image, which is already used in the validate job. Have to compile Kratos manually to make it accessible in the test job. Please take another look.

[aeneasr]

Thank you so much @kevgo !