From 84cc1a63c30cc6304d4e8d0cdd4a811dd3a8149d Mon Sep 17 00:00:00 2001 From: Johannes Ernst Date: Thu, 19 Dec 2024 20:13:49 -0800 Subject: [PATCH] Update for 0.5 --- content/_index.md | 2 +- .../malformed-messages.md | 2 +- content/quickstart/other-developer/_index.md | 2 +- content/quickstart/sandbox/_index.md | 159 ++++++++++++++++++ content/release-notes/0.5.md | 55 ++++++ 5 files changed, 217 insertions(+), 3 deletions(-) create mode 100644 content/quickstart/sandbox/_index.md create mode 100644 content/release-notes/0.5.md diff --git a/content/_index.md b/content/_index.md index 95d1fc4..fcffa33 100644 --- a/content/_index.md +++ b/content/_index.md @@ -40,7 +40,7 @@ Older {{% pageref "/blog" "posts "%}} and {{% pageref "/release-notes" "release {{% /box %}} {{% note %}} -FediTest is at version 0.4. Given this version number, don't expect perfection 😎 +FediTest is at version 0.5. Given this version number, don't expect perfection 😎 But it is beginning to be useful. {{% /note %}} diff --git a/content/faq/faq-feditest-tests-fediverse/malformed-messages.md b/content/faq/faq-feditest-tests-fediverse/malformed-messages.md index 0473058..4cc36c2 100644 --- a/content/faq/faq-feditest-tests-fediverse/malformed-messages.md +++ b/content/faq/faq-feditest-tests-fediverse/malformed-messages.md @@ -12,7 +12,7 @@ a {{% gl Node %}}, and that can be instructed to do various terrible things, suc as emitting malformed messages. We tentatively call this {{% gl app %}} the "Hungarian Babelbox" (think of it as Douglas Adams' [Babelfish](https://hitchhikers.fandom.com/wiki/Babel_Fish) using Monty Python's -[Hungarian Phrasebook](https://www.youtube.com/watch?v=grA5XmBRC6g) +[Hungarian Phrasebook](https://www.youtube.com/watch?v=grA5XmBRC6g).) It would essentially: diff --git a/content/quickstart/other-developer/_index.md b/content/quickstart/other-developer/_index.md index bb2468e..b2b8e91 100644 --- a/content/quickstart/other-developer/_index.md +++ b/content/quickstart/other-developer/_index.md @@ -1,7 +1,7 @@ --- title: Quickstart for developers using protocols that aren't part of the ActivityPub-based Fediverse (NOT YET) breadcrumbtitle: For other developers -weight: 30 +weight: 40 display: hide --- diff --git a/content/quickstart/sandbox/_index.md b/content/quickstart/sandbox/_index.md new file mode 100644 index 0000000..4387187 --- /dev/null +++ b/content/quickstart/sandbox/_index.md @@ -0,0 +1,159 @@ +--- +title: 'Understanding FediTest: the toy "sandbox" protocol' +breadcrumbtitle: Sandbox protocol +weight: 30 +--- + +The FediTest testing framework can be used for testing any distributed system, using +any protocol, not just the ActivityPub-based Fediverse. + +To illustrate how, we created an extremely trivial example protocol, the "sandbox" protocol, +that hopefully makes it easy to understand how FediTest works and how it can be applied to +other protocols. + +## Getting the code + +* Get FediTest: + + ``` + $ pip install feditest + ``` + +* Get the tests that test the Sandbox protocol (for readability, shown on several lines): + + ``` + $ git clone --recurse-submodules \ + https://github.com/fediverse-devnet/feditest-tests-sandbox.git + $ cd feditest-tests-sandbox + $ git checkout v$(feditest version) + ``` + +## The Sandbox protocol + +This trivial toy protocol is a client-server protocol that enables a client to ask +a server to perform a very simple multiplication on its behalf. The key interfaces +are contained in +[this file](https://github.com/fediverse-devnet/feditest/blob/develop/src/feditest/protocols/sandbox/__init__.py). + +### Client-Server interaction + +The protocol used "in production" (well, this is a toy example, so there is no production, +but there could be), is simply this: + +``` +class SandboxMultServer(Node): + def mult(self, a: float, b: float) -> float: + ... +``` +When a client invokes the `mult` method on the server, the server will calculate the product of the two +variables `a` and `b` and return the result. + +From a testing perspective, we want to know: + +* Does a server that implements this interface implement it correctly? Does it return + the product of the two values, or something else? We want to know this for many + combinations of values, including corner cases. + +* Does a client that uses this interface invoke the interface correctly? For example, + it might invoke the API with a number outside of the supported range (e.g. positive + numbers only). + +To be able to test this, we need two things: + +* We need to be able to write a test script that "makes the client" invoke the server, + and return the results to the test script. This test script then can test that + 2 * 7 indeed returns 14 and the server is doing the right thing for this and other + values. + +* We need to be able to "instrument the server" to log how the client invoked the server, + and what values, to make sure the client is doing the right thing, and not, for + example, dropping minus signs. + +### Extra interfaces + +To make this possible, extra instrumentation operations are provided on the `Nodes` in +the Sandbox protocol, specifically: + +On the server {{% gl Node %}}, activate logging and return the log of invocations by +the client: + +``` +class SandboxMultServer(Node): + def start_logging(self): + ... + + def get_and_clear_log(self) -> List[SandboxLogEvent]: + ... +``` + +On the client {{% gl Node %}}, make the client invoke the server with particular +arguments and return the result: + +``` +class SandboxMultClient(Node): + def cause_mult(self, server: SandboxMultServer, a: float, b: float) -> float: + ... +``` + +(This may be a little confusing as this operation appears to make the Sandbox "client" a +"server". It does, but only with respect to FediTest and the tests run by FediTest. +These tests need to be able "make the client do things". The client still remains only +a client with respect to the to-be-tested protocol and the Sandbox server.) + +### Running Sandbox tests + +Using the above Git repo, you can run the Sandbox tests against two different +implementations of the server interface -- one of which is correct, and one of which is +faulty (so you can see what happens). + +The easiest way to do this is to run `make`, like this: + +``` +$ make -f Makefile.create +``` + +This will create the various {{% pageref "/reference/json-files/" %}} used by FediTest. +Then, to run the actual tests: + +``` +$ make -f Makefile.run +``` + +The generated HTML reports are in `examples/testresults/`. + +(To clean up, there is also a "clean" target for both Makefiles.) + +## Applying this to a real-world protocol + +We suggest you follow an approach such as this: + +* Identify the roles of {{% gls Node %}} that you have in your system. You may have a traditional + client-server system, or you may have a P2P system, or a more complex architecture + (such as a [Peer Computing Architecture](https://peercomputing.net./) as in the + ActivityPub-based Fediverse). + +* Identify the interaction patterns between the {{% gls Node %}} with the different roles. + For example, as in case of ActivityPub, you may have some HTTPS POST interactions. + +* Identify some tests cases, along the lines of 2 * 7 = 14, as above: Which Node is + going to interact with which other Nodes, saying what? What is the expected behavior, and + what would be faulty behavior? How can it be determined whether it was correct or + faulty? (This might be to check a return value as in the Sandbox example, or it might + take a database query to look whether some data was updated correctly.) + +* Write down the test cases similarly to how we wrote the tests cases for the Sandbox + protocol. You will notice that to automate that, you need "extra methods" on your + {{% gls Node %}} similarly to how we added the "Extra interfaces" for the Sandbox + protocol above. Some will be "control" methods: "make the system do something", and some + will be "observe" methods: "check what happened". Sometimes the same method can do both. + +* How many of those methods you want to define depends on a tradeoff between test coverage + enabled by those methods, vs effort to create and maintain them. THat's ultimately up to + you, and you can certainly implement them incrementally. + +* And then, depending on how your {{% gls Node %}} can be deployed so they can be tested, + implement a suitable {{% gl NodeDriver %}}. In case of the Sandbox protocol above, + this was trivial: the implementations of the {{% gls Node %}} are simply Python code + that is run in the same address space as FediTest. In the real world, this may be more + complicated and you may need to implement something based on {{% gl UbosGears %}} as + we did for Fediverse software such as Mastodon. diff --git a/content/release-notes/0.5.md b/content/release-notes/0.5.md new file mode 100644 index 0000000..03fded5 --- /dev/null +++ b/content/release-notes/0.5.md @@ -0,0 +1,55 @@ +--- +title: FediTest 0.5 release +date: 2024-12-19 +--- + +FediTest 0.5 has been released. This is primarily a bug fix release. It also incorporates +feedback from users, and feedback from an accessibility review and a security review. + +## To upgrade: + +If you have installed FediTest via pip, upgrade with: + +``` +% pip install --upgrade feditest +``` + +Once you've upgraded successfully, you can now check your currently installed version with: + +``` +% feditest version +``` + +It should report 0.5. + +## Major changes + +* A security review was completed, and the results were incorporated. No major issues + were found. (Related issues at tagged "security" in the + [bug tracker](https://github.com/fediverse-devnet/feditest/milestone/7?closed=1)). + +* An accessibility review was completed, and most of the results were incorporated. + (Related issues tagged "accessibility" in the + [bug tracker](https://github.com/fediverse-devnet/feditest/milestone/7?closed=1)). + There's one [open issue](https://github.com/fediverse-devnet/feditest/issues/435) + we could use help with from a CSS wizard. + +* HTML report generation was improved in various ways. For example, the reports now + emit the name of the tested application even if that name was not known at the + beginning of the test run. + +* Better command-line prompts when tests are run interactively without a fully + automated {{% gl NodeDriver %}}. + +* There are more unit tests for FediTest itself. + +* Better error messages for a variety of conditions. + +* Test results on this website have been updated. + +* Documentation on this website has been improved. This includes more details on how + to extend FediTest. There is now also {{% pageref "/quickstart/sandbox/" %}} which + should help simplify the explanation for how FediTest works and how developers + can adapt it to their own needs. + +List of [all closed issues](https://github.com/fediverse-devnet/feditest/milestone/7?closed=1).