Create end-to-end tests for openAPI

[drasko]

A set of e2e tests should be created to check the sanity of the system before merging new PRs.

Test should:

• Act as user - use SDK to register and login user
• Use SDK to provision things and channels
• Act as a Mainflux thing (device or app) - use SDK to send messages that will be written via Writters to various DBs (C*, InfluxDB, MongoDB)
• Use SDK to read messages from various Readers
• Compare and validate read messages

This way whole Mainflux system will be treated as a blck box, and tests will proven API sanity and that main functionalities are preserved.

[alwindoss]

e2e tests are usually run on a deployed software. Are you hinting towards a deployment on docker to run the e2e tests and then the developer can raise a PR?

[mandric]

Is this something a newbie could work on? I have experience with e2e testing tools in the NodeJS world and little Golang experience but am motivated to learn.

[drasko]

@mandric yes, definitely you can give it a try. I think it should start by selecting a correct technology. @chombium and @anovakovic01 mentioned Pact before, and there is a Go version: https://github.com/pact-foundation/pact-go

I am not sure at this point what thecnology would be best to do this, some research on technology choices is needed.

[mandric]

Pact looks interesting, I am not a Ruby person either, I have Python experience though. But if you give me the set of tools I can play around with it and see how far I get.

[mandric]

There appears to be python version, not sure if that's interesting. https://github.com/pact-foundation/pact-python

[mandric]

Oh sorry, it would obviously be written in Go! That could be a fun way to learn!

[mandric]

Well I looked a little at pact-go and the APIs seemed a little complicated, so I just wrote something up that was quickest for me using bash and bats. I thought I would post here to get some initial feedback. I figured it would be useful even if not as a long term solution but maybe as a way to quickly get some coverage and maybe help think about the problem. I would be happy to go back to pact-go but it would take more time to get something up and running.

https://github.com/mandric/mainflux-e2e-bats

The next steps if you wanted to continue on the bats path would probably be 1) obviously adding more coverage 2) hooking up to CI.

[mandric]

This might be a better option that has more automation using swagger, postman and newman: e.g. https://github.com/gothinkster/realworld/tree/master/api

I haven't researched yet but I imagine GoLang has equivalent toolset.

[chombium]

@mandric this looks really interesting. I guess we'll have to add some glue code which will go through all of our swagger files and generate the whole api spec + some test data, before testing the api itself

[drasko]

Yes, this is interesting. I researched already go-swagger before.

In any case, fining an equivalent to this would be interesting: https://github.com/postmanlabs/newman

[nmarcetic]

Check out Kubernetes methodology overview Also Ginkgo Looks great.

[nmarcetic]

@mainflux/maintainers I took this issue to do research.

[drasko]

Related to https://github.com/mainflux/mainflux/issues/1149

[drasko]

• https://medium.com/@you54f/protecting-your-api-development-workflows-with-swagger-openapi-pact-io-5ba8dca8b9cf
• https://pactflow.io/blog/the-curious-case-for-the-provider-driven-contract/
• Support external schemas such as OAS/Swagger, JSON Schema, GraphQL or .proto pact-foundation/pact-specification#76
• https://docs.pact.io/faq/#can-i-generate-my-pact-file-from-something-like-swagger
• https://groups.google.com/g/pact-support/c/pNaz1mN9fZQ?pli=1

[drasko]

TBH, having all this in mind, I am leaning mostly to something like this:

• https://www.youtube.com/watch?v=1lo7idI7uq8
• https://github.com/schemathesis/schemathesis

As I mentioned before, I really like Property-Based Testing approach - as referenced in https://github.com/mainflux/mainflux/issues/643#issuecomment-687667599

Nice articles about Schematesis' PBT approach:

• https://code.kiwi.com/schemathesis-property-based-testing-for-api-schemas-52811fd2b0a4
• https://dzone.com/articles/schemathesis-property-based-testing-for-api-schema

Probably this nice Haskell tool is also worth exploring: https://github.com/rodrigosetti/swagger-test

Having in mind Mainflux lightweight mindset, I would not be surprised that this Haskell tool would be the best fit :). Just that it supports OpenAPI-only, and we'll need AsyncAPI as well (for MQTT purposes) - read more about it here.

[Stranger6667]

Hello, all!
The Schemathesis author here. Happy to see that you are considering Schemathesis :) Let me know if I can provide any support from my side, e.g., missing features or something not working as expected :)

[drasko]

@Stranger6667 thanks for your work on Schemathesis, I think it is ideated in the right directions, alongside the idea of PBT and QuickREST article.

One thing I see now missing for our purpose would be AsyncAPI, bacause in IoT MQTT is a predominant protocol, and in general lot of PUB/SUB paradigm is used.

Is there any plans to add this support from your side, and how hard would be to add it?

[Stranger6667]

Thank you!

Is there any plans to add this support from your side, and how hard would be to add it?

AsyncAPI is new to me. As far as I see, this spec is very close to Open API, and it should be relatively easy to adjust the existing Open API code to generate messages or other structures since they follow the JSON Schema spec (as far as I see).

However, to support a complete testing flow we need wider transport support. Schemathesis is centered around HTTP, and building other transports might require some more investigation and time. I will take a deeper look there, but from my side, AsyncAPI is definitely something I'd like to support in Schemathesis :)

At the moment I am working on making Schemathesis more spec-agnostic tool and available as a service - not sure about the bandwidth I can allocate now, but I think that supporting AsyncAPI would be a logical next step for Schemathesis after I am done with refactoring :) I expect that I'll have time to bring some proof of concept prototype during October

[drasko]

Adding John Hughes talk (on QuickCheck) for the reference: https://www.youtube.com/watch?v=zi0rHwfiX1Q

And another one: https://www.youtube.com/watch?v=hXnS_Xjwk2Y