Use OpenAPI fuzzing to find mistakes in specifications and general errors in the code

[provokateurin]

How to use GitHub

• Please use the 👍 reaction to show that you are interested into the same feature.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Is your feature request related to a problem? Please describe.
Bugs exist ;)

Describe the solution you'd like
An OpenAPI fuzzer can generate inputs and execute requests against API endpoints as described by the specification.
This can both find problems in the specifications as well as errors in the application code itself.
I already ran a small experiment against https://github.com/matusf/openapi-fuzzer which resulted in a few issues found in the core specification and code.
There is also https://github.com/microsoft/restler-fuzzer which is a lot more complex but also does more advanced mutations and checks for problems in the application logic. I haven't been able to get it properly working yet though.
One major thing I'm interested in is validation of response bodies as I often find problems myself (by hand basically), so an automation there would be great. I haven't yet found a fuzzer that seems to do that, but maybe restler can do it and they just don't mention it anywhere (the other one definitely only checks the status code, i read that part of the code).

A small script to run tests against the specifications in server can be found here: https://gist.github.com/provokateurin/3150b1c98049368bb6f00e7b7f76b1c6 (can be adapted to work with restler later)

[provokateurin]

I found https://github.com/schemathesis/schemathesis while looking for something unrelated and it produces pretty good results, including response schema validation which is really helpful for making sure OpenAPI specifications are correct.

[sorbaugh]

Would be a nice quality improvement!