v5.8.0: New error code for bad serialization

This release adds a new error code, malformed, which the server returns if a client's message is not deserializable without errors. This error gets a 400 HTTP Status Code. This should make it easier to filter out bad hand-written clients from server error monitoring.

This was #182.