Client: Validate all metadata files against the schema on repository update

[AlexanderDzhoganov]

This will help us avoid chasing non-existent bugs related to broken metadata files.

[AlexanderDzhoganov]

I have implemented this in #83 but there is an issue. The json parser doesn't understand schema version v4 and so the validation throws. How should we go about this?

[pjf]

The good news is that we're already partially protected against this. Travis runs a full scan over CKAN-meta any time there's an update (or incoming pull request), so it should be very hard to get non-validating data into the default repository.

Part of this can be mitigated simply by us having stronger typing in the client itself. Things like the license field being an enum rather than a simple string, and so on. Some of those types could be automatically generated from the spec itself; our build process already does things like inserting version numbers, it would be a pretty easy change to have it write out some enums and the like.

I've gone looking, but can't find any v4 validators for .NET.

We could try to re-write the schema for v3. It would mean a bunch of things which allow either a string or an an array (licenses, authors, etc) would need to always have arrays, which isn't quite as pretty or human friendly. However my biggest concern is that v3 doesn't support an explicit definitions block, which improves readability and maintenance significantly.

I'm happy for us to improve our internal types for now, and revisit this if we start seeing people hosting their own non-validating repositories. By that time thee might be a working v4 validator for .NET. ;)

[HebaruSan]

Netkan itself (bot included) checks new modules against the schema as of #2788.

I think running this validation in the client should not be done. First, it would be a lot more work during repo updates with a huge performance penalty. Second, if a non-conformant module was detected, the end user probably neither caused this problem nor knows how to do anything about it. Keeping the metadata repo clean accomplishes the same goal in a better way.