Validate TDE schemas on load

[rjrudin]

If your TDE schema is invalid, you won't find out right away, as it's just being loaded via /v1/documents. This can be a real pain because if your schema is invalid, it won't be applied on any loaded data, and you won't realize it until you try out your schema. And now you have to fix it and re-index all of your data.

I think we need an option (enabled by default) so that anything under "/tde/" that's loaded into a schemas database is first run through /v1/eval with a script to make sure it's valid. If it's not, the load should fail.

[rjrudin]

One challenge here is that the command for loading schemas is given a DatabaseClient that allows it to load schemas directly into the schemas database (typically one that talks to 8000). But in order to validate a schema, the code must be evaluated against a database that points to the given schemas database as its schemas database.

This can be done with xdmp.invokeFunction, but it requires knowing the name of a related database. ml-gradle can't easily figure that out either. With the Manage API, it would have to retrieve the properties for each database until it finds one that points to the schemas database. It could also look at all the database files, but I don't know if that's 100% certain to work (a file isn't guaranteed to exist that defines a database that points to the schemas database).

The easiest approach, but requires user configuration, is for a developer to define a property that defines databases that points to schema databases. For example, if I want to define related databases for two DHF databases, I would do this:

mlSchemaDatabaseMapping=data-hub-staging-SCHEMAS,data-hub-STAGING,data-hub-final-SCHEMAS,data-hub-FINAL

Of course those standard database names aren't always used, and not every app uses DHF, so ml-gradle wouldn't be able to assume that the above mapping exists. But the pattern would be "schema database 1,associated content database 1,schema database 2, associated database 2", etc.

That would give ml-gradle the info it needs to be able to validate a TDE template without having to make any additional calls to the Manage API (which would also keep it pretty fast).

[dmcassel]

[tde:validate](https://docs.marklogic.com/tde:validate) doesn't require the template to have been loaded. You could use the DatabaseClient that talks to 8000 and use invoke-function to call tde:validate on the template passed in as text, right?

[rjrudin]

@dmcassel The testing I've done in qconsole indicates that an error is thrown when I run tde.validate with a content source whose database doesn't have a schemas database itself. I might be missing something though.

Specifically, I get this error (and this is against an in-memory template, not a persisted one):

[javascript] TDE-NOSCHEMADB: tde.validate([Document({"template":{"context":"/example", "collections":["example"], "rows":[...]}})]) -- No schema database for TDE resource: no schema database configured for current database

The docs for tde.validate also state that the validation process "Validate against stored templates". I am guessing that's what requires a valid schema database to be associated with whatever database is being executed against.

[dmcassel]

Oh, I see.

[rjrudin]

Now thinking it's highly likely that a database file will define a relationship between a content database and a schema database, so going with that approach. In the event it can't be determined, TDE validation won't happen.