Register externally-managed files via the client

[danielballan]

Tiled enables registration of externally-managed files with a catalog. In main, this is a server-side activity where the user interacts directly with the SQL database via Python, and thus requires the database credentials. In this PR, this becomes a client-side activity where the user interacts with a new HTTP endpoint, POST /register/{path} and must hold a new scope, register.

The module tiled.catalog.register becomes tiled.client.register.

The CLI tiled catalog register <DATABASE_URI> becomes tiled register <TILED_URL> --api-key <API_KEY>.

Why didn't we do this in the first place?

We originally considered doing this client-side from the start. I had two concerns:

1. Security. This PR enables clients to ask the server to expose files on the server. This is naturally a security concern. However, giving the file-registering program direct access to the database is also a security concern, arguably a worse one. And we have the following protections:

   • In multiple-user deployments, the required scope, register, is given only to Tiled admins and service accounts that they elect to grant it to.
   • The server configuration must explicitly list areas of its local storage where files may be registered, via its readable_storage configuration. Anything outside of that will be rejected. This is enforced at read time as well, to be sure that even with a corrupted database there is no way that Tiled would serve an arbitrary file off of the system.

2. Latency. When scanning large directories, this may be slow. But there is a path to making it faster. In a future PR, we can add POST /nodes/metadata/{path} and POST /nodes/register/{path} for bulk creation of nodes, with internally- and externally-managed data respectively.

I have stacked @dylanmcreynolds commits from #654 on top of my commits here. My intent is to update and ideally simplify it before merging, but I'll need some input from @dylanmcreynolds and @Wiebke on that.

[padraic-shafer]

TODO: After #660 gets merged, will need to apply these analogous updates to test_container_fields.py.

tiled/tiled/_tests/test_container_files.py

Line 21 in 053d3d2

await register(client, tmpdir)

[dylanmcreynolds]

Excellent!

Should we consider adding cleint-side register text to the how-to document rather than replacing it? Even with this, are there cases where we would want to support both?

[danielballan]

Ideally I'd like to add whatever convenience APIs we need to reduce that chunk of Python to something pithier. I am aware of these use cases:

• Walk a directory. We have that covered.
• Register a specific file manually providing details (e.g. custom specs...) instead of taking whatever the automatic discovery generates. We need to document and perhaps streamline this.

Can you go into more detail about what needs to be addressed?

[dylanmcreynolds]

I think I was trying to ask a different question. My doc descrbed a labor-intensive but effective method for registering files in code that has direct access to the database, while this PR adds functionality to do the same through the client. Do we want to keep the documentation of the more involved server-side register, adding the client side as an alternative? Or do we want to abandon my documentation in favor of new client-side register text?

[padraic-shafer]

I like this plan. At a high level the code changes look good so far.

[dylanmcreynolds]

After talking to @danielballan and @Wiebke , I'm pretty excited about this PR. To clarify my previous understanding was incorrect, and this moves the register function from the server to the client.

[dylanmcreynolds]

Can we add sentence here?

Sometimes it is necessary to take more manual control of this registration
process. For example, if you want to add a `spec` or custom `metadata` to the data set, you will want to do that programmatically usign the Tiled client.

[danielballan]

I missed this specific suggest but I'm glad I capture the gist of it.

[dylanmcreynolds]

Quick suggestion for the register.md file.

[dylanmcreynolds]

Great

[dylanmcreynolds]

Great

[padraic-shafer]

This looks great. One small change noted in the comment.

[danielballan]

An embarrassing number of commits later, this docs section is finally right :-D

[danielballan]

....Plus one last commit for s/demonstrate/demonstrates/.

[padraic-shafer]

Thanks for updating the docs!

[Wiebke]

I am late to the party, but this is great!

[padraic-shafer]

@Wiebke Thanks for adding your support! The stacked PRs were starting to pile up, so with one approval from ALS + another from NSLS-II, I made a call that we were ok to merge this without further review. I'll aim to not make a habit of that though. :)