feat(gateway, cargo-shuttle): implement pagination for project list

[AlphaKeks]

Please write a summary of your changes and why you made them.

Closes #845. This adds pagination functionality to the project list and deployment list subcommands for cargo-shuttle and changes the API endpoints to accommodate that.

How Has This Been Tested (if applicable)?

Unit tests in gateway/src/service.rs and deployer/src/persistence/mod.rs

[Kazy]

Regarding the number of results returned, by default the API will keep returning every single result, since we believe the endpoint is also used in the web console. And to go with it, there is also no limitation of the limit parameter as well.

The CLI though does pagination by default, returning the first 10 items, which is a breaking change. We wanted to also improve the experience on that on two fronts but didn't to first get your opinion on how we should solve them (maybe in an other PR). First there is no mention that more pages are available. Second in the case of a limit of 0 or of a page that goes beyond the actual number of pages; there is a mention that no deployment/project exists, which isn't strictly correct.

The easy solution is just to rework the messages. We can for instance say that there might be more results available in the next page, even if that isn't the case. The other solution would be to fetch the total number of items, using either a second SQL request, or a single one but with a count(*) over() to get the total count. A less expensive call could be done by just fetching if there is a new row or not (lead() over()).

[AlphaKeks]

As I mentioned in our private conversation, we could very well make the --limit flag default to u32::MAX which would effectively keep the old behavior.

As for the potential issue with supplying a --page value that's too high, I would probably run 2 queries, returning early if the specified page is too high. But I'm open to do something else as well.

[iulianbarbu]

Thanks @AlphaKeks ! The PR looks really good, it's almost there. Just left a few questions that are unclear on my side.

[iulianbarbu]

Hey @Kazy and @AlphaKeks ! How is everything going? Let me know if you need any help on this.

[iulianbarbu]

Also FYI, this project was just changed to a medium one, because of the changed requirements. Let us know if you have any questions.

[Kazy]

Hey @iulianbarbu ! I've updated the branch with some of the changes we discussed.

Regarding the created_at column, I assumed everything was using Postgres, but this isn't the case when developing locally where instead Sqlite is used. I don't know if you actually support both at the same time (PSQL in prod, Sqlite in dev) or if you also use Sqlite in the context of the gateway and the deployer in production ?

In the meantime I've made the changes to use project_name for ordering in the gateway and last_updated_at (latest updated first, as it makes more sense when using the CLI) for the deployer. I've also added a text that an another page might be available: happy to rewrite it in any way.

[iulianbarbu]

Hey @iulianbarbu ! I've updated the branch with some of the changes we discussed.

Regarding the created_at column, I assumed everything was using Postgres, but this isn't the case when developing locally where instead Sqlite is used. I don't know if you actually support both at the same time (PSQL in prod, Sqlite in dev) or if you also use Sqlite in the context of the gateway and the deployer in production ?

We use sqlite in prod as well. We use it for both gateway and deployer at this moment.

In the meantime I've made the changes to use project_name for ordering in the gateway and last_updated_at (latest updated first, as it makes more sense when using the CLI) for the deployer. I've also added a text that an another page might be available: happy to rewrite it in any way.

Thanks, taking a look rn.

[iulianbarbu]

Looks good so far @Kazy ! I think the only thing left is to add a migration for the projects table, where we alter the table by adding the created_at column. This column should be changed when creating a new project, info persisted in the gateway sqlite. And then, when iterating through the account projects, we should ORDER BY created_at DESC, project_name. What do you think?

[Kazy]

Hey @iulianbarbu ! Thank you for the review !

I've added the created_at column to the gateway DB. I didn't use a DEFAULT statement when updating the column because with SQLite you cannot add a non constant default value for a column (in our case CURRENT_TIMESTAMP).

[Kazy]

Hey @iulianbarbu, I've fixed the tests, everything should be good for this PR now. Let me know if there is anything left to do :)

[oddgrd]

This looks really great, thank you for all your work on this @Kazy and @AlphaKeks! 😍