Configurable amount of leading 0s for migration files

[rafabulsing]

I've noticed that the amount of leading zeros is hard-coded to 3.

While 999 is a lot, for a long-lived project, it's quite possible that it won't be enough. I did a test, and after going over that limit, tern new a creates 1000_a.sql, but then running tern new a again fails with the following message:

Error loading migrations:
  Missing migration 100

Configuration could be done via cli flag or .conf file. It could also be inferred from the other files in the migration folder (would require manual editing of the first migration to set the wanted number of leading zeros, but would work automatically from then on).

If nothing else, at least a better error message for the 1001st migration would be nice, as it now is it's a bit misleading.

I could try my hand at implementing this feature, if need be!

[jackc]

It would be an improvement to support more than 999 migrations.

But I'm not sure if a configuration or looking at the existing migrations for number of digits is the best approach. Both those require changing it at the beginning of a project to avoid having to rename all existing migrations. And if they have to rename migrations then I think the fix is to support any number of digits simultaneously without any configuration. When someone hits migration 1000 they can use a shell script to rename the old migrations with an additional zero if they want it to be lexicographically sortable, but it should just work with no intervention.

[rafabulsing]

I see your point. I would argue, though, that there is already at least one instance of a configuration that, if changed, requires manual operations by the user: the migrations input folder config. If I set it to something, then change my mind later, I'd have to move the existing migrations to the new directory. Not like it's a huge hassle or anything, just pointing out that "philosophically" it's the same thing.

So I don't think a config for leading zeros would go feel out of place or weird, considering this. It's natural/expected that changing a config mid-project would require you to adapt some things to work with the new configs. And if you set it "correctly" (not ever need to expand it over the life of the project) you're good to go, no need to rename things ever.

Still, I appreciate the want to keep the config options more on the minimal side. Your suggestion would definitely be a nice improvement as well. So if the above has not convinced you, I'm happy to leave it at that :)

And like I said before, I'm willing to give it a shot - either my or your suggestion, whichever you judge more appropriate for the project.

[jackc]

And like I said before, I'm willing to give it a shot - either my or your suggestion, whichever you judge more appropriate for the project.

Thanks! I would prefer the "no config" option.

[rafabulsing]

Hey, sorry for the late response! Life got in the way a bit, but I finally got around to taking a crack at it.

I managed to find the problem, which actually I guess might not a problem, given your preferred solution. I'll explain.

The situation, it seems, is that FindMigrationsEx assumes the files are lexicographically sorted. It takes all files from the migration folder using fs.ReadDir, then iterates over them, checking to see that all migrations are there. First file should be 1_foo.sql, second should be 2_bar.sql, etc. If the ith file isn't named i_name.sql, it returns an error saying "Missing migration $i".

But fs.ReadDir sorts 1000_name.sql before 100_name.sql, so at the 100th loop, the function errors out.

If the SQL files are renamed to include the required leading zeros for proper sorting, the error disappears.

Given that, I guess it could be understandable to consider that "working as intended", but I think at least a warning somewhere in the docs would be warranted, saying that the files have to be properly sorted, so if you hit 1000 you should rename the files to add the extra 0.

Conversely, I could try to change the way FindMigrationsEx works, so that even if the files are not properly sortable, it still works if the entire sequence is there.

The way I picture that, would be creating a slice with found := make([]bool, len(paths)), then for every i_name.sql assigning found[i-1] = true (regardless of in which loop that i came up), then checking at the end if the entire found slice is true. Something like that, plus some extra logic for excluding subdirectories, and checking that there aren't repeats.

So, what do you think?

[jackc]

Conversely, I could try to change the way FindMigrationsEx works, so that even if the files are not properly sortable, it still works if the entire sequence is there.

👍