DBAL-642: Generated IDs are not guaranteed to be unique over the table's lifetime in SQLite

[doctrinebot]

Jira issue originally created by user flack:

In

http://docs.doctrine-project.org/en/2.0.x/reference/basic-mapping.html#identifier-generation-strategies

it says that MySQL and SQLite use AUTO_INCREMENT by default. The generated SQL for creating an ID field with GENERATOR_TYPE_AUTO looks like this (abbreviated
for readability):

CREATE*TABLE*TEST (id INTEGER DEFAULT 0 NOT NULL, PRIMARY KEY(id))

http://www.sqlite.org/faq.html#q1 states:

If you declare a column of a table to be INTEGER PRIMARY KEY, then whenever you insert a NULL into that column of the table, the NULL is automatically converted into an integer which is one greater than the largest value of that column over all other rows in the table, or 1 if the table is empty. [...] Note that the integer key is one greater than the largest key that was in the table just prior to the insert. The new key will be unique over all keys currently in the table, but it might overlap with keys that have been previously deleted from the table.

So in other words, if you remove an entity and then create a new one, the new one will have the same id as the previously deleted one. If you do both operations on the same entitymanager, id references (in proxies f.x.) will suddenly get confused and point to something else (at least that's my current theory..)

The point is: SQLite doesn't act like MySQL as the documentation implies, and IMHO SQLite's current behaviour makes it useless for more complex scenarios. I've found some reference to this problem here:

#66

Unfortunately, it doesn't mention any solution. The problem is that you can't override the columnDefinition, because it would have to be "INTEGER PRIMARY KEY AUTOINCREMENT", but then, you get an exception, because the Platform appends ", PRIMARY KEY(id)", so it's defined twice

[doctrinebot]

Comment created by @deeky666:

As far as I understand there is a conflict in SQLite between having an autoincrement primary key and having a composite primary because you can only choose either way. The PR you referenced removed the autoincrement behaviour in favour of having the opportunity to define composite primary keys. So what would you expect to be the solution here?

[doctrinebot]

Comment created by flack:

Well, from a user's point of view, it would be nice if the SQLite Platform implementation would make full use of the possibilities of SQLite. That is to say: If someone uses composite primary, they get the behaviour Doctrine has right now, and those that use the simple case (which is recommended all over the documentation), get the behaviour previous to the pull request, i.e. autoincrement that works like in MySQL (which the documentation implies). As far as I understood the discussion in the pull request, the author was looking for a solution to implement this, but then the PR was merged before the issue was solved.

[doctrinebot]

Comment created by flack:

I guess what is bothering me is that the current behaviour breaks assumptions that I think many applications using Doctrine make. At least I know that in my own code, I never planned for the possibility of a database primary key being re-used for a different object, especially not during the same request. And like I wrote above, I suspect that Doctrine itself is not totally prepared for that situation either (also mentioned here: #66 (comment)). So IMHO the IDENTIY generator strategy for SQLite seems broken, or at least is behaving unexpectedly. The documentation says

{quote}
AUTO (default): Tells Doctrine to pick the strategy that is preferred by the used database platform. The preferred strategies are IDENTITY for MySQL, SQLite and MsSQL and SEQUENCE for Oracle and PostgreSQL. This strategy provides full portability.
{quote}

I don't really see how the current behaviour can be said to provide full portability (at least with MySQL, which supposedly uses the same preferred strategy)

[doctrinebot]

Comment created by @deeky666:

Yeah I get your point. But it's always hard and error prone to work around the vendors lack of common features. A possibility COULD be to implement a trigger in columns declared as autoincrement which simulates the behaviour of an auotincrement column on inserts. Oracle uses a similar workaround with a trigger and a sequence to simulate autoincrement columns. But this is just an idea and has to evaluated for usability first.

[doctrinebot]

Comment created by @deeky666:

Yes that's true. The documentation is misleading here. I guess that it was written before the issue came up and then was not updated. Unfortunately SQLite does not support native sequences which would make life a lot easier. But I will keep that in mind and investigate a solution for this.

[doctrinebot]

Comment created by @beberlei:

I think there is no fix for this, this is just how SQLite works, and we cannot really keep the last ids somewhere. IMHO its a documentation issue.

[doctrinebot]

Comment created by flack:

Well, you cannot fix it for cases with multiple id columns (but the Doctrine documentation already suggests that they should be avoided where possible), but for single integer columns (which is the normal case, as suggested by documentation), SQLite provides all the necessary functionality, as long as you create the column with INTEGER PRIMARY KEY AUTOINCREMENT. So IMHO the best solution would be if support for this could be implemented somehow in the SQL Platform driver.

[doctrinebot]

Comment created by flack:

Case in point: I implemented exactly this in a Doctrine adapter I'm working on:

https://github.com/flack/midgard-portable/blob/master/src/midgard/portable/storage/subscriber.php#L136

Granted, this is a very ugly workaround that only works because I know all my ID columns are actually called 'id' (and will never change), but I'm fairly sure that a more general solution could be built with reasonable effort.

[doctrinebot]

Comment created by @beberlei:

[~flack] We removed the Sqlite AUTOINCREMENT for some weird reason. I am inclined to add this again, however I need to find out what the reasons for removing this have been.

[doctrinebot]

Comment created by @deeky666:

[~beberlei] I checked that lately and I came to the same conclusion. The reason why it was removed is to support composite primary keys which was not possible before somehow. We could add autoincrement if only a single column integer primary key is given I think...

[doctrinebot]

Comment created by flack:

Just noticed that the link I posted above is broken now. Here's a stable one:

https://github.com/flack/midgard-portable/blob/04d473356d804c7f64e940ef82dd1538c39fccdd/src/storage/subscriber.php#L170

The workaround is a bit less project-specific now, and might serve as the basis for a real solution I guess (basically, only checks for column type and composite keys would need to be added)

[flack]

I think this can be closed (implemented by #3141)

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.