Imported data from wakatime doesn't match the actual data.

[Fom123]

Describe the bug
When I am trying to import the data from wakatime , I am not getting the full information, just part of it. It can be seen by checking the total time. On wakatime I get the following output
, while on my self hosted instance of wakapi this.

Same goes for the whole statistics, it doesn't match the data provided in the wakatime dashboard.
It seems, it can't insert all the imported hearbeats into sqlite database.

Mar 08 11:43:17 ip-******* wakapi[26507]: 2022-03-08T11:43:17.82956296Z [WARN ] failed to insert imported heartbeat, already existing? - too many SQL variables
Mar 08 11:43:18 ip-******* wakapi[26507]: 2022-03-08T11:43:18.111405612Z [WARN ] failed to insert imported heartbeat, already existing? - too many SQL variables
Mar 08 11:43:18 ip-******* wakapi[26507]: 2022-03-08T11:43:18.119789307Z [WARN ] failed to insert imported heartbeat, already existing? - too many SQL variables
Mar 08 11:43:18 ip-******* wakapi[26507]: 2022-03-08T11:43:18.425061063Z [INFO ] downloaded 31872 heartbeats for user 'Fom123' (6134 actually imported)

Same problem was in the #206 issue. It was resolved by setting app.import_batch_size to 50, so I have tried to set it to 20 and still I haven't got the whole statistics.

Now I see the following in the logs:
Mar 08 13:18:43 ip-******* wakapi[27378]: 2022-03-08T13:18:43.827091656Z [INFO ] downloaded 31872 heartbeats for user 'Fom123' (28440 actually imported) but with no warnings.

Why is this actually imported thing happens?

System information

• Wakapi version: 2.2.4
• Linux: Ubuntu 20.04.3 LTS
• Docker: no
• Database: sqlite

[muety]

Thanks for that super detailed bug report! First of all, please note that WakaTime and Wakapi will never show the same total time. That is due to the fact that they estimate that time differently, see our FAQ. Unfortunately, there is no right or wrong here and we can't for sure know precisely what the actual, true coding time was, only judging by heartbeats. Calculations are just heuristic.

If there is a discrepancy between heartbeats downloaded and heartbeats actually inserted, but no warning, it might be due to the fact that duplicates have been filtered, see here:

wakapi/services/heartbeat.go

Line 62 in 85515d6

if _, ok := hashes[hb.Hash]; !ok {

Two heartbeats are considered duplicates of each other if the following of their properties have identical values: user_id, entity, type, category, project, branch, language, is_write, time. Would be interesting to know if WakaTime API returns duplicates (by that definition), but I don't see any other reason.

Btw., WakaTime import will (hopefully) be refactored soon, see #323.

Moreover, if it's not too much of a hassle for you, I'd recommend to use MySQL or Postgres as a database in favor of SQLite as there have been a few problem with SQLite in the past (including the one you came across).

[Fom123]

Thanks, I understand now. However, I think there should be a warning about using sqlite in FAQ

[muety]

It's not really like you shouldn't use SQLite, but I'd rather prefer a "proper" DMBS as the recommended way (mentioned here) for people with technical affinity.