Data dump / exporting functionality

[boehs]

Before the change where old data is deleted, I would like to export my data, but I can't find an option in the settings. It's very likely I'm blind, but I could use a little bit of guidance :)

[boehs]

I saw

To forward data to another Wakapi instance, use https:///api/compat/wakatime/v1 as a URL.

I just want to download this data locally. Not use another instance.

[muety]

Hi @boehs, downloading data locally is not supported, only migrating to another (local) Wakapi instance.

[boehs]

So does that make this a feature request?

[muety]

A "workaround" would be to (temporarily) host your own Wakapi instance, run the import and then just export your database's heartbeats table to CSV (example). Would that help?

I'm considering to add dumping / exporting functionality as a feature, but would like to spend a bit more time on planning this out. Best would be to simply mimic WakaTime's /dump endpoint, but I'll have to investigate to what extent their data format is compatible with Wakapi's data model.

Relates to #323.

[muety]

Here's an example for a heartbeats dump from WakaTime: sample_dump.json.

Looks quite straightforward. We won't have most of the user attributes, but that's not a problem. Instead of machine_name_id, and user_agent_id, well have the actual plain string instead, respectively.

And here's an example for the "daily totals" export: sample_daily_dump.json.

We're not tracking dependencies at all, but apart from that, the export could probably be computed from daily summaries for the most parts.

Note: large parts of the original data in the above samples was removed / redacted for brevity.

[boehs]

I feel like this is important to do before the data is deleted from past years

[muety]

Agree, but won't promise I'll get to this before end of month. Spinning up your own Wakapi instance and importing data to there would be a safer bet for now.

[boehs]

Sounds good

[muety]

Agree, but won't promise I'll get to this before end of month. Spinning up your own Wakapi instance and importing data to there would be a safer bet for now.

I'll try my best, though! Maybe a temporary solution would be a script to run locally, that just pulls data from Wakapi's standard API endpoints. I'll keep you posted!

[boehs]

Thanks for all the thought man.

[muety]

I created this script to download raw heartbeats from Wakapi. Hope that helps!

[theAkito]

I created this script to download raw heartbeats from Wakapi. Hope that helps!

I took the liberty of translating this script into Nim.

[muety]

Very cool, thanks a lot! 🙌

[muety]

I decided to not build exporting functionality into Wakapi directly, but instead have the above mentioned download script as the official way to dump heartbeats. Will add instructions to the readme.