Text encoding issues migrating Home Assistant data

[maxlyth]

Thank you for the this fork and the work to add the logic for HomeAssistant migrations. It got me 80% of the way there but I still experienced significant issues while migrating my Influx2 HA dataset.

My issues were related to text encoding and your version of the code not correctly complying to the Influx Line Protocol Standard, especially the section about escaping and quoting.

I have modified the code to correctly escape the names and values of keys and the value columns is now quoted for strings with escaped ", /r, /n as these would break the line protocol. I have a pretty diverse dataset and managed to migrate over 4GB without any problems.

I have forked this repo with documentation and significant changes, including a Dev Container definition which should make it a lot easier for anyone to run a migration when not familiar with Python.

[frli4797]

Great stuff that you found this useful. I'm amazed that the encoding problem didn't give me any issues when migrating my dataset. (Why is character encoding still such a mess?)

I'd love if you'd create a PR for your changes to merge back to my repo, to ensure that my mistakes don't spread to other unfortunate users of my hack. Thanks!

[maxlyth]

Hi Frederik Thanks for being so responsive. One of the reasons I forked is because I have lost so much hard work on PRs that never merge. I also rewrote the documentation and created a Dev Container that would help people unfamiliar with Python venvs and the command line. If you are happy with those level of changes then I can do a PR from my main branch to yours and if you make me a joint admin I can close down my fork Max... -------- Max Lyth 18 Grove Mews, London W6 7HS, UK Voice/iMessage: +44 777 44 222 40 On 30 Aug 2024, at 09:32, Fredrik ***@***.***> wrote: Great stuff that you found this useful. I'm amazed that the encoding problem didn't give me any issues when migrating my dataset. (Why is character encoding still such a mess?) I'd love if you'd create a PR for your changes to merge back to my repo, to ensure that my mistakes don't spread to other unfortunate users of my hack. Thanks! — Reply to this email directly, view it on GitHub<#3 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAQYDSELXKJC4ION3AD7KC3ZUAU2XAVCNFSM6AAAAABNKR4PCCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRQGQ4DQMJRHE>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[frli4797]

Hey! I'm not the one to nitpick when people are good-hearted enough to share their time and effort to make a piece of code better, or even less bad, given that I'm not a very good developer anymore. You're more than welcome to help me, and more important any other unfortunate soul that could venture into using this code.

I'm very likely to just merge your PR, would you be willing to create one, as I no longer possess a data set to test with. But given your tests and review, I am sure that it would improve things just as it is.

[frli4797]

Had a boring afternoon and created a PR off your changes, then did some minor alterations in the readme and merged back to this repo. Made sure to mention you/your fork as the originator of these fixes.

[frli4797]

Fixed by #5

[maxlyth]

Hi Frederik Thanks for merging the pull request, you say you are not a good developer but your code was good enough to get me to the finish line. I worked for a machine vision a few years ago which was my only previous experience with Python, I barely remember enough to be dangerous but it reminded me how cool Pandas is for ETL. I had a busy week so got distracted but I saw your suggestion to merge my fork back into your original repo and given you are more responsive than me then that makes sense. I am going to admit that I have never done that before so not sure of the steps and what volume of work that will take on your end if I don’t get it right first time. You could still give me commit perms on your repo if you want me to self serve.. I successfully completed my migration on Tuesday so I am looking to close this off by next week before it is lost forever so expect some Github messages. I’m terrible at email so if you need to contact me to get something finished then text me on the number below. Max... -------- Max Lyth 18 Grove Mews, London W6 7HS, UK Voice/iMessage: +44 777 44 222 40 On 1 Sep 2024, at 10:57, Fredrik ***@***.***> wrote: Had a boring afternoon and created a PR off your changes, then did some minor alterations in the readme and merged back to this repo. Made sure to mention you/your fork as the originator of these fixes. — Reply to this email directly, view it on GitHub<#3 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAQYDSDFLIW52AFMPDJ7BE3ZULQJ7AVCNFSM6AAAAABNKR4PCCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRTGI3DGOJRGA>. You are receiving this because you authored the thread.Message ID: ***@***.***>