Pub/Sub to BigQuery Errors PayloadString is not valid JSON

[fra-c]

The errors from the Cloud Pub/Sub Subscription to BigQuery template aren't saved as valid JSON in the PayloadString column and I am unable to replay them.

It looks like this:
{event={userId=1234, sessionEvent={sessionId=DSFG, ua=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36, browser={name=Chrome, version=74.0.3729.131, major=74}}}

If this is intended, how do I convert it and push it back to Pub/Sub?

[sabhyankar]

Hi fra-c - This is due to how the insertErrors are wrapper here with the main issue being in this method. There is a PR that is currently being reviewed. Once the PR is merged and a release is cut, it should solve this problem as well.

[bddckr]

We've been hit by this same issue and are looking at plenty of gigabytes of error data to parse ourselves due to this. The payloadBytes has the exact same issue (it's the same string ultimately).

What's the status of this issue/the linked PR?

[Hechamon]

We just ran into this problem as well and have a few million rows of errors that we would really like to recover. Did any of you write a regex or something that can recreate valid Json?
Any language, any technology is fine. Please?

[github-actions]

This issue has been marked as stale due to 180 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the issue at any time. Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.