switch to new json library

[ebadi]

Describe the bug

I updated logmessaged.py as below to see why I keep receiving an error message when I execute manager.py.

    if level >= log_level:
      print("record:::",record)    
      log_handler.emit(record)

Below is the result, the json message that is sent by logmessaged.py is not formatted correctly and https://github.com/commaai/openpilot/blob/master/common/logging_extra.py#L100 cannot parse it. (this part is not valid: {"created": 1652444715,3936491, ...)

((openpilot) ) hamid@hamid:~/openpilot/selfdrive/manager$ PASSIVE=0 NOSENSOR=1 USE_WEBCAM=1 ./manager.py
selfdrive/loggerd/bootlog.cc: bootlog to /home/hamid/.comma/media/0/realdata/boot/2022-05-13--14-25-13
missing public key: /home/hamid/.comma/persist/comma/id_rsa.pub
selfdrive/logmessaged.py: needs update
record::: {"created": 1652444715,3936491, "ctx": {"daemon": "soundd", "device": "pc", "dirty": true, "dongle_id": "UnregisteredDevice", "version": "0.8.14"}, "filename": "selfdrive/ui/soundd/sound.cc", "funcname": "Sound::Sound(QObject *)", "levelnum": 20, "lineno": 16, "msg": "default audio device:  \"\""}
--- Logging error ---
Traceback (most recent call last):
  File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/logging/__init__.py", line 1085, in emit
    msg = self.format(record)
  File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/logging/__init__.py", line 929, in format
    return fmt.format(record)
  File "/home/hamid/openpilot/common/logging_extra.py", line 100, in format
    v = json.loads(record)
  File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 24 (char 23)
Call stack:
  File "./manager.py", line 205, in <module>
    main()
  File "./manager.py", line 182, in main
    manager_thread()
  File "./manager.py", line 134, in manager_thread
    ensure_running(managed_processes.values(), False, params=params, CP=sm['carParams'], not_run=ignore)
  File "/home/hamid/openpilot/selfdrive/manager/process.py", line 313, in ensure_running
    p.start()
  File "/home/hamid/openpilot/selfdrive/manager/process.py", line 245, in start
    self.proc.start()
  File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/popen_fork.py", line 75, in _launch
    code = process_obj._bootstrap(parent_sentinel=child_r)
  File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hamid/openpilot/selfdrive/manager/process.py", line 43, in launcher
    getattr(mod, 'main')()
  File "/home/hamid/openpilot/selfdrive/logmessaged.py", line 29, in main
    log_handler.emit(record)
Unable to print the message and arguments - possible formatting error.
Use the traceback above to help find the error.
record::: {"msg": "System time valid", "ctx": {}, "level": "INFO", "levelnum": 20, "name": "swaglog", "filename": "manager.py", "lineno": 31, "pathname": "./manager.py", "module": "manager", "funcName": "manager_init", "host": "hamid", "process": 11907, "thread": 140079618852672, "threadName": "MainThread", "created": 1652444713.9832933}
record::: {"msg": "missing public key: /home/hamid/.comma/persist/comma/id_rsa.pub", "ctx": {}, "level": "WARNING", "levelnum": 30, "name": "swaglog", "filename": "manager.py", "lineno": 83, "pathname": "./manager.py", "module": "manager", "funcName": "manager_init", "host": "hamid", "process": 11907, "thread": 140079618852672, "threadName": "MainThread", "created": 1652444714.1361263}
record::: {"msg": "starting process ui", "ctx": {"dongle_id": "UnregisteredDevice", "version": "0.8.14", "dirty": true, "device": "pc"}, "level": "INFO", "levelnum": 20, "name": "swaglog", "filename": "manager.py", "lineno": 171, "pathname": "./manager.py", "module": "manager", "funcName": "main", "host": "hamid", "process": 11907, "thread": 140079618852672, "threadName": "MainThread", "created": 1652444714.969782}
record::: {"msg": "preimporting selfdrive.locationd.calibrationd", "ctx": {"dongle_id": "UnregisteredDevice", "version": "0.8.14", "dirty": true, "device": "pc"}, "level": "INFO", "levelnum": 20, "name": "swaglog", "filename": "manager.py", "lineno": 102, "pathname": "./manager.py", "module": "manager", "funcName": "manager_prepare", "host": "hamid", "process": 11907, "thread": 140079618852672, "threadName": "MainThread", "created": 1652444714.9746444}

p.s: the manger seems to continue working without any issue.

OS Version

20.04

openpilot version or commit

No response

Additional info

No response

[pd0wm]

The create time in the cloudlog json is a double, which is converted to a string using snprintf here: https://github.com/commaai/openpilot/blob/master/third_party/json11/json11.cpp#L60

The problem is that this probably takes into account your current locale, producing output with a comma as a decimal separator. Which is not valid json.

a workaround would be to switch your system locale to something that uses a . as decimal separator.

relevant json11 issue: dropbox/json11#38

[ebadi]

@pd0wm thanks, I confirm that the issue is correctly identified and your workaround fixed the issue. Feel free to close the issue if the issue is too minor to be fixed.
Thanks again.

[adeebshihadeh]

Looks like json11 is no longer maintained. rapidjson seems like a good and mature alternative that may also give us a small speed improvement too.

[adeebshihadeh]

https://github.com/nlohmann/json seems like a good one too

[adeebshihadeh]

Bounty is to fix this as simply as possible, and it seems like we'll need to switch libraries to do it (confirm this though). If we absolutely have to switch, https://github.com/nlohmann/json seems like the best options.

[adeebshihadeh]

First attempt in #31093 had an issue with thneed.

[FiReLScar]

I’m new, is this still a bounty, if not can I still work on it. I’m looking for a software development job and rediscovered this company. I’d love to work here.

[FiReLScar]

And maybe I don't fully understand but why not just force a locale?