Skip to content
This repository has been archived by the owner on Apr 3, 2024. It is now read-only.

fix: Unicode course data causes errors when sending to ClickHouse #23

Merged
merged 3 commits into from
Aug 3, 2023

Conversation

bmtcril
Copy link
Contributor

@bmtcril bmtcril commented Aug 3, 2023

Description: Fixes unicode errors when sending to ClickHouse by explicitly encoding the CSV output as utf-8.

ISSUE:
Closes: #22

Installation instructions: In a Tutor Aspects local setup you can:

  • Change OPENEDX_EXTRA_PIP_REQUIREMENTS for this package to "git+https://github.com/openedx/openedx-event-sink-clickhouse.git@bmtcril/fix_unicode",

Testing instructions:

After installing from this branch:

  • Rebuild the openedx image: tutor images build openedx
  • tutor local start -d
  • Create some unicode in your course name, problem / video block names, etc.
  • Publish the course
  • Browse the course, watch the video or try the problem
  • Confirm in the cms-worker logs that there are no errors
  • Confirm in ClickHouse that the changes appear correctly in the event_sink.course_overviews / event_sink.course_blocks tables (note that a lot of database tools don't handle utf-8 well, if there are "weird characters" in the output you can assume this passes and go to the next step)
  • Check Superset to make sure that the UTF-8 characters show up correctly in the reports and filters

@bmtcril bmtcril changed the title fix: Unicode course data caused errors in requests fix: Unicode course data causes errors when sending to ClickHouse Aug 3, 2023
@bmtcril bmtcril merged commit 7ba7521 into main Aug 3, 2023
8 checks passed
@bmtcril bmtcril deleted the bmtcril/fix_unicode branch August 3, 2023 16:07
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bug: Unicode in xblocks causes errors sending to Clickhouse
2 participants