Add syntax highlighting to code tags #2015

Plastikmensch · 2022-12-16T22:09:54Z

Fixes #2011

Important changes:

Adds a new dependency: highlight.js
Changes advanced_text_formatter to allow data attribute on code tags
Adds data and title attribute to code blocks
Modifies MASTODON_STRICT

Preview:

Some considerations:
As this adds a title attribute, when hovering it shows the specified language.
I think that's a good idea, as it might not always be obvious which language code is actually written in.
This also means it has to be sanitised to not show anything a user specifies.
Therefore it is checked, whether it's a valid language or not and if it isn't, the title attribute gets removed.
~~I didn't want to change MASTODON_OUTGOING directly, as I don't know where it's all used and I didn't want to accidentally mess with other stuff.~~
I chose the github-dark theme as it's what's most people will be familiar with.
Ideally more styles would be included and users can choose in which style code is shown, but I considered this out of scope for the initial implementation.

Plastikmensch · 2022-12-16T23:39:07Z

Ah, I see one issue with this approach:
~~title attribute doesn't federate, therefore highlighting only works on the instance it was posted to~~ :/
It actually does, but is missing on remote toots, that's why highlighting doesn't work on them.

Edit:
Seems like I figured it out. 'code' => %w(title) needs to be added to MASTODON_STRICT instead, as it is used in html_aware_formatter.rb

prplecake · 2022-12-17T02:07:58Z

Contrast needs to be fixed in light themes.

prplecake · 2022-12-17T02:09:37Z

And in fact this doesn't fix #2011 as line highlighting has been so far forgotten.

Plastikmensch · 2022-12-17T08:23:39Z

Seems like I figured it out. 'code' => %w(title) needs to be added to MASTODON_STRICT instead, as it is used in html_aware_formatter.rb

I'm not too happy about modifying MASTODON_STRICT, but the alternative would be to create another modified constant in html_aware_formatter.rb and at that point I might as well edit it directly.

Contrast needs to be fixed in light themes.

How people can use light themes without their eyes burning is beyond me, but fixed.

And in fact this doesn't fix #2011 as line highlighting has been so far forgotten.

"Fixes" is just a closing keyword used to link issues. I could've also used close(s|d)?|resolve(s|d)?, but that's semantics and irrelevant.

app/javascript/flavours/glitch/components/status_content.js

ClearlyClaire · 2022-12-20T11:31:11Z

app/lib/advanced_text_formatter.rb

      <<~HTML
-        <pre><code>#{ERB::Util.h(code).gsub("\n", '<br/>')}</code></pre>
+        <pre><code title="#{language}">#{ERB::Util.h(code).gsub("\n", '<br/>')}</code></pre>


I'm not sure title is the best option here, but I don't know what would be better. I thought about lang but programming languages are explicitly out of scope.

I was also considering using a ~~style~~ class attribute instead, but as mentioned, using the title attribute has the advantage of showing which code language a code block was written in when hovering it.
I think title is the best option.

Hmmh... Now that highlighting requires the class attribute anyway, I think it's actually better to also change title to class. I would still like the element to have a title attribute though.

Ah, that doesn't work, because vanilla also sanitises class names. So while setting a title in the backend, converting it to class and then removing it again in the frontend isn't ideal, that's the only approach that works and doesn't allow arbitrary classes.

Maybe use a data-codelang or something? Do you know if other fediverse software have such a feature, and if so, how they communicate the language being used?

Using a data attribute sounds good.
It seems Misskey has syntax highlighting, but looking at their implementation, I can't figure out which attribute they use, if any.

Ah, it looks like Misskey just uses a class attribute: class="language-<langcode>" on both pre and code elements, which gets then interpreted by their highlighter prism.js, and a "lang" prop internally.
I'm not sure that's what you wanted to know though.

github-actions · 2023-01-18T14:35:50Z

This pull request has merge conflicts that must be resolved before it can be merged.

ClearlyClaire · 2023-01-22T22:08:31Z

app/javascript/flavours/glitch/styles/highlight.scss

+
+  Outdated base version: https://github.com/primer/github-syntax-dark
+  Current colors taken from GitHub's CSS
+*/.hljs{color:#c9d1d9;background:#0d1117}.hljs-doctag,.hljs-keyword,.hljs-meta .hljs-keyword,.hljs-template-tag,.hljs-template-variable,.hljs-type,.hljs-variable.language_{color:#ff7b72}.hljs-title,.hljs-title.class_,.hljs-title.class_.inherited__,.hljs-title.function_{color:#d2a8ff}.hljs-attr,.hljs-attribute,.hljs-literal,.hljs-meta,.hljs-number,.hljs-operator,.hljs-selector-attr,.hljs-selector-class,.hljs-selector-id,.hljs-variable{color:#79c0ff}.hljs-meta .hljs-string,.hljs-regexp,.hljs-string{color:#a5d6ff}.hljs-built_in,.hljs-symbol{color:#ffa657}.hljs-code,.hljs-comment,.hljs-formula{color:#8b949e}.hljs-name,.hljs-quote,.hljs-selector-pseudo,.hljs-selector-tag{color:#7ee787}.hljs-subst{color:#c9d1d9}.hljs-section{color:#1f6feb;font-weight:700}.hljs-bullet{color:#f2cc60}.hljs-emphasis{color:#c9d1d9;font-style:italic}.hljs-strong{color:#c9d1d9;font-weight:700}.hljs-addition{color:#aff5b4;background-color:#033a16}.hljs-deletion{color:#ffdcd7;background-color:#67060c}


Do we have permission to use this CSS bit?

It's part of highlight.js, which is licensed under BSD 3-Clause, which permits usage.
GitHub's CSS itself is licenced under MIT.
I'm not sure if the license file included in the module is sufficient compliance or if we would need to add an additional copyright notice.

~~I copied the styles from the module, because I couldn't import them directly.~~
Actually I tried it again and importing from the module worked this time.
Maybe I had the wrong path or something when I initially tried it.
But this also means including the style files is unnecessary.

ClearlyClaire · 2023-01-26T18:38:10Z

app/lib/advanced_text_formatter.rb

      <<~HTML
-        <pre><code>#{ERB::Util.h(code).gsub("\n", '<br/>')}</code></pre>
+        <pre><code data-codelang="#{language}">#{ERB::Util.h(code).gsub("\n", '<br/>')}</code></pre>


I don't suppose language is guaranteed to be escaped?

I'm not sure what kind of escaping would be suitable here.
At least in my testing I couldn't cause any unexpected behaviour, nevertheless I agree accepting user input as is is never a good idea.
Ideally, I would check here if language is a valid code language, but this would create overhead.
What I could do though is checking that language only contains allowed characters via regex: /^[a-zA-Z0-9.#+-]+$/

ClearlyClaire · 2023-01-26T18:39:22Z

lib/sanitize_ext/sanitize_config.rb

@@ -81,6 +81,7 @@ module Config
        'blockquote' => %w(cite),
        'ol'         => %w(start reversed),
        'li'         => %w(value),
+        'code'       => [:data],


We may want to only allow data-codelang, I can imagine arbitrary data- attributes causing other issues.

Ideally yes and I tried to limit it to only data-codelang, but nothing I tried worked, it just removed all data- attributes.

nvm I must have had a typo in last time I tried or there was some kind of other issue which prevented it from working.

github-actions · 2023-02-26T14:08:33Z

This pull request has merge conflicts that must be resolved before it can be merged.

github-actions · 2023-02-26T15:44:26Z

This pull request has resolved merge conflicts and is ready for review.

github-actions · 2023-03-15T08:00:37Z

This pull request has merge conflicts that must be resolved before it can be merged.

github-actions · 2023-03-15T09:18:33Z

This pull request has resolved merge conflicts and is ready for review.

github-actions · 2023-03-16T22:12:19Z

This pull request has merge conflicts that must be resolved before it can be merged.

github-actions · 2023-03-16T22:20:26Z

This pull request has resolved merge conflicts and is ready for review.

Plastikmensch · 2023-03-17T17:09:40Z

Just noticed that highlighting is missing in the reply-indicator, because it doesn't use the status_content component and while I could move the highlightCode function to a util and reuse it in reply_indicator.jsx, the reply indicator uses a lighter background by default which messes with the styling, but this also means invalid codelangs are not removed.

github-actions · 2023-03-18T10:20:10Z

This pull request has merge conflicts that must be resolved before it can be merged.

github-actions · 2023-04-04T11:20:11Z

This pull request has merge conflicts that must be resolved before it can be merged.

Adds a new node dependency. Also adds required scss. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

This creates a new constant used to sanitize the html tree, which allows title attributes on code tags. Adds title attribute specifying language to code tags. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Adds a new html util which converts html entities to their symbols Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Adds a new function which runs on contentHtml to add styling to code blocks. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Adds new check which makes sure title attribute is set correctly Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Removes modified sanitation and adds it directly to `MASTODON_STRICT` This is needed so that the title attritube is not being stripped for remote toots. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Removes the ".scss" extension from the import of the highlight style Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Removed convertToChar from `utils/html.js` Let highlight.js run directly on DOM instead of using RegEx. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Instead of using RegEx to replace br tags, use DOM Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Use data-codelang instead of title attribute for user set code language. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Little oversight. Changes the wording of the test case to use language instead of title, as title isn't used anymore. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Having user specified input display anywhere without sanitizing and removing it when invalid is a bad idea. This removes the data attribute when it's not a valid or supported code language. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

These are third-party styles, which introduce lots of lint problems as they don't adhere to style rules. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Removes highlight.scss files. Removes entries in ignoreFiles. Stylesheets are imported from highlight.js module directly. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

This style change was previously skipped and I missed it while rebasing. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Instead of passing arbitrary user input to the frontend and checking if it's valid there, use Sanitize to strip invalid code languages. For that a yaml file was added containing all valid language codes, which needs to be kept in sync with highlight.js. This is inconvenient, but shouldn't be a problem. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Messed up while rebasing... Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

github-actions · 2023-07-17T15:14:09Z

This pull request has resolved merge conflicts and is ready for review.

github-actions · 2023-10-27T16:17:11Z

This pull request has merge conflicts that must be resolved before it can be merged.

ClearlyClaire requested changes Dec 20, 2022

View reviewed changes

Plastikmensch requested a review from ClearlyClaire December 21, 2022 12:39

github-actions bot added the rebase needed 🚧 label Jan 18, 2023

Plastikmensch force-pushed the add-syntax-highlighting branch 2 times, most recently from 862b738 to cde4ad3 Compare January 18, 2023 16:39

github-actions bot removed the rebase needed 🚧 label Jan 18, 2023

ClearlyClaire reviewed Jan 22, 2023

View reviewed changes

ClearlyClaire requested changes Jan 26, 2023

View reviewed changes

github-actions bot added the rebase needed 🚧 label Feb 26, 2023

Plastikmensch force-pushed the add-syntax-highlighting branch from 9e886f8 to cd173f9 Compare February 26, 2023 15:44

github-actions bot removed the rebase needed 🚧 label Feb 26, 2023

github-actions bot added the rebase needed 🚧 label Mar 15, 2023

Plastikmensch force-pushed the add-syntax-highlighting branch from 55cd774 to d9ac464 Compare March 15, 2023 09:18

github-actions bot removed the rebase needed 🚧 label Mar 15, 2023

github-actions bot added the rebase needed 🚧 label Mar 16, 2023

Plastikmensch force-pushed the add-syntax-highlighting branch from 4190e7e to 1775188 Compare March 16, 2023 22:20

github-actions bot removed the rebase needed 🚧 label Mar 16, 2023

github-actions bot added the rebase needed 🚧 label Apr 4, 2023

Plastikmensch added 19 commits July 17, 2023 16:11

Add highlight.js

ab8134a

Adds a new node dependency. Also adds required scss. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Allow title attribute on code tags

20d6a54

This creates a new constant used to sanitize the html tree, which allows title attributes on code tags. Adds title attribute specifying language to code tags. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Add new util function

ffe0fa9

Adds a new html util which converts html entities to their symbols Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Add syntax highlighting to code blocks

186e1ca

Adds a new function which runs on contentHtml to add styling to code blocks. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Update advanced_text_formatter spec

b1f97e6

Adds new check which makes sure title attribute is set correctly Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Use github-light for highlighting in light theme

e3dc1d7

Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Fix remote toots not being highlighted

c12b06d

Removes modified sanitation and adds it directly to `MASTODON_STRICT` This is needed so that the title attritube is not being stripped for remote toots. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Remove file extension from import

9894b6e

Removes the ".scss" extension from the import of the highlight style Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Replace RegEx with DOM

049fe35

Removed convertToChar from `utils/html.js` Let highlight.js run directly on DOM instead of using RegEx. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Replace line breaks with DOM instead of RegEx

e8ed035

Instead of using RegEx to replace br tags, use DOM Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Use data attribute instead of title

af981da

Use data-codelang instead of title attribute for user set code language. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Change wording of test

9cef121

Little oversight. Changes the wording of the test case to use language instead of title, as title isn't used anymore. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Add highlight.scss to stylelint ignore

bb5fd53

These are third-party styles, which introduce lots of lint problems as they don't adhere to style rules. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Import stylesheets from module

ba60ac4

Removes highlight.scss files. Removes entries in ignoreFiles. Stylesheets are imported from highlight.js module directly. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Only allow codelang data attribute

e4b84e6

Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Apply style change

c5cf5a0

This style change was previously skipped and I missed it while rebasing. Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Update highlight.js to 11.8.0

9c48ed6

Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Plastikmensch force-pushed the add-syntax-highlighting branch from 17c35ce to 9c48ed6 Compare July 17, 2023 14:37

Plastikmensch added 2 commits July 17, 2023 16:53

Fix lint issues

1949a53

Messed up while rebasing... Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

Fix import order

f881d8d

Signed-off-by: Plastikmensch <plastikmensch@users.noreply.github.com>

github-actions bot removed the rebase needed 🚧 label Jul 17, 2023

github-actions bot added the rebase needed 🚧 label Oct 27, 2023

Plastikmensch mentioned this pull request Nov 3, 2023

Add syntax highlighting to code tags #2457

Open

Plastikmensch closed this Nov 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add syntax highlighting to code tags #2015

Add syntax highlighting to code tags #2015

Plastikmensch commented Dec 16, 2022 •

edited

Loading

Plastikmensch commented Dec 16, 2022 •

edited

Loading

prplecake commented Dec 17, 2022

prplecake commented Dec 17, 2022

Plastikmensch commented Dec 17, 2022 •

edited

Loading

ClearlyClaire Dec 20, 2022

Plastikmensch Dec 20, 2022 •

edited

Loading

Plastikmensch Dec 21, 2022 •

edited

Loading

ClearlyClaire Dec 22, 2022

Plastikmensch Dec 22, 2022

Plastikmensch Dec 22, 2022

github-actions bot commented Jan 18, 2023

ClearlyClaire Jan 22, 2023

Plastikmensch Jan 22, 2023 •

edited

Loading

ClearlyClaire Jan 26, 2023

Plastikmensch Mar 3, 2023 •

edited

Loading

ClearlyClaire Jan 26, 2023

Plastikmensch Jan 26, 2023

Plastikmensch Mar 3, 2023

github-actions bot commented Feb 26, 2023

github-actions bot commented Feb 26, 2023

github-actions bot commented Mar 15, 2023

github-actions bot commented Mar 15, 2023

github-actions bot commented Mar 16, 2023

github-actions bot commented Mar 16, 2023

Plastikmensch commented Mar 17, 2023 •

edited

Loading

github-actions bot commented Mar 18, 2023

github-actions bot commented Apr 4, 2023

github-actions bot commented Jul 17, 2023

github-actions bot commented Oct 27, 2023

Add syntax highlighting to code tags #2015

Add syntax highlighting to code tags #2015

Conversation

Plastikmensch commented Dec 16, 2022 • edited Loading

Plastikmensch commented Dec 16, 2022 • edited Loading

prplecake commented Dec 17, 2022

prplecake commented Dec 17, 2022

Plastikmensch commented Dec 17, 2022 • edited Loading

Choose a reason for hiding this comment

Plastikmensch Dec 20, 2022 • edited Loading

Choose a reason for hiding this comment

Plastikmensch Dec 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jan 18, 2023

Choose a reason for hiding this comment

Plastikmensch Jan 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Plastikmensch Mar 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 26, 2023

github-actions bot commented Feb 26, 2023

github-actions bot commented Mar 15, 2023

github-actions bot commented Mar 15, 2023

github-actions bot commented Mar 16, 2023

github-actions bot commented Mar 16, 2023

Plastikmensch commented Mar 17, 2023 • edited Loading

github-actions bot commented Mar 18, 2023

github-actions bot commented Apr 4, 2023

github-actions bot commented Jul 17, 2023

github-actions bot commented Oct 27, 2023

Plastikmensch commented Dec 16, 2022 •

edited

Loading

Plastikmensch commented Dec 16, 2022 •

edited

Loading

Plastikmensch commented Dec 17, 2022 •

edited

Loading

Plastikmensch Dec 20, 2022 •

edited

Loading

Plastikmensch Dec 21, 2022 •

edited

Loading

Plastikmensch Jan 22, 2023 •

edited

Loading

Plastikmensch Mar 3, 2023 •

edited

Loading

Plastikmensch commented Mar 17, 2023 •

edited

Loading