Allow highlighting in code samples

[ArgentEnergy]

We use code samples (have a specific Word styling for code samples) for HTTP requests and one of the pain points we have is GW doesn't offer the ability to highlight text inside them. The mark HTML tag is stripped if manually adding into the source code under the code HTML tag. This looks to be a dependency issue relying on TinyMCE's code sample plugin that relies on Prism.js.

Is it possible to fix this issue with having a custom Prism.js file to not strip the mark tags?
https://www.tiny.cloud/docs/plugins/opensource/codesample/#codesample_global_prismjs
PrismJS/prism#879

We have to manually do the highlighting in Word. This becomes tedious if you did the manual work already and a report change happens that makes us regenerate the report again.

[chrismaddalena]

Hey @ArgentEnergy, I will look into this. If TinyMCE can't do it, there might be an easy way to stop using the codesample plugin and create a custom HTML wrapper. Ghostwriter's reporting engine only cares about the <code></code> tags to know to switch to your code block style for the paragraph.

[ArgentEnergy]

Thank you @chrismaddalena! :)

This might be another issue but I noticed that code samples, bullet lists, numbered lists, and inline code aren't following the styling in Word. I checked my template and they all have font justified for the font styles but we have to manually do this after exporting. TinyMCE covers the font justify issue for most but not code samples and the lists (bullet or numbered).

I don't think it's my template but here's our format:

{%p if finding.affected_entities_rt %} 
{{p finding.affected_entities_rt }} 
{%p else %} 

{%p endif %}

We're on v3.2.6 right now and plan on moving to version 4 so I'm not sure if this was resolved in a future version yet.

[felix-caboff]

In a related note, web browsers that have dark-mode enabled struggle to render the highlighting that TinyMCE can apply to non-code elements. I'm not really sure why though - I would imagine that highlighting yellow would always result in it being yellow. If you look closely, you can see there is a slight shift from dark grey to slightly lighter grey, but not much of one.

[domwhewell-sage]

@ArgentEnergy did you find out how to highlight lines of code within the codesample plugin?
I've got a work around at the moment where I can add multiple lines of inline code and it applies the "CodeBlock" style allowing us to highlight lines of code.
But it would be nice if the codesample plugin and gw supported this.

[ArgentEnergy]

@domwhewell-sage No, right now we have to do it manually in the Word document until Ghostwriter supports it. Not sure of the ETA on this feature. You'll have to ask @chrismaddalena.

[chrismaddalena]

I haven't looked into yet. As mentioned above, we'll have to investigate what we can do with TinyMCE. We can't highlight or otherwise style code inside a code block today because it's all wrapped in pre tags by TinyMCE. There is no styling inside those tags. I expect we'll need to customize the code block to it looks the same but doesn't apply pre tags or use TinyMCE's source code input modal.

[github-actions]

This issue has been labeled as stale because it has been open for 30 days with no activity.

[github-actions]

This issue is closed because it has been inactive for 14 days since being labeled stale. Feel free to re-open the issue with a comment. If this needs further discussion (e.g., a feature request), it might be better to open a topic under the Discussions tab.

[ArgentEnergy]

@chrismaddalena Is this on a roadmap to implement at some point in the near future?

[domwhewell-sage]

Hi @ArgentEnergy this might help,

I have achieved this by modifying the text() function and added a insert_code() function to ooxml.py

Essentially in the editor you add in a block of text and a inline code the same way and the insert_code() function uses the while current_el loop to determine if it is a paragraph containing only <code> tags or not.

If the paragraph contains other text/tags aswell as a <code> tag it can be considered as inline_code and that style is applied. However if the paragraph contains all <code> tags it should not be treated as inline_code and the CodeBlock paragraph style should be applied.

This way any styles that are children of <code> blocks are applied allowing for highlighting of code.

def text(self, el, *, par=None, style=None, **kwargs):
        if par is None:
            # Text without a paragraph. If this is just some trailing whitespace, ignore it, otherwise
            # report an error.
            if el.strip():
                raise ValueError(
                    "found text node that was not enclosed in a paragraph or other block item: {!r}".format(el.text)
                )
            return
        if style and style.get("inline_code"):
            self.insert_code(par, el, style)
        else:
            run = par.add_run()
            self.text_tracking.append_text_to_run(run, el)
            self.style_run(run, style or {})

def insert_code(self, par, el, style=None):
        """
        Add some code with formatting into the report.

        If no other code blocks exist within the parent element, the code will be styled as a code block. Otherwise, it will be styled as inline code.
        """
        current_el = el
        while current_el is not None and current_el.name != "p":
            current_el = current_el.parent

        if all(child.name in ['code', 'br'] for child in current_el.children):
            style["inline_code"] = False
            par.style = "CodeBlock"

        self.text_tracking.force_emit_pending_segment_break()
        run = par.add_run(text)
        self.style_run(run, style)

Our main use case for this is pasting in HTTP requests/responses from burpsuite and allowing us to highlight areas of the request/response that are important.

Here is the source of an rich text editor

<p>In the extract below a request was sent to <code>http://host.example.com/user/profile/update</code></p>
<p><code>POST /user/profile/update HTTP/2</code><br /><code>Host: host.example.com</code><br /><code>User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.0) Gecko/20100101 Firefox/127.0</code><br /><code>Accept: application/json, text/javascript, */*; q=0.01</code><br /><code>Accept-Language: en-GB,en;q=0.5</code><br /><code>Accept-Encoding: gzip, deflate, br</code><br /><code>Content-Type: application/json</code></p>
<p><code>{</code><br /><code>    <span class="highlight" style="background-color: yellow;">"FirstName":"Test",</span></code><br /><code>    "LastName":"User",</code><br /><code>    "Email":"testuser@example.com",</code><br /><code>}</code></p>

This when generated looks something like this

[chrismaddalena]

Yes, this is on the road map for us to look at.

@domwhewell-sage Is that all the code you added for highlighting in the code sample?

[chrismaddalena]

It looks like this works after changing run = par.add_run(text) to run = par.add_run(el.text). How are you going about inserting the code sample in this case?

You have multiple code elements separated by br elements inside one p element. Did you do that manually?

[chrismaddalena]

One option is adding forced_root_block: false, to the TinyMCE config. That will cause TinyMCE to insert <br /> every time you press Enter. Pressing SHIFT+Enter will start a new p element. That would make inserting code blocks like this easier, but would also require pressing SHIFT+Enter any time you want the next line to begin a new paragraph (important for Word).

[domwhewell-sage]

Hi @chrismaddalena, yes these are the changes I made to the report writer module to get it to process <span formatting tags (such as highlight) inside <code> tags.

I am pasting the code into the editor straight from burpsuite so perhaps that is where it's creating the <br> tags from once pasted in as plain text I just select it and pressing inline code and the source looks like this. I didn't add the <br> elements manually though.

From here I'm not sure if we can modify the code sample plugin to insert the code like this or we need todo a custom plugin to insert the code like this which will also allow is todo syntax highlighting.

[chrismaddalena]

OK, interesting! We can probably include this in a release so it's possible to do, even if how you add your code sample may add newlines or not. I'm going to think about the br aspect. I lean toward not wanting to surprise people with needing to press SHIFT+Enter for a new paragraph, but I do like the flexibility.

Our goal is writing reports, not HTML webpages, so it's unlikely you'd ever want to break a line in the middle of a paragraph. That makes me want to explore the custom code block/sample plugin idea.

[ColonelThirtyTwo]

OK, interesting! We can probably include this in a release so it's possible to do, even if how you add your code sample may add newlines or not. I'm going to think about the br aspect. I lean toward not wanting to surprise people with needing to press SHIFT+Enter for a new paragraph, but I do like the flexibility.

Our goal is writing reports, not HTML webpages, so it's unlikely you'd ever want to break a line in the middle of a paragraph. That makes me want to explore the custom code block/sample plugin idea.

I think a better way - which I'm gonna try to implement - is to wrap the code segment in a blockquote, and turn the <blockquote> into a code snippet iff all the paragraphs within are either empty or contain just <code> enclosed text. That would provide a definitive section for the code snippet without having to worry about <p> vs <br/>.

A custom TinyMCE plugin would be cool, but I'm not sure if it's worth the effort.

[chrismaddalena]

Alex worked on a different approach with PR #487. Here is an example they shared of it:

You can style with "rich code" to add formatting to a code block.

@domwhewell-sage If you have some time to check it out, please let us know if this does what you're looking for. I think it does what you wanted and then some.

[domwhewell-sage]

Hi both,
Thank you for working on this @ColonelThirtyTwo that approach is definitely better than mine 😜
I have tested it out with test cases

1. Apply rich code formatting, paste in HTML/HTTP request from burp and highlight line
2. Paste in HTML/HTTP request from burp, highlight line and apply rich code formatting

In both cases it generated correctly appearing how it did in the editor

Could we swap the "Rich code" menu item onto the menu bar in place of the "Code Sample", with the {;} icon (Moving code sample to the formatting menu item or removing "Code Sample" altogether as rich code replaces its function)?
Also am i mistaken or did <> used to be labeled "Inline Code" on the menu bar?

[ColonelThirtyTwo]

Yeah, I'm not opposed to removing the built-in code sample functionality. The only argument I can see against it is that code samples theoretically support automatic syntax highlighting, but I don't think we ever set that up.

[domwhewell-sage]

Yeh the built-in code sample plugin does syntax highlighting in the webapp

But this doesn't appear in the source of the editor and couldn't pass through into the report writer module

<pre class="text-evidence language-markup"><code>&lt;pre lang="JavaScript" line="1"&gt;
&lt;html&gt;
&lt;head&gt;
&lt;title&gt;Clickjacking PoC&lt;/title&gt;
&lt;/head&gt;
&lt;body&gt;
&lt;h2&gt;Your Web Application Can be Mounted within an iFrame and is vulnerable to ClickJacking!&lt;/h2&gt;
&lt;iframe src="https://example.com" height="450" width="1000"&gt;&lt;/iframe&gt;
&lt;/body&gt;
&lt;/html&gt;
&lt;/pre&gt;</code></pre>

I suppose if it becomes a requirement todo syntax highlighting as well it would be better to have some client side javascript to update the source of the editor with <span style="color: #000000;">blah</span> for each of the tags then it can be interpreted by the reportwriter module without any code changes

[ColonelThirtyTwo]

We could also run a syntax highligher on the server. But that's for the future.

(Also we'd probably keep the code snippet plugin loaded, to support existing uses, but we can hide the buttons)

@chrismaddalena Thoughts?

[ColonelThirtyTwo]

Based on Chris's feedback, I've updated the PR to replace the toolbar Code Snippet with the new Rich Code. The Code snippet feature is still available via the Insert menu.

[chrismaddalena]

I merged the PR and the feature will be in the next release.