feature_request(superfences): Tab indentation in code blocks

[Kristinita]

1. Summary

If I generate Markdown GFM code blocks to HTML via SuperFences:

    I get unexpected spaces in code blocks.

2. Argumentation

Premices:

• I use tabs, not spaces for indentation,
• SuperFences generate HTML with spaces, not tabs indentation,
• js-beautify doesn't convert spaces to tabs in <pre> blocks.

Conclusion:

• I get errors, when I check my HTML files, that them contains tabs for indentation, not spaces.

3. Configuration

Example file SashaSuperFences.md:

```json
{
	"name": "SashaSuperFences",
}
```

4. Steps to reproduce

I print in console:

python -m markdown -x pymdownx.superfences SashaSuperFences.md

5. Expected behavior

Tab before <span class="nt">.

<div class="highlight"><pre><span></span><span class="p">{</span>
	<span class="nt">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;SashaSuperFences&quot;</span><span class="p">,</span>
<span class="p">}</span>
</pre></div>

6. Actual behavior

4 spaces before <span class="nt">.

<div class="highlight"><pre><span></span><span class="p">{</span>
    <span class="nt">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;SashaSuperFences&quot;</span><span class="p">,</span>
<span class="p">}</span>
</pre></div>

Thanks.

[facelessuser]

Yes, this is generally a problem with the approach Python Markdown takes with whitespace. All tabs are normalized to the default indentation (which is 4). This is applied to the entire file usually before any processor is applied. You'll notice this behavior in indented code blocks as well. I believe this is done to make patterns and indention detection more straight forward in the code. In most places, this replacement isn't a problem....but with code blocks, especially if you are using tabs to align certain things, can be a problem.

With all of that said, I might be able be able address this for SuperFences. It will take a bit of work, but I believe it is possible. This would not apply to indented code blocks; only fenced blocks. If I do this, I might leave the current whitespace behavior as default for consistency with the rest of Python Markdown, but have an option to respect tabs.

In general though, I'd bring this issue up on Python Markdown, but I can probably alter SuperFences' fenced code block behavior in the future.

[facelessuser]

Thinking about this a little more. If I do this, I'll have to account for some special cases. SuperFences expects fenced blocks to align. If we have inconsistent indentation, some tabs will have to be converted to spaces to preserve alignment.

    ```
  \tcontent
\t```

In this case the opening and closing fence markers are equivalent as tabs are equivalent to 4 spaces when parsing. But the content line is preceded by 2 spaces and 1 tab. Indentation should be 4 spaces or 1 tab, but we have 2 spaces and a tab that is equivalent to 4 spaces that crosses the indentation boundary. In this case, we'd probably have to split the tab as 2 spaces for indentation and 2 spaces in content.

On the other hand, in this case we could also just have indentation absorb the entire tab. This might create unexpected formatting.

In short, to preserve tabs, it would be expected that tabs are wholly contained within the fenced content region or wholly contained in the indentation region. If they cross boundaries, they'd get translated to spaces.

Considering how Python Markdown can not currently handle fenced blocks as block processors, they are actually implemented as a preprocessor. This means that we sometimes convert blocks that are actually in indented code blocks and then have to restore them. But if we are trying to preserve tabs, our logic could potentially cause some weird results in indented code blocks when we restore.

In short, whatever we do may cause unintended side effects depending how it is implemented. We'll have to see if the effort required is worth it.

[facelessuser]

This looks to be doable when moved before the normalize_whitespace step, but unfortunately introduces an issue. We use place holders in the same way that Python Markdown's other extensions do. The placeholders utilize a special key and number surrounded by the control characters STX and ETX.

normalize_whitespace strips out all STX and ETX control characters, but we have to execute before normalize_whitespace. We'll probably have to look for something different to pull this off. I'd like to not introduce a complex normalize_whitespace replacement.

[facelessuser]

Experimental branch contains a working tab preserving feature. I've barely tested it, but it in limited testing, it appears to work. See commit for notes.

[facelessuser]

My understanding of the tab conversion process was incorrect. Python Markdown actually uses expandtabs method to expand tabs to spaces. This converts tab logic appropriately. If a tab length is 4 and you are presented with ..- (and . is a space and - is a tab) the tab will be covered to 2 spaces while - would be converted to 4. The following commit address this logic and now matches Python Markdown: 3d6fe65.

[facelessuser]

Looks like we finally have something pretty solid via pull request #279.

[Kristinita]

@facelessuser , we wait new release.

Thanks.

[facelessuser]

This has been released. If you run into tab related parsing issues, let me know, but hopefully it just works.

[Kristinita]

Status: ✔️ Fixed for me

Thanks.