pymdownx.highlight extension causes crash at startup

[jstafford5380]

Contribution guidelines

• I've read the contribution guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the problem doesn't occur with the mkdocs or readthedocs themes
• ... the problem persists when all overrides are removed, i.e. custom_dir, extra_javascript and extra_css
• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

I'm trying to get code highlighting to work but when including the documented option to enable, the site crashes

markdown_extensions:
  - pymdownx.highlight:
      anchor_linenums: true

Expected behaviour

The site should start up and code blocs should be highlighted

Actual behaviour

When serving the site, the crashes

mkdocs serve
INFO     -  Building documentation...
INFO     -  Cleaning site directory
ERROR    -  Error reading page 'index.md': format() argument must be a formatter instance, not a class
Traceback (most recent call last):
  File "C:\Python310\lib\site-packages\pygments\__init__.py", line 61, in format
    formatter.format(tokens, realoutfile)
  File "C:\Python310\lib\site-packages\pygments\formatter.py", line 94, in format
    return self.format_unencoded(tokensource, outfile)
  File "C:\Python310\lib\site-packages\pygments\formatters\html.py", line 981, in format_unencoded
    source = self.wrap(source)
TypeError: BlockHtmlFormatter.wrap() missing 1 required positional argument: 'outfile'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Python310\Scripts\mkdocs.exe\__main__.py", line 7, in <module>
  File "C:\Python310\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Python310\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Python310\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Python310\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Python310\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Python310\lib\site-packages\mkdocs\__main__.py", line 181, in serve_command
    serve.serve(dev_addr=dev_addr, livereload=livereload, watch=watch, **kwargs)
  File "C:\Python310\lib\site-packages\mkdocs\commands\serve.py", line 63, in serve
    config = builder()
  File "C:\Python310\lib\site-packages\mkdocs\commands\serve.py", line 58, in builder
    build(config, live_server=live_server, dirty=dirty)
  File "C:\Python310\lib\site-packages\mkdocs\commands\build.py", line 292, in build
    _populate_page(file.page, config, files, dirty)
  File "C:\Python310\lib\site-packages\mkdocs\commands\build.py", line 174, in _populate_page
    page.render(config, files)
  File "C:\Python310\lib\site-packages\mkdocs\structure\pages.py", line 175, in render
    self.content = md.convert(self.markdown)
  File "C:\Python310\lib\site-packages\markdown\core.py", line 268, in convert
    newRoot = treeprocessor.run(root)
  File "C:\Python310\lib\site-packages\pymdownx\highlight.py", line 491, in run
    code.highlight(
  File "C:\Python310\lib\site-packages\pymdownx\highlight.py", line 392, in highlight
    code = highlight(src, lexer, formatter)
  File "C:\Python310\lib\site-packages\pygments\__init__.py", line 82, in highlight
    return format(lex(code, lexer), formatter, outfile)
  File "C:\Python310\lib\site-packages\pygments\__init__.py", line 69, in format
    raise TypeError('format() argument must be a formatter instance, '
TypeError: format() argument must be a formatter instance, not a class

Steps to reproduce

1. Create a new docs project
2. Add provided configuration
3. Attempt to serve mkdocs serve

Package versions

• Python: python --version
  3.10.2

• MkDocs: mkdocs --version
  1.3.0

• Material: pip show mkdocs-material | grep -E ^Version
  8.2.10

Configuration

site_name: My Site
theme:
  name: material
markdown_extensions:  
  - pymdownx.highlight:
      anchor_linenums: true

System information

• Operating system: Windows 10
• Browser: Tested in Brave and Chrome

[facelessuser]

This is already fixed and released in 9.4

[facelessuser]

Of pymdownx-extensions that is

[giswqs]

Just ran into the same issue with the following package versions.

mkdocs-material-8.2.10
pymdown-extensions-9.4
Pygments-2.12.0

Tried to pin the pygments version with pygments>=2.10,<2.12, then got the error Pymdownx Highlight requires at least Pygments 2.12+ if enabling Pygments

[facelessuser]

@giswqs Your issue is not the same. Pymdown Extensions 9.4 fixes the issue with Pygments 2.12 and requires Pygments 2.12, so if you pin Pygments to less than 2.12, then you need to also use Pymdown Extensions 9.3. If you want to use the latest of both, don't pin them.

[giswqs]

@facelessuser I understand. Initially, I did not pin versions, and I got the error TypeError: format() argument must be a formatter instance, not a class. Then I pinned pygments>=2.10,<2.12 and got the error Pymdownx Highlight requires at least Pygments 2.12+ if enabling Pygments. So I guess I also need to pin pymdown-extensions<9.4 as a temporary workaround
https://github.com/giswqs/geemap/runs/6150893473?check_suite_focus=true#step:11:212

It would be great if mkdocs-material can fix the issue so that I don't need to pin versions.

[facelessuser]

@giswqs

So I guess I also need to pin pymdown-extensions<9.4 as a temporary workaround

No, I don't see why you need to pin <9.4. You need to use the latest of both. It is possible maybe the packages were out of sync when you tested, and you didn't get the latest of both.

I was able to test on a py3.7 - p3.10 with the latest Pygments and Pymdown Extensions with no issues. I was able to deploy the documentation for the project with the latest Pygments, Pymdown Extensions, and MkDocs Material.

TypeError: format() argument must be a formatter instance, not a class

This error is specifically caused because you are using the latest Pymdown Extensions but pinning an older Pygments version.

It would be great if mkdocs-material can fix the issue so that I don't need to pin versions.

From what I see, there are no more issues, and mkdocs-material doesn't need to fix any issues.

Can you provide more specific details? I'm happy to fix anything on my end that I can, but what you've posted is a known error when using incompatible Pygments and Pymdown Extensions versions. You need to use the lastest of both.

[giswqs]

@facelessuser No, it fails to build if I don't pin versions even with the latest mkdocs-material. See the following three scenarios. Only the third scenario with pin versions of the both pygments and pymdown-extensions works.

requirements_docs.txt

First scenario:

mkdocs-material 

Commit: https://github.com/giswqs/geemap/pull/1038/files (no changes to requirements_docs.txt)
Error: https://github.com/giswqs/geemap/runs/6150893473?check_suite_focus=true#step:11:213

Second scenario:

mkdocs-material 
pygments>=2.10,<2.12

Commit: gee-community/geemap@5cb3d12
Error: https://github.com/giswqs/geemap/runs/6150975478?check_suite_focus=true#step:11:64

Third scenario:

mkdocs-material 
pygments>=2.10,<2.12
pymdown-extensions<9.4

commit: gee-community/geemap@7049f86
pass: https://github.com/giswqs/geemap/runs/6151239311?check_suite_focus=true

[giswqs]

For the first scenario above, it installed all packages with the latest version. However, it still throws the error TypeError: format() argument must be a formatter instance, not a class.

mkdocs-material-8.2.10
pymdown-extensions-9.4
Pygments-2.12.0

Pygments: https://github.com/giswqs/geemap/runs/6150893473?check_suite_focus=true#step:10:77
mkdocs-material: https://github.com/giswqs/geemap/runs/6150893473?check_suite_focus=true#step:10:51
pymdown-extensions: https://github.com/giswqs/geemap/runs/6150893473?check_suite_focus=true#step:10:168

[facelessuser]

@giswqs But this isn't a Mkdocs Material issue or a Pymdown Extensions issue. You have a conflict as I believe Juypter is also requiring Pygments. It's using a Mistune markdown parser as well.

https://github.com/giswqs/geemap/runs/6150975478?check_suite_focus=true#step:11:64

This error clearly fails due to Pymdown Extensions needed the most recent Pygments, and it is not there. pygments-2.11.2 pymdown-extensions-9.4.

https://github.com/giswqs/geemap/runs/6150893473?check_suite_focus=true#step:11:213

This one isn't even failing in Mkdocs Material, it is failing in Jupyter. This is not a Material issue. You have a lot of libraries, and they have incompatible requirements. Juypter seems to be running an entirely different Markdown parser (Mistune) than Mkdocs is using.

You have a special case where you need to pin both pymdown-extensions and pygments, but that is because of additional 3rd party things you are pulling in. That has nothing to do with Material specifically though.

[giswqs]

It installed Pygments 2.12.0, which is the latest version, newer than 2.11.2 you mentioned.

https://github.com/giswqs/geemap/runs/6150893473?check_suite_focus=true#step:10:77

This error clearly fails due to Pymdown Extensions needed the most recent Pygments, and it is not there. pygments-2.11.2 pymdown-extensions-9.4.

[facelessuser]

I can't tell what version of Pygments is installed as I see this:

Requirement already satisfied: Pygments>=1.6 in /opt/hostedtoolcache/Python/3.9.12/x64/lib/python3.9/site-packages (from grip->-r requirements_docs.txt (line 4)) (2.12.0)

Also, the failure is not in Mkdocs Material or Pymdown Extensions, it is in Jupyter somewhere via mkdocs_jupyter. That's a Mkdocs Jupyter/Jupyter issue.

[jstafford5380]

Hey so I got a little lost in here... are we off topic or did I miss a turn somewhere? I don't use Jupyter and I just installed mkdocs-material using the docs on the site. I need version 9.4? I followed the instructions which led to 8.*

[giswqs]

The version number (2.12.0) can be found at the end of the line you pasted. The error message is the same as @jstafford5380 posted.

Once I pinned pygments and pymdown-extensions versions, then it built successfully. I agreed that it is probably not a mkdocs-material issue, but I don't think it is a mkdocs-jupyter either @danielfrg.

pygments>=2.10,<2.12
pymdown-extensions<9.4

[facelessuser]

@jstafford5380 sorry, we are a bit off topic. The failure was in pymdown-extensions, which I maintain. Version 9.4 of that package now works with Pygments 2.12+ only, but should fix the failure.

The off topic part is that some plugins for MkDocs may not support the latest pygments. That is not specifically a Material issue though. If you need to pin packages because some other external MkDocs plugin has a conflict still, then you'll have to pin multiple things. If not, you can use the latest Pygments with the latest pymdown-extensions.

[facelessuser]

@giswqs It is not the same

The opening post originates in pymdownx.highlight, something that Material uses, that is why it is relevant:

  File "C:\Python310\lib\site-packages\pymdownx\highlight.py", line 491, in run
    code.highlight(
  File "C:\Python310\lib\site-packages\pymdownx\highlight.py", line 392, in highlight
    code = highlight(src, lexer, formatter)
  File "C:\Python310\lib\site-packages\pygments\__init__.py", line 82, in highlight
    return format(lex(code, lexer), formatter, outfile)
  File "C:\Python310\lib\site-packages\pygments\__init__.py", line 69, in format
    raise TypeError('format() argument must be a formatter instance, '
TypeError: format() argument must be a formatter instance, not a class

Yours originates in Mistune/Jupyter, a completely different Markdown parser originating from MkDocs Jupyter. The final line is the same, but that does not mean the same libraries are responsible.

  File "/opt/hostedtoolcache/Python/3.9.12/x64/lib/python3.9/site-packages/mistune.py", line 1141, in output_loose_item
    body += self.tok()
  File "/opt/hostedtoolcache/Python/3.9.12/x64/lib/python3.9/site-packages/mistune.py", line 1063, in tok
    return getattr(self, 'output_%s' % t)()
  File "/opt/hostedtoolcache/Python/3.9.12/x64/lib/python3.9/site-packages/mistune.py", line 1085, in output_code
    return self.renderer.block_code(
  File "/opt/hostedtoolcache/Python/3.9.12/x64/lib/python3.9/site-packages/mkdocs_jupyter/nbconvert2.py", line 219, in block_code
    return highlight(code, lexer, formatter)
  File "/opt/hostedtoolcache/Python/3.9.12/x64/lib/python3.9/site-packages/pygments/__init__.py", line 82, in highlight
    return format(lex(code, lexer), formatter, outfile)
  File "/opt/hostedtoolcache/Python/3.9.12/x64/lib/python3.9/site-packages/pygments/__init__.py", line 69, in format
    raise TypeError('format() argument must be a formatter instance, '
TypeError: format() argument must be a formatter instance, not a class
Error: Process completed with exit code 1.

Pymdown Extensions, which Material relies on, is fully tested with Pygments 2.12: https://github.com/facelessuser/pymdown-extensions/actions/runs/2217033680. I even test that it builds with the latest MkDocs Material: I even test that it builds with MkDocs Material: https://github.com/facelessuser/pymdown-extensions/runs/6149575142?check_suite_focus=true#step:7:47.

If you can provide a minimal example showing otherwise, I'm happy to dig in and try to fix it, but the trace you've linked to seems to indicate this fails in the MkDocs Juypter plugin, it does not originate in MkDocs Material or Pymdown Extensions (what the opening post showed).