Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flexible styling of declarations #4289

Open
jakobandersen opened this issue Dec 11, 2017 · 5 comments
Open

Flexible styling of declarations #4289

jakobandersen opened this issue Dec 11, 2017 · 5 comments
Labels
type:enhancement enhance or introduce a new feature

Comments

@jakobandersen
Copy link
Contributor

Currently there are very few options for styling declarations, essentially only a few desc_* nodes in sphinx/addnodes.py. For some languages (e.g., C++) declarations may contain many keywords and built-in types that should be coloured to help readability. It would be cool if themes could define custom styles.

I'm not sure what the best way to go about this is, but how about the following.
From Pygments (pygments.tokens) I found the following types of tokens:
'Comment', 'Error', 'Escape', 'Generic', 'Keyword', 'Literal', 'Name', 'Number', 'Operator', 'Other', 'Punctuation', 'STANDARD_TYPES', 'String', 'Text', 'Token', 'Whitespace'
To addnodes.py we could add additional desc_* nodes for those we don't have yet, and that makes sense. Each should be constructible with a domain name. In terms of CSS each of them should emit two classes; a generic one and a domain-specific one (if a domain is specified).

  • Can this be done in a format-agnostic way with Docutils? (via the classes attribute?)
  • Can we make the nodes inherit from a common class so writers can opt-in to support this?
  • As this overlaps a lot with Pygments, is there anything we can reuse? E.g., the style maps?
  • Is there a better way to implement this?
  • Are people against this feature?
@jakobandersen jakobandersen added the type:enhancement enhance or introduce a new feature label Dec 11, 2017
jakobandersen added a commit to jakobandersen/sphinx that referenced this issue Dec 13, 2017
Try with::

	.. cpp:class:: template<typename T> MyClass
jakobandersen added a commit to jakobandersen/sphinx that referenced this issue Dec 23, 2017
Try with::

	.. cpp:class:: template<typename T> MyClass
@jakobandersen
Copy link
Contributor Author

In https://github.com/jakobandersen/sphinx/tree/4289_decl_styling there is a proof-of-concept implementation, based on the ideas outlined above (The current changes simply colours the C++ keyword template blue in the HTML output):

  • There is a single new node type for text elements in declarations (to avoid a lot of boiler-plate code in each writer).
  • The node type is not supposed to be constructed directly, but by functions for each syntactic construct.
  • The node always adds 2 or 3 elements to the classes attribute of it self (are the class names appropriate?):
    • d, a generic class for all description elements.
    • A class specific for the syntactic construct (e.g., k for keywords).
    • If a domain is specified, also <e>-<domain> where <e> is the is the element above, and <domain> is the given domain (e.g., k-cpp for a keyword in the C++ domain).
  • Only the HTML formats are actually affected by these changes, though I expect a set of macros for the Latex format could be introduced, controlled by checking the classes attribute in the Latex writer.
    Maybe @jfbu already have some thoughts of how to do this best?
  • The nodes desc_name and desc_addname can be deprecated and be made as their own syntactic constructs.

Modifying a domain to use this new node type is a lot of work, so before making a larger pull-request: are people ok with this change?

jfbu added a commit to jfbu/sphinx that referenced this issue Dec 30, 2017
This goes for the quick solution, hence uses the pre-existing
architecture underlying the `\PYG` macro from the Pygments
highlighting code.

Each domain can add its own style to LaTeX by defining suitable macro
for supporting css styles.

For example, here we added

    \@namedef{PYG@tok@k-cpp}{%
         \def\PYG@tc##1{\textcolor[RGB]{0,0,"FF}{##1}}%
    }%

to sphinx.sty to emulate the CSS

    code.k-cpp {
        color: #0000FF;
    }

(previous commit uses `#0000AA` but I chose `#0000FF` for better seeing
things...)

For a fully developed example all the `\@namedef` would be in some
`sphinxcpp.sty` style file and either the document will do
`\usepackage{sphinxcpp.sty}` or `sphinx.sty` will do
`\RequirePackage{sphinxcpp.sty}`. The user could override this file via
`latex_additional_files`.

In absence of such file, the macro definitions will be lacking but this
will raise no error. For the example here (from last commit message) the
latex output contains

    \PYG{d+k+k-cpp}

and the `\PYG` macro nests "action of d(of k(of k-cpp...)))". Turns out
that Pygments already has some macro for "k", which uses bold face (as
can be seen in PDF output) and changes the text color, but has "k-cpp"
is deeper nested it is executed last and sets the good text color.

Thus it would perhaps be better if the latex writer produces something
like

    \PYG{cssd+cssk+cssk-cpp}

to avoid name clashes with Pygments pre-defined styles. Or here it is OK
that "k" is styled boldface by Pygments in LaTeX.

Important: class names must not contain the `+` character which is use
by Pygments LaTeX code as separator.

There are some issues in the Pygments code with the scope of some
changes; this is already corrected by Sphinx (refs: sphinx-doc#4250), but some
more scrutiny is perhaps needed because so far the Pygments code is
exercised only inside Verbatim environments where each code-line is a
scope in itself so border effects do not propagate, hence I have not
checked closely the Pygments LaTeX code, but perhaps sphinx-doc#4250 fixes already
all problems.
@jfbu
Copy link
Contributor

jfbu commented Dec 30, 2017

Maybe @jfbu already have some thoughts of how to do this best?

@jakobandersen can you check https://github.com/jfbu/sphinx/tree/4289_decl_styling?

It is not "how to do it best" but rather "how to do it fast" ;-)

(I have amended the commit message)

edit: in my commit I do a manual definition of LaTeX styling macro. But from brief look at Pygments, the Formatters have get_style_defs() methods which produce automatically the styling (css for html, latex commands for latex), starting from a Style.styles. I have not checked if Pygments offer a user interface to extend the list of STANDARD_TYPES (in pygments.token), but this can surely be done. Thus producing CSS style and LatTeX macros can be obtained from Pygments itself. Currently Sphinx LaTeX Builder uses Pygments on the fly to produce again on each build the highlighting macros. It should not be too involved to get this to include the added highlighting macros for cpp domain.

jfbu added a commit to jfbu/sphinx that referenced this issue Dec 30, 2017
This goes for the quick solution, hence uses the pre-existing
architecture underlying the `\PYG` macro from the Pygments
highlighting code.

Each domain can add its own style to LaTeX by defining suitable macro
for supporting css styles.

For example, here we added

    \@namedef{PYG@tok@k-cpp}{%
         \def\PYG@tc##1{\textcolor[RGB]{0,0,"FF}{##1}}%
    }%

to sphinx.sty to emulate the CSS

    code.k-cpp {
        color: #0000FF;
    }

(previous commit uses `#0000AA` but I chose `#0000FF` for better seeing
things... also syntax `\textcolor[HTML]{0000FF}` is possible))

For a fully developed example all the `\@namedef` would be in some
`sphinxcpp.sty` style file and either the document will do
`\usepackage{sphinxcpp}` or `sphinx.sty` will do
`\RequirePackage{sphinxcpp}`. The user could override this file via
`latex_additional_files`.

In absence of such file, the macro definitions will be lacking but this
will raise no error. For the example here (from last commit message) the
latex output contains

    \PYG{d+k+k-cpp}

and the `\PYG` macro nests "action of d(of k(of k-cpp...)))". Turns out
that Pygments already has some macro for "k", which uses bold face (as
can be seen in PDF output) and changes the text color, but as "k-cpp"
is deeper nested it is executed last and sets the good text color.

Thus it would perhaps be better if the latex writer produced something
like

    \PYG{cssd+cssk+cssk-cpp}

to avoid name clashes with Pygments pre-defined styles. Or here it is OK
that "k" is styled boldface by Pygments in LaTeX.

Important: class names must not contain the `+` character which is used
by Pygments LaTeX code as separator.

There are some issues in the Pygments code with the scope of some
changes; this is already corrected by Sphinx (refs: sphinx-doc#4250), but some
more scrutiny is perhaps needed because so far the Pygments code is
exercised only inside Verbatim environments where each code-line is a
scope in itself so border effects do not propagate, hence I have not
checked closely the Pygments LaTeX code, but perhaps sphinx-doc#4250 fixes already
all problems.
@jfbu
Copy link
Contributor

jfbu commented Dec 30, 2017

The commit message at jfbu@441c3de makes a mention of + character to be avoided but anyhow it is not legit (without escaping) in CSS class names or selectors, so this is (mainly) not an issue.

(I have again amended the commit message)

edit: the commit message at jfbu@cc632ba discussed some issue of usage of \sphinxcode LaTeX macro. I have pushed jfbu@88cf9be to address that.

jfbu added a commit to jfbu/sphinx that referenced this issue Dec 30, 2017
This goes for the quick solution, hence uses the pre-existing
architecture underlying the `\PYG` macro from the Pygments
highlighting code.

Each domain can add its own style to LaTeX by defining suitable macro
for supporting css styles.

For example, here we added

    \@namedef{PYG@tok@k-cpp}{%
         \def\PYG@tc##1{\textcolor[RGB]{0,0,"FF}{##1}}%
    }%

to sphinx.sty to emulate the CSS

    code.k-cpp {
        color: #0000FF;
    }

(previous commit uses `#0000AA` but I chose `#0000FF` for better seeing
things... also syntax `\textcolor[HTML]{0000FF}` is possible))

For a fully developed example all the `\@namedef` would be in some
`sphinxcpp.sty` style file and either the document will do
`\usepackage{sphinxcpp}` or `sphinx.sty` will do
`\RequirePackage{sphinxcpp}`. The user could override this file via
`latex_additional_files`.

In absence of such file, the macro definitions will be lacking but this
will raise no error. For the example here (from last commit message) the
latex output contains

    \PYG{d+k+k-cpp}{\sphinxcode{template}}

The `\PYG` macro first does preparation for the styling in the order "d"
then "k" then "k-cpp", the last one can override any preliminary config.

Turns out that Pygments Sphinx latex style has no pre-defined "d" style
but it has one for "k", which activates bold face and sets the text to a
certain color. As "k-cpp" comes last it will set the blue color.

It would perhaps be better if the latex writer produced something
like

    \PYG{cssd+cssk+cssk-cpp}

to avoid name clashes with Pygments pre-defined styles. Or here it is OK
that "k" is styled boldface by Pygments in LaTeX. (to be examined more)

- The `\sphinxcode` main role is to switch to monospace font
  (`\texttt`), and it does things to allow wrapping long inline literal
  and prevent some TeX ligatures and make quotes straight. Certainly
  this `\sphinxcode` should be removed here, else `\texttt` will always
  be there and win. We can't move `\sphinxcode` to outer level, because
  it makes the `-` active, which is for "k-cpp" class name..

- We could thus remove the `\sphinxcode` and do rather

      \@namedef{PYG@tok@d}{\def\PYG@ff##1{\texttt{##1}}}%

  Then it can be modified by other classes, as the "d" gets executed
  first. If the other classes to not modify the font family, the
  `\texttt` will have effect.

- class names must not contain the `+` character which is used by
  Pygments LaTeX code as separator. But as `+` must be escaped in CSS
  classes/selectors names, this is probably not much of a worry here.
@jakobandersen
Copy link
Contributor Author

@jfbu, thanks, looks quite nice. I will take a more thorough look when I find more time.
A few thoughts:

  • It sounds like Pygments already have all the functionality for format-agnostic style specification. I vote for directly using that as much as possible.
  • Though, we should be able to use the highlighting directly, without the lexer.
  • If we anyway use Pygments, then we can simply have a normal attribute on the Docutils node, instead of using the classes attribute.
  • We have a few more types than Pygments due to the desc_name and desc_addname types,
    probably we can add some default values when a traditional Pygments style is selected.
  • There should probably be a global style config variable, but the possibility to overwrite for specific domains.
  • Each of those config variables should work very similarly to the pygments_style variable.
  • This means that multiple Pygments styles needs to be included in the same document, and used in different contexts. This seems to be possible via the get_style_defs method (at least for HTML and Latex).
  • There should be builtin styles that emulate the current look.
  • I guess custom styles by users can be defined rather easily through extensions already?

jfbu added a commit to jfbu/sphinx that referenced this issue Jan 3, 2018
This goes for the quick solution, hence uses the pre-existing
architecture underlying the `\PYG` macro from the Pygments highlighting
code.

Each domain can add its own style to LaTeX by defining suitable macro
for supporting css styles. For example, here we added

    \@namedef{PYG@tok@k-cpp}{%
         \def\PYG@tc##1{\textcolor[HTML]{0000FF}{##1}}%
    }%

to sphinx.sty to emulate the CSS

    code.k-cpp {
        color: #0000FF;
    }

(previous commit uses `0000AA` for html output but I chose `0000FF` for
latex output in this proof of concept... )

For a fully developed example all the `\@namedef` would be in some
`sphinxcpp.sty` style file and either the document will do
`\usepackage{sphinxcpp}` or `sphinx.sty` will do
`\RequirePackage{sphinxcpp}`. The user could override this file via
`latex_additional_files`.

In absence of such file, the macro definitions will be lacking but this
will raise no error. For the example here (from last commit message) the
latex output contains

    \PYG{d+k+k-cpp}{\sphinxupquote{template}}

The `\PYG` macro first does preparation for the styling in the order "d"
then "k" then "k-cpp", the last one can override any preliminary config.

Turns out that Pygments Sphinx latex style has no pre-defined "d" style
but it has one for "k", which activates bold face and sets the text to a
certain color. As "k-cpp" comes last it will set the blue color.

It would perhaps be better if the latex writer produced something
like

    \PYG{cssd+cssk+cssk-cpp}

to avoid name clashes with Pygments pre-defined styles. Or here it is OK
that "k" is styled boldface by Pygments in LaTeX. (to be examined more)

- The merged master branch has refactored the `\sphinxcode` to only do
  `\texttt`. The fancy stuff about getting straight quotes was moved to
  `\sphinxupquote` and this is why `\sphinxupquote` macro appears in
  the LaTeX macros output from latex.visit_desc_element().

- There is no `\sphinxcode` in latex.visit_desc_element(), because its
  use was moved into the `\PYG@tok@d` styling macro.

      \@namedef{PYG@tok@d}{%
           \def\PYG@ff##1{\sphinxcode{##1}}% default \sphinxcode is \texttt
      }%

- Class names must not contain the `+` character which is used by
  Pygments LaTeX code as separator. But as `+` must be escaped in CSS
  classes/selectors names, this is probably not much of a worry here.
@jfbu
Copy link
Contributor

jfbu commented Jan 3, 2018

(briefly to say that for the LaTeX aspects, I have merged current master in my forked 4289_decl_styling branch because master now includes #4370 which was motivated by needs of this discussion)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:enhancement enhance or introduce a new feature
Projects
None yet
Development

No branches or pull requests

3 participants