Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add space glyph to Symbols fonts #297

Closed
wants to merge 1 commit into from
Closed

Conversation

Mange
Copy link

@Mange Mange commented Oct 2, 2018

Description

Added the space glyph from DejaVu Sans Mono Regular into both Nerd
Font Complete symbol fonts and saved them as TTF.

I used fontforge and generated the font as a TrueType with default options.

This could, in theory, fix #275.

Requirements / Checklist

  • Read the Contributing Guidelines
  • Read or at least glanced at the FAQ
  • Read or at least glanced at the Wiki
  • Scripts execute without error (if necessary):
    • If any of the scripts were modified they have been tested and execute without error, e.g.:
      • ./font-patcher Inconsolata.otf --fontawesome --octicons --pomicons
      • ./gotta-patch-em-all-font-patcher\!.sh Hermit
  • Extended the README and documentation if necessary, e.g. You added a new font please update the table

What does this Pull Request (PR) do?

It adds the space character to the symbols fonts files.

How should this be manually tested?

On a Linux machine, check this output with the normal symbol fonts installed (and a fontconfig that automatically falls back to the symbols):

pango-view -t "Playing some  for you right now"

Screenshot showing rendering problem with the space after the symbol character

Then install this new version of the font, reset the cache and try again:

cp src/glyphs/Symbols-2048-em\ Nerd\ Font\ Complete.ttf ~/.local/share/fonts
fc-cache -fv
pango-view -t "Playing some  for you right now"

Screenshot showing that the rendering problem has now disappeared

Any background context you can provide?

Pango is one of the most common font rendering libraries on Linux. It's used by GTK/GNOME and a lot of standalone apps, like Rofi, Polybar, and a lot of terminals.

Pango looks for fallback fonts when the current font doesn't contain a given glyph, but it won't check spaces or any other very common "invisible" characters with the assumption that all fonts should have spaces in them.
This makes it so that after Pango switches to Nerd Fonts for a glyph, it won't switch back until it gets a non-space character after it, making it impossible to have spaces after a symbol.
I usually try to work around it by wrapping symbols in parenthesis, or adding a colon after them, but in some apps this isn't really possible to do anything about.

Issues to look at:

Would fix icon rendering in a lot of programs, including:

  • Kitty terminal
  • Polybar
  • Rofi
  • Almost all GNOME apps
  • Some Ubuntu-specific apps

What are the relevant tickets (if any)?

Screenshots (if appropriate or helpful)

See "Any background" above for some examples.

Caveats / problems

  1. I assume that it was fine to copy the space character from DejaVu Sans Mono Regular. (From a licensing perspective)
  2. Running "gotta patch 'em all" for the Hermit font does not lead to any diff in the repo; I assume because the space character is not included in any of the character ranges.
  3. I exported the TTF without knowing what I was doing. I picked "TrueType (TTF)" and left everything at their defaults in fontforge.
  4. I searched and searched to see if there was any script that generated the TTF files, but could not find any.
  5. I tried changing the .sfd files too, but they got no meaningful changes ("last modified" and "window size" information only).
  6. Trying to diff the TTF files is not easy, but it looks like some metadata got lost.
› diff -y <(git show master:./Symbols-1000-em\ Nerd\ Font\ Complete.ttf | xxd) <(xxd Symbols-1000-em\ Nerd\ Font\ Complete.ttf) | tail -n 40
000c0260: 0001 0e79 0001 0004 0000 0002 0000 0001  ...y...... | 000c0260: 7405 7675 656a 7307 7861 6d61 7269 6e0f  t.vuejs.xa
000c0270: 0000 0001 0000 0000 0000 0001 0000 0000  .......... | 000c0270: 7861 6d61 7269 6e2d 6f75 746c 696e 6516  xamarin-ou
000c0280: d5a4 2708 0000 0000 d463 d07e 0000 0000  ..'......c | 000c0280: 796f 7574 7562 652d 6372 6561 746f 722d  youtube-cr
000c0290: d6df 4d79 0001 0000 0000 0002 6663 6d74  ..My...... | 000c0290: 7374 7564 696f 0e79 6f75 7475 6265 2d67  studio.you
000c02a0: 0000 0018 666c 6f67 0000 0130 0001 010f  ....flog.. | 000c02a0: 616d 696e 6700 0000 0000 0001 ffff 0002  aming.....
000c02b0: 5061 7463 6865 6420 7769 7468 2027 4e65  Patched wi | 000c02b0: 0001 0000 000c 0000 0016 001e 0002 0001  ..........
000c02c0: 7264 2046 6f6e 7473 2050 6174 6368 6572  rd Fonts P | 000c02c0: 0001 0e7a 0001 0004 0000 0002 0000 0001  ...z......
000c02d0: 2720 2868 7474 7073 3a2f 2f67 6974 6875  ' (https:/ | 000c02d0: 0000 0001 0000 0000 0000 0001 0000 0000  ..........
000c02e0: 622e 636f 6d2f 7279 616e 6f61 7369 732f  b.com/ryan | 000c02e0: d76f f76b 0000 0000 d463 d07e 0000 0000  .o.k.....c
000c02f0: 6e65 7264 2d66 6f6e 7473 290a 0a2a 2057  nerd-fonts | 000c02f0: d7d9 74de                                ..t.
000c0300: 6562 7369 7465 3a20 6874 7470 733a 2f2f  ebsite: ht <
000c0310: 7777 772e 6e65 7264 666f 6e74 732e 636f  www.nerdfo <
000c0320: 6d0a 2a20 5665 7273 696f 6e3a 2032 2e30  m.* Versio <
000c0330: 2e30 0a2a 2044 6576 656c 6f70 6d65 6e74  .0.* Devel <
000c0340: 2057 6562 7369 7465 3a20 6874 7470 733a   Website:  <
000c0350: 2f2f 6769 7468 7562 2e63 6f6d 2f72 7961  //github.c <
000c0360: 6e6f 6173 6973 2f6e 6572 642d 666f 6e74  noasis/ner <
000c0370: 730a 2a20 4368 616e 6765 6c6f 673a 2068  s.* Change <
000c0380: 7474 7073 3a2f 2f67 6974 6875 622e 636f  ttps://git <
000c0390: 6d2f 7279 616e 6f61 7369 732f 6e65 7264  m/ryanoasi <
000c03a0: 2d66 6f6e 7473 2f62 6c6f 622f 6d61 7374  -fonts/blo <
000c03b0: 6572 2f63 6861 6e67 656c 6f67 2e6d 6400  er/changel <
000c03c0: 0000 0000 0001 010f 5061 7463 6865 6420  ........Pa <
000c03d0: 7769 7468 2027 4e65 7264 2046 6f6e 7473  with 'Nerd <
000c03e0: 2050 6174 6368 6572 2720 2868 7474 7073   Patcher'  <
000c03f0: 3a2f 2f67 6974 6875 622e 636f 6d2f 7279  ://github. <
000c0400: 616e 6f61 7369 732f 6e65 7264 2d66 6f6e  anoasis/ne <
000c0410: 7473 290a 0a2a 2057 6562 7369 7465 3a20  ts)..* Web <
000c0420: 6874 7470 733a 2f2f 7777 772e 6e65 7264  https://ww <
000c0430: 666f 6e74 732e 636f 6d0a 2a20 5665 7273  fonts.com. <
000c0440: 696f 6e3a 2032 2e30 2e30 0a2a 2044 6576  ion: 2.0.0 <
000c0450: 656c 6f70 6d65 6e74 2057 6562 7369 7465  elopment W <
000c0460: 3a20 6874 7470 733a 2f2f 6769 7468 7562  : https:// <
000c0470: 2e63 6f6d 2f72 7961 6e6f 6173 6973 2f6e  .com/ryano <
000c0480: 6572 642d 666f 6e74 730a 2a20 4368 616e  erd-fonts. <
000c0490: 6765 6c6f 673a 2068 7474 7073 3a2f 2f67  gelog: htt <
000c04a0: 6974 6875 622e 636f 6d2f 7279 616e 6f61  ithub.com/ <
000c04b0: 7369 732f 6e65 7264 2d66 6f6e 7473 2f62  sis/nerd-f <
000c04c0: 6c6f 622f 6d61 7374 6572 2f63 6861 6e67  lob/master <
000c04d0: 656c 6f67 2e6d 6400 0000 0000            elog.md... <

I think you might want to clean this up somehow; maybe repeat the process yourself..? Alternatively, if I could get some instructions on how to properly just add the glyph to the TTF file without any additional changes I could repeat that process and replace the commits in this PR.

@ryanoasis
Copy link
Owner

Thanks @Mange . It'll take me a bit to get to this but I'll review it

@Mange
Copy link
Author

Mange commented Nov 14, 2018

Trust me, I'm not trying to stress you right now, but I just wanted to do a friendly bump in case you forgot about this. (I know how it is sometimes)

If you get this notification and feel "Ugh, I really don't have time" you don't have to comment here and I'm sorry. I'm just sending you a notification in case you forgot. If not, feel free to archive the email and not click/tap on anything.

@ryanoasis
Copy link
Owner

No worries @Mange. I want to get things moving again. I should have some more time now but still won't be as much as before but at the very least I want to make sure PRs get in

@Mange
Copy link
Author

Mange commented Jan 18, 2019

Friendly bump. :)

Let me know if there's anything I can help you with to give you more time/motivation (bug triaging, for example).

@ryanoasis
Copy link
Owner

Thanks @Mange will be looking at it soon!

@ryanoasis
Copy link
Owner

@Mange

I think you might want to clean this up somehow; maybe repeat the process yourself..? Alternatively, if I could get some instructions on how to properly just add the glyph to the TTF file without any additional changes I could repeat that process and replace the commits in this PR.

Yeah you are right there are a bunch of differences besides just the added space glyph. Not sure what happened.

You are correct there isn't any script currently that generates those symbol only ttf files

@ryanoasis
Copy link
Owner

Actually I did document the symbol font generation process, it is here: https://github.com/ryanoasis/nerd-fonts/wiki/Release-Workflow#ad-hoc-updating-of-the-glyph-only-fonts

Probably not obvious..

@ryanoasis
Copy link
Owner

Something is up with those symbol ttf fonts (will look into it later)

However I think actually what you want to do might be as simple as adding 0x0020 to the list of characters that get copied/patched into all the fonts and it would only do so if that font didn't have a space. See:

{ 'Enabled': True, 'Name': "Seti-UI + Custom", 'Filename': "original-source.otf", 'Exact': False,'SymStart': 0xE4FA, 'SymEnd': 0xE52E, 'SrcStart': 0xE5FA,'SrcEnd': 0xE62E,'ScaleGlyph': None, 'Attributes': SYM_ATTR_DEFAULT },

@Mange
Copy link
Author

Mange commented Feb 20, 2019

I'll take a look. Thank you!

Having this space available will help with rendering on Linux and allows
renderers to render space symbols without switching fonts when icons
have spaces between them.
@Mange
Copy link
Author

Mange commented Mar 4, 2019

I tried that way, and it seems to work better with the diff. Thank you for pointing me towards the correct places to look in.

However, my test case does not work anymore with this version. Maybe I did the config wrong. Would you mind taking a look at the diff?

If I look at the generated font, it looks like the space character has become part of the font.

# Package-installed font
› fc-query --format=%{charset} /usr/share/fonts/TTF/Nerd\ Font\ Complete.ttf
23fb-23fe 2665 26a1 2b58 e000-e00a e0a0-e0a3 e0b0-e0c8 e0ca e0cc-e0d2 e0d4 e200-e2a9 e300-e3e3 e5fa-e62e e700-e7c5 f000-f00e f010-f01e f021-f03e f040-f04e f050-f05e f060-f06e f070-f07e f080-f08e f090-f09e f0a0-f0ae f0b0-f0b2 f0c0-f0ce f0d0-f0de f0e0-f0ee f0f0-f0fe f100-f10e f110-f11e f120-f12e f130-f13e f140-f14e f150-f15e f160-f16e f170-f17e f180-f18e f190-f19e f1a0-f1ae f1b0-f1be f1c0-f1ce f1d0-f1de f1e0-f1ee f1f0-f1fe f200-f20e f210-f21e f221-f23e f240-f24e f250-f25e f260-f26e f270-f27e f280-f28e f290-f29e f2a0-f2ae f2b0-f2be f2c0-f2ce f2d0-f2de f2e0 f300-f31c f400-f4a9 f500-fd46

# Newly generated font
› fc-query --format=%{charset} Symbols-1000-em\ Nerd\ Font\ Complete.ttf     
20 23fb-23fe 2665 26a1 2b58 e000-e00a e0a0-e0a3 e0b0-e0c8 e0ca e0cc-e0d2 e0d4 e200-e2a9 e300-e3e3 e5fa-e62e e700-e7c5 f000-f00e f010-f01e f021-f03e f040-f04e f050-f05e f060-f06e f070-f07e f080-f08e f090-f09e f0a0-f0ae f0b0-f0b2 f0c0-f0ce f0d0-f0de f0e0-f0ee f0f0-f0fe f100-f10e f110-f11e f120-f12e f130-f13e f140-f14e f150-f15e f160-f16e f170-f17e f180-f18e f190-f19e f1a0-f1ae f1b0-f1be f1c0-f1ce f1d0-f1de f1e0-f1ee f1f0-f1fe f200-f20e f210-f21e f221-f23e f240-f24e f250-f25e f260-f26e f270-f27e f280-f28e f290-f29e f2a0-f2ae f2b0-f2be f2c0-f2ce f2d0-f2de f2e0 f300-f31c f400-f4a9 f500-fd46

Note how 20 appeared.

I suspect that it's my reproduction of the demo that isn't working, but I just wanted to check with you. I'll update if I manage to see a pango view rendered correctly.

@Mange
Copy link
Author

Mange commented Mar 4, 2019

I got it to work. The problem was really embarrassing; I had no .local/share/fonts directory, so I just copied the TTF to a file named fonts. No wonder it wasn't found.

However, the result doesn't look very good. It's almost like the space has zero width, or the symbol taking up the space. The symbol is also aligned in a weird way.

New result (Ignore the blue background, that's unrelated)

screenshot from 2019-03-04 23-03-52

Previous result:

I'll try to patch the space from some other source font until it looks good.

@Mange
Copy link
Author

Mange commented Mar 5, 2019

I've tried it with all the included fonts and also with a copy of DejaVu Sans Mono. I figure that the font-patcher rescales the space to take up no space.

I tried defining a ScaleGlyph that tries to set the same scale on it, but I always get a DivisionByZero.

SPACE_SCALE_LIST  = { 'ScaleGlyph': 0x0020, 'GlyphsToScale': [ 0x0020 ] }

I've also tried to hack in that font_scale should be 1 for spaces, but that doesn't fix the exception.

It's almost like if the space symbol in all the fonts have 0 width or something.

@sdushantha
Copy link

@ryanoasis just thought of pinging :)

It would be very nice if this PR could be merged.

@ghost
Copy link

ghost commented Jun 15, 2019

I'm also experiencing this issue in kitty. I tried to work around this by setting symbol_map U+0020 Fira Code in the kitty config which utilizes kitty's ability to use custom fonts for specific codepoints (I also did symbol_map U+E000-U+E00D,U+E0A0-U+E0A2,etc... Symbols Nerd Font instead of using a patched font). Even after doing this, it continues to display the boxes next to icons.

@Mange

Pango looks for fallback fonts when the current font doesn't contain a given glyph, but it won't check spaces or any other very common "invisible" characters with the assumption that all fonts should have spaces in them.
This makes it so that after Pango switches to Nerd Fonts for a glyph, it won't switch back until it gets a non-space character after it, making it impossible to have spaces after a symbol.

Is this accurate for kitty's implementation of Pango as well? I would assume that kitty tries to render the icon with Symbols Nerd Font and then tries to render the space with Fira Code, especially because I mapped U+0020 to it, or does Pango's method of rendering spaces take precedence over symbol_map?

@ghost
Copy link

ghost commented Jun 15, 2019

Here's my attempt at putting the Fira Code U+0020 in Symbols-2048-em Nerd Font Complete.ttf. While it fixed the issue with the rectangles (and has not appeared to cause any other issues), the file is substantially different from the original: 866K vs 827K

NerdFontsSymbols-Fixed.zip

@Mange
Copy link
Author

Mange commented Jun 15, 2019

Pango looks for fallback fonts when the current font doesn't contain a given glyph, but it won't check spaces or any other very common "invisible" characters with the assumption that all fonts should have spaces in them.
This makes it so that after Pango switches to Nerd Fonts for a glyph, it won't switch back until it gets a non-space character after it, making it impossible to have spaces after a symbol.

Is this accurate for kitty's implementation of Pango as well? I would assume that kitty tries to render the icon with Symbols Nerd Font and then tries to render the space with Fira Code, especially because I mapped U+0020 to it, or does Pango's method of rendering spaces take precedence over symbol_map?

I have not read kitty's source code enough to comment on that. I just know that pango itself has this bug and has had it for years.

If kitty uses pango, it's likely the cause for the problem in that app as well. If kitty doesn't use pango, it could still be the same bug but inside some other software project/package.

@ryanoasis
Copy link
Owner

Just to get myself caught up I think the original change was to the actual source file of 'original-source.otf' but now we are just replacing the space character from the 'original-source.otf' ?

@Mange
Copy link
Author

Mange commented Jul 2, 2019

I started out by copying the glyph from a common font into the symbols font, which worked well but removed metadata from the symbols font file.

Now I've tried to get it to copy it straight in the patcher, but that didn't work no matter which way I tried to get it to work.

I'm not a pythonista, and I'm really not familiar with this codebase either so I've mostly tried to cargo cult a solution in here.

The end-goal is to just have a completely blank glyph in the 0x20 position inside the symbols font, without making the font file weird.

@ryanoasis
Copy link
Owner

Thanks @Mange . So I am a little confused, it looks like this change of copying 0x20 you have to font-patcher should fix the issue?

If so I think we are good to merge...

@Mange
Copy link
Author

Mange commented Jul 27, 2019

But it didn't work anymore with the change. Space characters still shows up badly.

However, the result doesn't look very good. It's almost like the space has zero width, or the symbol taking up the space. The symbol is also aligned in a weird way.

New result (Ignore the blue background, that's unrelated)

screenshot from 2019-03-04 23-03-52

Previous result:

@ryanoasis
Copy link
Owner

Thanks for the update. I'll try to look into this...

@Mange
Copy link
Author

Mange commented Sep 27, 2019

Pango 1.44 does not have this issue anymore.
https://blogs.gnome.org/mclasen/2019/07/27/more-text-rendering-updates/

pango-view -t "Playing some  for you right now"

Screenshot from 2019-09-27 08-38-43

A lot of Linux distros will be slow to upgrade since 1.44 had some issues with other programs because they changed font rendering quite a lot, but it will slowly become deployed everywhere over time.

I think we can close this PR, unless we want to fix this for LTS releases or other distros that could take years to upgrade.

@zeorin
Copy link
Contributor

zeorin commented Sep 27, 2019

Please don't close.

@ryanoasis
Copy link
Owner

Thanks for the update @Mange


Please don't close.

@zeorin Can you elaborate?

It would be nice to have this glyph for those cases where it isn't provided perhaps, but other than that I don't know how many people are running into this use case.

@ryanoasis
Copy link
Owner

House keeping: Closing this PR, if anyone has strong objections please comment and we can discuss re-opening. Thanks all

@ryanoasis ryanoasis closed this Jan 19, 2020
Finii added a commit that referenced this pull request Sep 26, 2022
[why]
It could be beneficial to have at least the 'blank' glyph defined, next
to all the special glyphs (symbols).

This is needed by some (old) font rendering engines (Pango, IE11).

Fixes: #297

Signed-off-by: Fini Jastrow <ulf.fini.jastrow@desy.de>
@Finii Finii mentioned this pull request Sep 26, 2022
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Include space in source / icons-only font
5 participants