Character bounding boxes and negative offsetx

[indigoviolet]

What did you do?

I'm trying to get character level bounding boxes for some text I'm generating, following #3921, and @nulano 's comment, this stackoverflow answer etc. Also tagging #4724 in the hope that this issue will be addressed by it.

def test_bbox(font, txt):
    img = Image.new(mode='RGB', size=(300,100), color='white')
    draw = ImageDraw.Draw(img)
    xref, yref = 50, 5
    draw.text((xref, yref), txt, font=font, fill='black')
    for i, c in enumerate(txt):
        wd0, ht0 = font.getsize(txt[:i+1])
        (wd, ht), (offsetx, offsety) = font.font.getsize(txt[:i+1])
        cmask_wd, cmask_ht = font.getmask(c).size
        char_wd, char_ht = font.getsize(c)
        
        x2, y2 = wd0, char_ht
        x1, y1 = x2 - cmask_wd, y2 - cmask_ht
        draw.rectangle((x1 + xref, y1 + yref, x2 + xref, y2 + yref), fill=None, outline='red', width=1)
        draw.rectangle((x1 + offsetx + xref, y1 + yref, x2 + offsetx + xref, y2 + yref), fill=None, outline='green', width=1)
    return img
italic_font = ImageFont.truetype('fonts/google/fonts-master/ofl/baskervville/Baskervville-Italic.ttf', 60, layout_engine=ImageFont.LAYOUT_BASIC) 

# show_grid just uses matplotlib ax.imshow to display the image
show_grid(test_bbox(italic_font, 'fa'))
show_grid(test_bbox(italic_font, 'af'))

What did you expect to happen?

I expected the red bounding box to encompass the a in the first image (fa).

What actually happened?

1. It was shifted to the right.

2. Also, while flailing around, I realized that the green bounding box, which is created by an additional offsetx shift, does appear to encompass the a. I don't understand why this would be, but it might just be coincidence.

3. Neither of these completely bound the a on the top edge. Why would that be?

What are your OS, Python and Pillow versions?

• OS: Ubuntu 18.04
• Python: 3.8.1
• Pillow: 7.1.2

[nulano]

Ah, I think you've found an issue I missed when writing some of my previous comments. It seems that the C code is adding (actually subtracting) the xoffset twice! Once at the start of the loop, and once at the end.

This means that getsize is actually correct for getting the text width, it is the private C function that is wrong here! (Not so for height, getsize returns the bottom coordinate while the C function returns the bounding-box height.)

I'll have a look at what effects this finding has for #4724 regarding getsize (which rewrites the C function) some other time.

---

The reason you need to add the x-offset a second time is that ImageDraw.Draw.text uses this offset to shift the text origin (i.e. slanted text such as your f can extend to the left of the passed xy parameter). The box for f is still wrong due to the issue above, using char_wd instead of cmask_wd should fix that.

You can access the offset with font.getoffset without using the private API of font.font (which could change without notice in future versions). Instead of cmask_ht you can use ht0 - offsety.

Why is the top edge slightly off? I would guess this #1668.

Here is the code I have come up with that should work for you: (click to expand)

def test_bbox(font, txt):
    img = Image.new('RGB', (300, 100), 'white')
    draw = ImageDraw.Draw(img)
    xref, yref = 50, 5
    draw.line((0, yref, 300, yref), 'gray')
    draw.line((xref, 0, xref, 100), 'gray')
    draw.text((xref, yref), txt, 'black', font)
    for i, c in enumerate(txt):
        text_wd, text_ht = font.getsize(txt[:i+1])
        text_ox, text_oy = font.getoffset(txt[:i+1])
        text_ht -= text_oy  # text_ht is unused

        text_x1, text_y1 = xref + text_ox, yref

        wd, ht = font.getsize(c)
        ox, oy = font.getoffset(c)
        ht -= oy

        x1, y1 = text_x1 + text_wd - wd, text_y1 + oy
        x2, y2 = text_x1 + text_wd, text_y1 + oy + ht

        draw.rectangle((x1, y1, x2, y2), outline='red')
    return img


italic_font = ImageFont.truetype('Baskervville-Italic.ttf', 60, layout_engine=ImageFont.LAYOUT_BASIC)

test_bbox(italic_font, 'fa').show()
test_bbox(italic_font, 'af').show()

Perhaps this (or a variation) should be added to the docs? It seems character bounding boxes are a common question.

---

In #4724 I proposed new functions to make this much simpler to understand. An (untested) example for how this should work using the proposed functions getbbox and getlength:

(click to expand)

def test_bbox(font, txt):
    img = Image.new(mode='RGB', size=(300,100), color='white')
    draw = ImageDraw.Draw(img)
    xref, yref = 50, 5
    draw.text((xref, yref), txt, font=font, fill='black')

    x0, y0 = xref, yref
    for i, c in enumerate(txt):
        x1, y1, x2, y2 = font.getbbox(c)
        draw.rectangle((x0 + x1, y0 + y1, x0 + x2, y0 + y2), fill=None, outline='green', width=1)
        x0 += font.getlength(c)

    return img

italic_font = ImageFont.truetype('fonts/google/fonts-master/ofl/baskervville/Baskervville-Italic.ttf', 60, layout_engine=ImageFont.LAYOUT_BASIC) 

# show_grid just uses matplotlib ax.imshow to display the image
show_grid(test_bbox(italic_font, 'fa'))
show_grid(test_bbox(italic_font, 'af'))

[indigoviolet]

Thank you for the detailed answer, @nulano. Your code does indeed solve my problem. I look forward to having the improved APIs from #4724.

[gofr]

Regarding @indigoviolet's point 3:

I expected the red bounding box to encompass the a in the first image (fa).
(...)
3. Neither of these completely bound the a on the top edge. Why would that be?

I don't think @nulano's explanation is correct:

Why is the top edge slightly off? I would guess this #1668.

#1668 actually makes the bottom and right edges of the red bounding boxes to be off by one. To work around that, this line in @nulano's first example code:

x2, y2 = text_x1 + text_wd, text_y1 + oy + ht

Would have to be changed to this:

x2, y2 = text_x1 + text_wd - 1, text_y1 + oy + ht - 1

That of course does not address why the top edge is off. This seems to have something to do with the subpixel font rendering and rounding. If I change the text from af to ab, the a shifts down about half a pixel while the red bounding box remains the same. I guess that shift is a separate issue entirely? Taking the above - 1 into account that then means the bottom of the a is slightly outside the bounding box, not the top.

[nulano]

I did not originally notice that the a actually extends outside the bbox, as IIRC I tested this locally with a different font. I have tested this again just now, and I now believe the top-edge issue is a duplicate of #4569 (except in this case the clipping is invisible due to the extra margin above the text), and is in fact also fixed by #4910.

From master:

From #4910:

I was only noting that due to #1668, it is expected that the top edge will be slightly obscured by the bounding box, while the bottom edge won't. I was hinting at the opposite idea of decrementing x1, y1 by 1, instead of decrementing x2, y2. Which method is better will obviously depend on what is the use case for this, so I left that as an exercise for the reader :).

[nulano]

Pillow 8.0.0 has now been released supporting the new API:

def test_bbox(font, txt):
    img = Image.new(mode='RGB', size=(300,100), color='white')
    draw = ImageDraw.Draw(img)
    xref, yref = 50, 5
    draw.text((xref, yref), txt, font=font, fill='black')
    x0, y0 = xref, yref
    for i, c in enumerate(txt):
        draw.rectangle(draw.textbbox((x0, y0), c, font=font), fill=None, outline='green', width=1)
        x0 += draw.textlength(c, font=font)
    return img

italic_font = ImageFont.truetype('Baskervville-Italic.ttf', 60, layout_engine=ImageFont.LAYOUT_BASIC) 

test_bbox(italic_font, 'fa').show()
test_bbox(italic_font, 'af').show()

This example does not adjust for kerning. Kerning currently has no effect in LAYOUT_BASIC mode due to a bug that may be resolved in the future. See the docs to see how to adjust for kerning.

[radarhere]

Thank you for the detailed answer, @nulano. Your code does indeed solve my problem. I look forward to having the improved APIs from #4724.

I did not originally notice that the a actually extends outside the bbox, as IIRC I tested this locally with a different font. I have tested this again just now, and I now believe the top-edge issue is a duplicate of #4569 (except in this case the clipping is invisible due to the extra margin above the text), and is in fact also fixed by #4910.

It sounds like this issue is resolved then?

[nulano]

For anyone looking at this in the future, I have adapted my comment above using the new functions into a Stack Overflow answer here: https://stackoverflow.com/a/70636273/1648883 This version also has a guide on how to adjust for kerning.