Get the ElementText rectangle or bounding box when the layout calculation is complete

[Shen9ke]

Hi,
I am trying to develop a rich text editor with RmlUI, but the existing WidgetTextInput element cannot edit and selection elements with child nodes, but only plain text, so I want to try to implement Selection and Range methods based on browser principles.
In order to achieve cross-text node and element node selection, I had to recalculate the text node layout to get the line feed information, which was too difficult. It would be nice to have an API in place to retrieve the calculated TextLayout (which contains the Rectangle List information) at rendering time, so as to avoid compute the text layout repeatedly.
Or have any better suggestions for implementing the Selection and Range methods?

[mikke89]

Hey there!

It's an interesting challenge. I guess we at least could start with exposing the lines of text elements, from those you should be able to deduce the area of each line at least. This would be considered fairly low-level, I don't want to guarantee API stability here, because we want room to refactor. We could consider adding a more stable API to get the surrounding box of each line, or something like that.

Note that, we don't really store the position of each letter, which might be what you're after. You might want to take a look at how the WidgetTextInput does text selection, and generalize some of that so that it works for ElementText. The main challenge though is that for normal text layout, the layout engine does all the layouting, and that has a fairly complex state, considering all of the CSS properties and its layout intricacies, as I'm sure you can imagine. And we don't really want to add more state for text nodes that don't need selection. I would be a lot more concerned over adding more state, than the extra computation of recalculating some lines, if it comes to that.

You might also want to consider implementing a rich text editor using a custom element instead. That should give you a lot more control over everything, and probably wouldn't need any changes to the library. You lose the ability to take advantage of all the CSS layout features the library supports, if you need those. I guess it depends a bit on how much you want to dive into the library to add the necessary changes, and discuss the design around that in detail. Versus just get something up and running without too much effort and iterate on that, more independently of the library.

[Shen9ke]

Yeah, lines might work better, but as you pointed out, I noticed it is a private variable in the ElementText class. I don't want to use that through a awkward way. I prefer to rely on the standard RmlUi library and make no changes.
I noticed that the WidgetTextInput had used the FormatText method to handle the computation of lines and generate selection geometry. What confused me was why we were not directly using the layout that is finished by the layout engine. For the RmlUi layout stream, does it mean we need to layout text elements twice? Correct me if I'm wrong.
So it's going to be very contradictory. On one hand, I really agree with your view that it's bad to have extra computation for recalculating and we don't really want to change RmlUi itself, but on the other hand, we don't have a ready-made API to achieve this.

Custom elements are a good idea, and I will consider using them in the future to implement some extended functions for the rich text editor, such as mathematical formulas. However, I don't want to use custom elements here. As you mentioned, custom elements seem to lose the ability to leverage all the CSS layout features supported by the library. Moreover, the workload doesn't seem to be reduced. We prefer to utilize RmlUi's original features and methods to implement basic functionalities as much as possible.

Actually, my ideal solution would be to execute ElementUtilities::GetBoundingBox(rect, textElement_node, BoxArea::Auto) and obtain the rectangular information of textElement_node, so that I can use textElement_node->GenerateLine(...) to calculate again line breaks by limiting the boundaries of the text field. Unfortunately, it seems that GetBoundingBox can only retrieve the BoundingBox of textElement_node's parent, which is not what I want. This is because the parent might have many non-text children, which can affect the calculation of line break layouts. As a result, I can't get the desired outcome.

Therefore, in summary, I still need to recalculate the virtual layout of the text myself to obtain the relevant information. What I can do is to consider and handle as many scenarios as possible to ensure a normal text layout.

[mikke89]

Have you done some research on how rich editors are implemented on the web? I did consider making something like contenteditable at some point. But after some research, I see they are notoriously bad and hard to work with. My impression is that text editors these days only listen to input events from the browser, but otherwise do most of the layouting themselves.

This is why I mentioned custom elements previously, it might not seem like it at first glance, but I'm pretty sure that would be an easier approach. At the very least, you should get ready to get your hands real greasy and throw some wrenches into those cogwheels. It would be nice to have a general selection API, and rich text editing as you outline. So I don't want to discourage you to go down this road, I just want you to be ready for this trip.

There is also the EditContext API being developed for the web. I think this might be worth looking into. It could also nicely tie in to (or generalize) the input method editor interface being developed in #541.

For the RmlUi layout stream, does it mean we need to layout text elements twice? Correct me if I'm wrong.

No, the layout engine does not touch the inner formatting of the text inputs at all. The text input elements handle all the layouting themselves. However, they do share some code, e.g. through the ElementText::GenerateLine that you referred to.

Actually, my ideal solution would be to execute ElementUtilities::GetBoundingBox(rect, textElement_node, BoxArea::Auto) and obtain the rectangular information of textElement_node, so that I can use textElement_node->GenerateLine(...) to calculate again line breaks by limiting the boundaries of the text field.

The issue with this approach is that there is a whole lot more going on from the layout engine while it's making successive calls into this function. To match the layout exactly, you would have to reproduce all of that logic that sits on top of the call to GenerateLine, effectively recreating the full layout engine! Not something you want to do of course. For this approach to work, you have to make a lot of assumptions about the surrounding layout. And even then it will need painstaking iterative refinement to try to prune out formatting mismatches in various situations.

I think a more realistic approach is to have the layout engine output the information you need somehow. For example, store all the bounding boxes of all the letters in each text node. That should at least help you in making a selection (API). That opens up a lot of new questions of where to go next, and I certainly don't have all the answers.