Implement autocomplete on custom gradio apps

[oliveirabruno01]

Hi there! I am looking for a way to autocomplete on gradio Textbox inputs and this is the only implementation I know, which is a very good implementation. I need to implement a very similar autocomplete for a multimodal app I'm building, but I don't have any idea on where to start. I just fast read some files of this repo and it seems very intricated with the automatic1111's webui.

My idea is simple: I have a File input that accepts multiple images, and I want the autocomplete to suggest the loaded files' names, so the user don't need to write it manually, and is able to intercalate seamlessly any configuration of images+text.

Anyone has any idea on how to implement this on an arbitrary Textbox? Or at least where I should start?

Thanks in advance for any info. I will try to read through this codebase and if I make it I will return here to share the process, but am not confident.

The following is a screenshot of my app. For instance, the autocomplete should activate when typing '<img:' and show me all available images names.

[DominikDoom]

This is a pretty broad question, there are a lot of ways you can go about it.

First, I want to stress that my implementation here is in no way optimal. Many of the design decisions were made based on hard limitations posed by the webui, and many are severely outdated by now (e.g. that it had no API to use at the beginning). It's just too much work to rewrite it in a better way in my free time, lol.
Since it's your own app and you don't have to ensure compatibility with a third party, you have much more control over every step and won't need many of the tricks I had to resort to.

Here are some suggestions I can give in that regard:

---

Pure gradio implementations are possible. In fact, my very first version of autocomplete wasn't using any javascript and just populating an appropriately styled dataframe, which I abandoned because it required modifying the webui code.

Since your file list is already a gradio component, you could maybe even find a way to make them interactible directly and e.g. just inserting them into the text after you click on one (or have a button for each).
But if you want the typing suggestions, Gradio still has everything needed to react to text changes and display & fill a clickable completion list.

• Pros:
  • Can be pure python and directly work with gradio events
  • Makes it very easy to access local files or use a database connection if needed
• Cons:
  • Gradio doesn't allow that much freedom, so there would be some concessions to be made regarding interactivity and design
  • Keyboard navigation would be tricky

---

The second option, which is more similar to what I did with some of the workarounds removed, is a javascript frontend for displaying the suggestions and filling in the text if selected.
So anything visual in javascript and css, while the actual search for matching images happens on the server.
You can then create a REST API endpoint (e.g. using FastAPI, which Gradio already includes) and send a javascript fetch request there with what the user typed to get the current suggestions. Or, another option in your case would be to get all current images from the API (or even directly from the HTML DOM by parsing the file input) and filter that list in javascript based on the typed word.

For this, the main challenge will be to find the current "tag word" the user currently types. In your case this will be way easier than on my end since you have that fixed <img: syntax. That way you could just check for that with a simple regex. How I solved the issue of what changes belong together is roughly the following:
Textbox input event -> Get full text -> Split into words (e.g. by whitespace or commas) -> Calculate the difference between that array and the array from the last input event, preserving order and duplicates -> We now have the one most recently edited word in the text, which we can check for a required syntax (these lines in the code)
Then we only have to replace that "tagword" with the completed version.

You have many ways to make this way simpler for yourself, for example by only showing completion at the end of the line instead of everywhere in the text. Then you could assume that the most recent word is always the last word in the text.

You could also possibly use a finished javascript library for exactly this purpose in which all the work is done already (haven't looked around much here, but there should be some). The reason I didn't do that here is since I a) didn't want to introduce a big dependency at the time and b) required some custom solutions anyway for the webui. But again, since you have much more control, you can make your app work with one of these libraries instead of the other way around.

One catch either way is that you have to notify gradio of the client-side change so the text gets properly updated internally. From the webui:

// Simulate an `input` DOM event for Gradio Textbox component. Needed after you edit its contents in javascript, otherwise your edits
// will only visible on web page and not sent to python.
function updateInput(target) {
    let e = new Event("input", {bubbles: true});
    Object.defineProperty(e, "target", {value: target});
    target.dispatchEvent(e);
}

• Pros:
  • Very flexible, you pretty much only need gradio elements for the initial text
  • All calculations and file accesses can still happen in Python, since we communicate purely over API
  • Possibly not even an API needed since the file list already exists in the DOM
  • Depending on what you prefer and how you set up the API, text parsing and result filtering can happen in both python and javascript
• Cons:
  • Much harder to actually do yourself (but possibly already a usable library)

[DominikDoom]

Also, here is a much simpler autocomplete script: https://github.com/pythongosssss/ComfyUI-Custom-Scripts/blob/main/web/js/common/autocomplete.js
It's for ComfyUI, which doesn't use gradio, but the principle stays the same. It also has less features than mine, but should be more than enough for your case since you only need one type of completion. Potentially, you could even use the TextAreaAutoComplete class from that file directly with some adjustments. It's MIT license too, so you can freely use it.

Edit: didn't even see it at first, but that is based on https://github.com/yuku/textcomplete, so maybe that is a potential library to have a look at.

[oliveirabruno01]

Thanks for taking the time and write this comprehensive response. I managed to do the second option after reading some of the early commits of this extension and a1111 web-ui.

To load the javascript file it overwrites gradio.routes.templates.TemplateResponse as in:

if __name__ == "__main__":
    with open("script.js", "r", encoding="utf8") as jsfile:
        javascript = jsfile.read()


    def template_response(*args, **kwargs):
        res = gradio_routes_templates_response(*args, **kwargs)
        res.body = res.body.replace(b'</head>', f'<script>{javascript}</script></head>'.encode("utf8"))
        res.init_headers()
        return res


    gradio_routes_templates_response = gradio.routes.templates.TemplateResponse
    gradio.routes.templates.TemplateResponse = template_response

    demo.launch()

Where script.js contains parts of a111 code responsible for getting the gradio app element and update cycle, and the code responsible for adding the autocomplete popup and input.

To get the names of the files first I add them to a hidden gradio Textbox input on File input's update event, and on the javascipt side, we get the value of the hidden textbox by the elem-id.

Thanks for the help! For reference the script.js and complete implementation can be found here

edit: typos