ArtInsight is an iOS app to support parents who are blind or have low-vision (BLV) to engage with their sighted children’s artwork. The app includes an accessible camera view. When a photo of artwork is taken using the app, a description of the artwork is generated using GPT. Following findings from formative research, the app includes a human-in-the-loop system to support children and parents editing AI descriptions as they feel appropriate. Descriptions will be saved in the app for future reference.
Cite the ArtInsight IUI 2025 paper:
Arnavi Chheda-Kothary, Ritesh Kanchi, Chris Sanders, Kevin Xiao, Aditya Sengupta, Melanie Kneitmix, Jacob O. Wobbrock, and Jon E. Froehlich. 2025. ArtInsight: Enabling AI-Powered Artwork Engagement for Mixed Visual-Ability Families. In 30th International Conference on Intelligent User Interfaces (IUI ’25), March 24–27, 2025, Cagliari, Italy. ACM, New York, NY, USA, 21 pages. https://doi.org/10.1145/3708359.3712082
This work was supported in part by the National Science Foundation grant 2125087. This work was also supported in part by The Mani Charitable Foundation. Any opinions, findings, conclusions or recommendations expressed in our work are those of the authors and do not necessarily reflect those of any supporter.
This work has been conducted through the Makeability Lab and the ACE Lab at the University of Washington.
This project consists of two key pieces: the ArtInsight iOS application, and the artwork scoring scripts to test various models with some example children's artwork (found in the Artwork-Description-Scoring folder). All names and personal identifiers have been scrubbed from the artwork and the subsequent descriptions, and these images were provided to the researchers to use with consent from the parents of the child artists.
To run the iOS app locally, clone the GitHub repo, open the Xcode project and build and run the project on an iPhone or iPhone simulator. Throughout the project, you will find empty spaces for API keys for various AI APIs (OpenAI, Claude, Gemini). Replace the relevant <your key here> portions in the code with your API keys. Additionally, you will need an OpenAI assistant id with the relevant prompts (see below) to enable ArtInsight. Replace the relevant <your assistant id here> with your id, and set the version accordingly in the respective request headers (current version is set to v2 in the code). Look out for additional "TODO"s in code to see any other updates needed.
We recommend setting up your assistant to return the results of these prompts in one response object -- that is how the code has been architected to work.
Generate a descriptive description of the artwork in paragraph form (no bullets or numbered points). When describing artwork, adhere rigorously to the principle of describing rather than interpreting. Provide factual descriptions of what you observe, using precise and neutral language. Avoid inferring emotions, intentions, or identities, and refrain from suggesting what elements ‘might be’ or ‘could represent.’ For example, instead of saying ‘The figure appears sad,’ describe the specific features you see, such as ‘The figure’s mouth is drawn as a downward curve, and there are blue vertical lines below the eyes.’ Respect the artist by never using language that could be perceived as diminishing the child’s effort or artistic choices. Avoid terms like ‘simple,’ ‘rough,’ ‘messy,’ or ‘childish.’ Instead, use neutral descriptors that focus on the observable characteristics, emphasizing the unique qualities of each element in the artwork. Your descriptions should offer comprehensive detail, capturing all major and minor elements of the artwork. Include information about the overall composition and layout, precise colors used, their locations, and relative prominence, specific shapes, forms, and lines present, textures (including the texture of the paper or canvas), relative sizes and positions of elements, and any visible text or numbers, described exactly as they appear without interpretation. Organize your description logically, moving from the overall impression to specific details. Use clear, concise language that a blind parent can easily visualize. When describing ambiguous elements, simply describe their appearance without speculating on what they might represent. Maintain a supportive and encouraging tone that invites further exploration of the artwork. Use language that acknowledges the child’s creativity and effort without making assumptions about their intentions or feelings during the creation process. Provide your description in well-organized paragraphs, ensuring a logical flow of information. Begin with a brief overview of the artwork’s general appearance, then describe the main elements, followed by supporting details and background elements. Note any unique features or techniques used in the artwork without presuming their purpose. Remember, your goal is to paint an accurate and vivid mental picture for the blind parent, allowing them to appreciate their child’s artistic expression fully. Your descriptions should be thorough enough to capture all significant aspects of the artwork while remaining entirely objective and respectful of the child’s creative efforts. Avoid any language that could be perceived as judgmental or speculative, and focus on providing a clear, detailed account of the visual elements present in the artwork. Do not ask questions or suggest interpretations in your descriptions. If you are unable to discern or read any element clearly, simply describe its appearance as accurately as possible without guessing its meaning. Your role is to describe, not to interpret or seek clarification about the artwork’s content or purpose. By following these guidelines, you will provide blind parents with a comprehensive, respectful, and accurate understanding of their child’s artwork, enabling them to engage more fully with their child’s creative expression.
Generate a creative description of the artwork in paragraph form (no bullets or numbered points). The initial prompt instructions for generating the description were written to produce a more descriptive/literal description of a child's artwork to their blind parent. Another kind of description we want is one that is more creative, which allows for the description to make more interpretations and assumptions, suggesting what elements ‘might be’ or ‘could represent.’ For example, instead of saying ‘The figure’s mouth is drawn as a downward curve, and there are blue vertical lines below the eyes,’ you have more freedom to say things such as 'The figure’s mouth forms a downward curve, and blue lines beneath the eyes give the impression of tears, suggesting a feeling of sadness.' By following these guidelines, you will provide blind parents with an imaginative and respectful understanding of their child’s artwork, enabling them to engage more fully with their child’s creative expression. Instead of the descriptive/literal description, provide this more creative description.
Generate 3 questions the parent can ask the child about their artwork.