Naoto Inoue*
Kento Masui*
Wataru Shimoda*
Kota Yamaguchi (*: equal contribution)
CyberAgent
Workshop on Graphic Design Understanding and Generation (at CVPR2024)
🤔 Automatic generation of graphic designs has recently received considerable attention.
😦 However, the state-of-the-art approaches are complex and rely on proprietary datasets, which creates reproducibility barriers.
🔥 In this paper, we propose an open framework for automatic graphic design called OpenCOLE, where we build a modified version of the pioneering COLE [Jia+, arXiv'23] and train our model exclusively on publicly available datasets.
🚀 Based on GPT4V evaluations, our model shows promising performance comparable to the original COLE. We release the pipeline and training results to encourage open development.
poetry install
OpenCOLE dataset (v1) is available at cyberagent/opencole in HuggingFace dataset hub.
- text_to_image:
cyberagent/opencole-stable-diffusion-xl-base-1.0-finetune - typography_lmm:
cyberagent/opencole-typographylmm-llava-v1.5-7b-lora
Some part requires additional environment variables. We recommend to use direnv. Please copy the template in .envrc.example and modify it on your own.
cp .envrc.example .envrcPlease refer to inference.md.
We provide a script for GPT4V-based evaluation on generated images.
poetry run python -m opencole.evaluation.eval_gpt4v --input_dir <INPUT_DIR> --output_path <OUTPUT_PATH>Please refer to training.md.
If you find this code useful for your research, please cite our paper:
@inproceedings{inoue2024opencole,
title={{OpenCOLE: Towards Reproducible Automatic Graphic Design Generation}},
author={Naoto Inoue and Kento Masui and Wataru Shimoda and Kota Yamaguchi},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
year={2024},
}
This repository has been migrated from the internal repo. Despite the fact that commit logs are not visible, all the contributors have made significant contributions to the repository.
- @proboscis: OpenCOLE dataset construction
- @shimoda-uec: TypographyLMM
- @kyamagu: renderer
- @naoto0804: other (bunch of) stuffs