加载下载好的FLUX.1-schnell模型提示not a string

[deepeye]

System Info / 系統信息

Python: 3.10.14
CUDA Version: 12.2
OS: Centos7.9

Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？

• docker / docker
• pip install / 通过 pip install 安装
• installation from source / 从源码安装

Version info / 版本信息

Release: v0.15.3

The command used to start Xinference / 用以启动 xinference 的命令

XINFERENCE_HOME=/data/llms/inference/temp nohup xinference-local -H 0.0.0.0 > worker.log 2>&1 &

Reproduction / 复现过程

1、下载模型
git clone https://www.modelscope.cn/AI-ModelScope/FLUX.1-schnell.git

2、UI配置model path

3、后台报如下错误
2024-10-11 17:15:13,633 xinference.core.worker 1265 INFO [request 529686a2-87b1-11ef-bd78-00163e1c05ef] Enter launch_builtin_model, args: <xinference.core.worker.WorkerActor object at 0x7f0579440900>, kwargs: model_uid=FLUX.1-schnell-1-0,model_name=FLUX.1-schnell,model_size_in_billions=None,model_format=None,quantization=None,model_engine=None,model_type=image,n_gpu=auto,request_limits=None,peft_model_config=None,gpu_idx=None,download_hub=None,model_path=/data/llms/inference/models/FLUX.1-schnell
Keyword arguments {'lora_model_paths': None} are not expected by FluxPipeline and will be ignored.
Loading pipeline components...: 0%| | 0/7 [00:00<?, ?it/s]2024-10-11 17:15:18,364 transformers.configuration_utils 19255 INFO loading configuration file /data/llms/inference/models/FLUX.1-schnell/text_encoder_2/config.json
2024-10-11 17:15:18,365 transformers.configuration_utils 19255 INFO Model config T5Config {
"_name_or_path": "google/t5-v1_1-xxl",
"architectures": [
"T5EncoderModel"
],
"classifier_dropout": 0.0,
"d_ff": 10240,
"d_kv": 64,
"d_model": 4096,
"decoder_start_token_id": 0,
"dense_act_fn": "gelu_new",
"dropout_rate": 0.1,
"eos_token_id": 1,
"feed_forward_proj": "gated-gelu",
"initializer_factor": 1.0,
"is_encoder_decoder": true,
"is_gated_act": true,
"layer_norm_epsilon": 1e-06,
"model_type": "t5",
"num_decoder_layers": 24,
"num_heads": 64,
"num_layers": 24,
"output_past": true,
"pad_token_id": 0,
"relative_attention_max_distance": 128,
"relative_attention_num_buckets": 32,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.44.2",
"use_cache": true,
"vocab_size": 32128
}

2024-10-11 17:15:18,365 transformers.modeling_utils 19255 INFO loading weights file /data/llms/inference/models/FLUX.1-schnell/text_encoder_2/model.safetensors.index.json
2024-10-11 17:15:18,365 transformers.modeling_utils 19255 INFO Instantiating T5EncoderModel model under default dtype torch.float16.
Loading checkpoint shards: 100%|█████████████████| 2/2 [00:02<00:00, 1.12s/it]
2024-10-11 17:15:20,654 transformers.modeling_utils 19255 INFO All model checkpoint weights were used when initializing T5EncoderModel.

2024-10-11 17:15:20,654 transformers.modeling_utils 19255 INFO All the weights of T5EncoderModel were initialized from the model checkpoint at /data/llms/inference/models/FLUX.1-schnell/text_encoder_2.
If your task is similar to the task the model of the checkpoint was trained on, you can already use T5EncoderModel for predictions without further training.
Loading pipeline components...: 14%|█▋ | 1/7 [00:02<00:13, 2.31s/it]2024-10-11 17:15:20,658 transformers.tokenization_utils_base 19255 INFO loading file spiece.model
2024-10-11 17:15:20,658 transformers.tokenization_utils_base 19255 INFO loading file tokenizer.json
2024-10-11 17:15:20,658 transformers.tokenization_utils_base 19255 INFO loading file added_tokens.json
2024-10-11 17:15:20,658 transformers.tokenization_utils_base 19255 INFO loading file special_tokens_map.json
2024-10-11 17:15:20,658 transformers.tokenization_utils_base 19255 INFO loading file tokenizer_config.json
2024-10-11 17:15:20,659 transformers.models.t5.tokenization_t5_fast 19255 WARNING You set add_prefix_space. The tokenizer needs to be converted from the slow tokenizers
Loading pipeline components...: 29%|███▍ | 2/7 [00:02<00:05, 1.16s/it]
2024-10-11 17:15:20,661 xinference.core.worker 1265 ERROR Failed to load model FLUX.1-schnell-1-0
Traceback (most recent call last):
File "/data/llms/inference/env/lib/python3.10/site-packages/xinference/core/worker.py", line 893, in launch_builtin_model
await model_ref.load()
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/backends/context.py", line 227, in send
return self._process_result_message(result)
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/backends/context.py", line 102, in _process_result_message
raise message.as_instanceof_cause()
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/backends/pool.py", line 659, in send
result = await self._run_coro(message.message_id, coro)
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/backends/pool.py", line 370, in _run_coro
return await coro
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/api.py", line 384, in on_receive
return await super().on_receive(message) # type: ignore
File "xoscar/core.pyx", line 558, in on_receive
raise ex
File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.on_receive
async with self._lock:
File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
result = await result
File "/data/llms/inference/env/lib/python3.10/site-packages/xinference/core/model.py", line 309, in load
self._model.load()
File "/data/llms/inference/env/lib/python3.10/site-packages/xinference/model/image/stable_diffusion/core.py", line 192, in load
self._model = AutoPipelineModel.from_pretrained(
File "/data/llms/inference/env/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/diffusers/pipelines/auto_pipeline.py", line 382, in from_pretrained
return text_2_image_cls.from_pretrained(pretrained_model_or_path, **kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 876, in from_pretrained
loaded_sub_model = load_sub_model(
File "/data/llms/inference/env/lib/python3.10/site-packages/diffusers/pipelines/pipeline_loading_utils.py", line 700, in load_sub_model
loaded_sub_model = load_method(os.path.join(cached_folder, name), **loading_kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2271, in from_pretrained
return cls._from_pretrained(
File "/data/llms/inference/env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2505, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/transformers/models/t5/tokenization_t5_fast.py", line 119, in init
super().init(
File "/data/llms/inference/env/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 131, in init
slow_tokenizer = self.slow_tokenizer_class(*args, **kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/transformers/models/t5/tokenization_t5.py", line 150, in init
self.sp_model.Load(vocab_file)
File "/data/llms/inference/env/lib/python3.10/site-packages/sentencepiece/init.py", line 961, in Load
return self.LoadFromFile(model_file)
File "/data/llms/inference/env/lib/python3.10/site-packages/sentencepiece/init.py", line 316, in LoadFromFile
return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
TypeError: [address=0.0.0.0:35200, pid=19255] not a string
2024-10-11 17:15:21,461 xinference.core.worker 1265 ERROR [request 529686a2-87b1-11ef-bd78-00163e1c05ef] Leave launch_builtin_model, error: [address=0.0.0.0:35200, pid=19255] not a string, elapsed time: 7 s
Traceback (most recent call last):
File "/data/llms/inference/env/lib/python3.10/site-packages/xinference/core/utils.py", line 69, in wrapped
ret = await func(*args, **kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/xinference/core/worker.py", line 893, in launch_builtin_model
await model_ref.load()
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/backends/context.py", line 227, in send
return self._process_result_message(result)
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/backends/context.py", line 102, in _process_result_message
raise message.as_instanceof_cause()
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/backends/pool.py", line 659, in send
result = await self._run_coro(message.message_id, coro)
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/backends/pool.py", line 370, in _run_coro
return await coro
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/api.py", line 384, in on_receive
return await super().on_receive(message) # type: ignore
File "xoscar/core.pyx", line 558, in on_receive
raise ex
File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.on_receive
async with self._lock:
File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
result = await result
File "/data/llms/inference/env/lib/python3.10/site-packages/xinference/core/model.py", line 309, in load
self._model.load()
File "/data/llms/inference/env/lib/python3.10/site-packages/xinference/model/image/stable_diffusion/core.py", line 192, in load
self._model = AutoPipelineModel.from_pretrained(
File "/data/llms/inference/env/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/diffusers/pipelines/auto_pipeline.py", line 382, in from_pretrained
return text_2_image_cls.from_pretrained(pretrained_model_or_path, **kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 876, in from_pretrained
loaded_sub_model = load_sub_model(
File "/data/llms/inference/env/lib/python3.10/site-packages/diffusers/pipelines/pipeline_loading_utils.py", line 700, in load_sub_model
loaded_sub_model = load_method(os.path.join(cached_folder, name), **loading_kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2271, in from_pretrained
return cls._from_pretrained(
File "/data/llms/inference/env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2505, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/transformers/models/t5/tokenization_t5_fast.py", line 119, in init
super().init(
File "/data/llms/inference/env/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 131, in init
slow_tokenizer = self.slow_tokenizer_class(*args, **kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/transformers/models/t5/tokenization_t5.py", line 150, in init
self.sp_model.Load(vocab_file)
File "/data/llms/inference/env/lib/python3.10/site-packages/sentencepiece/init.py", line 961, in Load
return self.LoadFromFile(model_file)
File "/data/llms/inference/env/lib/python3.10/site-packages/sentencepiece/init.py", line 316, in LoadFromFile
return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
TypeError: [address=0.0.0.0:35200, pid=19255] not a string
2024-10-11 17:15:21,466 xinference.api.restful_api 1151 ERROR [address=0.0.0.0:35200, pid=19255] not a string
Traceback (most recent call last):
File "/data/llms/inference/env/lib/python3.10/site-packages/xinference/api/restful_api.py", line 967, in launch_model
model_uid = await (await self._get_supervisor_ref()).launch_builtin_model(
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/backends/context.py", line 227, in send
return self._process_result_message(result)
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/backends/context.py", line 102, in _process_result_message
raise message.as_instanceof_cause()
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/backends/pool.py", line 659, in send
result = await self._run_coro(message.message_id, coro)
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/backends/pool.py", line 370, in _run_coro
return await coro
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/api.py", line 384, in on_receive
return await super().on_receive(message) # type: ignore
File "xoscar/core.pyx", line 558, in on_receive
raise ex
File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.on_receive
async with self._lock:
File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
result = await result
File "/data/llms/inference/env/lib/python3.10/site-packages/xinference/core/supervisor.py", line 1032, in launch_builtin_model
await _launch_model()
File "/data/llms/inference/env/lib/python3.10/site-packages/xinference/core/supervisor.py", line 996, in _launch_model
await _launch_one_model(rep_model_uid)
File "/data/llms/inference/env/lib/python3.10/site-packages/xinference/core/supervisor.py", line 975, in _launch_one_model
await worker_ref.launch_builtin_model(
File "xoscar/core.pyx", line 284, in __pyx_actor_method_wrapper
async with lock:
File "xoscar/core.pyx", line 287, in xoscar.core.__pyx_actor_method_wrapper
result = await result
File "/data/llms/inference/env/lib/python3.10/site-packages/xinference/core/utils.py", line 69, in wrapped
ret = await func(*args, **kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/xinference/core/worker.py", line 893, in launch_builtin_model
await model_ref.load()
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/backends/context.py", line 227, in send
return self._process_result_message(result)
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/backends/context.py", line 102, in _process_result_message
raise message.as_instanceof_cause()
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/backends/pool.py", line 659, in send
result = await self._run_coro(message.message_id, coro)
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/backends/pool.py", line 370, in _run_coro
return await coro
File "/data/llms/inference/env/lib/python3.10/site-packages/xoscar/api.py", line 384, in on_receive
return await super().on_receive(message) # type: ignore
File "xoscar/core.pyx", line 558, in on_receive
raise ex
File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.on_receive
async with self._lock:
File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
result = await result
File "/data/llms/inference/env/lib/python3.10/site-packages/xinference/core/model.py", line 309, in load
self._model.load()
File "/data/llms/inference/env/lib/python3.10/site-packages/xinference/model/image/stable_diffusion/core.py", line 192, in load
self._model = AutoPipelineModel.from_pretrained(
File "/data/llms/inference/env/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/diffusers/pipelines/auto_pipeline.py", line 382, in from_pretrained
return text_2_image_cls.from_pretrained(pretrained_model_or_path, **kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 876, in from_pretrained
loaded_sub_model = load_sub_model(
File "/data/llms/inference/env/lib/python3.10/site-packages/diffusers/pipelines/pipeline_loading_utils.py", line 700, in load_sub_model
loaded_sub_model = load_method(os.path.join(cached_folder, name), **loading_kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2271, in from_pretrained
return cls._from_pretrained(
File "/data/llms/inference/env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2505, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/transformers/models/t5/tokenization_t5_fast.py", line 119, in init
super().init(
File "/data/llms/inference/env/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 131, in init
slow_tokenizer = self.slow_tokenizer_class(*args, **kwargs)
File "/data/llms/inference/env/lib/python3.10/site-packages/transformers/models/t5/tokenization_t5.py", line 150, in init
self.sp_model.Load(vocab_file)
File "/data/llms/inference/env/lib/python3.10/site-packages/sentencepiece/init.py", line 961, in Load
return self.LoadFromFile(model_file)
File "/data/llms/inference/env/lib/python3.10/site-packages/sentencepiece/init.py", line 316, in LoadFromFile
return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
TypeError: [address=0.0.0.0:35200, pid=19255] not a string

Expected behavior / 期待表现

成功加载

[deepeye]

上面问题是因为缺少文件导致，目前解决了，不过又提示2024-10-11 17:33:32,030 xinference.core.worker 1265 ERROR [request 9e9169b2-87b3-11ef-bd78-00163e1c05ef] Leave launch_builtin_model, error: [address=0.0.0.0:38445, pid=23153] unable to mmap 9949328904 bytes from file </data/llms/inference/models/FLUX.1-schnell/transformer/diffusion_pytorch_model-00002-of-00003.safetensors>: Cannot allocate memory (12), elapsed time: 111 s

[qinxuye]

显存不够，试下加额外选项，quantize_text_encoder，值是 text_encoder_2

[deepeye]

图生图，提示：RuntimeError: Failed to variants the images, detail: [address=0.0.0.0:35146, pid=24884] AutoPipeline can't find a pipeline linked to FluxPipeline for flux

[deepeye]

pip show diffusers
Name: diffusers
Version: 0.30.2
Summary: State-of-the-art diffusion in PyTorch and JAX.
Home-page: https://github.com/huggingface/diffusers
Author: The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/diffusers/graphs/contributors)
Author-email: diffusers@huggingface.co
License: Apache 2.0 License
Location: /data/llms/inference/env/lib/python3.10/site-packages
Requires: filelock, huggingface-hub, importlib-metadata, numpy, Pillow, regex, requests, safetensors
Required-by: matcha-tts

[qinxuye]

看了下，flux 的图生图还没有发版。需要从源码安装。

pip install git+https://github.com/huggingface/diffusers.git

[deepeye]

提示这个：You are trying to load the model files of the variant=fp16, but no such modeling files are available

[qinxuye]

我测试 flux.1-dev 可以正常工作，有空我再看下 flux1.-schnell

[deepeye]

我测试 flux.1-dev 可以正常工作，有空我再看下 flux1.-schnell

我测试了flux.1-dev，仍然提示：You are trying to load the model files of the variant=fp16, but no such modeling files are available

[github-actions]

This issue is stale because it has been open for 7 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 5 days since being marked as stale.

[yanmao2023]

我运行FLUX.1-dev时同样报错误：ValueError: [address=0.0.0.0:42657, pid=85416] You are trying to load the model files of the variant=fp16, but no such modeling files are available.
运行环境为ubuntu22.04，xinference0.16.1，pip install 安装，运行环境设置了两个环境变量XINFERENCE_MODEL_SRC=modelscope，HF_ENDPOINT=https://hf-mirror.com，模型是在图形界面launch时默认下载的，应该是走的modelscope，quantize_text_encoder= text_encoder_2参数已设置。感觉是缺了啥模型文件。运行flux1.-schnell也是同样问题。