RAM optimization round 2

[AUTOMATIC1111]

Description

• prevents torch from storing weights twice when loading model weights from state_dict
• also sends weights to GPU earlier, if possible
• on my machine, peak RAM usage when loading SDXL according to the chart is 7GB, and it goes to 3.3GB after being loaded
• gains for --medvram are less, because the model needs to be stored in memory; gains for checkpoints in fp32 are also less.
• does some rather dangerous things so I'd like people to test it before it's merged
• intended to work on the version of torch required by the repo, torch 2.0.1
• additionally, prevents Loras from making weight backups until they are needed, which also makes the program use less RAM but only until you start using Loras.

Loading SDXL 1.0 fp16 without low/medvram:

[rltgjqmcpgjadyd]

torch 2.1 also works fine

no low/medvram

on webui startup ram usage when loading SDXL checkpoint & SDXL vae

before

after

SDXL image generation : work fine
SDXL image generation with hires fix : work fine
SDXL image generation with refiner(extension) : work fine
SDXL image generation with refiner(built-in) : work fine
SDXL image generation with refiner(built-in) + hires fix : work fine

SD1.5 image generation : work fine
SD1.5 image generation with hires fix : work fine
SD1.5 image generation with hires fix + lora : work fine
SD1.5 image generation with hires fix + multiple lora : work fine

[rltgjqmcpgjadyd]

i found some error

if --no-half is entered as an argument, an error occurs when loading the checkpoint

both SDXL and SD1.5 will occur

dev branch does not occur

here`s log

SDXL

changing setting sd_model_checkpoint to bluePencilXL_v010.safetensors [6ea7884bc7]: RuntimeError
Traceback (most recent call last):
  File "J:\SDWebUI\modules\options.py", line 133, in set
    self.data_labels[key].onchange()
  File "J:\SDWebUI\modules\call_queue.py", line 14, in f
    res = func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\modules\initialize_util.py", line 151, in <lambda>
    shared.opts.onchange("sd_model_checkpoint", wrap_queued_call(lambda: sd_models.reload_model_weights()), call=False)
                                                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\modules\sd_models.py", line 718, in reload_model_weights
    load_model(checkpoint_info, already_loaded_state_dict=state_dict)
  File "J:\SDWebUI\modules\sd_models.py", line 621, in load_model
    sd_model.cond_stage_model_empty_prompt = get_empty_cond(sd_model)
                                             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\modules\sd_models.py", line 506, in get_empty_cond
    d = sd_model.get_learned_conditioning([""])
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\modules\sd_models_xl.py", line 31, in get_learned_conditioning
    c = self.conditioner(sdxl_conds, force_zero_embeddings=['txt'] if force_zero_negative_prompt else [])
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\repositories\generative-models\sgm\modules\encoders\modules.py", line 141, in forward
    emb_out = embedder(batch[embedder.input_key])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\modules\sd_hijack_clip.py", line 234, in forward
    z = self.process_tokens(tokens, multipliers)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\modules\sd_hijack_clip.py", line 273, in process_tokens
    z = self.encode_with_transformers(tokens)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\modules\sd_hijack_clip.py", line 349, in encode_with_transformers
    outputs = self.wrapped.transformer(input_ids=tokens, output_hidden_states=self.wrapped.layer == "hidden")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\transformers\models\clip\modeling_clip.py", line 823, in forward
    return self.text_model(
           ^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\transformers\models\clip\modeling_clip.py", line 741, in forward
    encoder_outputs = self.encoder(
                      ^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\transformers\models\clip\modeling_clip.py", line 655, in forward
    layer_outputs = encoder_layer(
                    ^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\transformers\models\clip\modeling_clip.py", line 385, in forward
    hidden_states, attn_weights = self.self_attn(
                                  ^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\transformers\models\clip\modeling_clip.py", line 274, in forward
    query_states = self.q_proj(hidden_states) * self.scale
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\extensions-builtin\Lora\networks.py", line 429, in network_Linear_forward
    return originals.Linear_forward(self, input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half

SD 1.5

changing setting sd_model_checkpoint to etherBluMix_etherBluMix5.safetensors [6965f33b20]: RuntimeError
Traceback (most recent call last):
  File "J:\SDWebUI\modules\options.py", line 133, in set
    self.data_labels[key].onchange()
  File "J:\SDWebUI\modules\call_queue.py", line 14, in f
    res = func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\modules\initialize_util.py", line 151, in <lambda>
    shared.opts.onchange("sd_model_checkpoint", wrap_queued_call(lambda: sd_models.reload_model_weights()), call=False)
                                                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\modules\sd_models.py", line 718, in reload_model_weights
    load_model(checkpoint_info, already_loaded_state_dict=state_dict)
  File "J:\SDWebUI\modules\sd_models.py", line 621, in load_model
    sd_model.cond_stage_model_empty_prompt = get_empty_cond(sd_model)
                                             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\modules\sd_models.py", line 509, in get_empty_cond
    return sd_model.cond_stage_model([""])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\modules\sd_hijack_clip.py", line 234, in forward
    z = self.process_tokens(tokens, multipliers)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\modules\sd_hijack_clip.py", line 273, in process_tokens
    z = self.encode_with_transformers(tokens)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\modules\sd_hijack_clip.py", line 326, in encode_with_transformers
    outputs = self.wrapped.transformer(input_ids=tokens, output_hidden_states=-opts.CLIP_stop_at_last_layers)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\transformers\models\clip\modeling_clip.py", line 823, in forward
    return self.text_model(
           ^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\transformers\models\clip\modeling_clip.py", line 741, in forward
    encoder_outputs = self.encoder(
                      ^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\transformers\models\clip\modeling_clip.py", line 655, in forward
    layer_outputs = encoder_layer(
                    ^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\transformers\models\clip\modeling_clip.py", line 385, in forward
    hidden_states, attn_weights = self.self_attn(
                                  ^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\transformers\models\clip\modeling_clip.py", line 274, in forward
    query_states = self.q_proj(hidden_states) * self.scale
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\extensions-builtin\Lora\networks.py", line 429, in network_Linear_forward
    return originals.Linear_forward(self, input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "J:\SDWebUI\venv\Lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half