Peter/cogvlm

[probicheaux]

Description

Add in CogVLM to roboflow inference server (with chat history!)

See examples/cogvlm/cog_client.py for an example script.
Docker image available at: roboflow/roboflow-inference-server-cog

I ran the image with the command:

docker run --gpus=all --rm -it -v cog:/cache --net=host --env NVIDIA_DISABLE_REQUIRE=1 roboflow/roboflow-inference-server-cog

Dependencies added:

transformers<=4.35.2
sentencepiece<=0.1.99
einops<=0.7.0
xformers<=0.0.22
accelerate<=0.25.0
bitsandbytes<=0.41.2.post2

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

How has this change been tested, please provide a testcase or example of how you tested the change?

Ran on a T4.

Any specific deployment considerations

Note the env variables COG_LOAD_8BIT and COG_LOAD_4BIT. COG_LOAD_4BIT must be set to true to run on a T4 (this is the current default).

Docs

• Docs updated? What were the changes:

[paulguerrie]

Looking great!