Newly created Collection available for all the users when authentication is enabled

[llmwesee]

When authentication is enabled, I want to create a shared collection, such as UserData, that remains accessible to all users at all times, regardless of who created it.

I encountered an issue when attempting to create a new collection (UserData2). Here’s the process I followed:

1. I first created the collection using the UI.
2. I then ran the following command:

python generate.py --base_model=meta-llama/Llama-2-13b-chat-hf --score_model=None --langchain_modes="['UserData',UserData2,LLM,MyData]" --user_path=user_path --use_auth_token=True --max_seq_len=4096 --max_max_new_tokens=2048 --max_new_tokens=2048 --min_new_tokens=128 --prompt_type=llama2 --enable_stt=False --enable_tts=False --auth_filename=$auth_filename --auth_access=open --guest_name=avser --auth="[(admin, admin)]"

The issue is that the newly created collection (UserData2) is only accessible to the user who created it, similar to how MyData functions. However, I would like this collection to behave like UserData, where it is available to all users globally, even when authentication is enabled.

However When authentication is disabled during collection creation, the collection is accessible to all users as expected.

so Could you please provide guidance on how to create a shared collection like UserData that remains accessible to all users, even when authentication is enabled?

[pseudotensor]

How did you create it in the UI? The box takes a few args, including the collection type (assumed to be personal if no passed).

[llmwesee]

Document Selection >> Add Collection then type UserData2, shared, userpath then put UserData2 in the
--langchain_modes="['UserData',UserData2,LLM,MyData]"

[llmwesee]

However i also created the collection with src/make_db.py: by adding all the files in the folder user_path3 then
python src/make_db.py --user_path=user_path3 --collection_name=UserData3 --langchain_type=shared

python generate.py --base_model=meta-llama/Llama-2-13b-chat-hf --score_model=None --langchain_mode='UserData' --user_path=user_path --use_auth_token=True --max_seq_len=4096 --max_max_new_tokens=2048 --batch_size=16 --prompt_type=llama2 --langchain_modes=['UserData','UserData3','MyData','LLM'] --auth_filename=$auth_filename --auth_access=open --guest_name=avser --auth="[(admin, admin)]"

then still didn't showing UserData3 in the collections for all the users although the embedding are stored in db_dir_UserData3 folder

And when adding --langchain_modes=['UserData','UserData3'] --langchain_mode_paths={'UserData':'user_path','UserData3':'user_path3'} --langchain_mode_types={'UserData':'shared','UserData3':'shared'}

to the command like:

python generate.py --base_model=meta-llama/Llama-2-13b-chat-hf --score_model=None --langchain_mode='UserData' --user_path=user_path --use_auth_token=True --max_seq_len=4096 --max_max_new_tokens=2048 --batch_size=16 --prompt_type=llama2 --langchain_modes=['UserData','UserData3'] --langchain_mode_paths={'UserData':'user_path','UserData3':'user_path3'} --langchain_mode_types={'UserData':'shared','UserData3':'shared'} --auth_filename=$auth_filename --auth_access=open --guest_name=avser --auth="[(admin, admin)]"

then it showing the following error:

File "/home/xxxx/src/gen.py", line 1383, in main langchain_mode_paths = str_to_dict(langchain_mode_paths) File "/home/xxxx/src/utils.py", line 1863, in str_to_dict raise ValueError("Invalid str_to_dict for %s" % x) ValueError: Invalid str_to_dict for UserData3:user_path3

Note: I created the authentication server through the LDAP
@pseudotensor please help me regarding this!

[hartysoly]

This worked well for me hopefully that helps, i am a beginner so good luck :)

python generate.py --base_model=XYZ-brrrr --prompt_type=brrrr--max_max_new_tokens=2048 --max_new_tokens=1024 --max_seq_len=8094 --max_quality=True --langchain_modes="['UserData','brrrData']" --user_path=user_path --langchain_mode_types="{'UserData':'shared','brrrData':'shared'}" --function_server=True --function_server_workers=10 --multiple_workers_gunicorn=True --async_output=True --num_async=10 --auth_access=closed --admin_pass=meadmin --auth="[('brrryaa','winkwink')]" --auth_freeze