Recording Transfer Failed with Shared Storage

[pritamnanda]

I have installed Pilos on a Standalone BBB server for Testing.

I am using the below Paths

Pilos Spool DIR /mnt/storage/recordings-spool/ (Post_publish Ruby script should Transfer the Archived file here)

Pilos Recording DIR /mnt/storage/recordings/ (To be used by Pilos for storing Recordings)

pilos_post_publish.rb is working but It's failing to Transfer to the Spool DIR, Do I have to modify this Ruby script to be used with Shared Storage instead of SSH Transfer ?

Post Publish Log as below

I, [2024-09-22T11:53:44.181674 #972501] INFO -- : Task: Getting meeting metadata
I, [2024-09-22T11:53:44.183439 #972501] INFO -- : Recording transferring to PILOS starts
I, [2024-09-22T11:53:44.183534 #972501] INFO -- : Creating recording archive for video of meeting XXXXXXXXXXXXXXXXXXXXXXX
I, [2024-09-22T11:53:44.186756 #972501] INFO -- : Transferring recording archive for video of meeting XXXXXXXXXXXXXXXXXXXXX
I, [2024-09-22T11:53:44.192227 #972501] INFO -- : Rescued
I, [2024-09-22T11:53:44.192400 #972501] INFO -- : Failed to transfer recording archive
I, [2024-09-22T11:53:44.192553 #972501] INFO -- : Recording transferring to PILOS ends
I, [2024-09-22T11:53:44.986402 #972524] INFO -- : Recording Ready Notify for [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] starts
I, [2024-09-22T11:53:44.986486 #972524] INFO -- : Task: Getting meeting metadata
I, [2024-09-22T11:53:44.988818 #972524] INFO -- : Recording Ready notify ends

BigBlueButton user has access to Write on the Spool DIR
Pilos can Read & Delete as well

Should I try to find replacement of this below part of the Post_publish script with "mv" command as it is just moving from Temporary Archived File location to the Spool DIR ?

Transfer to spool_dir

    BigBlueButton.logger.info("Transferring recording archive for #{format} of meeting #{meeting_id} to #{spool_dir}")
    system('rsync', '--verbose', '--remove-source-files', '--chmod=664', *extra_rsync_opts, archive_file, spool_dir) \
      || raise('Failed to transfer recording archive')

[pritamnanda]

I have installed Pilos on a Standalone BBB server for Testing.

I am using the below Paths

Pilos Spool DIR /mnt/storage/recordings-spool/ (Post_publish Ruby script should Transfer the Archived file here)

Pilos Recording DIR /mnt/storage/recordings/ (To be used by Pilos for storing Recordings)

pilos_post_publish.rb is working but It's failing to Transfer to the Spool DIR, Do I have to modify this Ruby script to be used with Shared Storage instead of SSH Transfer ?

Post Publish Log as below

I, [2024-09-22T11:53:44.181674 #972501] INFO -- : Task: Getting meeting metadata I, [2024-09-22T11:53:44.183439 #972501] INFO -- : Recording transferring to PILOS starts I, [2024-09-22T11:53:44.183534 #972501] INFO -- : Creating recording archive for video of meeting XXXXXXXXXXXXXXXXXXXXXXX I, [2024-09-22T11:53:44.186756 #972501] INFO -- : Transferring recording archive for video of meeting XXXXXXXXXXXXXXXXXXXXX I, [2024-09-22T11:53:44.192227 #972501] INFO -- : Rescued I, [2024-09-22T11:53:44.192400 #972501] INFO -- : Failed to transfer recording archive I, [2024-09-22T11:53:44.192553 #972501] INFO -- : Recording transferring to PILOS ends I, [2024-09-22T11:53:44.986402 #972524] INFO -- : Recording Ready Notify for [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] starts I, [2024-09-22T11:53:44.986486 #972524] INFO -- : Task: Getting meeting metadata I, [2024-09-22T11:53:44.988818 #972524] INFO -- : Recording Ready notify ends

BigBlueButton user has access to Write on the Spool DIR Pilos can Read & Delete as well

Should I try to find replacement of this below part of the Post_publish script with "mv" command as it is just moving from Temporary Archived File location to the Spool DIR ?

Transfer to spool_dir

    BigBlueButton.logger.info("Transferring recording archive for #{format} of meeting #{meeting_id} to #{spool_dir}")
    system('rsync', '--verbose', '--remove-source-files', '--chmod=664', *extra_rsync_opts, archive_file, spool_dir) \
      || raise('Failed to transfer recording archive')

Update>>>>

Initially I thought issue is with Transfer , But I can't see any .tar file generated in the Temporary Working Directory of the BBB server.

work_dir: /var/bigbluebutton/recording/pilos (This location is empty)

So may be the issue isn't with the Transfer, .tar file isn't being generated for the Recordings

@SamuelWei Could you please assist in case I am missing something?

Thanks a lot in advance !!

[pritamnanda]

I can manually do it using a bash script as below

Recording imported into Pilos, Not sure what's wrong with Ruby Script

#!/bin/bash

# Parameters
RECORDING_ID=$1  # The subdirectory/recording ID (passed as an argument)
PUBLISHED_DIR="/var/bigbluebutton/published/video"
TEMP_DIR="/var/bigbluebutton/recording/pilos"
DEST_DIR="/mnt/storage/recordings-spool"

# Validate input
if [ -z "$RECORDING_ID" ]; then
  echo "Error: Recording ID is required."
  exit 1
fi

# Check if the recording directory exists
RECORDING_PATH="${PUBLISHED_DIR}/${RECORDING_ID}"
if [ ! -d "$RECORDING_PATH" ]; then
  echo "Error: Recording directory ${RECORDING_PATH} does not exist."
  exit 1
fi

# Create the archive file name (Meeting ID-video.tar)
ARCHIVE_FILE="${RECORDING_ID}-video.tar"

# Log message
echo "Creating archive for recording ${RECORDING_ID}"

# Create the .tar archive and store it temporarily in the TEMP_DIR
tar --create --file="${TEMP_DIR}/${ARCHIVE_FILE}" --directory="${PUBLISHED_DIR}" "${RECORDING_ID}" \
  || { echo "Failed to create tar archive for ${RECORDING_ID}"; exit 1; }

echo "Archive ${ARCHIVE_FILE} created successfully in ${TEMP_DIR}."

# Move the .tar file to the destination directory
echo "Transferring archive to ${DEST_DIR}"
mv "${TEMP_DIR}/${ARCHIVE_FILE}" "${DEST_DIR}/" \
  || { echo "Failed to transfer archive to ${DEST_DIR}"; exit 1; }

echo "Archive transferred successfully to ${DEST_DIR}/${ARCHIVE_FILE}."

# Optional: Set permissions for the transferred file
chmod 664 "${DEST_DIR}/${ARCHIVE_FILE}" \
  || { echo "Failed to set permissions on transferred archive"; exit 1; }

echo "Permissions set successfully for ${DEST_DIR}/${ARCHIVE_FILE}."

[SamuelWei]

@pritamnanda could you please format the code sections in your comments as markdown code sections to make them readable, thanks

[pritamnanda]

There is no issue with bash script, Just horizon-1 is failing to delete the .tar file automatically from the Spool Directory after it is imported to Pilos. Log as below

horizon-1 | [2024-09-22 16:47:06] production.ERROR: Deleting recording file XXXXXXXXXXXXXXXX-video.tar failed
horizon-1 | [2024-09-22 16:47:07] production.ERROR: Deleting recording file YYYYYYYYYYYYYYYYY-video.tar failed
horizon-1 | [2024-09-22 16:47:07] production.ERROR: Deleting recording file ZZZZZZZZZZZZZZZZ-video.tar failed

But I can manually delete with below command so there's no Permission issue I beleive.

docker compose exec app rm storage/recordings-spool/XXXXXXXXXXXXXXXXXXXX-video.tar

I also tried to Modify the Post Publish Script a little with better Logging for every step & it seems that the issue is with creating the .tar file with Ruby Script.

@SamuelWei Thanks for your attention, I am not good enough with Ruby. If we fail to figure it out I will try some workaround with Bash-script.

Additional Contexts:

1. I have only a Video Format Enabled, No Presentation format.
2. Transferring from One Directory to another on the Same BBB Server(I believe it's the same as Shared Storage)
3. I am not sure what Recording Ready Callback does to Pilos. How does it help? Recording only appears after it's imported right?

BBB Server version 2.7.12 (751)

BBB Post Publish Log as below

Task: Getting meeting metadata
Recording transferring to PILOS starts
Creating recording archive for video of meeting XXXXXXXXXXX
Rescued
Failed to create tar archive for XXXXXXXXXXXXXXX
Recording transferring to PILOS ends
Recording Ready Notify for XXXXXXXXXXXXXXXXXXXXXX starts
Task: Getting meeting metadata
Recording Ready notify ends

Changed Part in the Post Publish Ruby Script

begin
    # Check if meeting is from PILOS, otherwise do nothing
    if origin == "PILOS"
        # Create work_dir if it doesn't exist yet
        FileUtils.mkdir_p(work_dir)

        # Check if the published files directory exists
        published_path = File.join(published_dir, format, meeting_id)
        unless Dir.exist?(published_path)
          raise "Published directory for meeting #{meeting_id} not found: #{published_path}"
        end

        # Create archive file
        BigBlueButton.logger.info("Creating recording archive for #{format} of meeting #{meeting_id}")
        
        # Make sure the command is correct, and check its success
        unless system('tar', '--create', '--file', archive_file, '--directory', published_dir, "#{format}/#{meeting_id}")
          raise "Failed to create tar archive for #{meeting_id}"
        end
        
        BigBlueButton.logger.info("Archive file created: #{archive_file}")

        # Transfer to spool_dir
        BigBlueButton.logger.info("Transferring recording archive for #{format} of meeting #{meeting_id} to #{spool_dir}")
        
        unless system('rsync', '--verbose', '--remove-source-files', '--chmod=664', *extra_rsync_opts, archive_file, spool_dir)
          raise "Failed to transfer recording archive to #{spool_dir}"
        end

        BigBlueButton.logger.info('Create sender.done file')
        File.write("#{recording_dir}/status/published/#{meeting_id}-sender.done", "Published #{meeting_id}")
    else
        BigBlueButton.logger.info("Meeting #{meeting_id} from origin #{origin} - not a Pilos recording")
    end

[SamuelWei]

I assume that there is a permissions problem. When executing docker compose exec app rm, the command is executed as root. However, the queue in the Horizon container is executed under the www-data user.

The bash script you run on the BBB server is probably run as root user, but the post-publishing script is run as bigbluebutton user.

To solve this permission problem, we try to set the file/folder permissions so that both users are in a group with the same group ID and the group permission is set to read and write.

Please check the docs: https://thm-health.github.io/PILOS/docs/administration/advanced/recording#permissions

[pritamnanda]

Yes @SamuelWei There was a Permission problem, but not on the mentioned path.

I already verified the ownership of the Spool Directory.

But I manually created the Temporary Work DIR(/var/bigbluebutton/recording/pilos) & It was owned by the Root instead of BigBlueButton.

Finally, now Recordings are successfully Acrchieved, Transferred & Imported by Pilos.

But the only remaining issue is the Failure of Automatic Deletion of the .tar files from Spool Directory.

It is currently owned by bigbluebutton:pilos-spool

drwxr-xr-x 2 bigbluebutton pilos-spool 4096 Sep 23 10:36 recordings-spool

Should I add the www-data User to the pilos-spool group ?

I can't see any user as pilos-spool , Do I have to create one ? Otherwise this below command in the Instruction isn't effective
chown pilos-spool:pilos-spool /mnt/storage/recordings-spool

[SamuelWei]

The container should add the www-data user to this group on startup. There is a env setting to set the group id

[pritamnanda]

The container should add the www-data user to this group on startup. There is a env setting to set the group id

Finally the issue is resolved with following action

I had to create the pilos-spool user first manually in the OS with below command

sudo useradd -g pilos-spool pilos-spool

Then

chown pilos-spool:pilos-spool /mnt/storage/recordings-spool
chmod 775 /mnt/storage/recordings-spool

After this it worked.

Maybe mentioning the IDs of www-user & group in the .env isn't relevant to this issue.

Thanks @SamuelWei for your time and attention.

You may delete this Discussion if necessary.

[SamuelWei]

Great, however I think you don't need a pilos-spool user, just a group as mentioned here: https://thm-health.github.io/PILOS/docs/administration/advanced/recording#permissions

addgroup pilos-spool --gid 2000
chown :pilos-spool /mnt/storage/recordings-spool
chmod 775 /mnt/storage/recordings-spool