playlist sometimes plays files out of order after reloading

[vitoyucepi]

Describe the bug

playlist could break the order of files if files are updated using non-atomic writes.

• 1.mp3
• 2.mp3
• 3.mp3
• 4.mp3

Reload

• 2.mp3
• 1.mp3
• 3.mp3
• 4.mp3

Reload

• 1.mp3
• 2.mp3
• 3.mp3
• 4.mp3

To Reproduce

1. main.liq

   s = playlist("playlist.txt", mode="normal", prefetch=4, reload_mode="watch")
   s.on_track(print)
   output.dummy(s, fallible=true)

2. playlist1.txt

   1.mp3
   2.mp3
   3.mp3
   4.mp3
   5.mp3
   6.mp3
   7.mp3
   8.mp3
   9.mp3
   

3. Generate files
   for i in $(seq 1 9);do ffmpeg -f lavfi -i "sine=frequency=1000:duration=1" -c:a libmp3lame -b:a 128k -y $i.mp3;done.
4. Run liquidsoap main.liq.
5. Exec cat playlist1.txt > playlist.txt.
6. Repeat the previous action several times with a delay.
7. Stop liquidsoap.

Expected behavior

The playlist doesn't break the order.

Version details

• OS: ubuntu:22.04 in docker
• Version: 2.2.0

Install method
Deb package from liquidsoap releases at github

Common issues
#3342

[kingpalethe]

I am trying to write the playlist "atomically" with python, but in my case LiquidSoap is not recognizing the newly written file -- or at least is not using the changes.

I can reproduce this behavior:

1. Write the playlist with ...

with open(video_playlist_path, "w") as f:
            f.write(playlist_string)

.... and liquidsoap uses the changes in the playlist file.....

BUT, if I use this function instead....

def atomic_write(file_path, content, permissions=0o666):
    # Create a temporary file in the same directory as the target file
    temp_dir = os.path.dirname(file_path)
    temp_file = tempfile.NamedTemporaryFile(delete=False, dir=temp_dir)

    try:
        # Write the content to the temporary file
        temp_file.write(content.encode("utf-8"))
        temp_file.flush()

        # Atomically replace the target file with the temporary file
        os.replace(temp_file.name, file_path)

        # Set permissions on the newly written file
        os.chmod(file_path, permissions)
    finally:
        # Clean up the temporary file if something goes wrong
        if os.path.exists(temp_file.name):
            os.remove(temp_file.name)

..And liquidsoap does not bring in the changes -- even though I can confirm in a text editor that the changes have been made....

I also tried this library:

https://pypi.org/project/atomicwrites/

.. and the behavior is the same.

I wonder how liquidsoap has implemented the "watch" functionality.....

[kingpalethe]

Further to this -- has anyone found a rock-solid and reliable way to provide new playlist files for liquidsoap, and guarantee that those playlist files are played in order?

I tried making making http requests for a playlist file, directly from liquidsoap, as suggested like in the manual (p. 182) like...

Making HTTP requests to other sites. We recall that Liquidsoap has integrated
support distant files, in particular through the http and https protocols. This
means that you can load a playlist on some web server by writing something
like
radio = playlist("http://www.some-server.com/playlist")

... But found that this is too dangerous to use -- if the server doesn't respond quickly, it backs up the entire liquidsoap process and causes problems....

... So this is why I implemented a solution with python running in a separate docker container, doing all the communication with the server, and then writing a new playlist file.... But (as shown in this thread) that has some serious pitfalls also -- liquidsoap doesn't seem to recognize changes in "atomically" written files, and if a file is not "atomically" written, then it might be played out of order.

Has anyone found a reliable way to write new playlist files and have them picked up without problems by liquidsoap?

[vitoyucepi]

Reproduced in 2.2.4-1.

1. main.liq

   settings.log.level := 1
   s = playlist("/tmp/test/playlist.m3u", mode="normal", prefetch=4, reload_mode="watch")
   
   s.on_track(fun(l) -> print(l["filename"]))
   
   output.dummy(s, fallible=true)

2. compose.yaml

   services:
     liquidsoap:
       image: savonet/liquidsoap:v2.2.4
       command:
         - /tmp/test/main.liq
       volumes:
         - ./:/tmp/test

3. for i in $(seq 1 9);do 
     ffmpeg -f lavfi -i "sine=frequency=1000:duration=1" \
       -c:a libmp3lame -b:a 128k \
       -y $i.mp3;
   done

4. playlist1.m3u

   1.mp3
   3.mp3
   4.mp3
   5.mp3
   6.mp3
   7.mp3
   8.mp3
   9.mp3
   

5. while true; do
     cat playlist1.m3u > playlist.m3u;
     sleep 3.5;
   done

6. docker compose up

Result may look like this:

/tmp/test/1.mp3
/tmp/test/2.mp3
/tmp/test/3.mp3

/tmp/test/2.mp3
/tmp/test/1.mp3
/tmp/test/2.mp3
/tmp/test/3.mp3

/tmp/test/1.mp3
/tmp/test/2.mp3
/tmp/test/3.mp3