[WIP] Migrate Away From Spotify Direct Download To Spotdl (yt-dlp backed)

Dockerfile

@@ -1,12 +1,21 @@
-FROM python:3.12-bookworm
+FROM python:3.13-bookworm
 
 ENV PYTHONDONTWRITEBYTECODE=1
 ENV PYTHONUNBUFFERED=1
 
 WORKDIR /code
-RUN apt-get update && apt-get install -y ffmpeg aria2
 
+# Update all image packages & install user-friendly cli editor
+RUN apt-get update && apt-get upgrade -y && apt install nano -y
+
+# Install deps
 COPY requirements.txt /code/
 RUN pip install -r requirements.txt
 
+# Pre-cache ffmpeg
+RUN spotdl --download-ffmpeg
+
+# Cleanup any APT leftovers
+RUN apt clean && rm -rf /var/cache/apt/archives /var/cache/apt/lists
+
 COPY ./spotify_library_sync/ /code/

README.md

@@ -73,17 +73,12 @@ An example docker-compose file is included in this repo that can be dropped into
 > [!NOTE]
 > This is currently only configured for running on Unix-based systems due to the default configuration folder, but if anyone wants to update this doc with the correct configuration I'd welcome the upload!
 >
-> If developing on Windows, using WSL should work just fine, installing python, ffmpeg, and aria2c via `apt`
-
-1. Install Python 3.7 or higher
-2. Add [FFmpeg](https://ffmpeg.org/download.html) to PATH
-    * Older versions of FFmpeg may not work
-3. Place your cookies in `/config/` as `cookies.txt`
-    * You can export your cookies by using this Google Chrome extension on Spotify website: https://chrome.google.com/webstore/detail/open-cookiestxt/gdocmgbfkjnnpapoeobnolbbkoibbcif. Make sure to be logged in.
-4. Install spotify-aac-downloader using pip
-    ```bash
-    pip install spotify-aac-downloader
-    ```
+> If developing on Windows, using WSL should work just fine, installing python via `apt`
+
+1. Install Python 3.10 or higher
+2. Place your cookies in `/config/` as `cookies.txt`
+    * You can export your cookies by using this Google Chrome extension on Spotify website: https://chromewebstore.google.com/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc. Make sure to be logged in.
+
 
 5. (Optional) I personally like setting huey into instant-run mode so as to not have a separate worker, but if you are using this for non-development I would advise against this due to degraded performance!
 This should be placed in `/config/` as `settings.yaml`
@@ -100,81 +95,19 @@ default:
 bash -c "python manage.py migrate && python manage.py runserver 0.0.0.0:5000"
 ```
 
-## Extracting A Manual WVD
-While the wvd that's hard-coded into this project _should_ work, you can still extract the native one from your device via the following instructions:
- - To get a .wvd file, you can use [dumper](https://github.com/wvdumper/dumper) to dump a L3 CDM from an Android device. Once you have the L3 CDM, use pywidevine to create the .wvd file from it.
-  1. Install pywidevine with pip
-    ```bash
-    pip install pywidevine pyyaml
-    ```
-  2. Create the .wvd file
-    ```bash
-    pywidevine create-device -t ANDROID -l 3 -k private_key.pem -c client_id.bin -o .
-    ```
-
-> [!NOTE]
-> This needs to be polished and likely will not work until the below Configuration section is merged into the `settings.yaml`, but should not impede the ability to use this.
-> 
-> Place your .wvd file in `/config/` as `device.wvd`, and specify this in the configuration
-
-## Configuration (IGNORE)
+## Configuration (IGNORE, Half-updated and docker is the intended use)
 > [!CAUTION]
 > TO BE MIGRATED TO `settings.yaml` -- THESE WILL CHANGE NOTHING PRESENTLY
 
 spotify-aac-downloader can be configured using the command line arguments or the config file. The config file is created automatically when you run spotify-aac-downloader for the first time at `~/.spotify-aac-downloader/config.json` on Linux and `%USERPROFILE%\.spotify-aac-downloader\config.json` on Windows. Config file values can be overridden using command line arguments.
 | Command line argument / Config file key                         | Description                                                           | Default value                                       |
 | --------------------------------------------------------------- | --------------------------------------------------------------------- | --------------------------------------------------- |
-| `-f`, `--final-path` / `final_path`                             | Path where the downloaded files will be saved.                        | `./Spotify`                                         |
-| `-t`, `--temp-path` / `temp_path`                               | Path where the temporary files will be saved.                         | `./temp`                                            |
 | `-c`, `--cookies-location` / `cookies_location`                 | Location of the cookies file.                                         | `./cookies.txt`                                     |
-| `-w`, `--wvd-location` / `wvd_location`                         | Location of the .wvd file.                                            | `null`                                              |
+| `-w`, `--po-token` / `po_token`                                 | PO Token for your Youtube Music account                                            | `null`                                              |
 | `--config-location` / -                                         | Location of the config file.                                          | `<home_folder>/.spotify-aac-downloader/config.json` |
-| `--ffmpeg-location` / `ffmpeg_location`                         | Location of the FFmpeg binary.                                        | `ffmpeg`                                            |
-| `--aria2c-location` / `aria2c_location`                         | Location of the aria2c binary.                                        | `aria2c`                                            |
-| `--template-folder-album` / `template_folder_album`             | Template of the album folders as a format string.                     | `{album_artist}/{album}`                            |
-| `--template-folder-compilation` / `template_folder_compilation` | Template of the compilation album folders as a format string.         | `Compilations/{album}`                              |
-| `--template-file-single-disc` / `template_file_single_disc`     | Template of the song files for single-disc albums as a format string. | `{track:02d} {title}`                               |
-| `--template-file-multi-disc` / `template_file_multi_disc`       | Template of the song files for multi-disc albums as a format string.  | `{disc}-{track:02d} {title}`                        |
-| `--download-mode` / `download_mode`                             | Download mode.                                                        | `ytdlp`                                             |
-| `-e`, `--exclude-tags` / `exclude_tags`                         | List of tags to exclude from file tagging separated by commas.        | `null`                                              |
-| `--truncate` / `truncate`                                       | Maximum length of the file/folder names.                              | `40`                                                |
 | `-l`, `--log-level` / `log_level`                               | Log level.                                                            | `INFO`                                              |
-| `-p`, `--premium-quality` / `premium_quality`                   | Download in 256kbps AAC instead of 128kbps AAC.                       | `false`                                             |
-| `-l`, `--lrc-only` / `lrc_only`                                 | Download only the synced lyrics.                                      | `false`                                             |
 | `-n`, `--no-lrc` / `no_lrc`                                     | Don't download the synced lyrics.                                     | `false`                                             |
-| `-s`, `--save-cover` / `save_cover`                             | Save cover as a separate file.                                        | `false`                                             |
 | `-o`, `--overwrite` / `overwrite`                               | Overwrite existing files.                                             | `false`                                             |
 | `--print-exceptions` / `print_exceptions`                       | Print exceptions.                                                     | `false`                                             |
 | `-u`, `--url-txt` / -                                           | Read URLs as location of text files containing URLs.                  | `false`                                             |
 | `-n`, `--no-config-file` / -                                    | Don't use the config file.                                            | `false`                                             |
-
-### Tag variables
-The following variables can be used in the template folder/file and/or in the `exclude_tags` list:
-- `album`
-- `album_artist`
-- `artist`
-- `comment`
-- `compilation`
-- `copyright`
-- `cover`
-- `disc`
-- `disc_total`
-- `isrc`
-- `label`
-- `lyrics`
-- `media_type`
-- `rating`
-- `release_date`
-- `title`
-- `track`
-- `track_total`
-
-### Download mode
-> [!NOTE]
-> If using the docker image, both ytdlp and aria2c will work by default.
-
-The following download modes are available:
-* `ytdlp`
-* `aria2c`
-    * Faster than `ytdlp`
-    * Can be obtained from here: https://github.com/aria2/aria2/releases

requirements.txt

@@ -9,6 +9,8 @@ huey
 jinja2
 pybase62
 pylint-django
-pywidevine
 pyyaml
-yt-dlp
+tinytag
+# spotdl
+git+https://github.com/Kyrluckechuck/spotify-downloader.git@add-yt-dlp-extractor-support#egg=spotdl
+

spotify_library_sync/downloader/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.0.0.1"

spotify_library_sync/downloader/default_download_settings.py

@@ -0,0 +1,78 @@
+DEFAULT_DOWNLOAD_SETTINGS = {
+    "client_id": "5f573c9620494bae87890c0f08a60293",
+    "client_secret": "212476d9b0f3472eaa762d90b19b0ba8",
+    "auth_token": None,
+    "user_auth": False,
+    "headless": True,
+    "cache_path": None,
+    "no_cache": True,
+    "max_retries": 3,
+    "use_cache_file": False,
+    "audio_providers": [
+        "youtube-music"
+    ],
+    "lyrics_providers": [
+        "genius",
+        "azlyrics",
+        "musixmatch"
+    ],
+    "genius_token": "alXXDbPZtK1m2RrZ8I4k2Hn8Ahsd0Gh_o076HYvcdlBvmc0ULL1H8Z8xRlew5qaG",
+    "playlist_numbering": False,
+    "playlist_retain_track_cover": False,
+    "scan_for_songs": False,
+    "m3u": None,
+    "output": "/mnt/music_spotify/{artist}/{album}/{artists} - {title}.{output-ext}",
+    "overwrite": "skip",
+    "search_query": None,
+    "ffmpeg": "ffmpeg",
+    "bitrate": "disable",
+    "ffmpeg_args": None,
+    "format": "m4a",
+    "save_file": None,
+    "filter_results": True,
+    "album_type": None,
+    "threads": 4,
+    "cookie_file": None,
+    "restrict": None,
+    "print_errors": True,
+    "sponsor_block": False,
+    "preload": False,
+    "archive": None,
+    "load_config": True,
+    "log_level": "INFO",
+    "simple_tui": True,
+    "fetch_albums": False,
+    "id3_separator": "/",
+    "ytm_data": False,
+    "add_unavailable": False,
+    "generate_lrc": False,
+    "force_update_metadata": False,
+    "only_verified_results": False,
+    "sync_without_deleting": False,
+    "max_filename_length": None,
+    "yt_dlp_args": None,
+    "detect_formats": None,
+    "save_errors": None,
+    "ignore_albums": None,
+    "proxy": None,
+    "skip_explicit": False,
+    "log_format": None,
+    "redownload": False,
+    "skip_album_art": False,
+    "create_skip_file": False,
+    "respect_skip_file": False,
+    "sync_remove_lrc": False,
+    "web_use_output_dir": False,
+    "port": 8800,
+    "host": "localhost",
+    "keep_alive": False,
+    "enable_tls": False,
+    "key_file": None,
+    "cert_file": None,
+    "ca_file": None,
+    "allowed_origins": None,
+    "keep_sessions": False,
+    "force_update_gui": False,
+    "web_gui_repo": None,
+    "web_gui_location": None
+}

spotify_library_sync/downloader/downloader.py

@@ -0,0 +1,109 @@
+import functools
+import json
+import re
+
+from library_manager.models import Album, Artist, ContributingArtist, DownloadHistory, Song, TrackedPlaylist
+from . import utils
+
+from spotdl.utils.spotify import SpotifyClient
+
+class Downloader:
+    def __init__(self, spotipy_client: SpotifyClient):
+        self.spotipy_client = spotipy_client
+
+    def get_artist_albums(self, artist_gid: str) -> list[Album]:
+        """Get all albums (including EPs and Singles) for this artist
+
+        Args:
+            artist_gid (str): The artist GID as supplied by Spotify
+
+        Returns:
+            list[str]: an array of urls to each album for the artist
+        """
+        artist = Artist.objects.get(gid=artist_gid)
+        albums_to_create_or_update: list[dict] = []
+        artist_uri = utils.gid_to_uri(artist_gid)
+
+        album_iterator = self.spotipy_client.artist_albums(artist_uri, limit=50)
+
+        while album_iterator is not None:
+            for album in album_iterator['items']:
+                new_or_updated_album_data: dict = {
+                    'spotify_gid': album['id'],
+                    'artist': artist,
+                    'spotify_uri': album['uri'],
+                    'total_tracks': album['total_tracks'],
+                    'name': album['name'],
+                }
+
+                albums_to_create_or_update.append(new_or_updated_album_data)
+
+            album_iterator = self.spotipy_client.next(album_iterator)
+
+        if len(albums_to_create_or_update) == 0:
+            return []
+
+        albums: list[Artist] = Album.objects.bulk_create(
+            [Album(**album) for album in albums_to_create_or_update],
+            update_conflicts=True,
+            unique_fields=["spotify_gid"],
+            update_fields=albums_to_create_or_update[0].keys(),
+        )
+        return albums
+
+    def get_track(self, track_id: str) -> dict:
+        return self.spotipy_client.track(track_id)
+
+    def create_album(self, album_id: str, artist: Artist) -> Album:
+        album_details = self.get_album(album_id)
+        album = Album.objects.create(
+            spotify_gid=album_details['id'],
+            artist=artist,
+            spotify_uri=album_details['uri'],
+            total_tracks=album_details['total_tracks'],
+            name=album_details['name'],
+        )
+        return album
+
+    @functools.lru_cache()
+    def get_album(self, album_id: str) -> dict:
+        album = self.spotipy_client.album(album_id)
+        album_track_iterator = self.spotipy_client.next(album["tracks"])
+
+        while album_track_iterator is not None:
+            album["tracks"]["items"].extend(album_track_iterator["items"])
+            album_track_iterator = self.spotipy_client.next(album_track_iterator)
+        return album
+
+    def get_playlist(self, playlist_id: str) -> dict:
+        playlist = self.spotipy_client.playlist(playlist_id)
+        playlist_iterator = self.spotipy_client.next(playlist["tracks"])
+
+        while playlist_iterator is not None:
+            playlist["tracks"]["items"].extend(playlist_iterator["items"])
+            playlist_iterator = self.spotipy_client.next(playlist_iterator)
+        return playlist
+
+    def get_download_queue(self, url: str) -> list[dict]:
+        uri = re.search(r"(\w{22})", url).group(1)
+        download_queue = []
+        if "album" in url:
+            download_queue.extend(self.get_album(uri)["tracks"]["items"])
+        elif "track" in url:
+            download_queue.append(self.get_track(uri))
+        elif "playlist" in url:
+            raw_playlist = self.get_playlist(uri)["tracks"]["items"]
+            for i in raw_playlist:
+                i['track']['added_at'] = i['added_at']
+            download_queue.extend(
+                [i["track"] for i in raw_playlist]
+            )
+        else:
+            raise Exception("Not a valid Spotify URL")
+        return download_queue
+
+    def get_song_core_info(self, metadata: dict) -> str:
+        return {
+            'song_gid': utils.uri_to_gid(metadata['id']),
+            'song_name': metadata['name'],
+        }