New Release?

[aarondvail]

Checklist

• I'm asking a question
• I've looked through the README and FAQ for similar questions
• I've searched the bugtracker for similar questions including closed ones

Question

Preface:
There have been a number of fixes done since the last release (2021-12-17) {over a years worth} The error messages refer using youtube-dl -U (which is already in place). All the errors I am experiencing are already closed issues, but no no release so youtube-dl -U won't fix it. Issue #31583

Question:
Can either the instructions be updated (and the error message refering to youtube-dl -U be dropped) to "install" via a github pull from Master

or

Can a new release be cut?

Thanks

[dirkf]

A new release is overdue but depends on solving certain cross-platform issues in the CI build process. Meanwhile for the main platforms, see #31530 for update hints.

[Grub4K]

What are those issues that need sorting out? Is there an issue/PR tracking this? Creating a PR could bring in people that are willing to help with the process (like myself)

[dirkf]

#30644, but it should probably be refreshed now.

This is very far down my stack today, but as I recall there were various possible blockers:

• change to pytest for Py3.10+ (mostly achieved in my test branch with a pytest build that should run on the supported platforms)
• especially Jython was tricky, if anyone cares (but we have found a Py2.6 user today)
• various workflow upgrades because of GH dependencies
• the usual maze of twisty build system dependencies, all alike.

Obviously I need to dive back into this especially in view of yt-dlp/yt-dlp#6220, which I would also like to acquire, if only for more regular versioned releases.

[dirkf]

But isn't the auto-update installer already there, depending on the how the yt-dl instance was installed? Once a build has been deployed to the release page and to PyPi, it can be pulled by almost all installation types.

The problem being addressed is really to get the small number of builds in the release tested on all the different platforms.

[Grub4K]

Will #30644 keep tracking this and should I post my remarks on there?

I think maybe instead of perfecting the release process it might make sense to release something so as to get updates to the end user. If releases were not automated, whats problematic about building like that, then passing those executables to the user to be tried on various platforms? The related release can be marked as pre-release if need be, but that way there at least are executables which will work for most people while we can get reports for the edge cases and squash those specifically.

I would also advise to look into the release.yml/release-nightly.yml workflow in yt-dlp/yt-dlp#6220. It is crafted so that it creates a release in the yt-dlp style regardless of build output, adding all of the build artifacts from build.yml, it's checksums and the _update_spec. A build.yml satisfying simplistic conditions can be created at first, which would build some more general artifacts, gathering info about possible incompatibilities.

These are the repository configuration options of the release and nighly action:

vars.PUSH_VERSION_COMMIT: Update master branch with a push
vars.BUILD_NIGHTLY: Build a nightly release on every commit

# Both need to be set to push build to archive repo as release
vars.ARCHIVE_REPO: repo like `yt-dlp/yt-dlp-builds`
secrets.ARCHIVE_REPO_TOKEN: token with contents:write

# Push to PyPI and brew if each is set
secrets.PYPI_TOKEN: Token for PyPI
secrets.BREW_TOKEN: Token for brew

update-version.py would have to be backported since the format changed and it's output is now a bit different. Maybe the naming could also be changed to update_version.py?

make_changelog.py, while configurable, is most likely not configurable enough to be able to support the ydl changelog. The author assumption would have to be corrected (Authored by: ) or the changelog would have to be crafted and read manually for now (would be simple to do a shim that just reads from ChangeLog for now).

• change to pytest for Py3.10+ (mostly achieved in my test branch with a pytest build that should run on the supported platforms)

Are the test adjustments needed for the release for now? Is that preparation to be testing if the builds would work as expected? If so, I'd think that can be moved back for now?

• especially Jython was tricky, if anyone cares (but we have found a Py2.6 user today)

Do we need to supply youtube-dl builds that embed JPython? Otherwise I can't really see the direct relation of that and a release.

• various workflow upgrades because of GH dependencies

This should be handled by 6220. I can help with the remaining blockers there.

• the usual maze of twisty build system dependencies, all alike.

Yes, understandable. If we encounter anything in particular we might need to push that back until somebody with experience on that system or with those dependencies can chime in. I think those issues will become apparent once we create/try creating the actual build.yml.

The steps that I would propose for now are:

• Create the release.yml and adjust to the format we would want
• Fix/port update_version.py and make_changelog.py
• Create a simplified build.yml handling the most common targets.
• Publish a pre-release with the created executables (people will be _{^{sort of}} happy here)
• Fix problems, improve and extend build
• Employ extended testing and the sort

Feel free to contact me via E-Mail if you would like to move to a different communication channel for this. Alternatively let me know of anything else I can do.

[dirkf]

Just to show that a release might appear quite soon, and not in the far future.

From #31530 (comment)

... I was wonder why so big deal to create new release. I came across to #30568 and #31535 and I hope I understand. @dirkf decided to support old unsupported versions of python and probably this is reason why is not possible to use continuous integration to build and release new version.

No, although it's true that the GH workflow ecosystem forces you into an upgrade race as various dependencies are moved to new versions, much as in the rest of the s/w world. Actually, for yt-dl itself, Py2.7 is fine and dandy for the most part: the #31530 fix did reveal some 2.6 hold-outs but also surprisingly many verbose logs showing 2.7 still in use.

The previous build and release system, not well documented, depended on some local setup that had been passed down through the former maintainers. Lured by the gleaming promise of online builds, I completely failed to reproduce the necessary local setup. That brings us to last month.

Now, I find that GH is about to junk ("deprecated") versions of Linux that support the Pythons needed for yt-dl's CI testing. So we'll have to rework that completely using our own runners, etc. This is too hard for now and the existing tests will have to do: boo to Python >= 3.10 (though I don't expect significant issues there). Therefore the build and release workflow doesn't depend on the test workflow.

I took the yt-dlp scheme as a pattern, as suggested, rather than building directly on the excellent work in #30644.

Thus build.yml is a simplified script merging the one from #30644 with the yt-dlp-derived version, removing the bits that are handled elsewhere in the yt-dlp-derived workflows. For now, only the existing targets will be built, ie: POSIX (Linux/macOS/BSD/etc) self-extracting executable, matching .tar.gz, Windows self-extracting executable with built-in Python 3.4.4. As the unofficial nightly builds seem unproblematic I don't expect to see a lot of build issues.

Users of new macOSes that have no Python will still have to install a Python and arrange for it to be found by the shebang of the POSIX build.

Currently a tricky part is make_changelog.py. yt-dl wants a plain-text changelog, but the Changelog class in the script can be specialised and the punch-line becomes:

print((Changelog if args.markdown else PlainChangelog)(commits.groups(), args.repo))

Less simple, the 200 commits since the last release aren't as disciplined as the yt-dlp script expected. I think the only practical design for grouping these commits automatically is to use git commands to identify what each commit changed and then map back from there. For instance:

• git log --format= --dirstat=files commit gives a count of changed files by directory that can be sorted to pick the significant directory, eg if the top directory in the commit is youtube_dl/extractors, the group should be CommitGroup.EXTRACTOR
• git log --format= --numstat commit gives a count of lines changed in each file that can be sorted to give the most changed file (should be relative to the line count of the file) and the unsuffixed basename of that file is a useful input to classification.

Given this, the solution is to take some output of an as-good-as-can-be-expected hack on the existing script and modify the CI process to accept some manually tweaked changelog delta that will be prepended with its version ID to the ChangeLog file and uploaded in the release notes.

Obviously any other suggestions would be happily considered.

[ReenigneArcher]

Good to hear there is a path forward!

Now, I find that GH is about to junk ("deprecated") versions of Linux

Yes, I believe the 18.04 runner is being removed in April if I remember correctly. I assume you're aware of this action to setup Python? https://github.com/actions/setup-python ... Example: https://github.com/LizardByte/Themerr-plex/blob/master/.github/workflows/CI.yml#L36-L49

Ubuntu 20.04 and 22.04 also still have python 2.7 that can be manually installed. Here's a build in an Ubuntu 22.04 dockerfile that uses Python 2.7 for reference: https://github.com/LizardByte/PlexyGlass/blob/nightly/Dockerfile#L4-L24 ... Debian/Ubuntu has too many packages that still rely on Python 2.x to completely remove it.

I didn't see a mention of publishing to PyPi in your comment. Are you still going to publish there?

[Grub4K]

Is there a new PR, some branch or similar that I can take a look at and implement missing factors to help out?

I think the only practical design for grouping these commits automatically is to use git commands to identify what each commit changed and then map back from there.

This is why I suggested shimming it for now, as manual mapping will be required anyways. Regarding future commits, it might make sense to adjust the commit message to be closer to dlp format OR (I thought about it before but scrapped it since its easier to just rely on message only) search for the files using file or text search if the format should be kept the same ([YouTube] => extractor/youtube.py => EXTRACTOR). Let me know of any changes to that script that I could do.

I didn't see a mention of publishing to PyPi in your comment. Are you still going to publish there?

Assuming that this is building on the release workflow of dlp the PyPI publish is integrated in those steps

It might make sense to diff with dlp build workflow as some new changes were coming. A relevant branch can be found here (though many changes were directly related to nightly and --update-to and are therefore irrelevant)

[MikeMcQuaid]

Homebrew project leader here 👋🏻. I think even getting a tagged prerelease here with failing CI will allow Homebrew to update to a version that will work for downloading from YouTube and stop you getting so many issues filed by our users.

[dirkf]

Thanks for your interest!

Almost any commit to master is meant to be reliable and its CI test will have succeeded, having been exercised on another branch first.

If you would like a special tag applied to some commit, that's easy. If you're able to take an unofficial nightly build (from a downstream), which has the advantage of reporting a different version and showing a disclaimer, that's even easier.

[dirkf]

Is there a new PR, some branch or similar that I can take a look at and implement missing factors to help out?

Upcoming.

This is why I suggested shimming it for now, as manual mapping will be required anyways. ...

I suppose that a manually adjusted changelog delta could be offered to the workflow as a var, even if it's ~15kB?

I didn't see a mention of publishing to PyPi in your comment. Are you still going to publish there?

Assuming that this is building on the release workflow of dlp the PyPI publish is integrated in those steps

For now I commented out the package management interfaces. The existing PyPi update process would have been manual AFAIK with any account details held by former maintainers; presumably we can regenerate them if they can't be recalled.

It might make sense to diff with dlp build workflow as some new changes were coming. A relevant branch can be found here (though many changes were directly related to nightly and --update-to and are therefore irrelevant)

The workflow files

	new file:   .github/workflows/build.yml
	new file:   .github/workflows/publish.yml
	new file:   .github/workflows/release.yml
	new file:   devscripts/make_changelog.py
	modified:   devscripts/make_issue_template.py
	new file:   devscripts/update_formulae.py
	new file:   devscripts/update_version.py
	new file:   devscripts/utils.py

all started out from the yt-dlp equivalents in a meld session. So yes.

[MikeMcQuaid]

Thanks for your interest!

Thanks for your great software that I personally use!

If you would like a special tag applied to some commit, that's easy.

Yes, our hope would be any sort of tag applied to a release that would show up in https://github.com/ytdl-org/youtube-dl/releases that we could update to 🙇🏻

[dirkf]

Which release asset is used to create the Homebrew package?

[MikeMcQuaid]

The pypi URL: https://github.com/Homebrew/homebrew-core/blob/a44ce7e234fddf054ba589cc682d73d9d8499771/Formula/youtube-dl.rb#L6

[kloczek]

It would be really nice to have regular release with version git tag in git repo ..

[dirkf]

A nightly release stays up unless some terrible bug is found that means downloading its assets would be a bad idea. In that sense they're all better than the 2021 release. Ofc you can still check out the bad tag and run the broken release if you want.

[rofl0r]

A nightly release stays up unless some terrible bug is found that means downloading its assets would be a bad idea.

it's not about bugs but about a reproducible build. i'm maintaining a distro and a package build script contains an upstream URL for the source tarball, a checksum of the tarball, and a few script instructions to build and install it. that means if a nightly tarball is only available for half a week the distro package recipe is basically useless.

[dirkf]

Check out the nightly release pages. I think you'll see that what you want is available, so long as GitHub keeps hosting the pages. Obvs don't build in a brand-new release in case some anomaly ((c) Post Office Ltd) has got through the CI and local tests.

[rofl0r]

yeah, looks good. the nightlies from 2023 are still available. i guess it'd be good to make a link on the homepage or make a "final" tag here where you refer to the nightlies repo, so everyone seeing the new "release" here gets the news.

[Mis012]

hm, couldn't find this by searching the issues...

It seems that some if not all of the issues mentioned are with making a binary release, but a new tag without any artifacts would be more than sufficient for having something for linux distros to upgrade to if they don't want to chase nightlies. People who use the prebuilts are presumably going to use nightlies anyway, they don't really care whether something is an official release.

[ddelange]

easiest installation currently (does not require git) is to go:

pip install https://github.com/ytdl-org/youtube-dl/archive/refs/heads/master.zip

or to add

youtube-dl @ https://github.com/ytdl-org/youtube-dl/archive/refs/heads/master.zip

to your requirements.txt or your library requirements

[Mis012]

I don't have an issue installing youtube-dl, it's just that the README says that "nothing can be done" when a distro has outdated package, but making a release tag is definitely something that has to be done if the distro only packages release tags

I'm pretty sure that at least in the single case of Tumbleweed, the reason the package is stuck on the 2021 version is that from their point of view there is no newer version.

as a Linux user, I obviously don't have issues with using git since I can just install git using the package manager like god intended (and I already have it installed anyway), obviously on windows installing git is kinda annoying though so is installing python fwiw

[dirkf]

There's nothing to stop anyone packaging a nightly build (it has tags, etc) but it'll immediately be obsolete because of web site turnover.

This is why the hassle of porting back the build process from the nightly repo hasn't seemed worthwhile.

... obviously on windows installing git is kinda annoying though so is installing python fwiw

Unlike the POSIX-ish single file bundled executable, the single file Windows build contains a Python executable and only needs the user to install ffmpeg for a good experience. The POSIX-ish targets are expected to have or be able to install a python program.

[ddelange]

@dirkf if you set contents: write permissions, the github actions pipeline could also create releases and upload (binary) assets to them 🤔 for instance this step uploads to existing release, or creates a release (named after run_id) and uploads to it

[dirkf]

I'm not sure what problem that addresses. There is already an Actions workflow that creates nightly releases. A new release for every commit is not wanted. If you're suggesting cross-posting nightly releases, it's not just a matter of posting the release in the main repo: the releases are linked to the repo, so the nightly release updates from the nightly repo. It would be confusing for people to download a release and then have it update from a different source.

[Mis012]

hm, indeed https://github.com/ytdl-org/ytdl-nightly/releases has source tarballs... I guess it's just not on the radar for some packagers :/

[Grub4K]

If the nightly releases are the source of truth, this repository loses its value. It on the other hand also does not correlate to to a release being blocked on this repo. Nightly builds are being made and tested and using a workflow like that on this repository will require little changes. You can omit the duplication that does not match the actual development state since the two repositories are diverged. Instead, consider creating a prerelease on this repository, with future posibility of making it a regular release. Automation can be fine tuned to achieve this easily.

I will offer my help on this topic again: Feel free to contact me through email if you want any help or someone to discuss with in private

[dirkf]

Technically not much would have to be done to pull the nightly workflow into the main repo, changing it to make pre-releases, but it would actually have to be done and with more detailed work, at least initially and especially when designating such a pre-release as a stable release. First off, the actions in the nightly workflow need to be brought up-to-date in GitHub's version hell actions ecosystem.

Otherwise I just suggest that, if a packager needs an up-to-date tagged commit to ingest into its workflow, the nightly repo is a good choice; a packager can install the tar.gz as if from the main repo but presumably would have to patch the build process if delivering a single-file build for which ytdl_is_updatable() would normally be true regardless of the source repo.

[seproDev]

Replying to:

There have been releases since last year. — #32842 (comment)

No, nightlies tagged in a different repo are not releases. This is exactly what led to the current state:

• Alpine: removed
• Arch: removed
• Brew: marked deprecated
• Debian: installs yt-dlp instead
• FreeBSD: removed
• Gentoo: removed
• NixOS: marked unmaintained
• Ubuntu: installs yt-dlp instead

[bashonly]

Most downstream packagers are not going to be OK with arbitrarily selecting a pre-release from a repo other than the actual upstream repo; some have policies that forbid doing this. E.g. #31585 (comment)

[ReenigneArcher]

Replying to:

There have been releases since last year. — #32842 (comment)

No, nightlies tagged in a different repo are not releases. This is exactly what led to the current state:

* Alpine: [removed](https://gitlab.alpinelinux.org/alpine/aports/-/commit/6ffe7d0d310c249f1cd00f74a4a0b7aabeb95fe4)

* Arch: [removed](https://gitlab.archlinux.org/archlinux/packaging/state/-/commit/d9639dd376d28c9627f2ee25887216f2bf66125d)

* Brew: [marked deprecated](https://github.com/Homebrew/homebrew-core/commit/556b8ead35aa52c527a10d59f87e2b84ecd80bf1)

* Debian: [installs yt-dlp instead](https://salsa.debian.org/debian/youtube-dl/-/commit/2a825917bfe302784b03b8531b9f80b4c45adae4)

* FreeBSD: [removed](https://cgit.freebsd.org/ports/commit/www?id=336dff981d65c244979d7f8ceb8bcde06033420a)

* Gentoo: [removed](https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a91f98e856723f42c8636180f2f3a5bf829ff899)

* NixOS: [marked unmaintained](https://github.com/NixOS/nixpkgs/commit/9f82aef9b451e6ca6488a332ed407b98caf38129)

* Ubuntu: [installs yt-dlp instead](https://packages.ubuntu.com/mantic/youtube-dl)

Also, Kodi is struggling a lot to keep this updated https://github.com/xbmc/repo-scripts/commits/matrix/script.module.youtube.dl

[dirkf]

There are no pre-releases. A nightly release may be retrospectively marked pre-production and even have artefacts removed if it includes a significant bug that is fixed in a later release or by falling back to a previous one. But the proposal is to select a nightly release for packaging that is not brand new so as to avoid any possible lemon. As packagers have to tweak the updating and version code, it doesn't really matter whether they use the main repo or the nightly repo, since this is the only difference in the executable code.

The complaint linked above referred to the ytdl-patched/youtube-dl (superseded by the nightly build repo here). Unlike ytdl-patched/youtube-dl, the nightly build repo is clearly an adjunct of the development repo.

Maybe a nightly tag (s/t like 2024.07.11-nightly, on the understanding that this does not signify any sort of pre-release, despite SemVer) could be back-propagated to the equivalent main repo commit. After rebasing the equivalent commits have different SHAs (or am I gitting something wrong?): this might confuse someone who is not familiar with the code. The equivalent main repo commit is already shown in the --verbose output.

[Grub4K]

The complaints linked are about tagging a regular release in THIS repo. None other. The project is this repository, and like I said before, rebasing or adding a nightly commit means they are DISJOINTED, and can no longer be seen as the same project / state.

Creating a regular, stable git tag on this repository is trivial. Downstream packagers can do the remaining work. No binaries are required.

Using the nightly build workflow on this repository to create binaries is optional, albeit also trivial. Interestingly doing both on workflow_dispatch when state is deemed stable is already equivalent to a regular release.

Trying to reason against doing either is an interesting choice though

[dirkf]

So there is now https://github.com/ytdl-org/youtube-dl/tag/2024.07.11-nightly and so also https://github.com/ytdl-org/youtube-dl/archive/refs/tags/2024.07.11-nightly.tar.gz. Let's see if that helps. I have avoided, once GH allowed me to delete it, having an actual release page here since it isn't one.

Otherwise, or anyway, I stand by the previous comments

• the nightly repo is an "official" adjunct that would be perfectly adequate for packaging
• Mike McQuaid was specifically (and not unreasonably) rejecting an "unofficial" repo.

... doing both on workflow_dispatch when state is deemed stable is already equivalent to a regular release.

For packagers? I don't see how that applies otherwise.

[Grub4K]

For packagers? I don't see how that applies otherwise.

Tried to clarify in #32876. Running that workflow when the repository is deemed stable will produce a stable release.

[hudsonm62]

Will the pypi packages be updated with any missed Nightly releases? Hasn't been updated since 2021.
I want to use a requirements file, kept updated via dependabot or renovate, but I obviously don't want the version from all the way back then

[ReenigneArcher]

@hudsonm62 If you want @dependabot to keep it updated, you can do what I do. Add it as a submodule.

• gitmodules file: https://github.com/LizardByte/Themerr-plex/blob/master/.gitmodules
• the submodule: https://github.com/LizardByte/Themerr-plex/tree/master/third-party
• dependabot config: https://github.com/LizardByte/Themerr-plex/blob/c948caa73991e05cebe8b24b88bce5ccb6bede4c/.github/dependabot.yml#L43
• requirements file: https://github.com/LizardByte/Themerr-plex/blob/c948caa73991e05cebe8b24b88bce5ccb6bede4c/requirements.txt#L16

[hudsonm62]

ah good idea, thank you sir I shall look into this for my use case
totally forgot submodules existed tbh