[Discussion] State of repository and some thoughts on changes

[Ash258]

No description provided.

[Calinou]

There are some manifests, which does not look good and are really messy.

We could use a tool to format and sort properties in all manifests at once.

[Ash258]

Sure. .\bin\format.ps1 or ConvertTo-PrettyJson doing its job with some indentation / strings array formating. But sorting / organizing properties would require to parse json files into some Class (in best way) with all it's properties and then just print it to file in given order.

[leonyu]

Most of the sublime settings can be superseded by .editorconfig which is supported natively by number of IDEs/editors, and has plugins for most popular IDEs/editor.

IMO .vscode/settings.json should probably go as well, it can just be a CI hook, so as to work for everyone regardless of editor. (If your commit fails lint, it shouldn't be committed.)

Also I wish there is a point system for quality of the app, so we can promote and demote apps to and from extras and versions. Oddities I found:

• ConEmu is not in the main bucket, but its fork Cmder is
• python27 is in the versions bucket, but node-lts is in the main bucket. (Last I checked, Node actually requires Python 2.7 if you want to write C++ bindings)

But that last point might be a bigger undertaking.

[kiedtl]

That write-hosts are nice for users, but not much for "developers", because it's really pain to capture write-hosts.

I can't agree with that. Scoop should be built with ease of use for the end user, not a developer.

Yes, it's a pain to capture write-hosts, and prior to PSH 5, impossible. But there are plenty of ways around this.

[r15ch13]

Renaming examples:

• lukesampson/scoop-extras > ScoopInstaller/Extras
• scoopinstaller/scoop.sh > leave as is

Already changed:

• scoopinstaller > ScoopInstaller
• scoopinstaller/scoop-main > ScoopInstaller/Main
• scoopinstaller/php > ScoopInstaller/PHP
• scoopinstaller/versions > ScoopInstaller/Versions
• scoopinstaller/binary-mirror > ScoopInstaller/Binary
• scoopinstaller/shimexe > ScoopInstaller/Shim (extraction required)
• scoopinstaller/validator > ScoopInstaller/Validator (extraction required)
• scoopinstaller/install > ScoopInstaller/Install
• scoopinstaller/nightlies > ScoopInstaller/Nightlies (delete because it's unused or reactivate)
• scoopinstaller/awesome-scoop > ScoopInstaller/Awesome
• scoopinstaller/scoop-clone > ScoopInstaller/GitCloner
• se35710/scoop-java > ScoopInstaller/Java

[AndrewSav]

Not sure about yml. Having worked with it last two years, got a lot of problems from developers who find it difficult, to get the indentation right, get string escaping right and distinguish between lists and properties.
On the topic of strings there are nine different ways to write them with yaml, which, again, from experience, developers find confusing.

What are specific problems with json (apart from not being able to write comments?).

Toml could be a simpler alternative to yaml, however it does not seem to have adequate tooling for Powershell.

[Calinou]

What are specific problems with json (apart from not being able to write comments?).

JSON also doesn't support multi-line strings, so you have to emulate them with arrays. Since you have to quote all strings, you also have to escape any double quotes you're using in strings (which will be common occurrences when writing post-install scripts).

When it comes to multi-line strings with YAML, there's a website for it: yaml-multiline.info 🙂
There's also many YAML linting extensions for various editors such as Visual Studio Code.

[Ash258]

YAML is best suitable for manifest creation.

You do not quote any string in yaml. Unless it is number, which needs to be string

Here is comparison of pretty tough manifest with lots of code inside json and yaml
json
yml Completely valid yml file with block chomping and no need to escape quotes, no commas, no problems

[chawyehsu]

Toml could be a simpler alternative to yaml, however it does not seem to have adequate tooling for Powershell.

Acutally there are some toml libraries for .NET, such as Nett. And I prefer toml too (see, Rust/Cargo is great!), yaml is simpler though.

[shoogle]

YAML is best suitable for manifest creation.

You need to be careful with YAML. It certainly looks a lot nicer than JSON, but it's actually a hugely complicated format and a nightmare to parse, so most parsers don't implement the full feature set.

Consider the fact that YAML supports comments. At first glance this seems like an advantage, but let's see what would happen if you actually tried to use them in a manifest:

description: Example YAML manifest
version: 2.3.1
# really important comment that maintainers absolutely must read!
checkver:
  url: https://example.com/download

Now let's run autoupdate:

description: Example YAML manifest
version: 2.3.2
checkver:
  url: https://example.com/download

Oh dear, the comment has disappeared!

How can we avoid this happening? Like this:

description: Example YAML manifest
version: 2.3.1
_comment: really important comment that maintainers absolutely must read!
checkver:
  url: https://example.com/download

This is exactly how comments in JSON work. The fact that JSON doesn't support proper comments is a feature of the language. The situation with newlines is similar, though not a serious.

The problem with comments is not unique to YAML, but it demonstrates that having another feature is not necessarily a good thing (and YAML has a lot of features!). If you want to switch to YAML then you should restrict yourself to using a sensible subset of its features.

[AndrewSav]

Now let's run autoupdate:

What's autoupdate?

While I myself warned against unnescessary complexity that yaml brings, your comment does not see to apply to the scoop use case. Let's take comments. I cannot forsee a scenarion where comments in scoop yaml needs to be re-parsed and put back, this is just does not seem realistic. So based on your evidence there is no point in restricting yourself to using a feature set that excludes comments.

And if that's not an issue it's not clear to what subset you recommend to restrict to.

In scoop, the intention of using Yaml is to make the manifests easier to work with for humans, while not more difficult to work with for scoop (that is not requiring a huge re-write). The most prominent improvement mentioned, is the use of comments, and use of multi-line values. If you look at it from this perspective it's easy to see why Yaml is appealing. There is no intention to use some especialy complex feature set just for the sake of it, intention is to model the same things that are currently modelled in json, but without the aforementioned problematic facets.

I personally think that there are other facets of yaml which may make its use cumbersome, but I already touched in my prior comment.

[kiedtl]

Now let's run autoupdate:

description: Example YAML manifest
version: 2.3.2
checkver:
url: https://example.com/download

Oh dear, the comment has disappeared!

It should be possible to add a script that will automatically add these comments back.

What's autoupdate? I cannot forsee a scenarion where comments in scoop yaml needs to be re-parsed and put back, this is just does not seem realistic.

Checkver/autoupdate is a tool that bucket admins use to automatically update the manifests. E.g. if go-lang releases a new update, the autoupdate tool will automatically download the release and update the download URL, hash, and version of the manifest. In the process of doing so, some of the fields can be re-arranged and comments will disappear.

[shoogle]

@AndrewSav

What's autoupdate?

Autoupdate is a tool to ensure manifests always point to the latest version without requiring manual intervention by scoop maintainers. If you look through the bucket history you will see that the vast majority of commits are updating manifests, so autoupdate is central to how scoop operates. Comments, like this one in git's manifest, need to be preserved.

your comment does not see to apply to the scoop use case.

How can you claim that when you don't know what autoupdate is?

I personally think that there are other facets of yaml which may make its use cumbersome

I totally agree, but as you said elsewhere, there are no plans to use those features. I was pointing out that even the features that have been explicitly cited as advantages, and given as reasons to switch, have unintended consequences.

@kiedtl

It should be possible to add a script that will automatically add these comments back.

You are proposing to create a custom YAML parser that doesn't obey the standard, for no better reason than syntax highlighting.

If comments are important then they should be treated as part of the data. This is the point that JSON's inventor made when he deliberately removed comments from the JSON spec.

Fundamentally, what is the difference between comments and app descriptions? One is aimed at developers, the other at users, yet both are equally important. We don't comment out checkver and autoupdate just because users aren't supposed to see them.

[shoogle]

Ultimately I'm in favour of a switch to YAML, just as long as it is used sensibly.

Even if you end up treating YAML like JSON and doing this:

description: Example YAML manifest
version: 2.3.1
_comment: really important comment that maintainers absolutely must read!
checkver:
  url: https://example.com/download
post_install:
  - $doThis
  - $doThat
  - $doSomethingElse

I think we can all agree that is a massive improvement over:

{
    "description": "Example JSON manifest",
    "version": "2.3.1",
    "_comment": "really important comment that maintainers absolutely must read!",
    "checkver": {
        "url": "https://example.com/download"
    },
    "post_install": [
        "$doThis",
        "$doThat",
        "$doSomethingElse"
    ]
}

The problem with JSON is its unnecessary use of quotes and braces, not its less-is-more philosophy on features. Its just a pity that the YAML developers weren't content with simply sorting out JSON's problems and instead went on to create a bunch of problems that didn't previously exist.

[AndrewSav]

How can you claim that when you don't know what autoupdate is?

I'm right back with you. Now it makes total sense.

[mottosso]

As someone who recently discovered Scoop only a few days ago, I really can't comment much about which manifest format is best suited; however I wanted to see whether you've considered Powershell?

We're using a project similar to Scoop called Rez where YAML was the de-factor standard for the first few years, until it switched to Python (in which the project is also written). There was the argument, like with any configuration-like format, that using a scripting language gives the author too much power and invites complexity, but in reality what manifests do is complex. I think the issue of multiline is one example of this, and post_install which appear suitable for functions rather than plain-text. Only now are we unconstrained by the format and can leverage the Python community to encapsulate just about any software project.

As an example.

• https://github.com/predat/rez-packages/blob/master/softs/maya/2018.0.4/package.py

On the flip-side, parsing a JSON is really straightforward and fast, and importantly doesn't carry side-effects. Something the manifest author would have to take into account if something like scoop search is going to be running .ps1 scripts on the local (or remote!) machine just to get a hold of the metadata.

TLDR; a complex problem may rightfully require a complex solution

[shoogle]

The advantage of using a serialisation format rather than a scripting language is that it doesn't allow arbitrary code to be executed. I see this as a feature rather than a bug.

post_install is really an exception for applications that require extra configuration (e.g. to hook into a system service), but it is usually used (or abused) as a workaround for applications that have not been written properly (i.e. in a portable/generic way). I think scoop install should display a warning if a manifest contains a post_install step, and there should be an option to skip that step or even globally disable installation of applications that make use of it.

[kidonng]

Hi, I built another documentation site for Scoop. Ideas and suggestions are welcome!

https://scoop-docs.now.sh/

[dillonKneeland]

Hello, I've been using scoop for around a year now and something that I've always wished for was the ability to use scoop search -Bucket <bucket-name> to search for a program in a specific bucket or to be used without a query to list the apps in a bucket. It's something that I have working in my local install of scoop but is liable to break or get overwritten with every scoop update. I would be happy to make a PR for this feature if everyone agrees that it is as useful as I have found it to be.

[bb010g]

I really hope Scoop doesn't switch to YAML. JSON is much more sane to parse, and honestly to write. (YAML multiline strings even have their own reference website: https://yaml-multiline.info/.)

If you want to change off JSON to something more human-readable, I'd recommend TOML or eno. TOML is probably the better option here, and it has some nice .NET parsing options, including Tommy (MIT), a single-file parser that should be usable via Add-Type -TypeDefinition if necessary, and Tomlyn (BSD-2-Clause), which supports modification of the syntax tree with perfect style preservation if necessary and has a compatible minimum .NET Framework requirement of 4.5 (for shipping the DLL).

Example main\bucket\git.toml

version = "2.24.0.windows.2"
homepage = "https://gitforwindows.org"
description = "Distributed version control system."
license = "GPL-2.0-only"

[architecture.64bit]
url = "https://github.com/git-for-windows/git/releases/download/v2.24.0.windows.2/PortableGit-2.24.0.2-64-bit.7z.exe#/dl.7z"
hash = "353d0e1566d8897cb7afe2f6f9088bac17182ca43416feadec1c16f5c3bb9e0f"

[architecture.32bit]
url = "https://github.com/git-for-windows/git/releases/download/v2.24.0.windows.2/PortableGit-2.24.0.2-32-bit.7z.exe#/dl.7z"
hash = "6ac4761e2711cb584f2df1fbc27b677a03c745cfef0bae55604bc27a5c26d68b"

# Comment

Here's .bin as it is now, converted to TOML:

bin = [
  'cmd\git.exe',
  'cmd\gitk.exe',
  'cmd\git-gui.exe',
  'usr\bin\tig.exe',
  'git-bash.exe',
]

Note that this would need to be declared at the root, above [architecture].
(Tables cannot be defined more than once, and the root table doesn't even have
a name. It's more of an ambient key/value pair holder than a table.)

Some potentially more natural TOML encodings of .bin follow.

Encode .bin as a table (JSON object):

[bin.'cmd\git.exe']
[bin.'cmd\gitk.exe']
[bin.'cmd\git-gui.exe']
[bin.'usr\bin\tig.exe']
[bin.'git-bash.exe']

# Comment

This is equivalent to the following inline table:

bin = {
  'cmd\git.exe' = {},
  'cmd\gitk.exe' = {},
  'cmd\git-gui.exe' = {},
  'usr\bin\tig.exe' = {},
  'git-bash.exe' = {},
}

Which is similar to the current array-based method, but you gain the benefit
of properties, as shown in .shortcuts.

Encode .bin as an array of tables:

[[bin]]
path = 'cmd\git.exe'
[[bin]]
path = 'cmd\gitk.exe'
[[bin]]
path = 'cmd\git-gui.exe'
[[bin]]
path = 'usr\bin\tig.exe'
[[bin]]
path = 'git-bash.exe'

I think this still works better than the current weakly typed
"stringOrArrayOfStringsOrAnArrayOfArrayOfStrings" approach, but you have
to specify a path every time, so unless we need to make duplicate shims
available, it makes sense to use tables keying on the path.

With that said, we still don't have to lose that advantage with the
table of tables approach. Allow .bin[$path] to be an array of tables as
well as a table, and you'd get this for a potential multiply-shimmed
program "bar" in a package providing "foo", "bar", and "baz" shims:

[bin.'foo.exe']
[[bin.'bar.exe']]
[[bin.'bar.exe']]
alias = 'bar3.exe'
[[bin.'bar.exe']]
alias = 'bar2.exe'
arguments = ['--two-mode']
[bin.'baz.exe']

Which looks pretty good, I'd say.

(This is also a perfect time to convert to arguments as an array instead of
a weirdly-escaped string. (#2025))

We could put the path or name here in the key for .shortcuts, but I
used path for consistency with .bin.

[shortcuts.'git-bash.exe']
name = "Git Bash"
arguments = ['--cd-to-home']
[shortcuts.'cmd\git-gui.exe']
name = "Git GUI"
[shortcuts.'cmd\gitk.exe']
name = "gitk"

# Comment

We can drop the array of strings dance here (and in .pre_install, and in
.install.script), because TOML has multi-line strings. (And they're good.)
Here we're using a multi-line literal string.

post_install = '''
git config --global credential.helper manager'''

[checkver]
github = "https://github.com/git-for-windows/git"
regex = '''v([\w.]+)/PortableGit-(?<ver>[\d\w.]+)-64-bit'''

[env_set]
GIT_INSTALL_ROOT = "$dir"

[autoupdate]
# note = """
# I'm a sample autoupdate note. \
# This is still the first line.
# Here's my second line!"""

[autoupdate.architecture.64bit]
url = "https://github.com/git-for-windows/git/releases/download/v$version/PortableGit-$matchVer-64-bit.7z.exe#/dl.7z"

[autoupdate.architecture.32bit]
url = "https://github.com/git-for-windows/git/releases/download/v$version/PortableGit-$matchVer-32-bit.7z.exe#/dl.7z"

[autoupdate.hash]
url = "https://github.com/git-for-windows/git/releases/latest"
regex = "<td>$basename</td>\\s*<td>$sha256</td>"

An advantage of using a library like Tomlyn for Scoop manifests would be that machine formatting could be specified, beyond just order of declarations, down to the canonical table syntax or string type for each property. You could make things really consistent.

(CC: Ash258#2)

[shoogle]

Or there's StrictYAML. It keeps the readability of YAML but removes the unsafe and unnecessary features.

[r15ch13]

@dillonKneeland I was working on #3446 and this could be added. But I haven't had the time to finish it yet.

@bb010g TOML looks really promising. 👍

[dwettstein]

Hey @Ash258

As you have closed this issue/discussion, what is the latest status of the Scoop project? It looks like it is getting less and less maintained. Additionally, I saw that you put a lot of effort into your fork (https://github.com/Ash258/Scoop-Core).

Furthermore, why are the remaining parts of Scoop not moved to the https://github.com/ScoopInstaller organization?

BR

[jtagcat]

I'd think this is the best place to mention.

Persons are interested in business support for scoop/shovel. I've seen multiple companies/people pass on scoop due to there being no support available. From the point of 'if something is broken, call/chat/mail support to have it fixed', and not use up a workday fixing things.

[dominsights]

Late to the party. I can see the project is being abandoned. Thank you @dwettstein for pointing that out.

[sharpninja]

I would like to add support for placing output in a PSObject instead of writing to stdout. Would such a PR be welcome?

[rashil2000]

I would like to add support for placing output in a PSObject instead of writing to stdout. Would such a PR be welcome?

@sharpninja Yes! Please find the relevant issue (or create one if it doesn't exist), and raise a PR against it :)

[sharpninja]

I would like to add support for placing output in a PSObject instead of writing to stdout. Would such a PR be welcome?

@sharpninja Yes! Please find the relevant issue (or create one if it doesn't exist), and raise a PR against it :)

I just finished up a bunch of changes for my PS Snippets and am motivated to work on this again. I had these changes made at one point locally, but I'm not sure if they got lost with my laptop that died. Will look for them.