WIP: Easier handling of snippets in source code files

[mxmehl]

This PR relates to a discussion on spdx-legal about snippet handling in SPDX. It is supposed to go hand in hand with a specification update of REUSE for which we seek to make marking licensing and copyright of snippets in source code files easier.

The format we would aim for is the following, e.g. for a shell file:

# /bin/sh

# SPDX-License-Identifier: GPL-3.0-or-later
# SPDX-FileCopyrightText: 2020 John Doe

echo "This is John's script"

# SPDX-SnippetBegin
# SPDX-License-Identifier: CC-BY-SA-4.0
# SPDX-SnippetCopyrightText: 2020 Max Mehl <https://stackoverflow.com/users/4273755/mxmehl>
# Copyright © 2016 Original author of code Max developed further <contact@example.com>

echo "Here's Max' great addition to this script" > /dev/null

# SPDX-SnippetEnd

echo "John's script continues"

Note on SnippetCopyrightText and Copyright: per REUSE's specification, both is perfectly fine, so both should also be supported in snippets.

Following @goneall's request, I've added this to Annex E. However, I am wondering whether snippet-information.md also needs some update, e.g. to make clear that nesting is not allowed (see ML thread).

To be honest, I am bit lost here how SPDX specifies things, and whether they are different for SPDX documents than in source code files. Help and guidance is most welcome!

Variants

There are some variants we could discuss:

• SPDX-License-Identifier is a beautiful, well-known tag which I would prefer to also be valid for snippets. However, since we already have SnippetCopyrightText, one could think about SPDX-Snippet-License-Identifier. But I would see a big problem caused by the missing/additional dash in the tag name, something that is already quite confusing.
• I am impartial about SnippetBegin or Snippet-Begin. AFAIU, camel-case is preferred in SPDX, so I took that.
• To make it brutally simple for users, we could also replace SnippetCopyrightText by FileCopyrightText, but it's quite obvious that the scope would be wrong. As said above, it's always possible to use the traditional Copyright lines, and tools would just have to interpret it correctly, knowing that it related to the full file or just a snippet.

[swinslow]

Hi @mxmehl, thanks for getting this discussion going.

Personally, I'm indifferent on whether to add Snippet-specific tags at all. I guess I'm skeptical about whether they will be adopted in significant measure, but of course I could be wrong about this.

That said, I'm (at least currently) opposed to incorporating a different meaning for SPDX-License-Identifier into this proposal. Right now, one of the benefits of SPDX-License-Identifier as it's currently used is that it is greppable and not context-specific. This proposal seems to say that it could have a different meaning depending on whether it appears between SnippetBegin / SnippetEnd tags, or not.

I'd suggest that if this Snippet proposal is adopted (which as noted, I'm not convinced on), then something like SPDX-Snippet-License-Identifier should be used instead. If this is going to be net-new and isn't reflecting something that is already in use in the community, then it should be something that doesn't collide with existing use.

My final comment is just to note that as part of the 3.0 spec, the formats and names of the license-related fields will likely be changing to some degree, as part of the new 'licensing' profile and the 3.0 revamp. So whatever the decision is on this, I'd suggest not planning to put it into use until 3.0 is more settled since things may change significantly.

[zvr]

Simply also allowing SPDX-SnippetBegin-1, SPDX-SnippetBegin-2, etc. takes care of the nesting snippets, as well.

+1 to @swinslow reservations about wide adoption of this and patience for the 3.0 naming.

[mxmehl]

I get your point with the context problem, and so SPDX-Snippet-License-Identifier would be fine for me as well, with the little downsides mentioned above (damn dashes vs. CamelCase!).

Regarding measureable use, at REUSE we receive quite some demand for this feature. Re-using code from stackoverflow (under CC-BY-SA-4.0) seems to be a common practice, and allowing for a simply way to mark that would surely increase proper usage once more.

Regarding nesting, numbering these would also be fine, but I see two problems:

• If these are to be translated in an SPDX document, the current byte/line ranges could be a show-stopper
• Are there many use-cases of nested snippets? Here, I am not sure whether this would be overengineering

[silverhook]

Unsurprisingly, as I suggested this issue in REUSE, I’m in favour of this being adopted in SPDX, if it does not break anything. Especially in front-end web development, I see snippets being used a lot, both when writing, as well as when concatenating JS and CSS files.

Regarding nesting, I am slightly against it in order to keep it simple. But on the other hand, @zvr’s suggestion of numbered (or other unique IDs) snippets does have its benefits. Also of more easily debugging if one snippet is missing a beginning or ending tag.

[nicorikken]

Wauw, I like this idea a lot, regardless of the SPDX technicalities. I regularly add a couple of lines with a source and copyright annotation, but the annotation in SPDX is currently quite coarse.

[kestewart]

Based on ACT tooling meeting discussion - For consideration earlier in 2.3 vs. 3.0 - to be discussed.

[silverhook]

@kestewart , was there an outcome whether this would be in 2.3 or 3.0 yet?

In any case, did we decide whether we’ll use non-nesting generic SPDX-SnippetBegin or numbered/named SPDX-SnippetBegin-<$str> tags?

I’d like to use this ASAP even if it’s not part of the spec/standard yet.

[goneall]

@swinslow - Discussed on the tech call - we'd like your review. If you're good with this PR, we'll merge and include it into 2.3

[swinslow]

Thanks all. @goneall three comments from me, one procedural and two substantive:

Procedurally, I don't think I'm the sole voice on behalf of the Legal Team community about this, so I'll email the Legal Team mailing list about this shortly (similarly to the email I sent earlier today on the license namespaces draft annex). cc @jlovejoy and @pmadick as well in case they have direct input here.

Substantively, as currently drafted I'm a strong -1 on this and I don't think it should be merged. As I mentioned above, this appears to give a second meaning to SPDX-License-Identifier, inserting a context-specific meaning when it is enclosed by SnippetBegin / SnippetEnd tags. Given how broadly SPDX-License-Identifier has been implemented, and the current understanding of what it means to grep for it in a file without further context-sensitive meaning, I am not in favor of anything that changes this, even if only doing so in a subset of use cases.

If the identifier for this were changed to SPDX-Snippet-License-Identifier as I suggested above, then personally I would be +1 on this.

My other substantive question (and a more minor one) is, why is this limited to use of "third-party" code? I could see that use of third-party code could be the most common use case, but I could also see a situation where e.g. someone is combining their own code from multiple projects under multiple licenses into a single file, and wants to call them out as separately-licensed snippets. I guess I'd think that the format would work for designating separate snippets of code, whether that's for a third-party or for one's own code.

[swinslow]

Another follow-up question here:

As drafted, this is ambiguous as to whether the license identifier and/or the copyright text tags are each mandatory and/or each optional within a designated snippet. Personally I think they should both be optional and presumed to have the meaning NOASSERTION if not specified.

[silverhook]

Substantively, as currently drafted I'm a strong -1 on this and I don't think it should be merged. As I mentioned above, this appears to give a second meaning to SPDX-License-Identifier, inserting a context-specific meaning when it is enclosed by SnippetBegin / SnippetEnd tags. Given how broadly SPDX-License-Identifier has been implemented, and the current understanding of what it means to grep for it in a file without further context-sensitive meaning, I am not in favor of anything that changes this, even if only doing so in a subset of use cases.

I’m OK with either option.

Since AFAIK the default when finding several SPDX-License-Identifier tags is to assume AND in between, it seems to me like it continues to work as expected. The SnippetBegin / SnippetEnd tags would only locate which parts of the code the license applies to, and therefore add information, not change it.

Let’s assume a project with SPDX-License-Identifier: (MPL-1.1 OR MIT) and the analysed file includes a snippet that is marked as being under ISC

1. if a more complete tool understands the Snippet* tags, it would report the file being (MPL-1-1 OR MIT) AND ISC, and being able to identify the snippet that is under ISC

2. yet still, just a simple tool (e.g. grep SPDX-License-Identifier) would find the same (MPL-1-1 OR MIT) AND ISC, just say that this applies to the whole file …which TBH is what happens already now if you scan a file with e.g. FOSSology or ScanCode and it finds a snippet with its own license

If the identifier for this were changed to SPDX-Snippet-License-Identifier as I suggested above, then personally I would be +1 on this.

As said, personally, I don’t see a reason to add another *-License-Identifier tag, and see the potential that existing (simple) tools would not find snippet-level licensing if we introduce it. But if it fixes things, I’m OK with that option as well.

+1 for both of the tag options from my side.

My other substantive question (and a more minor one) is, why is this limited to use of "third-party" code? I could see that use of third-party code could be the most common use case, but I could also see a situation where e.g. someone is combining their own code from multiple projects under multiple licenses into a single file, and wants to call them out as separately-licensed snippets. I guess I'd think that the format would work for designating separate snippets of code, whether that's for a third-party or for one's own code.

On this point I fully agree. I don’t know how the “third party” limitation crept in, I don’t see any reason for such limitation either.

[mxmehl]

I agree to @silverhook's analysis regarding the SPDX-License-Identifier tag that it would actually match the AND sense that snippets create. I would also be fine with a separate tag but the the same issues: new tag may take long until it's understood by all tooling, and developers are likely to miss/confuse this.

As soon as we've clarified the tag question I can update the PR and remove the third-party thingy. No idea what I was thinking when adding it, probably it was rather meant as an example.

[swinslow]

Since AFAIK the default when finding several SPDX-License-Identifier tags is to assume AND in between, it seems to me like it continues to work as expected. The SnippetBegin / SnippetEnd tags would only locate which parts of the code the license applies to, and therefore add information, not change it.

@silverhook This is a really good point.

So, looking at the current spec and the way it describes short identifiers in source files:

• The current spec says that SPDX-License-Identifier "declares the license the file is under and should be placed at or near the top of the file in a comment."
• To me, this implies that the identifier should only appear once, at or near the top of the file; and if someone were to concatenate multiple files together which each contained these tags, they should replace it with a single combined tag at the top of the concatenated file.
• Of course, this often doesn't happen in real life, and files can end up containing lots of snippets of code with many SPDX-License-Identifier tags scattered throughout. (I've seen this for example in JavaScript files in particular.)

@silverhook I think your statement is to say that, in the absence of other information, when there's multiple SPDX-License-Identifier tags throughout a single file, the only reasonable way to handle them is to AND them together.

And so, the proposal here wouldn't change the effect of that approach for a person or tool that ignores the SnippetBegin / SnippetEnd tags. But for a person or tool that DOES parse those tags, they get the benefit of the extra contextual information.

Does all that make sense, and fit with your take on this? If so, I might be starting to be persuaded that I'm wrong on this one :)

I do think though that if this approach were taken, using the same SPDX-License-Identifier tag as for file-level statements, then the parts of the existing annex I referenced (referring to it only appearing at the top of a file) would also need to be addressed, I'd think.

I plan to bring this up on the Legal Team call this Thursday, for anyone who's available to join and participate in the discussion. Even if I'm getting on board with this, I'd like to see whether there's broader consensus from Legal Team participants on making this change.

[pombredanne]

If you are effectively tagging multiple source code snippets from different origins which is intrusive since it does modifies the code, could this exceptional use be better handled by suggesting to go a bit further in the modification and to split things in different files, one for each origin and license? In doing so, IMHO there is no need to further specify anything new, just some guidelines for this use case. What about this simpler way?

[silverhook]

Does all that make sense, and fit with your take on this? If so, I might be starting to be persuaded that I'm wrong on this one :)

You described exactly my train of thought, yes :)

I do think though that if this approach were taken, using the same SPDX-License-Identifier tag as for file-level statements, then the parts of the existing annex I referenced (referring to it only appearing at the top of a file) would also need to be addressed, I'd think.

Having them at the top does makes sense, but the question is of the should vs must here I think. If it’s a “should” (= in general, but there may be reasonable exceptions) it’s fine as is. If it’s a “must” then the spec is already different than real life examples.

I plan to bring this up on the Legal Team call this Thursday, for anyone who's available to join and participate in the discussion. Even if I'm getting on board with this, I'd like to see whether there's broader consensus from Legal Team participants on making this change.

I’d love to, but SPDX for the past few years regularly clash with some work meetings I have :/

[silverhook]

@pombredanne , you’re of course right!

But what we want to do here is not to encourage people to copy-paste stuff when it’s better to have it separate, but to help properly mark/document when this was already done.

I’m pretty sure you saw several files by now as well that were basically:

• this file is GPL’d by ProjectX
• but this snippet here is some legacy code from ProjectA under ISC
• oh, and this part here is by Jack, who while part of the team insists all his code is under MIT
• and Jill also pulled this ancient list of timezones from somewhere that’s claimed to be in public domain

…I would just like some order in that (and as @swinslow said, in JS this gets really ugly really fast)

[bsdimp]

At first I liked the idea of bringing increased semantic understanding for SPDX-License-Identifier tags.

However, i hate this idea. Why? Because one license per file is the easiest to manage, and #include is a trivial way to nest files with multiple licenses (and a case that anybody doing compliance will have to cope with anybody due to the wide-spread use of #include). Given such a trivial workaround, that puts my initial analysis at "no way, unless there's something compelling here". And yes, this is dictating policy, but sometimes that's a good thing.

And I don't think there is. It would create more confusion in the day to day maintenance of these files. Software isn't created once, but multiple times as each leaf grows or branch is pruned. Will all these markers get properly moved around? Will they all get updated correctly? Do I update the whole file copyright or not?

And is such a trivial inclusion even covered by copyright? It isn't always since its use may be fair, or the code in question may not have nay protectable content. As we delve down to sub-file levels of licensing, how does one opt-out of using that? How does one even know that the code compiles in and isn't eliminated by optimization. Etc.

So, not only does it complicate understanding what the license is, it does so in a way that's horrible to maintain.

I'm strongly against this and it would only create problems for FreeBSD.

[silverhook]

Will they all get updated correctly? Do I update the whole file copyright or not?

What do you intend to update? If you’re referring to copyright notices, may I refer you to this post I wrote?

I'm strongly against this and it would only create problems for FreeBSD.

In what way? You can easily ignore them and simply not use the snippet-level tags.

Requiring all code (also already written) to be (re-)written to use the include directive seems like a much bigger change and one that I don’t see happening. I do agree that whenever possible, it perhaps should be preferable, but it’s not always possible.

[jlovejoy]

hmmm... having now read this whole thread, I'm not sure how I feel. My gut is - why do we need this? and isn't this just over-complicating things?
To be clear, based on the file - this is ONLY a change to the Appendix on how to apply SPDX ids and NOT a change to the spec, is that right?

My leaning is not to add another license tag. Please no.

I also note the earlier comments about holding off on this until 3.0 but the tech team discussed and seems ok with the possibility for 2.3??

[silverhook]

I fear we’ve overcomplicated this proposal quite a bit through these years of slow discussion. The original proposal and its reasoning started very simple really. Let’s step back a bit. Also let’s please decide on the tags first and then we’ll modify the proposed draft PR. I volunteer to also either help draft a new PR if it is needed or draft the new one myself (of course in agreement with the rest of the REUSE team).

Since this is blocking REUSE spec’s new version release and has been stuck for about 2 years, I’d really appreciate it if we can push this forward. For this reason and to save everyone’s time, I’ve spend a whole day to summarise all that I could find and can think of on this topic so far.

Introduction

In REUSE our goal is to provide a spec that makes it easy to mark (your own) source code with appropriate license and copyright info. We try to rely on SPDX as much as possible in order to a) leverage and promote the power of the awesome standard it is (to the extent it’s easy to generate an SPDX document from a REUSE-compliant source code package), and b) not to introduce any clashes or ambiguities between the two specs.

This all began in 2019 with fsfe/reuse-docs#34 where, after stumbling upon this problem/need several times at work as well as noticing others in and around REUSE mentioning similar issues that could elegantly be solved by introduction of snippet-level tags, I suggested that we figure out a way how to properly equip snippets in source code, where their license and copyright info differs from the rest of the file, with such info.

I have seen use cases in the wild that cover both snippets of third party as well as first party code. And since SPDX already had optional tags for snippets since 2.1 version of its spec (i.e. 2016, IIRC), it seemed reasonable to try to use SPDX as much as possible. Especially since one of the main goals of REUSE is the license and copyright info in a REUSE-compliant source code can very easily be translated to a valid SPDX Document, without the need of tools that have to make assumptions, but simple and reliable one-on-one tag translation. So we wanted was to make sure that the snippet-level info could be just as easily translated to an SPDX Document.

In 2020 the discussion started on both SPDX Legal and REUSE mailing lists and also this issue we are all reading now.

What the proposal actually was/is

In REUSE we want to introduce a way to mark snippets – we hear and see the need for that.

In order to achieve that we need to identify three things:

• what is the right SDPX tag (for the author/code maintainer) to mark a license in a snippet in source code
• what is the right SPDX tag (for the author/code maintainer) to mark copyright statements in a snippet in source code
• what tag we should introduce (for the author/code maintainer) to mark the beginning and end of a snippet in source code (does not exist in SPDX yet)

We saw SPDX spec already has Clause 9: Snippet Information. But it does not answer all of our questions.

With the basics out of the way let’s dive into the individual options.

License tag options

SDPX offers the following options, that we could use in SPDX by leveraging Annex H: SPDX File Tags by prepending SPDX-:

1. License-Identifier:

   • by using SPDX-License-Identifier, which is already used in REUSE on file-level (its use is also acknowledged in both Annex E and Annex H), it would be more consistent within REUSE and easier to explain to the users
   • it also relies on the already established use of SPDX short IDs in source code files
   • even a simple tool (e.g. grep License-Identifier) that knows nothing about snippet-level tags, would still correctly catch the license ID. Without any other indication, finding several SPDX-License-Identifiers in the same file, one should already now reasonably assume that there is an AND in between several found licenses (in the absence of an explicit OR). Such a tool would not know which lines this license applies to, but it would still find the right license
   • potential problem: this tag is not explicitly intended for snippets
   • counter-argument: this tag is also not strictly speaking just about file-level info and there seems no reason for there to be different tags for package, file and snippet levels in the source code itself

2. LicenseInfoInSnippet:

   • issue that this indicates a license ID that was found in the snippet, which may be true when a tool/person goes through the source code and in its audit stores the info in an SPDX Document, but not when we mark the source code file with it. From the PoV of REUSE and the developer, who is marking the snippet info in the source code, it clearly is not info that they found in the snippet, but info they put in the snippet themselves
   • also loses the ability for simpler tools to find it, if they aren’t aware of snippet-level tags (some may thing that’s a plus, I think it’s a huge minus)

3. SnippetLicenseConcluded:

   • similarly, issue that this indicates a license ID that was found in the snippet by a tool that goes through the source code and a person concluded in its audit this is the right license and therefore stores the info in an SPDX Document, but not when we mark the source code file with it. From the PoV of REUSE and the developer, who is marking the snippet info in the source code, it clearly is not info that they concluded from info they found in the snippet, but info they put in the snippet themselves
   • also loses the ability for simpler tools to find it, if they aren’t aware of snippet-level tags (some may thing that’s a plus, I think it’s a huge minus)

Personally, I’m mostly in favour of 1), as I see several practical benefits, and no practical downsides, bar some theoretical ones, which are already solved through Annex H: SPDX File Tags. But at this stage, I’d just want to settle for a tag we can use.

Copyright tag options

SDPX offers the following options, that we could use in SPDX by leveraging Annex H: SPDX File Tags by prepending SPDX-:

1. FileCopyrightText:

   • intended to store copyright notices associated with the file

2. SnippetCopyrightText:

   • intended to store copyright notices associated with the snippet

Here I think we can agree that SPDX-SnippetCopyrightText is the tag that SPDX spec intends us to use.

Snippet beginning and end tags

This is the only part of the proposal where a new tag might need to be introduced into SPDX spec.

Obviously we need a way to mark the beginning and the end of a snippet in source code for any of this to be useful.

Furthermore and as a bonus to SPDX, what we propose to include is markers in source code, which a tool could directly translate into the already existing SnippetLineRange tag (and its startPointer and endPointer tags) in an SPDX Document, making that tag easier to use in practice as a consequence.

So there are two ways we can go about this in REUSE (we’ll replace “SPDX-” with “REUSE-” in the tags if SPDX wants none of this – thought this would be unfortunate, since it would mean the first tag in REUSE that would not be an SPDX- tag):

1. generic tag, e.g. SPDX-SnippetBegin/SPDX-SnippetEnd:

   • simple to explain and use (in REUSE)
   • problem with snippets nested within snippets (fixable in REUSE, but can be a bit messy)

2. named tag, e.g. SPDX-SnippetBegin-{$name_or_id}/SPDX-SnippetEnd-{$name_or_id}:

   • solves the problem of snippets nested within snippets (in both REUSE and as a consequence SPDX Document)
   • in addition al tool could translate the {$name_or_id} part of into SnippetName in a SPDX Document
   • but needs a bit more effort from the person marking the code, to come up with names or ids

I have a preference for 2) since IMHO it solves more problems and provides additional info to be translated into SPDX Documents, while the added complexity is not too much to ask.

Bonus question: How to mark origin/provinence

This is not part of this proposal, but something we are wondering within REUSE as well.

Namely, if the snippet is of a different origin, how to best provide the origin, as a link or description.

Reading the SPDX spec, it seems like SnippetLicenseComments seems to be the closest, so we could use something like SPDX-SnippetLicenseComments for this (and then a tool (or user) could then feed that to SnippetLicenseComments, if they wanted).

Not breaking SPDX

This should not break anything in SPDX, and so far I’ve not seen an argument that showed what it would break. If there is something, please let me know. We really don’t want to break anything in SPDX.

Whichever new (if any) tags SDPX adopts for this, they should be optional, just as snippet-level tags already are. If you don’t care about providing (or reading) snippet-level info, simply don’t, as Annex H: SPDX File Tags already indicates is the case for SPDX-* tags found in source code files.

If we want to manage snippets in source code files, regardless of which tag options we decide upon, we might also want to make the following line in H.2 Format a bit clearer still by emphasising that the file tags are most often at the top, but esp. if there are snippets, they could be elsewhere as well:

An SPDX file tag consists of a single line, generally as part of a comment near the top of the file, in the following format:

To re-iterate again, this is not just about third party snippets, and our intention is not to promote the method of copy-pasting third party code (just as Clause 9: Snippet Information of SDPX doesn’t), but instead to properly mark snippets that carry a different license and/or copyright notice than the rest of the file (just as Clause 9: Snippet Information of SPDX does).

[nicorikken]

Thanks @silverhook for the clear summary. Recent arguments seem to indicate the dilemma whether SPDX should support snippets at all or should nudge projects to branch off copied snippets to separate files. In my experience snippets are so small that the overhead of creating a dedicated file and doing the import is too much overhead and not worth it. Typical use-cases I encountered are complex regexes or clever one-liners. So I certainly see value in snippets and hope this idea makes it into SPDX.

I don't see a need for nested snippets. It is not a use-case I personally encountered. Also as source code is sequential by nature you can always have a sequence of snippet sections to indicate the different origins. Nesting can be harder to parse, so if that complexity can be avoided I think that would help the SPDX ecosystem.

[zvr]

Thanks, @silverhook for the analysis.

My "votes":

• License tag options, option 1, SPDX-License-Identifier
• Copyright tag options, option 2, SPDX-SnippetCopyrightText
• Snippet beginning and end tags, option both, SPDX-SnippetBegin[-id] where the -id part is optional. In most of the cases people might use the simple option, but in the case where you want nested snippets, you have the mechanism to do so. As you note, this is the only newly introduced change, and I believe that supporting both formats is not that difficult.

Please note that the id should be used uniquely per file, i.e., you cannot re-use the same id in another place in the same file.

In contrast with @nicorikken comment above, I have encountered nested snippets: for example, a 20-line inside a 100-line snippet.

[silverhook]

Was discussed in the SPDX Tech Team call (minutes) and the SPDX Legal Team call.

TL;DR:

1. SPDX-License-Identifier is the right option
2. SPDX-SnippetCopyrightText is the right option
3. SPDX-SnippetBegin and SPDX-SnippetEnd to be added to Annex H: SPDX File Tags – @silverhook is to create a pull request proposal for how to word that instead of the version in this PR

[goneall]

@silverhook - Have you updated or created the PR for this issue? If so, can you link here.

[silverhook]

@goneall Working on it. I’m battling with the question whether to introduce a new H.3 subsection and push the existing one into H.4 or if I need to be more creative in order not to break the existing order.

[silverhook]

@goneall I created #718 to replace this PR here.

[goneall]

Thanks @silverhook

[silverhook]

Also created #719 with an alternative, IMHO better, structure.

[silverhook]

Can be closed in favour of the approved #719

[goneall]

I'm going to go ahead and close this to avoid possible confusion.