SwiftDocC: use "portable" paths for file names #668

compnerd · 2023-07-22T18:09:23Z

Windows restricts a CharacterSet from use in the file name. Replace that set with _. This requires an associated change in the DocC renderer to perform the substitution when converting the URL to a file path.

Some file systems have restrictions on character sets which are valid file name characters. Add a filter for the Windows file system character set restrictions. We replace them with `_` to match the behaviour in the DocC bundle generation after swiftlang/swift-docc#668.

compnerd · 2023-07-22T18:26:04Z

This has the associated change in the docc-render application to permit the continued use of the characters in the URL.

compnerd · 2023-07-24T22:18:41Z

@swift-ci please test

d-ronnqvist · 2023-07-26T22:57:24Z

Sources/SwiftDocC/Infrastructure/NodeURLGenerator.swift

+                let components = name.components(separatedBy: ["<", ">", ":", "\"", "/", "\\", "|", "?", "*"])
+                if components.count > 1 {
+                    isURLModified = true
+                    name = components.joined(separator: "_")


I wonder if we should always append a hashed identifier to the name here to ensure that the file names don't collide.

I think that appending the hashed identifier seems reasonable. Definitely would avoid any possible conflicts. Would we do that for any case where isURLModified is true?

The code already mostly does that when modifying the URL, although if it's a specific path component that's longer than what a file name is allowed to be, then it shortens that path component and appends the hash to that component instead of later in the URL.

d-ronnqvist · 2023-07-26T23:06:56Z

Tests/SwiftDocCTests/Infrastructure/DocumentationContext/DocumentationContextTests.swift

-            "/tutorials/Technology-file:-with---whitespace,-and_punctuation/Volume_Section-Title:-with---various!-whitespace,-and/punctuation",
-            "/tutorials/Technology-file:-with---whitespace,-and_punctuation/Volume_Section-Title:-with---various!-whitespace,-and/punctuation/Chapter_Title:-with---various!-whitespace,-and/punctuation"
+            "/tutorials/Technology-file_-with---whitespace,-and_punctuation/Volume_Section-Title:-with---various!-whitespace,-and/punctuation",
+            "/tutorials/Technology-file_-with---whitespace,-and_punctuation/Volume_Section-Title:-with---various!-whitespace,-and/punctuation/Chapter_Title:-with---various!-whitespace,-and/punctuation"


Is it expected that the : towards the end of this string ("Chapter_Title:") isn't replaced with _?

No, that isn't expected, but how did the test pass 🤯. : is a ADS (alternate data stream, think of it as macOS file forks) separator, the file would just be truncated to the : and the rest would be the name of the stream.

d-ronnqvist · 2023-07-26T23:12:43Z

Tests/SwiftDocCTests/LinkTargets/LinkDestinationSummaryTests.swift

@@ -276,7 +276,7 @@ class ExternalLinkableTests: XCTestCase {
            let summary = node.externallyLinkableElementSummaries(context: context, renderNode: renderNode)[0]

            XCTAssertEqual(summary.title, "globalFunction(_:considering:)")
-            XCTAssertEqual(summary.relativePresentationURL.absoluteString, "/documentation/mykit/globalfunction(_:considering:)")
+            XCTAssertEqual(summary.relativePresentationURL.absoluteString, "/documentation/mykit/globalfunction(__considering_)")


I don't think this value should change. It represents the relative URL for this page. As far as I understood the URL would still contain the characters that aren't allowed in the file names.

Right, the URLs can contain the view, but the file paths would be replaced. I believe that the replativePresentationURL is still referencing the file path?

No, this is meant to refer to the web URL relative to where this documentation is hosted.

This API is used to bridge with other documentation systems which aren't expected to have access to the doccarchive directly. Those systems are expected to use the summary information to create links that navigate the reader to the hosted documentation.

It can also refer to a landmark on a page which wouldn't have its own file in the documentation archive.

d-ronnqvist · 2023-07-26T23:14:43Z

Tests/SwiftDocCTests/Indexing/RenderIndexTests.swift

@@ -96,7 +96,7 @@ final class RenderIndexTests: XCTestCase {
                            "type": "groupMarker"
                          },
                          {
-                            "path": "/documentation/mixedlanguageframework/bar/mystringfunction(_:)",
+                            "path": "/documentation/mixedlanguageframework/bar/mystringfunction(__)",


@ethan-kusters do you remember if this value represents the path of the web URL or the file path within the documentation archive? I thought that this is the path that the browser would load to access this page.

Some file systems have restrictions on character sets which are valid file name characters. Add a filter for the Windows file system character set restrictions. We replace them with `_` to match the behaviour in the DocC bundle generation after swiftlang/swift-docc#668.

Some file systems have restrictions on character sets which are valid file name characters. Add a filter for the Windows file system character set restrictions. We replace them with `_` to match the behaviour in the DocC bundle generation after swiftlang/swift-docc#668. Conditionally enable the new portable paths based upon a setting in `theme-settings.json` to provide a means for compatibility.

Windows restricts a CharacterSet from use in the file name. Replace that set with `_`. This requires an associated change in the DocC renderer to perform the substitution when converting the URL to a file path.

compnerd · 2024-05-09T18:12:02Z

@d-ronnqvist this would be nice to get in for 6.0

d-ronnqvist · 2024-05-10T08:42:10Z

This is not ready to be merged. I added the "needs forum discussion" tag because this previous Swift Forums thread or the Workgroup meeting where this topic was discussed didn't reach a specific conclusion for how to best solve this problem.

It seems like the solution would be to add an opt-in feature flag, communicate about the change and the possible disruption in the Swift Forums, and then reevaluate after one minor release if we should change the default behavior top opt-out instead of opt-in. However, it might be good to revive that thread just to confirm that everyone is happy with that solution. If so, what's missing from this PR is adding that flag and making the behavior conditional.

richardtop · 2024-08-21T16:00:14Z

Interested in this being implemented/merged.

d-ronnqvist

This enhancement needs an agreed upon solution.
So far this has been discussed in this Forums thread but AFAICT there's not an agreed upon solution yet.

compnerd requested review from d-ronnqvist, ethan-kusters and franklinsch July 22, 2023 18:09

compnerd mentioned this pull request Jul 22, 2023

utils: perform portable path sanitisation of URLs swiftlang/swift-docc-render#721

Open

compnerd force-pushed the portable branch from 6b10dbf to cb6264d Compare July 22, 2023 18:25

compnerd force-pushed the portable branch from cb6264d to 82bcf51 Compare July 24, 2023 21:42

compnerd requested a review from mportiz08 July 26, 2023 15:18

d-ronnqvist reviewed Jul 26, 2023

View reviewed changes

compnerd force-pushed the portable branch from 82bcf51 to f274579 Compare October 15, 2023 21:02

d-ronnqvist added the needs forum discussion Needs to be discussed in the Swift Forums label Dec 21, 2023

SwiftDocC: use "portable" paths for file names

74fbe3f

Windows restricts a CharacterSet from use in the file name. Replace that set with `_`. This requires an associated change in the DocC renderer to perform the substitution when converting the URL to a file path.

compnerd force-pushed the portable branch from f274579 to 74fbe3f Compare May 9, 2024 15:30

compnerd mentioned this pull request Jun 21, 2024

Swift's Windows installer is missing the features.json file for docc and clang swiftlang/swift#74634

Closed

compnerd mentioned this pull request Sep 9, 2024

features.json is missing from Windows installer for docc swiftlang/swift-installer-scripts#337

Open

d-ronnqvist requested changes Nov 27, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SwiftDocC: use "portable" paths for file names #668

SwiftDocC: use "portable" paths for file names #668

compnerd commented Jul 22, 2023

compnerd commented Jul 22, 2023

compnerd commented Jul 24, 2023

d-ronnqvist Jul 26, 2023

compnerd Jul 26, 2023

d-ronnqvist Jul 26, 2023

d-ronnqvist Jul 26, 2023

compnerd Jul 26, 2023

d-ronnqvist Jul 26, 2023

compnerd Jul 26, 2023

d-ronnqvist Jul 26, 2023

d-ronnqvist Jul 26, 2023

d-ronnqvist Jul 26, 2023

compnerd commented May 9, 2024

d-ronnqvist commented May 10, 2024

richardtop commented Aug 21, 2024

d-ronnqvist left a comment

SwiftDocC: use "portable" paths for file names #668

Are you sure you want to change the base?

SwiftDocC: use "portable" paths for file names #668

Conversation

compnerd commented Jul 22, 2023

compnerd commented Jul 22, 2023

compnerd commented Jul 24, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

compnerd commented May 9, 2024

d-ronnqvist commented May 10, 2024

richardtop commented Aug 21, 2024

d-ronnqvist left a comment

Choose a reason for hiding this comment