//> using file directives should allow URLs and git repository paths

[benwbooth]

It would be useful if the //> using file directives could accept URL and git repository paths in addition to local files. This would allow users to refer to their scala-cli module scripts from their executable scripts, no matter what paths the executable scripts were stored in:

//> using file "git+ssh://git@gitlab.com/benwbooth/scala-scripts.git?dir=lib/scriptdir/my-script.sc"

or

//> using file "https://raw.githubusercontent.com/benwbooth/scala-scripts/lib/scriptdir/my-script.sc"

Deno has a similar feature for pulling in dependencies from URLs.

[Gedochao]

Handling for remote sources is already in place for the command line, so this is a matter of porting this feature to the //> using file directive. Let me list some key points in the code for tackling this.

The standard entry point for inputs resolution is here:
https://github.com/VirtusLab/scala-cli/blob/main/modules/cli/src/main/scala/scala/cli/commands/shared/SharedOptions.scala#L688-L704

Where the interesting stuff starts here:

scala-cli/modules/build/src/main/scala/scala/build/input/Inputs.scala

Lines 319 to 333 in 4fb5c5b

	private def forNonEmptyArgs(
	args: Seq[String],
	cwd: os.Path,
	download: String => Either[String, Array[Byte]],
	stdinOpt: => Option[Array[Byte]],
	scriptSnippetList: List[String],
	scalaSnippetList: List[String],
	javaSnippetList: List[String],
	markdownSnippetList: List[String],
	acceptFds: Boolean,
	forcedWorkspace: Option[os.Path],
	enableMarkdown: Boolean,
	allowRestrictedFeatures: Boolean,
	extraClasspathWasPassed: Boolean
	)(using invokeData: ScalaCliInvokeData): Either[BuildException, Inputs] = {

Inputs (including remote ones) are validated here:

scala-cli/modules/build/src/main/scala/scala/build/input/Inputs.scala

Lines 225 to 232 in 4fb5c5b

	def validateArgs(
	args: Seq[String],
	cwd: os.Path,
	download: String => Either[String, Array[Byte]],
	stdinOpt: => Option[Array[Byte]],
	acceptFds: Boolean,
	enableMarkdown: Boolean
	)(using programInvokeData: ScalaCliInvokeData): Seq[Either[String, Seq[Element]]] =

All remote (virtual) inputs are then represented as instances of scala.build.input.Virtual:

scala-cli/modules/build/src/main/scala/scala/build/input/Element.scala

Lines 16 to 57 in 4fb5c5b

	sealed abstract class Virtual extends SingleElement {
	def content: Array[Byte]
	def source: String
	def subPath: os.SubPath = {
	val idx = source.lastIndexOf('/')
	os.sub / source.drop(idx + 1)
	}
	def scopePath: ScopePath =
	ScopePath(Left(source), subPath)
	}
	object Virtual {
	val urlPathWithQueryParamsRegex = "https?://.*/([^/^?]+)(/?.*)?$".r
	def apply(path: String, content: Array[Byte]): Virtual = {
	val filename = path match {
	case urlPathWithQueryParamsRegex(name, _) => name
	case _ => path.split("/").last
	}
	val wrapperPath = os.sub / filename
	if filename.endsWith(".scala") then VirtualScalaFile(content, path)
	else if filename.endsWith(".java") then VirtualJavaFile(content, path)
	else if filename.endsWith(".sc") then VirtualScript(content, path, wrapperPath)
	else if filename.endsWith(".md") then VirtualMarkdownFile(content, path, wrapperPath)
	else VirtualData(content, path)
	}
	}
	sealed abstract class VirtualSourceFile extends Virtual {
	def isStdin: Boolean = source.startsWith("<stdin>")
	def isSnippet: Boolean = source.startsWith("<snippet>")
	protected def generatedSourceFileName(fileSuffix: String): String =
	if (isStdin) s"stdin$fileSuffix"
	else if (isSnippet) s"${source.stripPrefix("<snippet>-")}$fileSuffix"
	else s"virtual$fileSuffix"
	}

The current implementation of the //> using file directive is defined here:

scala-cli/modules/directives/src/main/scala/scala/build/preprocessing/directives/Sources.scala

Lines 12 to 29 in 4fb5c5b

	@DirectiveGroupName("Custom sources")
	@DirectiveExamples("//> using file utils.scala")
	@DirectiveUsage(
	"`//> using file `_path_ | `//> using files `_path1_ _path2_ …",
	"""`//> using file` _path_
	|
	|`//> using files` _path1_ _path2_ …
	|""".stripMargin
	)
	@DirectiveDescription(
	"Manually add sources to the project. Does not support chaining, sources are added only once, not recursively."
	)
	@DirectiveLevel(SpecificationLevel.SHOULD)
	final case class Sources(
	@DirectiveName("file")
	files: DirectiveValueParser.WithScopePath[List[Positioned[String]]] =
	DirectiveValueParser.WithScopePath.empty(Nil)
	) extends HasBuildOptions {

Inputs from directives are then extracted here:

scala-cli/modules/build/src/main/scala/scala/build/CrossSources.scala

Lines 207 to 211 in 4fb5c5b

	val sourcesFromDirectives =
	preprocessedInputFromArgs
	.flatMap(_.options)
	.flatMap(_.internal.extraSourceFiles)
	.distinct

cc @tgodzik

[ivan-klass]

Oh, I didn't know "using file" is non-recursive. Then it seems much simpler