search term:

sbt plugin classpath isolation

A build performs an assortment of tasks, often reusing existing libraries and tools. The plugin mechanism in sbt enables reuse, by adding libraries to the metabuild. This works well for lightweight tasks, or for integrating with CLI tools like gpg. However, adding more libraries to the metabuild may not be desirable or possible. For example, tools like Scalafix or Coursier are written in some Scala version, which may not be compatible with the Scala version used by sbt 2.x or 1.x. For this reason, the idea of plugin classpath isolation comes up occasionally, and we’ll look into it in this post. See Isolate plugin classpath recipe for the full source.

overview

The gist of the idea is:

  1. Define a command-line app.
  2. Execute the command-line app from your plugin via run.

Because sbt’s run task uses a sandbox classloader instead of shelling out, it is effectively the same as having classpath isolation. The additional feature needed for classpath isolation has been around since sbt 0.13.13. In fact, I wrote about a similar approach in downloading and running app on the side in 2017. This is an improved version that doesn’t need sbt-sidedish.

synthetic subproject

A plugin can create a synthetic subproject by overriding extraProjects:

package example

import sbt.{ *, given }
import Keys.*

object BootstrapPlugin extends AutoPlugin:
  override lazy val requires = sbt.plugins.JvmPlugin

  lazy val bootstrapCs = project
    .settings(
      scalaVersion := "2.12.21",
      libraryDependencies += "io.get-coursier" %% "coursier-cli" % "2.1.14",
      Compile / run / mainClass := Some("coursier.cli.Coursier"),
      // applicable to sbt 2.x only
      clientSide := false,
    )

  override lazy val extraProjects = Vector(bootstrapCs)

  ....
end BootstrapPlugin

In this example, we add the Coursier CLI, which was built with Scala 2.12. Since it’s injected into the build, prefix the name with your plugin name, like bootstrapCs.

calling the command-line

Using the Coursier CLI, we can implement a task that generates a bootstrap JAR. Let’s start with the following keys:

  object autoImport:
    val packageBootstrap = taskKey[HashedVirtualFileRef]("packageBootstrap")
    val packageBootstrapArgs = settingKey[Seq[String]]("packageBootstrapArgs")
    val packageBootstrapOutput = settingKey[File]("packageBootstrapOutput")
  end autoImport
  import autoImport.*

The actual implementation looks as follows:

  override lazy val projectSettings = Vector(
    packageBootstrapOutput := target.value / "bootstrap" / s"${moduleName.value}.jar",
    packageBootstrapArgs := {
      val coord =
        s"${organization.value}:${name.value}_${scalaBinaryVersion.value}:${version.value}"
      val sv = scalaVersion.value
      Vector("bootstrap", "--verbose", "--bat=true",
        "--scala-version", sv,
        "-f", coord,
        "-o", packageBootstrapOutput.value.toString)
    },
    packageBootstrap := Def.uncached {
      // to process args before toTask, we need to use dynamic tasks
      (Def.taskDyn {
        // phase 1
        val c = fileConverter.value
        val args = packageBootstrapArgs.value
        val out = packageBootstrapOutput.value
        // phase 2
        val outVf: HashedVirtualFileRef = c.toVirtualFile(out.toPath())
        IO.createDirectory(out.getParentFile())
        // phase 3
        (bootstrapCs / Compile / run)
          .toTask(args.mkString(" ", " ", ""))
          .map(_ => outVf)
      }).value
    },
  )

In the above, packageBootstrapOutput and packageBootstrapArgs are settings to construct the command-line arguments that will be passed in to Coursier CLI. The packageBootstrap task uses Def.taskDyn, or a dynamic task, which lets us compose tasks sequentially (our encoding of flatMap).

The input into Coursier CLI is string arguments, and produces files as output. Its console output is automatically displayed to the terminal:

sbt:isolation-root> app/publishLocal
sbt:isolation-root> app/packageBootstrap
[info] running coursier.cli.Coursier bootstrap --verbose --bat=true --scala-version 3.8.4 -f com.example:hello_3:0.1.0-SNAPSHOT -o /.../isolation/target/out/jvm/scala-3.8.4/hello/bootstrap/hello.jar
  Dependencies:
com.example:hello_3:0.1.0-SNAPSHOT:
Wrote /.../isolation/target/out/jvm/scala-3.8.4/hello/bootstrap/hello.jar
Wrote /.../isolation/target/out/jvm/scala-3.8.4/hello/bootstrap/hello.jar.bat
[success] elapsed time: 3 s, cache 100%, 17 disk cache hits

This shows that app/packageBootstrap in sbt 2.x called Coursier CLI to build the bootstrap JAR.

a note on forking

When defining the bootstrapCs subproject, we intentionally set clientSide to false:

clientSide := false,

This overrides the default client-side run, so Coursier CLI will execute inside the same JVM as the sbt server. One caveat is that CLI programs often call sys.exit(1), which will shut down the sbt server:

sbt:isolation-root> bootstrapCs/run --help
[info] running coursier.cli.Coursier --help
Usage: coursier <COMMAND>
Coursier is the Scala application and artifact manager.
It can install Scala applications and setup your Scala development environment.
It can also download and cache artifacts from the web.
....
$

If you want to protect your build from sys.exit(1), you have to fork the run:

// default for sbtn
clientSide := true,
// for sbt --server
Compile / run / fork := true,

The tradeoff is that forking runs slower than the in-process run, due to JVM warmup, so it might depend on how many times the task gets called.

summary

sbt plugin classpath isolation can be implemented in both sbt 2.x and 1.x by creating a CLI program as a synthetic subproject in a Scala version of your choice. See Isolate plugin classpath recipe for the full source.