keeping promise in sbt
build.sbt is a DSL for defining a task graph to be used for automatic parallel processing. The message passing among the tasks are expressed using something.value
macro, which encodes Applicative composition (task1, task2) mapN { case (t1, t2) => .... }
.
One mechanism I've been thinking about is allowing some long-running task1
to communicate with task2
midway.
Normally, we would break down task1
into two subtasks. But it might not be as straight-forward to implement such thing. For example, how would be tell Zinc to compile something halfway, and resume later? Or tell Coursier to resolve, but fetch later?
As a starting point, we could think of a solution where task1
generates some JSON file, and task2
can try to wait until the file appears, and read from it. We can improve this by replacing JSON file with a concurrent data structure, such as Promise[A]
. Still there's the complication of waiting. sbt limits the number of tasks that would run in parallel, and it would be wasteful to use a slot for waiting. Daniel's Thread Pools post is informative in this regard. What we have is a blocking IO polling that won't do any work.
Def.promise
I've implemented a wrapper around scala.concurrent.Promise
called Def.promise
. Here's an example usage:
val midpoint = taskKey[PromiseWrap[Int]]("") val longRunning = taskKey[Unit]("") val task2 = taskKey[Unit]("don't call this from shell") val joinTwo = taskKey[Unit]("") // Global / concurrentRestrictions := Seq(Tags.limitAll(1)) lazy val root = (project in file(".")) .settings( name := "promise", midpoint := Def.promise[Int], longRunning := { val p = midpoint.value val st = streams.value st.log.info("start") Thread.sleep(1000) p.success(5) Thread.sleep(1000) st.log.info("end") }, task2 := { val st = streams.value val x = midpoint.await.value st.log.info(s"got $x in the middle") }, joinTwo := { val x = longRunning.value val y = task2.value } )
First, we create a PromiseWrap[Int]
task called midpoint
. This is still a task because we need a fresh promise for each command invocation. Next we have longRunning
task, which completes the promise halfway. task2
depends on midpoint.await.value
. This mean that sbt's task scheduler won't start task2
until midpoint
promise is completed.
To run both longRunning
and task2
together, we define joinTwo
task. This runs as follows:
sbt:promise> joinTwo [info] start [info] got 5 in the middle [info] end
As you can see above, we were able to run both tasks in parallel while letting longRunning
task pass message to task2
.
Warning: If you call task2
from the shell, it will be blocked forever and will not return. You'd have to Ctrl-C to cancel out of it.
summary
Def.promise
sbt/sbt#5552 is a draft proposal to allow long-running tasks to pass message to another task. One of the potential use case for this is pipelining of subproject compilations.
- Login to post comments