fast Scala 3 parsing with tree-sitter

4h9m
I was a Staff Engineer at Twitter’s Build/Bazel Migration team. After two incredible years, November 17 was my last day (I took the voluntary separation offer and resigned, not that it matters). Twitter has been a special place to work for, for its culture of excellence, diversity, and outpouring of care for all the people that made Flock the Flock. I am grateful that I got the opportunity to experience that firsthand, and be part of it.
Here’s a quick retrospective on my last two years. Info available here are based on publicly available talks and data. Just from our team 10+ members left Twitter after the buyout, so I’ve sprinkled this post with links to their LinkedIn profiles both current and former.
Hi everyone. On behalf of the sbt project, I am happy to announce sbt 1.8.0. This is the eighth feature release of sbt 1.x, a binary compatible release focusing on new features. sbt 1.x is released under Semantic Versioning, and the plugins are expected to work throughout the 1.x series. Please try it out, and report any issues you might come across.
Hi everyone. On behalf of the sbt project, I am happy to announce sbt 1.8.0-RC1. This is the eighth feature release of sbt 1.x, a binary compatible release focusing on new features. sbt 1.x is released under Semantic Versioning, and the plugins are expected to work throughout the 1.x series. Please try it out, and report any issues you might come across.
sbt 1.8.0 is a small release focused on upgrading scala-xml to 2.x. In theory this breaks the binary compatibility in the plugin ecosystem, but in practice there’s already a mixture of both 1.x and 2.x.
If you encounter a conflict in plugins, try putting the following in project/plugins.sbt
:
ThisBuild / libraryDependencySchemes += "org.scala-lang.modules" %% "scala-xml" % VersionScheme.Always
I’m happy to announce sbt 1.7.3 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.7.3
See 1.7.0 release note for the details on 1.7.x features.
I’m happy to announce sbt 1.7.2 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.7.2
See 1.7.0 release note for the details on 1.7.x features.
testQuick
task #6903 by @gontardsbt new
by default to use Giter8 0.15.0diagnosticCode
and diagnosticRelatedInforamation
(sic) to InterfaceUtil.problem(...)
#7006 by @ckipp01diagnosticCode
to BSP #6998 by @ckipp014h2m
I’m happy to announce sbt 1.7.1 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.7.1
See 1.7.0 release note for the details on 1.7.x features.
Hi everyone. On behalf of the sbt project, I am happy to announce sbt 1.7.0. This is the seventh feature release of sbt 1.x, a binary compatible release focusing on new features. sbt 1.x is released under Semantic Versioning, and the plugins are expected to work throughout the 1.x series. Please try it out, and report any issues you might come across.
Download the official sbt runner from cs setup
, SDKMAN, or download from https://github.com/sbt/sbt/releases/tag/v1.7.0.
The sbt version used for your build is upgraded by putting the following in project/build.properties
:
sbt.version=1.7.0
This mechanism allows that sbt 1.7.0 is used only for the builds that you want.
scopt 4.1.0 is released. To try new scopt 4.1.0:
libraryDependencies += "com.github.scopt" %% "scopt" % "4.1.0"
scopt 4.1.0 is cross published to the following build matrix:
Scala Version | JVM | JS (1.x) | JS (0.6.x) | Native (0.4.x) |
---|---|---|---|---|
3.x | ✅ | ✅ | n/a | ✅ |
2.13.x | ✅ | ✅ | ✅ | ✅ |
2.12.x | ✅ | ✅ | ✅ | ✅ |
2.11.x | ✅ | ✅ | ✅ | ✅ |
scopt is a little command line options parsing library. See https://eed3si9n.com/scopt4 or readme for the details on how to use scopt.
3h53m
I released Eval 0.1.0. Eval evaluates Scala 3 code. It’s a Scala 3 port of Eval
class used in sbt.
package example
import com.eed3si9n.eval.Eval
import com.eed3si9n.eval.EvalReporter
import java.nio.file.Paths
@main def main(): Unit =
val eval = Eval(
backingDir = Paths.get("/tmp/classes"),
mkReporter = () => EvalReporter.store
)
val result = eval.evalInfer("2")
println(result.tpe)
println(result.getValue(this.getClass.getClassLoader))
The above prints:
Int
2
Jar Jar Abrams 1.8.1 and sbt-assembly 1.2.0 are released.
Jar Jar Abrams is an experimental extension to Jar Jar Links, intended to shade Scala libraries.
sbt is simple, in a sense that it has a few concepts like settings and tasks, and it achieves a wide variety of things. An idea popped into my head today that could simplify sbt further. I don’t have an implementation on this yet.
I’m happy to announce sbt 1.6.2 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.6.2
See 1.6.0 release note for the details on 1.6.x features.
In the context of sbt, Bazel, and likely many other build tools, the term test could encompass various levels, and it’s useful to disamgibuate this, especially when we want to configure pre- and post-hooks and parallel execution. In other words, what do we mean when we say “test”?
There are four levels to test:
The top-most level is the test
command that the build tools provide to the users.
I’m happy to announce sbt 1.6.1 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.6.1
See 1.6.0 release note for the details on 1.6.x features.
Hi everyone. On behalf of the sbt project, I am happy to announce sbt 1.6.0. This is the sixth feature release of sbt 1.x, a binary compatible release focusing on new features. sbt 1.x is released under Semantic Versioning, and the plugins are expected to work throughout the 1.x series. Please try it out, and report any issues you might come across.
The headline features of sbt 1.6.0 are:
I’m happy to announce sbt 1.5.8 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.5.8
Hi everyone. On behalf of the sbt project, I am happy to announce sbt 1.6.0-RC2. This is the sixth feature release of sbt 1.x, a binary compatible release focusing on new features. sbt 1.x is released under Semantic Versioning, and the plugins are expected to work throughout the 1.x series. Please try it out, and report any issues you might come across.
The headline features of sbt 1.6.0 are:
I’m happy to announce sbt 1.5.7 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.5.7
I’m happy to announce sbt 1.5.6 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.5.6
I’m hacking on a small project called sudori, an experimental sbt. The initial goal is to port the macro to Scala 3. It’s an exercise to take the macro apart and see if we can build it from the ground up. This an advanced area of Scala 2 and 3, and I’m finding my way around by trial and error. This is part 3.
It’s been a while since I wrote part 2, but in between I’ve written intro to Scala 3 macros, which is sort of a sudori prequel.
Starlark is a dialect of Python, originally designed as a configuration language for the Bazel build tool. Currently there are implementations in Go, Java, and Rust. As far as I know, the main Java implementation of Starlark has only been available as Bazel’s source repo on GitHub.
Since it would be convenient to have a binary distribution, I’ve forked the repo, and published it as "com.eed3si9n.starlark" % "starlark" % "4.2.1"
(com.eed3si9n.starlark:starlark:4.2.1
) on Maven Central. The code is the same as Bazel 4.2.1.
Here’s a quick tutorial of how to test your project on JDK 17 using Ólaf’s olafurpg/setup-scala. As the starting point we’ll use the following setup, which is documented in Setting up GitHub Actions with sbt:
name: CI
on:
pull_request:
push:
jobs:
test:
strategy:
fail-fast: false
matrix:
include:
- os: ubuntu-latest
java: 11
jobtype: 1
- os: ubuntu-latest
java: 11
jobtype: 2
- os: ubuntu-latest
java: 11
jobtype: 3
runs-on: ${{ matrix.os }}
steps:
- name: Checkout
uses: actions/checkout@v1
- name: Setup
uses: olafurpg/setup-scala@v13
with:
java-version: "adopt@1.${{ matrix.java }}"
- name: Build and test
run: |
case ${{ matrix.jobtype }} in
1)
sbt -v "mimaReportBinaryIssues; scalafmtCheckAll; +test;"
;;
2)
sbt -v "scripted actions/*"
;;
3)
sbt -v "dependency-management/*"
;;
*)
echo unknown jobtype
exit 1
esac
shell: bash
Let’s say for jobtype
3 we’d like to use JDK 8, and for jobtype
1 and 2 we’d like to test on JDK 17. sbt-ci-release uses jabba to grab the JDKs, and at the moment the openjdk 17.0 distros are not available on jabba yet. However, Eclipse Adoptium fka AdoptOpenJDK does have the binary available, so we can use the custom JDK mode to use it as follows:
Macro is a fun and powerful tool, but overuse of the macro could cause harm as well. Please enjoy macros responsibly.
What is macro? A common explanation given is that a macro is a program that is able to take code as an input and output code. While it’s true, it might not immediately make sense since Scala programmers are often familiar with higher-order functions like (map {...}
) and by-name parameter, which on the surface it might seem like it is passing a block of code around.
Jar Jar Abrams 1.8.0 and sbt-assembly 1.1.0 are released.
Jar Jar Abrams is an experimental extension to Jar Jar Links, intended to shade Scala libraries. Thus far we have been using Pants team’s fork of Jar Jar Links, but now that it’s been abandaned, Eric Peters has in-sourced it to jarjar-abrams repo so we can patch it.
Our jarjar
fork is released under com.eed3si9n.jarjar
organization name and package name.
ShadeRules.keep
.sbt-assembly 1.1.0 upgrades the Jar Jar Abrams dependency to 1.8.0.
I’m hacking on a small project called sudori, an experimental sbt. The initial goal is to port the macro to Scala 3. It’s an exercise to take the macro apart and see if we can build it from the ground up. This an advanced area of Scala 2 and 3, and I’m finding my way around by trial and error. This is part 2.
Reference:
When we think of the build.sbt
macro, the first thing that comes to our mind is the Applicative do macro that it implements using .value
even though some may not use those terms exactly. The main driver for this imperative-to-functional is in the companion object for an oddly named Instance class:
I’m hacking on a small project called sudori, an experimental sbt. The initial goal is to port the macro to Scala 3. It’s an exercise to take the macro apart and see if we can build it from the ground up. This an advanced area of Scala 2 and 3, and I’m finding my way around by trial and error.
Reference:
I think I’ve identified a basic part called Convert, which doesn’t really depend on anything.
I’m happy to announce sbt 1.5.5 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.5.5
buildTarget/resources
support for BSP #6552 by @samuelClarencTeadsbuild.sbt
support for BSP import #6553 by @retronymNoClassDefFoundError
when launching sbt 1.4.0 - 1.4.2 launcher#98 by @eed3si9n_3
lm#383 by @eed3si9nI’m happy to announce sbt 1.5.4 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.5.4
Download the official sbt runner + launcher from SDKMAN or download from https://github.com/sbt/sbt/releases/.
In addition, the sbt version used for your build is upgraded by putting the following in project/build.properties
:
sbt.version=1.5.4
This mechanism allows that sbt 1.5.4 is used only for the builds that you want.
compilerJars.toList
(For Scala 3, this drops support for 3.0.0-M2) #6538 by @adpi2-release
flag zinc#982 by @retronymFor more details please see https://github.com/sbt/sbt/releases/tag/v1.5.4
In June of 2011, I started working on sbt-assembly for sbt 0.10, based on Coda Hale’s assembly-sbt from sbt 0.7, which in turn was probably inspired by maven-assembly-plugin. After ten years, I’m going to call this one 1.0.0. sbt-assembly 1.0.0 is published to Maven Central.
I’m happy to announce sbt 1.5.3 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.5.3
Download the official sbt runner + launcher from SDKMAN or download from https://github.com/sbt/sbt/releases/.
In addition, the sbt version used for your build is upgraded by putting the following in project/build.properties
:
sbt.version=1.5.3
This mechanism allows that sbt 1.5.3 is used only for the builds that you want.
scalacOptions
not getting forwarded to ScalaDoc in Scala 3 #6499 by @pikinier20inputFile
resolving to incorrect files when file specific globs are used io#319 by @eatkinsFor more details please see https://github.com/sbt/sbt/releases/tag/v1.5.3
syntax | Scala | Python |
---|---|---|
immutable variable | val x = 1 |
Starlark:REV = "1.1.0" |
lazy variable | lazy val x = 1 |
n/a |
mutable variable | var x = 1 |
in function:x = 1 |
if expression | if (x > 1) "a" else "b" |
"a" if x > 1 else "b" |
———————- | —————————- | —————————- |
function | def add3(x: Int): Int = x + 3 |
def add3(x): return x + 3 |
anonymous function | _ * 2 |
not in Starlark:lambda x: x * 2 |
———————- | —————————- | —————————- |
List | val xs = List(1, 2, 3, 4) |
xs = [1, 2, 3, 4] |
size | xs.size |
len(xs) |
empty test | xs.isEmpty |
not xs |
head | xs.head |
xs[0] |
tail | // List(2, 3, 4) xs.tail |
# [2, 3, 4] xs[1:] |
take | // List(1, 2) xs.take(2) |
# [1, 2] xs[:2] |
drop | // List(3, 4) xs.drop(2) |
# [3, 4] xs[2:] |
drop right | // List(1, 2, 3) xs.dropRight(1) |
# [1, 2, 3] xs[:-1] |
nth element | xs(2) |
xs[2] |
map | xs.map(_ * 2) for { x <- xs } yield x * 2 |
map(lambda x: x * 2, xs) [x * 2 for x in xs] |
filter | xs.filter(_ % 2 == 0) for { x <- xs if x % 2 == 0 } yield x |
filter(lambda x: not x % 2, xs) [x for x in xs if not x % 2 ] |
fold from left | // "a1234" xs.foldLeft("a") { _ + _ } |
from functools import reduce # "a1234" reduce(lambda a,x: a + str(x), xs, "a") |
membership | xs.contains(3) |
3 in xs |
———————- | —————————- | —————————- |
String | val s = "hello" |
s = "hello" |
variable interpolation | val count = 3 s"$count items" |
not in Starlark:count = 3 f"{count} items" |
split | // Array(1.2.3, M1) "1.2.3-M1".split("-") |
# ['1.2.3', 'M1'] "1.2.3-M1".split("-") |
substring test | s.contains("el") |
"el" in s |
———————- | —————————- | —————————- |
Map | val d = Map("a" -> 1, “b” -> 2) |
d = { "a": 1, "b": 2 } |
I’m picking up Python and its Bazel dialect Skylark lately. On the other hand, I’m familiar with Scala and its sbt dialect. Often I know exactly what I want to express, and I am fairly certain Python has the equivalent concept as Scala, but just don’t remember the exact incantation. For people starting Scala, maybe they could use this table in reverse.
There’s a long-standing bug that sbt maintainers have known for a while, which is that when sbt plugin is published to a Maven repository, the POM file sbt generates is not valid. From a mailing list thread titled [0.12] plan for instance, Mark McBride reported it in 2012:
On the maven note, the poms generated for plugins aren’t actually valid. Trying to upload them to artifactory without disabling pom consistency checks fails :/
I’m happy to announce sbt 1.5.2 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.5.2
Download the official sbt runner + launcher from SDKMAN or download from https://github.com/sbt/sbt/releases/.
In addition, the sbt version used for your build is upgraded by putting the following in project/build.properties
:
sbt.version=1.5.2
This mechanism allows that sbt 1.5.2 is used only for the builds that you want.
sbt new
leaving behind target
directory #6488 by @eed3si9nConcurrentModificationException
while compiling Scala 2.13.4 and Java sources zinc#974 by @lefou-client
by making it the same as --client
#6500 by @Nirvikalpa108-Duser.home
instead of $HOME
to download launcher JAR #6483 by @rdesgroppesFor more details please see https://github.com/sbt/sbt/releases/tag/v1.5.2
I’m happy to announce sbt 1.5.1 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.5.1. This post will also report the Bintray to JFrog Artifactory migration.
First and foremost, I would like to thank JFrog for their continued support of sbt project and the Scala ecosystem.
As sbt was taking off in the number of contributors and plugins, we had a Bintray-shaped problem. We wanted individuals to create Ivy-layout repository, publish sbt plugins, but somehow aggregate the resolution to them. Having Github sbt organization allowed fluid ownership of plugin sources, but distributing the binary files were challenge as sbt version was churning. We adopted Bintray in 2014 and it provided the distribution mechanism during our growth years. In addition, we used Bintray to host Debian and RPM installers for sbt, paid for by Lightbend.
Wrote herding cats: day 19 featuring FunctionK
, or Rúnar’s encoding of rank-2 polymorphic function, and Resource datatype, which he envisioned rank-N polymorphism would unlock back in 2010.
Hi everyone. On behalf of the sbt project, I am happy to announce sbt 1.5.0. This is the fifth feature release of sbt 1.x, a binary compatible release focusing on new features. sbt 1.x is released under Semantic Versioning, and the plugins are expected to work throughout the 1.x series.
The headline features of sbt 1.5.0 are:
Download the official sbt launcher from SDKMAN or download from https://github.com/sbt/sbt/releases/tag/v1.5.0. This installer includes the new Coursier-based launcher.
Hi everyone. On behalf of the sbt project, I am happy to announce sbt 1.5.0-RC2. This is the fifth feature release of sbt 1.x, a binary compatible release focusing on new features. sbt 1.x is released under Semantic Versioning, and the plugins are expected to work throughout the 1.x series.
The headline features of sbt 1.5.0 are:
I’m happy to announce sbt 1.4.9 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.4.9
Download the official sbt launcher from SDKMAN or download from https://github.com/sbt/sbt/releases/.
In addition, the sbt version used for your build is upgraded by putting the following in project/build.properties
:
sbt.version=1.4.9
This mechanism allows that sbt 1.4.9 is used only for the builds that you want.
sbt 1.4.9 is published to Sonatype OSS without going through Bintray.
I’m happy to announce sbt 1.4.8 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.4.8
Download the official sbt launcher from SDKMAN or download from https://github.com/sbt/sbt/releases/.
In addition, the sbt version used for your build is upgraded by putting the following in project/build.properties
:
sbt.version=1.4.8
This mechanism allows that sbt 1.4.8 is used only for the builds that you want.
sbt 1.4.8 is published to Sonatype OSS without going through Bintray.