search term:

sudori part 3

I’m hacking on a small project called sudori, an experimental sbt. The initial goal is to port the macro to Scala 3. It’s an exercise to take the macro apart and see if we can build it from the ground up. This an advanced area of Scala 2 and 3, and I’m finding my way around by trial and error. This is part 3.

It’s been a while since I wrote part 2, but in between I’ve written intro to Scala 3 macros, which is sort of a sudori prequel.

Starlark 4.2.1

Starlark is a dialect of Python, originally designed as a configuration language for the Bazel build tool. Currently there are implementations in Go, Java, and Rust. As far as I know, the main Java implementation of Starlark has only been available as Bazel’s source repo on GitHub.

Since it would be convenient to have a binary distribution, I’ve forked the repo, and published it as "com.eed3si9n.starlark" % "starlark" % "4.2.1" (com.eed3si9n.starlark:starlark:4.2.1) on Maven Central. The code is the same as Bazel 4.2.1.

JDK 17 on GitHub Actions

Here’s a quick tutorial of how to test your project on JDK 17 using Ólaf’s olafurpg/setup-scala. As the starting point we’ll use the following setup, which is documented in Setting up GitHub Actions with sbt:

name: CI
on:
  pull_request:
  push:
jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        include:
          - os: ubuntu-latest
            java: 11
            jobtype: 1
          - os: ubuntu-latest
            java: 11
            jobtype: 2
          - os: ubuntu-latest
            java: 11
            jobtype: 3
    runs-on: ${{ matrix.os }}
    steps:
    - name: Checkout
      uses: actions/checkout@v1
    - name: Setup
      uses: olafurpg/setup-scala@v13
      with:
        java-version: "adopt@1.${{ matrix.java }}"
    - name: Build and test
      run: |
        case ${{ matrix.jobtype }} in
          1)
            sbt -v "mimaReportBinaryIssues; scalafmtCheckAll; +test;"
            ;;
          2)
            sbt -v "scripted actions/*"
            ;;
          3)
            sbt -v "dependency-management/*"
            ;;
          *)
            echo unknown jobtype
            exit 1
        esac        
      shell: bash

    Read More…
  

intro to Scala 3 macros

Introduction

Macro is a fun and powerful tool, but overuse of the macro could cause harm as well. Please enjoy macros responsibly.

What is macro? A common explanation given is that a macro is a program that is able to take code as an input and output code. While it’s true, it might not immediately make sense since Scala programmers are often familiar with higher-order functions like (map {...}) and by-name parameter, which on the surface it might seem like it is passing a block of code around.

Jar Jar Abrams 1.8.0 and sbt-assembly 1.1.0

Jar Jar Abrams 1.8.0 and sbt-assembly 1.1.0 are released.

Jar Jar Abrams is an experimental extension to Jar Jar Links, intended to shade Scala libraries. Thus far we have been using Pants team’s fork of Jar Jar Links, but now that it’s been abandaned, Eric Peters has in-sourced it to jarjar-abrams repo so we can patch it.

Our jarjar fork is released under com.eed3si9n.jarjar organization name and package name.

sudori part 2

I’m hacking on a small project called sudori, an experimental sbt. The initial goal is to port the macro to Scala 3. It’s an exercise to take the macro apart and see if we can build it from the ground up. This an advanced area of Scala 2 and 3, and I’m finding my way around by trial and error. This is part 2.

Reference:

Instance

When we think of the build.sbt macro, the first thing that comes to our mind is the Applicative do macro that it implements using .value even though some may not use those terms exactly. The main driver for this imperative-to-functional is in the companion object for an oddly named Instance class:

sudori part 1

I’m hacking on a small project called sudori, an experimental sbt. The initial goal is to port the macro to Scala 3. It’s an exercise to take the macro apart and see if we can build it from the ground up. This an advanced area of Scala 2 and 3, and I’m finding my way around by trial and error.

Reference:

Convert

I think I’ve identified a basic part called Convert, which doesn’t really depend on anything.

sbt 1.5.5

I’m happy to announce sbt 1.5.5 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.5.5

Highlights

sbt 1.5.4

I’m happy to announce sbt 1.5.4 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.5.4

How to upgrade

Download the official sbt runner + launcher from SDKMAN or download from https://github.com/sbt/sbt/releases/.

In addition, the sbt version used for your build is upgraded by putting the following in project/build.properties:

sbt.version=1.5.4

This mechanism allows that sbt 1.5.4 is used only for the builds that you want.

sbt-assembly 1.0.0

In June of 2011, I started working on sbt-assembly for sbt 0.10, based on Coda Hale’s assembly-sbt from sbt 0.7, which in turn was probably inspired by maven-assembly-plugin. After ten years, I’m going to call this one 1.0.0. sbt-assembly 1.0.0 is published to Maven Central.

sbt 1.5.3

I’m happy to announce sbt 1.5.3 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.5.3

How to upgrade

Download the official sbt runner + launcher from SDKMAN or download from https://github.com/sbt/sbt/releases/.

In addition, the sbt version used for your build is upgraded by putting the following in project/build.properties:

sbt.version=1.5.3

This mechanism allows that sbt 1.5.3 is used only for the builds that you want.

Scala, Python quick reference

syntax Scala Python
immutable variable val x = 1 Starlark:
REV = "1.1.0"
lazy variable lazy val x = 1 n/a
mutable variable var x = 1 in function:
x = 1
if expression if (x > 1) "a" else "b" "a" if x > 1 else "b"
———————- —————————- —————————-
function def add3(x: Int): Int =
x + 3
def add3(x):
return x + 3
anonymous function _ * 2 not in Starlark:
lambda x: x * 2
———————- —————————- —————————-
List val xs = List(1, 2, 3, 4) xs = [1, 2, 3, 4]
size xs.size len(xs)
empty test xs.isEmpty not xs
head xs.head xs[0]
tail // List(2, 3, 4)
xs.tail
# [2, 3, 4]
xs[1:]
take // List(1, 2)
xs.take(2)
# [1, 2]
xs[:2]
drop // List(3, 4)
xs.drop(2)
# [3, 4]
xs[2:]
drop right // List(1, 2, 3)
xs.dropRight(1)
# [1, 2, 3]
xs[:-1]
nth element xs(2) xs[2]
map xs.map(_ * 2)

for {
x <- xs
} yield x * 2
map(lambda x: x * 2, xs)

[x * 2 for x in xs]
filter xs.filter(_ % 2 == 0)

for {
x <- xs if x % 2 == 0
} yield x
filter(lambda x: not x % 2, xs)

[x for x in xs if not x % 2 ]
fold from left // "a1234"
xs.foldLeft("a") { _ + _ }
from functools import reduce
# "a1234"
reduce(lambda a,x: a + str(x), xs, "a")
membership xs.contains(3) 3 in xs
———————- —————————- —————————-
String val s = "hello" s = "hello"
variable interpolation val count = 3
s"$count items"
not in Starlark:
count = 3
f"{count} items"
split // Array(1.2.3, M1)
"1.2.3-M1".split("-")
# ['1.2.3', 'M1']
"1.2.3-M1".split("-")
substring test s.contains("el") "el" in s
———————- —————————- —————————-
Map val d = Map("a" -> 1,
“b” -> 2)
d = { "a": 1, "b": 2 }

notes

I’m picking up Python and its Bazel dialect Skylark lately. On the other hand, I’m familiar with Scala and its sbt dialect. Often I know exactly what I want to express, and I am fairly certain Python has the equivalent concept as Scala, but just don’t remember the exact incantation. For people starting Scala, maybe they could use this table in reverse.

POM consistency for sbt plugins

There’s a long-standing bug that sbt maintainers have known for a while, which is that when sbt plugin is published to a Maven repository, the POM file sbt generates is not valid. From a mailing list thread titled [0.12] plan for instance, Mark McBride reported it in 2012:

On the maven note, the poms generated for plugins aren’t actually valid. Trying to upload them to artifactory without disabling pom consistency checks fails :/

sbt 1.5.2

I’m happy to announce sbt 1.5.2 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.5.2

How to upgrade

Download the official sbt runner + launcher from SDKMAN or download from https://github.com/sbt/sbt/releases/.

In addition, the sbt version used for your build is upgraded by putting the following in project/build.properties:

sbt.version=1.5.2

This mechanism allows that sbt 1.5.2 is used only for the builds that you want.

Bintray to JFrog Artifactory migration status and sbt 1.5.1

I’m happy to announce sbt 1.5.1 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.5.1. This post will also report the Bintray to JFrog Artifactory migration.

Bintray to JFrog Artifactory migration status

First and foremost, I would like to thank JFrog for their continued support of sbt project and the Scala ecosystem.

As sbt was taking off in the number of contributors and plugins, we had a Bintray-shaped problem. We wanted individuals to create Ivy-layout repository, publish sbt plugins, but somehow aggregate the resolution to them. Having Github sbt organization allowed fluid ownership of plugin sources, but distributing the binary files were challenge as sbt version was churning. We adopted Bintray in 2014 and it provided the distribution mechanism during our growth years. In addition, we used Bintray to host Debian and RPM installers for sbt, paid for by Lightbend.

sbt 1.5.0

Hi everyone. On behalf of the sbt project, I am happy to announce sbt 1.5.0. This is the fifth feature release of sbt 1.x, a binary compatible release focusing on new features. sbt 1.x is released under Semantic Versioning, and the plugins are expected to work throughout the 1.x series.

The headline features of sbt 1.5.0 are:

  • Scala 3 support
  • Eviction error
  • Deprecation of sbt 0.13 syntax
  • Coursier-based launcher

How to upgrade

Download the official sbt launcher from SDKMAN or download from https://github.com/sbt/sbt/releases/tag/v1.5.0. This installer includes the new Coursier-based launcher.

sbt 1.5.0-RC2

Hi everyone. On behalf of the sbt project, I am happy to announce sbt 1.5.0-RC2. This is the fifth feature release of sbt 1.x, a binary compatible release focusing on new features. sbt 1.x is released under Semantic Versioning, and the plugins are expected to work throughout the 1.x series.

  • If no serious issues are found by Saturday, April 3rd 2021, 1.5.0-RC2 will become 1.5.0 final.
  • If no serious issues are found by Saturday, March 27th 2021, 1.5.0-RC1 will become 1.5.0 final.

The headline features of sbt 1.5.0 are:

sbt 1.4.9

I’m happy to announce sbt 1.4.9 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.4.9

How to upgrade

Download the official sbt launcher from SDKMAN or download from https://github.com/sbt/sbt/releases/.

In addition, the sbt version used for your build is upgraded by putting the following in project/build.properties:

sbt.version=1.4.9

This mechanism allows that sbt 1.4.9 is used only for the builds that you want.

sbt 1.4.8

I’m happy to announce sbt 1.4.8 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.4.8

How to upgrade

Download the official sbt launcher from SDKMAN or download from https://github.com/sbt/sbt/releases/.

In addition, the sbt version used for your build is upgraded by putting the following in project/build.properties:

sbt.version=1.4.8

This mechanism allows that sbt 1.4.8 is used only for the builds that you want.

syntactic Scalafix rule for unified slash syntax

In sbt 1.1.0 I implemented unified slash syntax for sbt. Today I sent a pull request to deprecate the old sbt 0.13 shell syntax #6309.

Naturally, the topic of deprecating old syntax for build.sbt also came up.

This is because “unified” slash syntax is called so because it unifies the shell syntax and the build syntax together. Thus, it makes sense to deprecate the old build.sbt syntax that uses in like skip in publish or scalacOptions in (Compile, console), if we’re deprecating the old shell syntax.

git bisecting scala/scala

git bisecting is a useful technique to locate the source of a bug. For scala/scala in particular, bisect.sh can save a lot of time by using the pre-build compiler artifacts on the Scala CI Artifactory.

sbt 1.4.7

I’m happy to announce sbt 1.4.7 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.4.7

How to upgrade

Download the official sbt launcher from SDKMAN or download from https://github.com/sbt/sbt/releases/.

In addition, the sbt version used for your build is upgraded by putting the following in project/build.properties:

sbt.version=1.4.7

This mechanism allows that sbt 1.4.7 is used only for the builds that you want.

sbt 1.4.6

I’m happy to announce sbt 1.4.6 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.4.6

How to upgrade

Download the official sbt launcher from SDKMAN or download from https://github.com/sbt/sbt/releases/.

In addition, the sbt version used for your build is upgraded by putting the following in project/build.properties:

sbt.version=1.4.6

This mechanism allows that sbt 1.4.6 is used only for the builds that you want.

enforcing Semantic Versioning with sbt-strict-update

Rob wrote:

I want to tell sbt “this specific version breaks binary compatibility, so don’t resolve it via eviction, fail the build instead.” How do I do this? Complete answers only, I’m done trying to figure it out by following clues.

I wrote a small sbt plugin sbt-strict-update to do this.

Add this to project/plugins.sbt:

addSbtPlugin("com.eed3si9n" % "sbt-strict-update" % "0.1.0")

    Read More…
  

sbt 1.4.5

I’m happy to announce sbt 1.4.5 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.4.5

How to upgrade

Download the official sbt launcher from SDKMAN or download from https://github.com/sbt/sbt/releases/.

In addition, the sbt version used for your build is upgraded by putting the following in project/build.properties:

sbt.version=1.4.5

This mechanism allows that sbt 1.4.5 is used only for the builds that you want.

auto publish sbt plugin from GitHub Actions

This is a GitHub Actions version of auto publish sbt plugin from Travis CI.

In this post, we’ll try to automate the release of an sbt plugin using Ólaf’s olafurpg/sbt-ci-release. The README of sbt-ci-release covers the use case for a library published to Sonatype OSS. Read it thoroughly since this post will skip over the details that do not change for publishing sbt plugins.

Automated release in general is a best practice, but there’s one benefit specifically for sbt plugin releases. Using this setup allows multiple people to share the authorization to release an sbt plugin without adding them to Bintray sbt organization. This is useful for plugins maintained at work.

scopt 4

This post was first published in December 2018 together with 4.0.0-RC2. It’s updated to reflect the changes made in November 2020 for 4.0.0.

You can skip to the readme, if you’re in a hurry.

To try new scopt 4.0.0:

libraryDependencies += "com.github.scopt" %% "scopt" % "4.0.0"

scopt 4.0.0 is cross published for the following build matrix:

Weehawken-Lang1

about Weehawken-Lang

It’s a strange time we live in. We can’t just meet up and catch up and talk about coding. This also opens an opportunity to think more virtually about the idea of meetups.

I want to start Weehawken-Lang, a virtual meetup group about programming languages and tooling design (compilers, interpreters, build tools etc). It aims to be a casual place where people with different language backgrounds can exchange ideas about programming languages.

sbt 1.4.4

I’m happy to announce sbt 1.4.4 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.4.4

How to upgrade

Download the official sbt launcher from SDKMAN or download from https://github.com/sbt/sbt/releases/.

In addition, the sbt version used for your build is upgraded by putting the following in project/build.properties:

sbt.version=1.4.4

This mechanism allows that sbt 1.4.4 is used only for the builds that you want.

sbt 1.4.3

I’m happy to announce sbt 1.4.3 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.4.3

How to upgrade

Download the official sbt launcher from SDKMAN or download from https://github.com/sbt/sbt/releases/. This installer includes the sbtn binary.

In addition, the sbt version used for your build is upgraded by putting the following in project/build.properties:

sbt.version=1.4.3

This mechanism allows that sbt 1.4.3 is used only for the builds that you want.

sbt 1.4.2

I’m happy to announce sbt 1.4.2 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.4.2

How to upgrade

Download the official sbt launcher from SDKMAN or download from https://github.com/sbt/sbt/releases/. This installer includes the sbtn binary.

In addition, the sbt version used for your build is upgraded by putting the following in project/build.properties:

sbt.version=1.4.2

This mechanism allows that sbt 1.4.2 is used only for the builds that you want.

remote caching sbt builds with Bintray

The feature in sbt and Zinc 1.4.x that I spent most amount of time and energy probably is the virtualization of file, and lifting out timestamps. Combined together, we can liberate the Zinc state from machine-specificity and time, and become the foundation we lay towards building incremental remote caching for Scala. I blogged about this in cached compilation for sbt. This is part 2.

Now that sbt 1.4.x is out, there is a growing interest in this feature among people who want to try this out.

sbt 1.4.1

I’m happy to announce sbt 1.4.1 patch release is available. Full release note is here - https://github.com/sbt/sbt/releases/tag/v1.4.1

How to upgrade

Download the official sbt launcher from SDKMAN or download from https://www.scala-sbt.org/download.html. This installer includes the sbtn binary.

In addition, the sbt version used for your build is upgraded by putting the following in project/build.properties:

sbt.version=1.4.1

This mechanism allows that sbt 1.4.1 is used only for the builds that you want.

virtualizing a hackathon at ScalaMatsuri 2020

Here’s a report of running a virtual hackathon at ScalaMatsuri Day 2 Unconference. Someone proposed it for the Unconference, and I volunteered to be a facilitator on the day, so I went in without preparation. I booked the time originally for 4h (noon - 4pm JST, 11pm - 3am EDT) but it was successful so it got extended after some coffee break.

One thing I emphasize is The Law of Two Feet:

sbt 1.4.0

Hi everyone. On behalf of the sbt project, I am happy to announce sbt 1.4.0. This is the fourth feature release of sbt 1.x, a binary compatible release focusing on new features. sbt 1.x is released under Semantic Versioning, and the plugins are expected to work throughout the 1.x series.

The headline features of sbt 1.4.0 are:

  • build server protocol (BSP) support
  • sbtn: a native thin client for sbt
  • build caching
  • ThisBuild / versionScheme to take the guessing out of eviction warning

sbt 1.4.0-RC2

Hi everyone. On behalf of the sbt project, I am happy to announce sbt 1.4.0-RC2. This is the fourth feature release of sbt 1.x, a binary compatible release focusing on new features. sbt 1.x is released under Semantic Versioning, and the plugins are expected to work throughout the 1.x series.

  • If no serious issues are found by Saturday, October 3rd 2020, 1.4.0-RC2 will become 1.4.0 final.
  • If no serious issues are found by Saturday, September 19th 2020, 1.4.0-RC1 will become 1.4.0 final.

The headline features of sbt 1.4.0 are:

  • build server protocol (BSP) support
  • sbtn: a native thin client for sbt
  • build caching
  • ThisBuild / versionScheme to take the guessing out of eviction warning

parallel cross building sandwich

This is part 4 of the post about sbt-projectmatrix, an experimental plugin that I’ve been working to improve the cross building in sbt. Here’s part 1, part 2, and part 3. I’ve just released 0.6.0.

recap: building against multiple Scala versions

After adding sbt-projectmatrix to your build, here’s how you can set up a matrix with two Scala versions.

ThisBuild / organization := "com.example"
ThisBuild / scalaVersion := "2.12.12"
ThisBuild / version      := "0.1.0-SNAPSHOT"

lazy val core = (projectMatrix in file("core"))
  .settings(
    name := "core"
  )
  .jvmPlatform(scalaVersions = Seq("2.12.12", "2.13.3"))

    Read More…
  

joining Twitter

I’m excited to announce that I’m joining Twitter’s Build Team to work on the next generation of efficient build systems supporting thousands of Twitter developers worldwide. Today’s my first day.

This is the team that developed monorepo build tool Pants, and is transitioning to migrate the flock to Bazel. This presented a unique opportunity for me to work with a team of people passionate about developer experience and productivity, and I’m looking forward to getting to know the team, and learning the new challenges.

sbt-buildinfo 0.10.0

I’m happy to announce sbt-buildinfo 0.10.0. sbt-buildinfo is a small sbt plugin to generate BuildInfo object from your build definitions.

Since the last feature release was in 2018, there have been some pending contributions. I think the important thing is that it compiles with -Xlint and -Xfatal-warnings on both Scala 2.13.3 and 2.12.12.

auto publish sbt plugin from Travis CI

In this post, we’ll try to automate the release of an sbt plugin using Ólafur’s olafurpg/sbt-ci-release. The README of sbt-ci-release covers the use case for a library published to Sonatype OSS. Read it thoroughly since this post will skip over the details that do not change for publishing sbt plugins.

Automated release in general is a best practice, but there’s one benefit specifically for sbt plugin releases. Using this setup allows multiple people to share the authorization to release an sbt plugin without adding them to Bintray sbt organization. This is useful for plugins maintained at work.