search term:

semantics of dependency resolvers

The semantics of a dependency resolver determine the concrete classpath based on the user-specified dependency constraints. Typically the differences in the details manifest as different way the version conflicts are resolved.

  • Maven uses nearest-wins strategy, which could downgrade transitive dependencies
  • Ivy uses latest-wins strategy
  • Coursier generally uses latest-wins strategy, but it’s tries to enforce version range strictly
  • Ivy’s version range handling goes to the Internet, which makes the build non-repeatable
  • Coursier orders version string completely differently from Ivy

parallel cross building using sbt-projectmatrix

Last year I wrote an experimental sbt plugin called sbt-projectmatrix to improve the cross building in sbt. I’ve just released 0.2.0.

building against multiple Scala versions

After adding sbt-projectmatrix to your build, here’s how you can set up a matrix with two Scala versions.

ThisBuild / organization := "com.example"
ThisBuild / scalaVersion := "2.12.8"
ThisBuild / version      := "0.1.0-SNAPSHOT"

lazy val core = (projectMatrix in file("core"))
  .settings(
    name := "core"
  )
  .jvmPlatform(scalaVersions = Seq("2.12.8", "2.11.12"))

This will create subprojects coreJVM2_11 and coreJVM2_12. Unlike ++ style stateful cross building, these will build in parallel.

Read More…

splitting git repo

split a subdirectory into a new repo (simple case)

git clone --no-hardlinks --branch master originalRepoURL childRepo
cd childRepo
git filter-branch --prune-empty --subdirectory-filter path/to/keep master
git remote remove origin
git prune
git gc --aggressive

Change originalRepoURL, master, and path/to/keep to appropriate values. Use -- --all to handle all branches.

split a subdirectory into a new repo (complex case)

In case you have multiple paths you want to filter, you need to use --index-filter together with GNU xargs and GNU sed available via brew install gnu-sed findutils.

Read More…

all your JDKs on Travis CI using SDKMAN!

This is a second post on installing your own JDKs on Travis CI. Previously I’ve written about jabba.

Today, let’s look at SDKMAN!, an environment manager written by Marco Vermeulen (@marc0der) for JDKs and various tools on JVM, including Groovy, Spark, sbt, etc.

AdoptOpenJDK 11 and 8

  • Update 2020-09-23: Updated the regex of version number.
  • Update 2019-11-06: Added sdkman_auto_selfupdate to workaround the update prompt blocking the CI. Also it adds || true on the sdk install line.
  • Update 2019-07-08: Updated the script to detect patch version. See GitHub for the older version.

Here’s how we can use SDKMAN! on Travis CI to cross build using AdoptOpenJDK 11 and 8:

Read More…

'Ancillary Justice' by Ann Leckie

Ancillary Justice (Imperial Radch, #1)Ancillary Justice by Ann Leckie
My rating: 4 of 5 stars

‘Ancillary Justice’ is written in a first-person narrative as the AI of thousands of years old starship Justice of Toren of The Radchaai Empire. Another interesting part is since the Radchaai language is genderless, the narrator constantly describes everyone as “she,” but later you might discover that some character might be he. There are other small details here and there that narrator might say, but you start to question it as the story develops. In other words, Ann Leckie has done a stylish job of show-and-not-tell about the world that she’s built.

Read More…

Pamflet 0.8.0

Over the holiday break I’ve implemented left TOC for Pamflet, and released it as Pamflet 0.8.0.

Pamflet is a publishing application for short texts, particularly user documentation of open-source software.

Read More…

masking scala.Seq

As of Scala 2.13.0-M5, it’s planned that scala.Seq will change from scala.collection.Seq to scala.collection.immutable.Seq. Scala 2.13 collections rework explains a bit about why it’s been non-immutable historically. Between the lines, I think it’s saying that we should celebrate that scala.Seq will now be immutable out of the box.

Defaulting to immutable sequence would be good for apps and fresh code. The situation is a bit more complicated for library authors.

  • If you have a cross-built library, and
  • if your users are using your library from multiple Scala versions
  • and your users are using Array(...)

this change to immutable Seq could be a breaking change to your API.

Read More…

making conference a safer space for women

We need to change the culture around tech conferences to improve the inclusion of women (and people from other backgrounds too!). For that, there needs to be clear signaling and communication about two basic issues

  1. No, it’s not ok to hit on women at a conference.
  2. Assume technical competence, and treat women as professional peers. These points should be communicated over and over at each conference before the keynote takes place, and before socializing hours.

'Red Mars' by Kim Stanley Robinson

Red Mars (Mars Trilogy, #1)Red Mars by Kim Stanley Robinson
My rating: 4 of 5 stars

Red Mars is a science fiction classic written in 1992. This book is everything I wanted and more, starting with first hundred astronauts and cosmonauts migrating to Mars to build the infrastructure such that more people can migrate. By the middle of the book, thousands of people migrate to Mars. The book explores various dimensions the epic project, not just technological challenges, but psychological effects, personality differences, and political interests by superpowers. The geography of Mars is written in vivid, majestic details.

Read More…

super shell for sbt

I’ve implemented “super shell” feature for sbt over the weekend. The idea is to take over the bottom n lines of the termnial, and display the current tasks in progress.

the limitation of using log as status report

Logs are useful in many situations, and sometimes it’s the only viable tool to find out what’s going on. But on a console app like sbt, using logs to tell the build user what’s going on doesn’t always work.

Read More…

stricter Scala with -Xlint, -Xfatal-warnings, and Scalafix

Compile, or compile not. There’s no warning. Two of my favorite Scala compiler flags lately are "-Xlint" and "-Xfatal-warnings". Here is an example setting that can be used with subprojects:

ThisBuild / organization := "com.example"
ThisBuild / version      := "0.1.0-SNAPSHOT"
ThisBuild / scalaVersion := "2.12.6"

lazy val commonSettings = List(
  scalacOptions ++= Seq(
    "-encoding", "utf8",
    "-deprecation",
    "-unchecked",
    "-Xlint",
    "-feature",
    "-language:existentials",
    "-language:experimental.macros",
    "-language:higherKinds",
    "-language:implicitConversions",
    "-Ypartial-unification",
    "-Yrangepos",
  ),
  scalacOptions ++= (scalaVersion.value match {
    case VersionNumber(Seq(2, 12, _*), _, _) =>
      List("-Xfatal-warnings")
    case _ => Nil
  }),
  Compile / console / scalacOptions --= Seq("-deprecation", "-Xfatal-warnings", "-Xlint")
)

lazy val foo = (project in file("foo"))
  .settings(
    commonSettings,
    name := "foo",  
  )

what’s -Xlint?

-Xlint enables a bunch of compiler warnings. @smogami contributed a page called Scala Compiler Options so we can now read what’s in -Xlint.

Read More…

git gone: cleaning stale local branches

Working with GitHub and pull requests a lot, I end up accumulating stale branches that are no longer needed. In this post, we will look at how to clean the stale local branches.

There are mainly two strategies:

  • Pick a “master” branch, and delete what’s merged to it
  • Assuming branches are deleted first on GitHub, delete local branches that no longer exists on remote “origin”

Erik Aybar’s Git Tip: Deleting Old Local Branches takes the second approach.

Read More…

console games in Scala

I’ve been thinking about rich console applications, the kind of apps that can display things graphically, not just appending lines at the end. Here are some info, enough parts to be able to write Tetris.

ANSI X3.64 control sequences

To display some text at an arbitrary location on a termial screen, we first need to understand what a terminal actually is. In the middle of 1960s, companies started selling minicomputers such as PDP-8, and later PDP-11 and VAX-11. These were of a size of a refrigerator, purchased by “computer labs”, and ran operating systems like RT-11 and the original UNIX system that supported up many simultaneous users (12 ~ hundreds?). The users connected to a minicomputer using a physical terminal that looks like a monochrome screen and a keyboard. The classic terminal is VT100 that was introduced in 1978 by DEC.

Read More…

all your JDKs on Travis CI using jabba

Whether you want to try using OpenJDK 11-ea, GraalVM, Eclipse OpenJ9, or you are stuck needing to build using OpenJDK 6, jabba has got it all. jabba is a cross-platform Java version manager written by Stanley Shyiko (@shyiko).

AdoptOpenJDK 8 and 11

Here’s how we can use jabba on Travis CI to cross build using AdoptOpenJDK 8 and 11:

sudo: false
dist: trusty
group: stable

language: scala

scala:
  - 2.12.7

env:
  global:
    - JABBA_HOME=/home/travis/.jabba

matrix:
  include:
  - env:
      - TRAVIS_JDK=adopt@1.8.192-12
  - env:
      - TRAVIS_JDK=adopt@1.11.0-1

before_install:
  - curl -sL https://raw.githubusercontent.com/shyiko/jabba/0.11.0/install.sh | bash && . ~/.jabba/jabba.sh

install:
  - $JABBA_HOME/bin/jabba install $TRAVIS_JDK
  - unset _JAVA_OPTIONS
  - export JAVA_HOME="$JABBA_HOME/jdk/$TRAVIS_JDK" && export PATH="$JAVA_HOME/bin:$PATH" && java -Xmx32m -version

script: sbt -Dfile.encoding=UTF8 -J-XX:ReservedCodeCacheSize=256M ++$TRAVIS_SCALA_VERSION! test

before_cache:
  - find $HOME/.ivy2 -name "ivydata-*.properties" -delete
  - find $HOME/.sbt  -name "*.lock"               -delete

cache:
  directories:
    - $HOME/.ivy2/cache
    - $HOME/.sbt/boot
    - $HOME/.jabba/jdk

When the job runs you should see something like:

Read More…

bringing back power assert with Expecty

Last week I wrote about using source dependencies with sbt-sriracha for testing purpose. This week we’ll look into using Expecty to do power assert.

Power assert (or power assertion) is a variant of assert(...) function that that prints out detailed error message automatically. It was originally implemented by Peter Niederwieser (@pniederw) for Spock, and in 2009 it was merged into Groovy 1.7. Power assert has spread to Ruby, JavaScript, Rust, etc.

traditional assert statements

Let’s say you have something like a * b. Using a traditional assert, we would write:

Read More…

hot source dependencies using sbt-sriracha

Source dependencies is one of features that existed in sbt since ever, but hasn’t been documented well.

immutable source dependency

Here’s how to declare source dependency to the latest commit for scopt commandline option parsing library.

lazy val scoptJVMRef = ProjectRef(uri("git://github.com/scopt/scopt.git#c744bc48393e21092795059aa925fe50729fe62b"), "scoptJVM")

ThisBuild / organization := "com.example"
ThisBuild / scalaVersion := "2.12.2"

lazy val root = (project in file("."))
  .dependsOn(scoptJVMRef)
  .settings(
    name := "Hello world"
  )

When you start sbt and run compile, sbt will automatically clone scopt/scopt under the staging directory, and link the builds together.

Read More…

detecting Java version from Bash

Yesterday I wrote about cross JVM testing using Travis CI.

testing Scala apps on macOS using Travis CI

Here’s how we can test Scala apps on macOS using Travis CI. This is adapted from Lars and Muuki’s method: Testing Scala programs with Travis CI on OS X

dist: trusty

language: scala

matrix:
  include:
    ## build using JDK 8, test using JDK 8
    - script:
        - sbt universal:packageBin
        - cd citest && ./test.sh
      jdk: oraclejdk8

    ## build using JDK 8, test using JDK 8, on macOS
    - script:
        - sbt universal:packageBin
        - cd citest && ./test.sh
      ## https://github.com/travis-ci/travis-ci/issues/2316
      language: java
      os: osx
      osx_image: xcode9.2

    ## build using JDK 8, test using JDK 9
    - script:
        - sbt universal:packageBin
        - jdk_switcher use oraclejdk9
        - cd citest && ./test.sh
      jdk: oraclejdk8

    ## build using JDK 8, test using JDK 10
    - script:
        - sbt universal:packageBin
        - citest/install-jdk10.sh
        - cd citest && ./test.sh
      jdk: oraclejdk8

scala:
  - 2.10.7

before_install:
  # https://github.com/travis-ci/travis-ci/issues/8408
  - unset _JAVA_OPTIONS
  - if [[ "$TRAVIS_OS_NAME" = "osx" ]]; then
      brew update;
      brew install sbt;
    fi

cache:
  directories:
    - $HOME/.ivy2/cache
    - $HOME/.sbt/boot

before_cache:
  - find $HOME/.ivy2 -name "ivydata-*.properties" -delete
  - find $HOME/.sbt  -name "*.lock"               -delete

Normally you’d write jdk: oraclejdk8 at the top level, but since the macOS image does not have the jdk_switcher script travis/travis#2317, we need to add to all entries in the matrix except for the osx one.

Read More…

cross JVM testing using Travis CI

Oracle is moving to ship non-LTS JDK every 6 months, and LTS JDK every 3 years. Also it’s converging to OpenJDK. In this scheme, JDK 9 will be EOL in March 2018; JDK 10 will come out in March 2018, and EOL in September 2018; and LTS JDK 11 that replaces JDK 8 in September 2018 will stay with us until 2021.

As we will see quick succession of JDKs in the upcoming months, here’s a how-to on testing your app on JDK 8, JDK 9, and JDK 10 Early Access using Travis CI.

Read More…

Coursera machine learning memo

This holiday break, I somehow got into binge watching Coursera’s Stanford Machine Learning course taught by Andrew Ng. I remember machine learning to be really math heavy, but I found this one more accessible.

Here are some notes for my own use. (I am removing all the fun examples, and making it dry, so if you’re interested in machine learning, you should check out the course or its official notes.)

Intro

Machine learning splits into supervised learning and unsupervised learning.

Read More…

encoding file path as URI reference

In this post I am going to discuss an old new problem of encoding file path as Uniform Resource Identifier (URI) reference.

As of 2017, the authoritative source of information is RFC 8089 - The “file” URI Scheme written by Matthew Kerwin.

Future readers might also want to search for “file URI scheme RFC”, and find the latest version. If you’re a programmer, read the RFC. This post is to raise the awareness of the some of the issues around file to URI encoding, but it’s not a substitution.

Read More…

sbt server with Sublime Text 3

On Tech Hub blog I demonstrated how to use sbt server from VS Code to display compiler errors from a running sbt session. In this post, I’ll show how to do that for Sublime Text 3 in this post.

setting up Sublime Text 3 with sbt server

First, add tomv564/LSP plugin to Sublime Text 3.

  1. cd ~/Library/Application\ Support/Sublime\ Text\ 3/Packages
  2. git clone https://github.com/tomv564/LSP.git
  3. Run ‘Preferences > Package Control > Satisfy Dependencies’

Next, download sbt-server-stdio.js and save it to ~/bin/ or somewhere you keep scripts. sbt server by default uses Unix domain sockets on POSIX systems and named pipe on Windows, but editors seem to expect stdio. The script is a Node script that’s included as our VS Code extension that discovers the socket, and fronts it with stdio.

Read More…

Ergodox

Over the weekend I assembled an Ergodox.

  • Infinity ErgoDox Ergonomic Keyboard Kit via massdrop
  • Cherry MX Brown switches
  • Datamancer Infinity Ergodox Hardwood Case (Black Walnut / Original) via massdrop
  • Plum Blossom PBT Dye-Subbed Keycap Set (OEM, Blank) via massdrop

Scala language server using sbt

It’s been a month since sbt 1.0 shipped, and I can finally sit back and think about sbt server again. Using my weekends time, I started hacking on an implementation of Scala language server on top of sbt server.

what is a language server?

A language server is a program that can provide language service to editors like Visual Studio Code, Eclipse Che, and Sublime Text 3 via Language Server Protocol. A typical operation might be textDocument/didOpen, which tells the server that a source file was opened in the editor.

Read More…

Persistent Versioning

In this post, I’d like to introduce a version scheme that I call Persistent Versioning. Most of the ideas presented in this post are not new or my own. Let me know if there’s already a name for it.

In 2015, Jake Wharton (@JakeWharton) wrote a blog post titled Java Interoperability Policy for Major Version Updates:

auto publish (a website) from Travis-CI

GitHub Pages is a convenient place to host OSS project docs. This post explains how to use Travis CI to deploy your docs automatically on a pull request merge.

1. Generate a fresh RSA key in some directory

Make a directory outside of your project first. Pick a key name deploy_yourproject_rsa, so you can distinguish it from other keys.

$ mkdir keys
$ cd keys
$ ssh-keygen -t rsa -b 4096 -C "yours@example.com"
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/xxx/.ssh/id_rsa): deploy_website_rsa
Enter passphrase (empty for no passphrase):

Keep the passphrase empty.

Read More…

tray for Atreus

In the last post that I wrote about Atreus build, I noted that there’s an issue of keyboard positioning:

Even if I can overcome the layout and memorize the various symbol locations, there’s the issue of the placement. If I place the keyboard in between me and the laptop the screen becomes too far.

I solved this issue by making a tray for Atreus that I can position it on top of the MacBook Pro keyboard.

Read More…

Atreus

Last night I finished making my Atreus keyboard from a DYI kit that I got a while back. Here are some of the details:

  • I chose Matias Quiet Click switch option (gray slider). There’s no clicking.
  • The modifiers use Matias Quiet Linear switches (red slider).
  • There are 42 keys in split ortholinear layout.
  • Mahogany ply case.

The materials

The kit comes with almost everything you need to assemble the Arteus keyboard. You need lacquer, a soldering iron, solder, and wire cutters.

Read More…

Gigahorse 0.3.0

Gigahorse 0.3.0 is now released. See documentation on what it is.

OkHttp support

0.3.0 adds Square OkHttp support. Gigahorse-OkHttp is availble for Scala 2.10, 2.11, and 2.12.

According to the JavaDoc you actually don’t have to close the OkHttpClient instance.

scala> import gigahorse._, support.okhttp.Gigahorse
import gigahorse._
import support.okhttp.Gigahorse

scala> import scala.concurrent._, duration._
import scala.concurrent._
import duration._

scala> val http = Gigahorse.http(Gigahorse.config) // don't have to close
http: gigahorse.HttpClient = gigahorse.support.okhttp.OkhClient@23b48158
Read More…

downloading and running app on the side with sbt-sidedish

I’ve been asked by a few people on downloading JARs, and then running them from an sbt plugin. Most recently, Shane Delmore (@shanedelmore) asked me about this at nescala in Brooklyn.

During an unconference session I hacked together a demo, and I continued some more after I came home.

sbt-sidedish

sbt-sidedish is a toolkit for plugin authors to download and run an app on the side from a plugin. It on its own does not define any plugins.

Read More…

Contraband, an alternative to case class

Here are a few questions I’ve been thinking about:

  • How should I express data or API?
  • How should the data be represented in Java or Scala?
  • How do I convert the data into wire formats such as JSON?
  • How do I evolve the data without breaking binary compatibility?

limitation of case class

The sealed trait and case class is the idiomatic way to represent datatypes in Scala, but it’s impossible to add fields in binary compatible way. Take for example a simple case class Greeting, and see how it would expand into a class and a companion object:

Read More…

Gigahorse 0.2.0

Gigahorse 0.2.0 is now released. The new change is that it abstracts over two backends. @alexdupre contributed migration from AHC 1.9 to AHC 2.0, which is based on Netty 4 in #12.

In addition, there’s now an experimental Akka HTTP support that I added. #15

Please see Gigahorse docs for the details.

Read More…

Gigahorse 0.1.0

Update: please use Gigahorse 0.1.1

Gigahorse 0.1.0 is now released. It is an HTTP client for Scala with Async Http Client underneath. Please see Gigahorse docs for the details. Here’s an example snippet to get the feel of the library.

scala> import gigahorse._
scala> import scala.concurrent._, duration._
scala> Gigahorse.withHttp(Gigahorse.config) { http =>
         val r = Gigahorse.url("http://api.duckduckgo.com").get.
           addQueryString(
             "q" -> "1 + 1",
             "format" -> "json"
           )
         val f = http.run(r, Gigahorse.asString andThen {_.take(60)})
         Await.result(f, 120.seconds)
       }
Read More…

registry and reference pattern

There’s a “pattern” that I’ve been thinking about, which arises in some situation while persisting/serializing objects.

To motivate this, consider the following case class:

scala> case class User(name: String, parents: List[User])
defined class User

scala> val alice = User("Alice", Nil)
alice: User = User(Alice,List())

scala> val bob = User("Bob", alice :: Nil)
bob: User = User(Bob,List(User(Alice,List())))

scala> val charles = User("Charles", bob :: Nil)
charles: User = User(Charles,List(User(Bob,List(User(Alice,List())))))

scala> val users = List(alice, bob, charles)
users: List[User] = List(User(Alice,List()), User(Bob,List(User(Alice,List()))),
  User(Charles,List(User(Bob,List(User(Alice,List()))))))

The important part is that it contains parents field, which contains a list of other users. Now let’s say you want to turn users list of users into JSON.

Read More…

sjson-new and the prisoner of Azkaban

This is part 3 on the topic of sjson-new. See also part 1 and part 2.

Within the sbt code base there are a few places where the persisted data is in the order of hundreds of megabytes that I suspect it becomes a performance bottleneck, especially on machines without an SSD drive. Naturally, my first instinct was to start reading up on the encoding of Google Protocol Buffers to implement my own custom binary format.

Read More…