Bartosz Bąbol

Software engineering

Scala Macros: Part I - Def Macros

I. Introduction. What are scala macros?

Macros in scala allow you modify existing, or even generate new code before it gets compiled. They are used under the hood of many known libraries in scala ecosystem like shapeless, slick or play framework. They come in different flavors, one of them are: def macro and macro annotations.

In this blog post I will give you example of how you could easily start writing your own def macro. In Part 2 I will give you more practical usage of macro annotations and in Part 3 we will build intellij idea plugin to support our macros.

You will find code on my github, branch part-1

Let’s get started!

II. How macros work?

You might think about scala macros like about framework/api for generating/modifying Abstract Syntax Trees(AST).

A) What is AST?

Abstract syntax tree is core data structure used in compilers. Early stage of compiling code is lexical analysis. What it does is checking the sequence of code elements and it assigns them to tokens. Next stage is parsing code. What parsing does is checking sequence of tokens in terms of grammar and builds some data structure base on parsed code. This data structure is called parse tree and it includes each token found in parsed string. AST is similar but without redundant information. Let’s look at this simple string:

1
1 + (2+3)

after lexical analysis this string could look like this:

1
int(1) '+' '(' int(2) '+' int(3) ')'

Now compiler need to have some structure which will give him an information about grammar relationships between those tokens. Without redundant information like parenthesis etc. AST will provide it, and it could look like this:

images

Or maybe this ‘tree like’ structure, would be more intuitional:

images

After creating AST, it is further passed to compiler which makes use of it. What macro does is it “replaces” AST written by developer with AST produced by macro itself, and then compilation goes further. So we are not resigning from any advantages of static typed, compiled language assets.

III. Setup

Look at build.sbt:

build.sbt
1
2
3
4
5
6
7
8
9
10
11
12
13
name := "macros_examples_part_1"

version := "1.0"

scalaVersion := "2.11.7"

lazy val macros_implementations = project

lazy val root = (project in file(".")).aggregate(macros_implementations).dependsOn(macros_implementations)

val paradiseVersion = "2.1.0-M5"

addCompilerPlugin("org.scalamacros" % "paradise" % paradiseVersion cross CrossVersion.full)

Macros must be defined in other compilation phase(in other sbt project in our case). I’ve created new project called macros_implementations, and make our main project dependent on it. By the way this line:

build.sbt
1
lazy val macros_implementations = project

is also scala macro. According to sbt docs: “The name of the val is used as the project ID and the name of the base directory of the project” So we have 2 projects. Main one and dependent one called macros_implementations. To use macros we will also need compiler plugin. called macro paradise.

IV. Example no.1: def macros

Look at Main.scala. I’ve placed there example of macros usage.

Main.scala
1
2
3
object Main extends App{
  Welcome.isEvenLog(12)
}

From “outside” developer cannot specify if method is implemented with scala macros or not. Invoking of def macro is the same as any other method. What this macro does, it println some message depending on argument is even or odd. In main project directory run:

1
sbt run

to see result of this method. So let’s look closer at the implementation.

Welcome.scala
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import scala.reflect.macros.blackbox.Context
import scala.language.experimental.macros

object Welcome {
  def isEvenLog(number: Int): Unit = macro isEvenLogImplementation

  def isEvenLogImplementation(c: Context)(number: c.Tree): c.Tree = {
    import c.universe._

    q"""
      if ($number%2==0){
        println($number.toString + " is even")
      }else {
        println($number.toString + " is odd")
      }
    """
  }
}

There is few interesting lines, first we need to import macros:

1
import scala.language.experimental.macros

Next, we need some api to modify AST, we would like to have some typechecker, some error logger etc:

1
import scala.reflect.macros.blackbox.Context

Macros in scala comes in 2 flavors: blackbox and whitebox. What is important: use blackbox ;). Talking seriously, if macro faithfully follows its type signatures, and its implementation is not necessary to understand its behaviour then we call that macro blackbox. In other case, in whitebox macro type signature is only an approximation.

Next, implementation of macro:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
object Welcome {
  def isEvenLog(number: Int): Unit = macro isEvenLogImplementation

  def isEvenLogImplementation(c: Context)(number: c.Tree): c.Tree = {
    import c.universe._

    q"""
      if ($number%2==0){
        println($number.toString + " is even")
      }else {
        println($number.toString + " is odd")
      }
    """
  }
}
  1. To specify macro implementation we use keyword macro followed by implementation method name.

  2. This implementation method takes 2 parameters: first one is mentioned earlier Context, and second is parameter as element of Tree. Tree is node type of AST. Returning type is also Tree.

  3. Next thing is importing context universe which provide API for modifying AST.

  4. And the main part of out implementation: building a Tree. This q”“ interpolator is called quasiquotes. What it does provides developer an easier way to create/modify AST. I’ve said ‘easier’, so there should be also harder way. Let’s modify a little bit body of our macro to see what q”“ produces in our case.

Welcome.scala
1
2
3
4
5
6
7
8
9
10
11
     ...
     val result = q"""
       if ($number%2==0){
         println($number.toString + " is even")
       }else {
         println($number.toString + " is odd")
       }
     """
     println(showRaw(result))
     result
...

Now, run the project. Printed result looks like the following:

1
If(Apply(Select(Apply(Select(Literal(Constant(12)), TermName("$percent")), List(Literal(Constant(2)))), TermName("$eq$eq")), List(Literal(Constant(0)))), Apply(Ident(TermName("println")), List(Apply(Select(Apply(Select(Apply(Select(Literal(Constant("12")), TermName("$plus")), List(Literal(Constant("= ")))), TermName("$plus")), List(Select(Literal(Constant(12)), TermName("toString")))), TermName("$plus")), List(Literal(Constant(" and it is even")))))), Apply(Ident(TermName("println")), List(Apply(Select(Apply(Select(Apply(Select(Literal(Constant("12")), TermName("$plus")), List(Literal(Constant("= ")))), TermName("$plus")), List(Select(Literal(Constant(12)), TermName("toString")))), TermName("$plus")), List(Literal(Constant(" and it is odd")))))))

I hope that you see now, what quasiquotes give you ;) Each of Those If, Apply, Select objects are Tree subclasses, and you can build your AST using them instead of quasiquotes, but it would be frustrating to do this, when you can use q”“ and write human readable code.

To understand more what is going on in this example let’s modify our code a little bit. In Main.scala write this code:

Main.scala
1
2
3
4
5
object Main extends App{
  val x = 2
  val y = 3
  Welcome.isEvenLog(x + y)
...

And to see generated code change showRaw to showCode in Welcome.scala:

Welcome.scala
1
2
3
4
5
6
7
8
9
10
11
   ...
   val result = q"""
     if ($number%2==0){
       println($number.toString + " is even")
     }else {
       println($number.toString + " is odd")
   }
   """
   println(showCode(result))
   result
...

And now run this project again.

1
sbt run

You should see generated code in console. Without redundant syntax, generated code looks like the following:

1
2
3
4
5
     if((x+y)%2==0){
        println((x+y).toString + " is even")
     }else{
        println((x+y).toString + " is odd")
     }

So you maybe see now better that this (number: c.Tree) parameter of #isEvenLogImplementation is exactly only tree node. Not Int, it is just some token, you might say, which is placed in places where you specify it. Of course this is some kind of hidden bug(multiple evaluation), because you can imagine what will happen if you provide non deterministic argument, like random integer:

Main.scala
1
2
3
4
object Main extends App{
  val x = 2
  Welcome.isEvenLog(x + Random.nextInt())
...

Result will look like the following:

1
2
3
4
5
     if((x+Random.nextInt())%2==0){
        println((x+Random.nextInt()).toString + " is even")
     }else{
        println((x+Random.nextInt()).toString + " is odd")
     }

And resulted print statement often will be as smart as following:

1
17 is even

The simplest fix of this issue is to explicitly evaluate input arguments first, so assign them to val, and use those vals further in generating AST:

Welcome.scala
1
2
3
4
5
6
7
8
   val result = q"""
      val evaluatedNumber = $number
      if (evaluatedNumber%2==0){
        println(evaluatedNumber.toString + " is even")
      }else {
        println(evaluatedNumber.toString + " is odd")
      }
   """

For closer look at quasiquotes look at their docs.

V. Summary of part 1

I hope that this simple example will encourage you to experiment with def macros. If you have been using play framework, then probably you have been parsing json too, so look at source of Play json api and check how macros are used to reduce boilerplate in play framework. There was a post about this json api, available here. Few useful links, where you can find more extensive/formal answers to the topic:

Def macro in docs

Macro annotations in docs

Context in docs

In part 2 of this blog series I will describe macro annotations and give few more examples of scala macros usage.

Comments