How do Scala macro typechecks resolve identifiers to types?

Question

I'm trying to create an annotation macro which can only be applied to a certain type. When I run my tests I see a type not found error when the annotation is applied to top level objects only.

My macro code:

trait Labelled[T] {
  def label: T
}

@compileTimeOnly("DoSomethingToLabelled requires the macro paradise plugin")
class DoSomethingToLabelled extends StaticAnnotation {
  def macroTransform(annottees: Any*): Any = macro DoSomethingToLabelled.impl
}

object DoSomethingToLabelled {
  def impl(c: whitebox.Context)(annottees: c.Expr[Any]*): c.Expr[Any] = {
    import c.universe._

    annottees.map(_.tree).head match {
      case expr @ ModuleDef(mods: Modifiers, name: TermName, impl: Template) =>
        println(showRaw(impl.parents))
        val parentTypes = impl.parents.map(c.typecheck(_, c.TYPEmode))

        if (parentTypes.exists(_.tpe <:< typeOf[Labelled[_]])) {
          c.Expr[Any](expr)
        } else {
          c.abort(c.enclosingPosition, s"DoSomethingToLabelled can only be applied to a Labelled. Received types: $parentTypes")
        }
    }
  }
}

My test code:

class DoSomethingToLabelledSpec extends Specification {

  private def classPathUrls(cl: ClassLoader): List[String] = cl match {
    case null => Nil
    case u: java.net.URLClassLoader => u.getURLs.toList.map(systemPath) ++ classPathUrls(cl.getParent)
    case _ => classPathUrls(cl.getParent)
  }

  private def systemPath(url: URL): String = {
    Paths.get(url.toURI).toString
  }

  private def paradiseJarLocation: String = {
    val classPath = classPathUrls(getClass.getClassLoader)
    classPath.find(_.contains("paradise")).getOrElse {
      throw new RuntimeException(s"Could not find macro paradise on the classpath: ${classPath.mkString(";")}")
    }
  }

  lazy val toolbox = runtimeMirror(getClass.getClassLoader)
    .mkToolBox(options = s"-Xplugin:$paradiseJarLocation -Xplugin-require:macroparadise")

  "The DoSomethingToLabelled annotation macro" should {

    "be applicable for nested object definitions extending Labelled" in {
      toolbox.compile {
        toolbox.parse {
          """
            |import macrotests.Labelled
            |import macrotests.DoSomethingToLabelled
            |
            |object Stuff {
            |  @DoSomethingToLabelled
            |  object LabelledWithHmm extends Labelled[String] {
            |    override val label = "hmm"
            |  }
            |}
            |""".stripMargin
        }
      } should not (throwAn[Exception])
    }

    "be applicable for top level object definitions extending Labelled" in {
      toolbox.compile {
        toolbox.parse {
          """
            |import macrotests.Labelled
            |import macrotests.DoSomethingToLabelled
            |
            |@DoSomethingToLabelled
            |object LabelledWithHmm extends Labelled[String] {
            |  override val label = "hmm"
            |}
            |""".stripMargin
        }
      } should not (throwAn[Exception])
    }
  }
}

And my test log is:

sbt:macro-type-extraction> test
[info] Compiling 1 Scala source to C:\Users\WilliamCarter\workspace\macro-type-extraction\target\scala-2.11\classes ...
[info] Done compiling.
List(AppliedTypeTree(Ident(TypeName("Labelled")), List(Ident(TypeName("String")))))
List(AppliedTypeTree(Ident(TypeName("Labelled")), List(Ident(TypeName("String")))))
[info] DoSomethingToLabelledSpec
[info] The DoSomethingToLabelled annotation macro should
[info]   + be applicable for nested object definitions extending Labelled
[error] scala.tools.reflect.ToolBoxError: reflective compilation has failed:
[error]
[error] exception during macro expansion:
[error] scala.reflect.macros.TypecheckException: not found: type Labelled
[error]         at scala.reflect.macros.contexts.Typers$$anonfun$typecheck$2$$anonfun$apply$1.apply(Typers.scala:34)
[error]         at scala.reflect.macros.contexts.Typers$$anonfun$typecheck$2$$anonfun$apply$1.apply(Typers.scala:28)
[error]         at scala.reflect.macros.contexts.Typers$$anonfun$3.apply(Typers.scala:24)
[error]         at scala.reflect.macros.contexts.Typers$$anonfun$3.apply(Typers.scala:24)
[error]         at scala.reflect.macros.contexts.Typers$$anonfun$withContext$1$1.apply(Typers.scala:25)
[error]         at scala.reflect.macros.contexts.Typers$$anonfun$withContext$1$1.apply(Typers.scala:25)
[error]         at scala.reflect.macros.contexts.Typers$$anonfun$1.apply(Typers.scala:23)
[error]         at scala.reflect.macros.contexts.Typers$$anonfun$1.apply(Typers.scala:23)
[error]         at scala.reflect.macros.contexts.Typers$class.withContext$1(Typers.scala:25)
[error]         at scala.reflect.macros.contexts.Typers$$anonfun$typecheck$2.apply(Typers.scala:28)
[error]         at scala.reflect.macros.contexts.Typers$$anonfun$typecheck$2.apply(Typers.scala:28)
[error]         at scala.reflect.macros.contexts.Typers$class.withWrapping$1(Typers.scala:26)
[error]         at scala.reflect.macros.contexts.Typers$class.typecheck(Typers.scala:28)
[error]         at scala.reflect.macros.contexts.Context.typecheck(Context.scala:6)
[error]         at scala.reflect.macros.contexts.Context.typecheck(Context.scala:6)
[error]         at macrotests.DoSomethingToLabelled$$anonfun$2.apply(DoSomethingToLabelled.scala:19)
[error]         at macrotests.DoSomethingToLabelled$$anonfun$2.apply(DoSomethingToLabelled.scala:19)
[error]         at scala.collection.immutable.List.map(List.scala:284)
[error]         at macrotests.DoSomethingToLabelled$.impl(DoSomethingToLabelled.scala:19)

My debug printing tells me the extracted parent types are the same in each test but for some reason the top level object cannot resolve that the TypeName("Labelled") is actually a macrotests.Labelled. Is anyone able to help shed some light here? The macro appears to work outside of the testing context but I'd really like to understand what's going on so I can write some proper tests.

Answer 1

Try

toolbox.compile {
  toolbox.parse {
    """
      |import macrotests.DoSomethingToLabelled
      |
      |@DoSomethingToLabelled
      |object LabelledWithHmm extends macrotests.Labelled[String] {
      |  override val label = "hmm"
      |}
      |""".stripMargin
  }
}

or even

toolbox.compile {
  toolbox.parse {
    """
      |import macrotests.DoSomethingToLabelled
      |
      |@DoSomethingToLabelled
      |object LabelledWithHmm extends _root_.macrotests.Labelled[String] {
      |  override val label = "hmm"
      |}
      |""".stripMargin
  }
}

By the way, why do you need toolbox? Why not to write just

@DoSomethingToLabelled
object LabelledWithHmm extends Labelled[String] {
  override val label = "hmm"
}

in tests? Then the fact that the code compiles will be checked at compile time rather than at runtime with toolbox.

https://github.com/scala/bug/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+%28toolbox+%26%26+%28import+%7C%7C+package%29%29

https://github.com/scala/bug/issues/6393

@xeno-by said: It looks like we're doomed w.r.t this one.

The problem is that Scala reflection and reflective compiler (which is underlying toolboxes) use a different model of classfile loading than vanilla scalac does. Vanilla compiler has its classpath as a list of directories/jars on the filesystem, so it can exhaustively enumerate the packages on the classpath. Reflective compiler works with arbitrary classloaders, and classloaders don't have a concept of enumerating packages.

As a result, when a reflective compiler sees "math" having "import scala.; import java.lang." imports in the lexical context, it doesn't know whether that "math" stands for root.math, scala.math or java.lang.math. So it has to speculate and provisionally creates a package for root.math, which ends up being a wrong choice.

We could probably support a notion of "overloaded" packages, so that the compiler doesn't have to speculate and can store all the possible options, but that'd require redesign of reflection and probably of the typer as well.