Testing an assertion that something must not compile

Question

The problem

When I'm working with libraries that support type-level programming, I often find myself writing comments like the following (from an example presented by Paul Snively at Strange Loop 2012):

// But these invalid sequences don't compile:
// isValid(_3 :: _1 :: _5 :: _8 :: _8 :: _2 :: _8 :: _6 :: _5 :: HNil)
// isValid(_3 :: _4 :: _5 :: _8 :: _8 :: _2 :: _8 :: _6 :: HNil)

Or this, from an example in the Shapeless repository:

/**
 * If we wanted to confirm that the list uniquely contains `Foo` or any
 * subtype of `Foo`, we could first use `unifySubtypes` to upcast any
 * subtypes of `Foo` in the list to `Foo`.
 *
 * The following would not compile, for example:
 */
 //stuff.unifySubtypes[Foo].unique[Foo]

This is a very rough way of indicating some fact about the behavior of these methods, and we could imagine wanting to make these assertions more formal—for unit or regression testing, etc.

To give a concrete example of why this might be useful in the context of a library like Shapeless, a few days ago I wrote the following as a quick first attempt at an answer to this question:

import shapeless._

implicit class Uniqueable[L <: HList](l: L) {
  def unique[A](implicit ev: FilterAux[L, A, A :: HNil]) = ev(l).head
}

Where the intention is that this will compile:

('a' :: 'b :: HNil).unique[Char]

While this will not:

('a' :: 'b' :: HNil).unique[Char]

I was surprised to find that this implementation of a type-level unique for HList didn't work, because Shapeless would happily find a FilterAux instance in the latter case. In other words, the following would compile, even though you'd probably expect it not to:

implicitly[FilterAux[Char :: Char :: HNil, Char, Char :: HNil]]

In this case, what I was seeing was a bug—or at least something bug-ish—and it has since been fixed.

More generally, we can imagine wanting to check the kind of invariant that was implicit in my expectations about how FilterAux should work with something like a unit test—as weird as it may sound to be talking about testing type-level code like this, with all the recent debates about the relative merit of types vs. tests.

My question

The problem is that I don't know of any kind of testing framework (for any platform) that allows the programmer to assert that something must not compile.

One approach that I can imagine for the FilterAux case would be to use the old implicit-argument-with-null-default trick:

def assertNoInstanceOf[T](implicit instance: T = null) = assert(instance == null)

Which would let you write the following in your unit test:

assertNoInstanceOf[FilterAux[Char :: Char :: HNil, Char, Char :: HNil]]

The following would be a heck of a lot more convenient and expressive, though:

assertDoesntCompile(('a' :: 'b' :: HNil).unique[Char])

I want this. My question is whether anyone knows of any testing library or framework that supports anything remotely like it—ideally for Scala, but I'll settle for anything.

Answer 1

Not a framework, but Jorge Ortiz (@JorgeO) mentioned some utilities he added to the tests for Foursquare's Rogue library at NEScala in 2012 which support tests for non-compilation: you can find examples here. I've been meaning to add something like this to shapeless for quite a while.

More recently, Roland Kuhn (@rolandkuhn) has added a similar mechanism, this time using Scala 2.10's runtime compilation, to the tests for Akka typed channels.

These are both dynamic tests of course: they fail at (test) runtime if something that shouldn't compile does. Untyped macros might provide a static option: ie. a macro could accept an untyped tree, type check it and throw a type error if it succeeds). This might be something to experiment with on the macro-paradise branch of shapeless. But not a solution for 2.10.0 or earlier, obviously.

Update

Since answering the question, another approach, due to Stefan Zeiger (@StefanZeiger), has surfaced. This one is interesting because, like the untyped macro one alluded to above, it is a compile time rather than (test) runtime check, however it is also compatible with Scala 2.10.x. As such I think it is preferable to Roland's approach.

I've now added implementations to shapeless for 2.9.x using Jorge's approach, for 2.10.x using Stefan's approach and for macro paradise using the untyped macro approach. Examples of the corresponding tests can be found here for 2.9.x, here for 2.10.x and here for macro paradise.

The untyped macro tests are the cleanest, but Stefan's 2.10.x compatible approach is a close second.

Answer 2

ScalaTest 2.1.0 has the following syntax for Assertions:

assertTypeError("val s: String = 1")

And for Matchers:

"val s: String = 1" shouldNot compile

Answer 3

Do you know about partest in the Scala project? E.g. CompilerTest has the following doc:

/** For testing compiler internals directly.
* Each source code string in "sources" will be compiled, and
* the check function will be called with the source code and the
* resulting CompilationUnit. The check implementation should
* test for what it wants to test and fail (via assert or other
* exception) if it is not happy.
*/

It is able to check for example whether this source https://github.com/scala/scala/blob/master/test/files/neg/divergent-implicit.scala will have this result https://github.com/scala/scala/blob/master/test/files/neg/divergent-implicit.check

It's not a perfect fit for your question (since you don't specify your test cases in terms of asserts), but may be an approach and/or give you a head start.

Answer 4

Based on the links provided by Miles Sabin I was able to use the akka version

import scala.tools.reflect.ToolBox

object TestUtils {

  def eval(code: String, compileOptions: String = "-cp target/classes"): Any = {
    val tb = mkToolbox(compileOptions)
    tb.eval(tb.parse(code))
  }

  def mkToolbox(compileOptions: String = ""): ToolBox[_ <: scala.reflect.api.Universe] = {
    val m = scala.reflect.runtime.currentMirror
    m.mkToolBox(options = compileOptions)
  }
}

Then in my tests I used it like this

def result = TestUtils.eval(
  """|import ee.ui.events.Event
     |import ee.ui.events.ReadOnlyEvent
     |     
     |val myObj = new {
     |  private val writableEvent = Event[Int]
     |  val event:ReadOnlyEvent[Int] = writableEvent
     |}
     |
     |// will not compile:
     |myObj.event.fire
     |""".stripMargin)

result must throwA[ToolBoxError].like {
  case e => 
    e.getMessage must contain("value fire is not a member of ee.ui.events.ReadOnlyEvent[Int]") 
}

Answer 5

The compileError macro in µTest does just that:

compileError("true * false")
// CompileError.Type("value * is not a member of Boolean")

compileError("(}")
// CompileError.Parse("')' expected but '}' found.")