Dynamic order by in scala slick with several columns

Question

I have been learning scala, playframework and slick, but i have found a problem. I am trying to make a simple CRUD, with a list controllers that receives a custom filter field, some pagination info (page size and number) and a Seq of string tuples with the field name and the order (asc or desc), and everything is working fine, except for the order by seq, i can not make the order by dynamic.

I got the basic structure from Scadiddle blog. So, the basic code is as follows:

I have my basic color model:

case class Color(
  id: Int,
  name: String)

It is a simple table definition:

    class ColorsTable(tag: Tag) extends Table[Color](tag, "color") {
        def id = column[Int]("id", O.PrimaryKey, O.AutoInc)
        def name = column[String]("name")
        def * = (id, name) <> ((Color.apply _).tupled, Color.unapply)
      }

In my repo i have the search method:

def findAll(searchTerm: Option[String], page: Int, top: Int, sortsBy: Seq[(String, SortDirection)]): Future[Seq[Color]] = {
    var query = searchTerm match {
      case Some(term) => colors.filter(_.name like s"%$term%")
      case _ => colors
    }
    val offset = (page - 1) * top
    // This is building the sort clause, matching on each column in the sort
    sortsBy.foreach {
      sortTuple =>
        val (sortColumn, sortDirection) = sortTuple
        query = query.sortBy(sortColumn match {
          case "id" => if (sortDirection == Desc) _.id.desc else _.id.asc
          case _ => if (sortDirection == Desc) _.name.desc else _.name.asc
        })
    }
    // The "list" method actually executes the query
    val colorsQuery = query.drop(offset).take(top).result

    db.run(colorsQuery)
  }

The problem is, when I call the search method with this sequence:

val sortsBy = Seq[(String, SortDirection)](("name", Desc), ("id", Asc))
    colorService.getColors(None, 1, 10, sortsBy).map(colorList => Ok(Json.toJson(colorList)))

This query is generated:

select "id", "name" from "color" order by "id", "name" desc limit 10 offset 0

As you can see, the order of the sortBy got inverted (id and then name, instead of name and id as the sequence).

If I use a tuple instead the foreach the order is respected:

query = query.sortBy(
      s => (s.name.desc, s.id.asc)
    )

But there is no way to generate a dynamic-size tuple. To add some more confusionm the other thing that causes me trouble is this part in the slick documentation:

Be aware that a single ORDER BY with multiple columns is not equivalent to multiple .sortBy calls but to a single .sortBy call passing a tuple

So, in reality can i use the foreach and concatenate the orders? Or is because this limitation that the order gets reversed?

In case that only can tuples can be used for the sortBy, how can i achieve a dynamic sized order by?

PD: Thanks for the attention and sorry for the bad english

EDIT:

Thanks for the quick response, I tried your code and looks fine, sadly i have almost no idea how it works :( (scala is a very nice, but hard to learn language :S).

When i saw Higher Kinded Type Should be Enabled i just freaked out, searching for answers this did not gave me much hope to get an easy understanding, hopefully when I finish the Programming in Scala, 3rd ed book i will have more understanding and knowledge of what the hell is going on.

Just one more question, is this equivalent to do several sortBy calls? how this compare to use a tuple? I am still confused by this part of the slick docs:

a single ORDER BY with multiple columns is not equivalent to multiple .sortBy calls but to a single .sortBy call passing a tuple

I checked my method and by adding a reverse to the seq make it worked fine, of course not as functional and nice like your code, so i will be using your suggestion and working a way to make the rest of the filters with helpers and avoiding vars. (yes, in the other part I am still using var, but i will make it nicer when i get to understand more of Scala).

Confession: After more than 8 years programming in several languages (From JavaScript to Java, C#, Python and others) I have to repeat, Scala looks like a beautiful but very complex language, but I will not give up learning it

Answer 1

Let's define DynamicSortBySupport helper

object DynamicSortBySupport {
  import slick.ast.Ordering.Direction
  import slick.ast.Ordering
  import slick.lifted.Query
  import slick.lifted.ColumnOrdered
  import slick.lifted.Ordered
  type ColumnOrdering = (String, Direction) //Just a type alias
  trait ColumnSelector {
    val select: Map[String, Rep[_]] //The runtime map between string names and table columns 
  }
  implicit class MultiSortableQuery[A <: ColumnSelector, B, C[_]](query: Query[A, B, C]) {
    def dynamicSortBy(sortBy: Seq[ColumnOrdering]): Query[A, B, C]  =
      sortBy.foldRight(query){ //Fold right is reversing order
        case ((sortColumn, sortOrder), queryToSort) =>
          val sortOrderRep: Rep[_] => Ordered = ColumnOrdered(_, Ordering(sortOrder))
          val sortColumnRep: A => Rep[_] = _.select(sortColumn)
          queryToSort.sortBy(sortColumnRep)(sortOrderRep)
      }
  }
}

and redefine your Table adding a "sorting map"

class ColorsTable(tag: Tag) extends Table[Color](tag, "color") with DynamicSortBySupport.ColumnSelector {
  def id = column[Int]("id", O.PrimaryKey, O.AutoInc)
  def name = column[String]("name")
  def * = (id, name) <> ((Color.apply _).tupled, Color.unapply)
  val select = Map(
    "id" -> (this.id),
    "name" -> (this.name)
  )
}

and finally use the whole thing in your code:

object FindAll extends App {
  import DynamicSortBySupport._
  import slick.ast.Ordering.Direction
  import slick.ast.Ordering
  object colors extends TableQuery(new ColorsTable(_))
  val sortsBy = Seq[(String, Direction)](("name", Ordering.Desc), ("id", Ordering.Asc)) //Replaced 
  val db = Database.forURL("jdbc:h2:mem:test;DB_CLOSE_DELAY=-1", driver="org.h2.Driver") //Just for testing
  findAll(sortsBy)
  def findAll(sortsBy: Seq[(String, Direction)]): Future[Seq[Color]] = {
    val query = colors.dynamicSortBy(sortsBy).result
    db.run(query)
  }
}

Notes and comments:

• The runtime map Map[String, Rep[_]] between string names and table columns could be improved with error handling (now it just throws a runtime exception that must be managed properly) or automatic derivation from table definition itself;
• I have replaced SortDirection with proper slick.ast.Ordering.Direction, feel free to write a converter;
• You could also write an helper for optional filters, something like filterOption;
• Note the use of foldRight to reverse sorting order;
• If possible avoid vars and be functional :-)
• If you want to have ColumnSelector outside the data layer (it is a good practice actually) you could rewrite it requiring implicitly something like ColumnSelector[T];