Composing Database.Esqueleto queries, conditional joins and counting

Question

How can I compose Database.Esqueleto queries in a modular way such that after defining a "base" query and the corresponding result set, I can restrict the result set by adding additional inner joins and where expressions.

Also, how can I convert the base query that returns a list of entities (or field tuples) into a query that counts the result set since the base query is not executed as such, but a modified version of it with LIMIT and OFFSET.

The following incorrect Haskell code snippet adopted from the Yesod Book hopefully clarifies what I'm aiming at.

{-# LANGUAGE QuasiQuotes, TemplateHaskell, TypeFamilies, OverloadedStrings #-}
{-# LANGUAGE GADTs, FlexibleContexts #-}
import qualified Database.Persist as P
import qualified Database.Persist.Sqlite as PS
import Database.Persist.TH
import Control.Monad.IO.Class (liftIO)
import Data.Conduit
import Control.Monad.Logger
import Database.Esqueleto
import Control.Applicative

share [mkPersist sqlSettings, mkMigrate "migrateAll"] [persistLowerCase|
Person
    name String
    age Int Maybe
    deriving Show
BlogPost
    title String
    authorId PersonId
    deriving Show
Comment
    comment String
    blogPostId BlogPostId
|]

main :: IO ()
main = runStdoutLoggingT $ runResourceT $ PS.withSqliteConn ":memory:" $ PS.runSqlConn $ do
    runMigration migrateAll

    johnId <- P.insert $ Person "John Doe" $ Just 35
    janeId <- P.insert $ Person "Jane Doe" Nothing

    jackId <- P.insert $ Person "Jack Black" $ Just 45
    jillId <- P.insert $ Person "Jill Black" Nothing

    blogPostId <- P.insert $ BlogPost "My fr1st p0st" johnId
    P.insert $ BlogPost "One more for good measure" johnId
    P.insert $ BlogPost "Jane's" janeId

    P.insert $ Comment "great!" blogPostId

    let baseQuery = select $ from $ \(p `InnerJoin` b) -> do 
        on (p ^. PersonId ==. b ^. BlogPostAuthorId)
        where_ (p ^. PersonName `like` (val "J%"))
        return (p,b)

    -- Does not compile
    let baseQueryLimited = (,) <$> baseQuery <*> (limit 2)

    -- Does not compile
    let countingQuery = (,) <$> baseQuery <*> (return countRows)

    -- Results in invalid SQL 
    let commentsQuery = (,) <$> baseQuery
                <*> (select $ from $ \(b `InnerJoin` c) -> do
                        on (b ^. BlogPostId ==. c ^. CommentBlogPostId)
                        return ())

    somePosts <- baseQueryLimited
    count <- countingQuery
    withComments <- commentsQuery
    liftIO $ print somePosts
    liftIO $ print ((head count) :: Value Int)
    liftIO $ print withComments
    return ()

Answer 1

For LIMIT and COUNT, hammar's answer is entirely correct so I'll not delve into them. I'll just reiterate that once you use select you'll not be able to change the query in any way again.

For JOINs, currently you are not able to do a INNER JOIN with a query that was defined in a different from (nor (FULL|LEFT|RIGHT) OUTER JOINs). However, you can do implicit joins. For example, if you have defined:

baseQuery = 
  from $ \(p `InnerJoin` b) -> do 
  on (p ^. PersonId ==. b ^. BlogPostAuthorId)
  where_ (p ^. PersonName `like` val "J%")
  return (p, b)

Then you may just say:

commentsQuery = 
  from $ \c -> do
  (p, b) <- baseQuery
  where_ (b ^. BlogPostId ==. c ^. CommentBlogPostId)
  return (p, b, c)

Esqueleto then will generate something along the lines of:

SELECT ...
FROM Comment, Person INNER JOIN BlogPost
ON    Person.id = BlogPost.authorId
WHERE Person.name LIKE "J%"
AND   BlogPost.id = Comment.blogPostId

Not pretty but gets the job done for INNER JOINs. If you need to do a OUTER JOIN then you'll have to refactor your code so that all the OUTER JOINs are in the same from (note that you can do an implicit join between OUTER JOINs just fine).

Answer 2

Looking at the documentation and the type of select:

select :: (...) => SqlQuery a -> SqlPersistT m [r]

It's clear that upon calling select, we leave the world of pure composable queries (SqlQuery a) and enter the world of side effects (SqlPersistT m [r]). So we simply need to compose before we select.

let baseQuery = from $ \(p `InnerJoin` b) -> do 
      on (p ^. PersonId ==. b ^. BlogPostAuthorId)
      where_ (p ^. PersonName `like` (val "J%"))
      return (p,b)

let baseQueryLimited = do r <- baseQuery; limit 2; return r
let countingQuery    = do baseQuery; return countRows

somePosts <- select baseQueryLimited
count     <- select countingQuery

This works for limiting and counting. I haven't figured out how to do it for joins yet, but it looks like it should be possible.