We can use head with .SD
library(data.table)
dt <- data.table(mtcars)
> dt[, head(.SD, 3), by = "cyl"]
cyl mpg disp hp drat wt qsec vs am gear carb
1: 6 21.0 160.0 110 3.90 2.620 16.46 0 1 4 4
2: 6 21.0 160.0 110 3.90 2.875 17.02 0 1 4 4
3: 6 21.4 258.0 110 3.08 3.215 19.44 1 0 3 1
4: 4 22.8 108.0 93 3.85 2.320 18.61 1 1 4 1
5: 4 24.4 146.7 62 3.69 3.190 20.00 1 0 4 2
6: 4 22.8 140.8 95 3.92 3.150 22.90 1 0 4 2
7: 8 18.7 360.0 175 3.15 3.440 17.02 0 0 3 2
8: 8 14.3 360.0 245 3.21 3.570 15.84 0 0 3 4
9: 8 16.4 275.8 180 3.07 4.070 17.40 0 0 3 3
