This doesn't answer you question of why the compiler could not infer the type, but an easy work around would be to write your expression this way:
Expression> expression =
n => !fileTypeGroupID.HasValue || n.Document.MimeType.FileTypeGroupID == fileTypeGroupID.Value;