For a JFIF file, a JFIF header should follow immediately after FFD8. The JFIF header is contained within an APP0 marker. The specification doesn't say anything about padding though.
Without the JFIF header we can only guess what color format is used.