Vista introduced an interface: IFileDialog::SetFilter, which allows me to setup a filter that will be called for every potential filename to see if it should be shown to the
You need to initialise the callbacks for your CFileDialog. Then you need to process CDN_INCLUDEITEM notification code to include or exclude items.
You can also check this great article. The author uses some other approaches in addition to callbacks
As you have already discovered, starting in Windows 7 it is no longer possible to filter out files from being displayed based on content, only file extension. You can, however, validate that the user's selected file(s) are acceptable to you before allowing the dialog to close, and if they are not then display a message box to the user and keep the dialog open. That is the best you will be able to do unless you create your own custom dialog.