I submitted an application for review and I notice that the build that I submitted has an issue associated with it saying that Build 168 does not contain the correct b
Yes, you need to recreate provisioning profile and one more important thing is that you need to provision your app with App Store distribution provisioning profile.
Here is how it should look like when you click in Organizer "Export..." -> "Save for iOS App Store Deployment":

You can also convert .mobileprovision file to .xml with that command to see if there really is a beta-reports-active key in your entitlements.
security cms -D -i YourProfile.mobileprovision > YourProfile.xml