I am getting FOUC when loading css inside of my entry point when using webpack. If I remove my css from being loaded by webpack and just include it in my html file as a norm
ExtractTextWebpackPlugin will allow you to output your CSS as a separate file rather than having it embedded in your JS bundle. You can then include this file in your HTML, which as you said, prevents the flash of unstyled content.
I'd recommend only using this in production environments, as it stops hot-loading from working and makes your compile take longer. I have my webpack.config.js set up to only apply the plugin when process.env.NODE_ENV === "production"; you still get the FOUC when you're doing a development build/running the dev server, but I feel like this is a fair trade off.
For more information on how to set this up, take a look at SurviveJS's guide.
Update: As noted in the comments, ExtractTextWebpackPlugin has now been superceded by mini-css-extract-plugin - you should use that instead.