Although I am not sure if it is possible to enforce a build failure on layer dependency violation, there is at least a tool that can check and visualize these layers: Sonar. You can integrate it with your maven and jenkins builds as well as with eclipse (and maybe other IDEs).
Sonar features a dependency matrix view which visualized the interdependencies of packages.
EDIT: it seems to be possible to configure sonar to force the build to fail triggered by a violation: the build breaker plugin
