My layout structure is like this
LinearLayout
FrameLayout
ImageView
ImageView
FrameLayout
TextView
LinearLayout
I
Taken from the FrameLayout docs (link)
The size of the frame layout is the size of its largest child (plus padding), visible or not (if the FrameLayout's parent permits).
This seems to describe the fact that it'll strip margins out. Like boulder mentioned, you could try switching to padding as it can be used to produce a similar effect if done properly.
Out of curiosity, you mentioned that it does work fine when using a LinearLayout container, why the choice of FrameLayout?