Right now my solution is using a combination of SliverToBoxAdapter, Row, and ShrinkWrappingViewport to make the layout I want.
SliverToBoxAdapter
Row
ShrinkWrappingViewport