Listbox scrollbar thumb changes size when content is variable height

Question

I have a ListBox with many objects displayed, each of which can be a variable height, based on the number of values each object has. See my previous question that was answer

Answer 1

Why not switch off any size restrictions on the ListBox itself, let it size to contents and wrap it into a ScrollViewer, setting a proper size for the latter?

The markup should look like the following:

I saw no thumb size changings during scrolling if it was implemented this way.