I leave my comment here, although it's an old post, for others that might search for it.
We made huge improvements in memory handeling and GC calls with latest UWP 6.2.9 Nuget update while targeting >=RS3 (Win 10 1709). The complete release notes are here