Dictionary in Swift with Mutable Array as value is performing very slow? How to optimize or construct properly?

Question

I am trying to build a data structure in Swift that maps an Integer to an array of objects (a dictionary with int as key and array as value). The objects are extremely small

Answer 1

I had some work arounds until swift 4.2 came along

var countToColorMap = [Int: [CountedColor]]()

for (color, colorCount) in colorInfo {
    countToColorMap[colorCount, default: [CountedColor]()].append(CountedColor(color: color as! UIColor, colorCount: colorCount))
}

it is fast and readable