You are storing a struct by value which means that accession of that struct in the map gives you a copy of the value. This is why when you modify it, the struct in the map remains unmutated until you overwrite it with the new copy.
As RickyA pointed out in the comment, you can store the pointer to the struct instead and this allows direct modification of the struct being referenced by the stored struct pointer.
i.e. map[whatever]*struct instead of map[whatever]struct
