If I change the access modifier of the setter of a public property from private to public, is that causes any breaking change in the other assemblies that refer it?
This depends on how strict you want to be about "breaking change".
One definition is: does it break the build or execution of the referring code.
Another is, can the other code determine that you made the change.
By the first definition, this is not a breaking change. By the second definition, it is a breaking change.