I tried to install Oracle Entitlements Server Client. When I call
config.cmd -smConfigId Sample-SM -prpFileName C:\\oracle\\product\\11.1.2\\as_1\\oessm\\SM
keyser gave direction for answer in comment.
Problem was in key's length. In short: "Starting from 7u40, the use of x.509 certificates with RSA keys less than 1024 bits in length is restricted."
So the right way to solve this problem it is using certificates with key's length at least 2048 bits.