What is the difference between a sigmoid followed by the cross entropy and sigmoid_cross_entropy_with_logits in TensorFlow?

Question

When trying to get cross-entropy with sigmoid activation function, there is a difference between

1. loss1 = -tf.reduce_sum(p*tf.log(q), 1)

Answer 1

you can understand differences between softmax and sigmoid cross entropy in following way:

1. for softmax cross entropy, it actually has one probability distribution
2. for sigmoid cross entropy, it actually has multi independently binary probability distributions, each binary probability distribution can treated as two class probability distribution

so anyway the cross entropy is:

   p * -tf.log(q)

for softmax cross entropy it looks exactly as above formula，

but for sigmoid, it looks a little different for it has multi binary probability distribution for each binary probability distribution, it is

p * -tf.log(q)+(1-p) * -tf.log(1-q)

p and (1-p) you can treat as two class probability within each binary probability distribution