问题
Question
Why does setting the number of threads in the TensorFlow backend for Keras, intra_op_parallelism_threads and inter_op_parallelism_threads, to 1 negatively impact the accuracy and loss of a model?
Background
I am using Keras (2.1.6) to train an MNIST CNN with TensorFlow (1.7.0) as the backend. I was running some training in an AWS EC2 instance and noticed my accuracy decreased greatly after switching from a t2.medium instance to a t2.small. This happened without changing the code at all between the two instance types.
Given that the number of CPU cores changes from a t2.medium (2 cores) to a t2.small (1 core) I hypothesized that the decrease in accuracy had something to do with threading. In order to test this I forced TensorFlow to use a single thread and ran the training locally.
from keras import backend as K
config = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
sess = tf.Session(graph=tf.get_default_graph(),config=config)
K.set_session(sess)
Running with a single thread yielded much worse results on my local machine than running multi-threaded.
One single-threaded training run resulted in:
train_loss: 2.303228187561035 train_accuracy: 10.75%
Whereas a multi-threaded training run resulted in:
train_loss: 0.3670464503765106 train_accuracy: 88.00%
These results do not make sense to me since, in my mind, running a certain number of training epochs should result in the same amount of computational effort regardless of the number of threads used. My testing results seem to indicate that there is more training occurring with more threads rather than just parallelism being used for speed gains.
I have browsed the Keras repository in GitHub but I haven't found any code that jumps out to me why the results would be so different.
I am calling Keras' model.fit() to train the model and model.evaluate() to get the loss and accuracy.
Here are some of my hyperparameters:
loss_function: categorical_crossentropy
optimizer: Adadelta
epochs: 12
mini_batch_size: 128
train_size: 600
validate_size: 400
Update 6/26/2018
I wanted to test the increase in accuracy with more threads independently of my own code so I ran the Keras MNIST CNN with some changes based on the Keras FAQ on reproducible development. My testing code can be seen in this gist. The only difference between the two Python files in the gist are lines 80, 90, and 91. Uncommenting those lines forces the TensorFlow backend to run on a single thread.
I ran mnist_cnn_single_threaded.py three times and mnist_cnn_multi_threaded.py three times. The results can be seen in the same gist. They fit with the results I found running my own model yesterday locally and in EC2.
Testing Results
Here are my testing results per NPE's comment. Running with an explicitly set, but default tf.ConfigProto() yielded similar results to not setting the thread values at all. In the chart and table below "threads" equates to setting both intra_op_parallelism_threads and inter_op_parallelism_threads to the specified value.
Threads Test Loss Test Accuracy
1 2.300546171 0.1141
2 0.060040779 0.9806
4 0.060651763 0.9805
6 0.06015457 0.9808
8 0.057530957 0.9819
The only case that makes a noticeable difference in the accuracy and loss is explicitly setting the threads to one.
Update 6/27/2018
This appears to only be an issue with the TensorFlow backend to Keras. I tried the test using the Theano backend, with the options below to force a single thread, and I did not notice the accuracy/loss issue.
os.environ['OMP_NUM_THREADS'] = '1'
os.environ['THEANO_FLAGS'] = "device=cpu,force_device=True,openmp=False"
来源：https://stackoverflow.com/questions/51032845/single-thread-impacts-model-accuracy-and-loss-with-tensorflow-keras-backend