Caffe network getting very low loss but very bad accuracy in testing

Question

I'm somewhat new to caffe, and I'm getting some strange behavior. I'm trying to use fine tuning on the bvlc_reference_caffenet to accomplish an OCR task.

I've taken their pretrained net, changed the last FC layer to the number of output classes that I have, and retrained. After a few thousand iterations I'm getting loss rates of ~.001, and an accuracy over 90 percent when the network tests. That said, when I try to run my network on data by myself, I get awful results, not exceeding 7 or 8 percent.

The code I'm using to run the net is:

[imports]

net = caffe.Classifier('bvlc_reference_caffenet/deploy.prototxt', 'bvlc_reference_caffenet/caffenet_train_iter_28000.caffemodel',  
                       image_dims=(227, 227, 1))

input_image = caffe.io.load_image('/Training_Processed/6/0.png')    
prediction = net.predict([input_image])  # predict takes any number of images, and formats them for the Caffe net automatically    
cls = prediction[0].argmax()

Any thoughts on why this performance might be so poor?

Thanks!

PS: Some additional information which may or not be of use. When classifying as shown below, the classifier really seems to favor certain classes. Even though I have a 101 class problem, it seems to only assign a max of 15 different classes

PPS: I'm also fairly certain I'm not overfitting. I've been testing this along the way with snapshots and they all exhibit the same poor results.

Answer 1

Your code for testing the model you posted seem to miss some components:

1. It seems like you did not subtract the image's mean.
2. You did not swap channels from RGB to BGR.
3. You did not scale the inputs to [0..255] range.

Looking at similar instances of caffe.Classifier you may see something like:

net = caffe.Classifier('bvlc_reference_caffenet/deploy.prototxt',
                       'bvlc_reference_caffenet/caffenet_train_iter_28000.caffemodel', 
                       mean = NP.load( 'ilsvrc_2012_mean.npy' ),
                       input_scale=1.0, raw_scale=255,
                       channel_swap=(2,1,0),
                       image_dims=(227, 227, 1))

It is crucial to have the same input transformation in test as in training.