问题
I am working on a task to generate heatmap by guided backpropagation. I have overridden the original Relu and obtained the gradient for each parameter. However, I am not sure what should I do next. Your assistance is appreciated! Thank you!
Here is my code:
I first use @tf.RegisterGradient("GuidedRelu") like:
def _GuidedReluGrad(op, grad):
gate_f = tf.cast(op.outputs[0] > 0, "float32")
gate_R = tf.cast(grad > 0, "float32")
return gate_f * gate_R * grad
Then, I obtained grads by:
with g.gradient_override_map({"Relu": "GuidedRelu"}):
with tf.GradientTape() as tape:
logits = self.net(tf.cast(img, dtype=tf.float32))
xentropy = tf.nn.softmax_cross_entropy_with_logits(
labels=tf.cast(
tf.one_hot(predicted_class, depth=1000), dtype=tf.int32
),
logits=logits,
)
reduced = tf.reduce_mean(xentropy)
grads = tape.gradient(reduced, self.net.trainable_variables)
I found the grad for the first layer has shape (7, 7, 3, 64). But I don't know how to use this grad to generate a heatmap that has similar size to the input.
回答1:
It is similar to kernel visualization of the layer. Below is an example where I am visualizing Conv2D layer having (7, 7, 4, 4) shape. (7, 7, 4, 4) means the layer has 7*7 Kernels with 4 incoming filters(filters of previous layer) and last 4 is the outgoing filters of this layer.
So in your case, (7, 7, 3, 64) means you have 7*7 Kernels with 3 incoming filters(as it is your first layer, guessing your input is a color image) and 64 is your layers filters.
For your reference, I have print all the Convolution layers of my model. I am using the same code in visualization code to get the filter shapes of last layer .i.e. conv2d_3 (7, 7, 4, 4) and use it for visualization -
# summarize filter shapes
for layer in model.layers:
# check for convolutional layer
if 'conv' in layer.name:
# get filter weights
filters, biases = layer.get_weights()
print(layer.name, filters.shape)
Output -
conv2d_1 (3, 3, 3, 2)
conv2d_2 (3, 3, 2, 4)
conv2d_3 (7, 7, 4, 4)
We shall Visualize conv2d_3 (7, 7, 4, 4) as it is similar to your requirement. So basically we should have (incoming filters * outgoing filters = 16)16 images of 7*7 size.
Code for Visualization - You need to modify incoming_filters and outgoing_filters which are filters of previous layer(if it is first layer then the channel size of image) and filters of this layer respectively.
from matplotlib import pyplot
# filters will have details of last Conv layer .i.e. conv2d_3 (7, 7, 4, 4)
for layer in model.layers:
# check for convolutional layer
if 'conv' in layer.name:
# get filter weights
filters, biases = layer.get_weights()
# Fix the figure size
fig, ax = pyplot.subplots(figsize=(15, 15))
# Normalize filter values to 0-1 so we can visualize them
f_min, f_max = filters.min(), filters.max()
filters = (filters - f_min) / (f_max - f_min)
outgoing_filters, ix = 4, 1
for i in range(outgoing_filters):
# get the filter
f = filters[:, :, :, i]
# plot each channel separately
incoming_filters = 4
for j in range(incoming_filters):
# specify subplot and turn of axis
ax = pyplot.subplot(incoming_filters, outgoing_filters, ix)
ax.set_xticks([])
ax.set_yticks([])
# plot filter channel
# Use cmap='gray' for Gray scale image
pyplot.imshow(f[:, :, j])
ix += 1
# show the figure
pyplot.show()
Output -
Hope this answers your question. Happy Learning.
Edit - Made some more effort to capture gradients after every epoch and visualize them. Used below code to capture gradients after every epoch. I am using old way to capture gradients in Tensorflow 1.15.0 and not using tf.GradientTape. If you would like to know How to capture gradient using tf.GradientTape then you can refer our answer to this question.
In the below program, gradient is the array that has gradients captured after every epoch for every layer.
Code -
# (1) Importing dependency
%tensorflow_version 1.x
import tensorflow as tf
import keras
from keras import backend as K
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout, Flatten, Conv2D, MaxPooling2D, Conv3D
from keras.layers.normalization import BatchNormalization
import numpy as np
np.random.seed(1000)
# (2) Get Data
import tflearn.datasets.oxflower17 as oxflower17
x, y = oxflower17.load_data(one_hot=True)
# (3) Create a sequential model
model = Sequential()
# 1st Convolutional Layer
model.add(Conv2D(filters=2, input_shape=(224,224,3), kernel_size=(3,3), strides=(4,4), padding='Same'))
model.add(Activation('relu'))
# 2nd Convolutional Layer
model.add(Conv2D(filters=4, kernel_size=(3,3), strides=(1,1), padding='Same'))
model.add(Activation('relu'))
# 3rd Convolutional Layer
model.add(Conv2D(filters=4, kernel_size=(7,7), strides=(1,1), padding='Same'))
model.add(Activation('relu'))
# Passing it to a dense layer
model.add(Flatten())
# 1st Dense Layer
model.add(Dense(100))
model.add(Activation('relu'))
# Output Layer
model.add(Dense(17))
model.add(Activation('softmax'))
model.summary()
# (4) Compile
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
epoch_gradient = []
def get_gradient_func(model):
grads = K.gradients(model.total_loss, model.trainable_weights)
inputs = model.model._feed_inputs + model.model._feed_targets + model.model._feed_sample_weights
func = K.function(inputs, grads)
return func
# Define the Required Callback Function
class GradientCalcCallback(tf.keras.callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
get_gradient = get_gradient_func(model)
grads = get_gradient([x, y, np.ones(len(y))])
epoch_gradient.append(grads)
epoch = 4
model.fit(x, y, batch_size=64, epochs= epoch, verbose=1, validation_split=0.2, shuffle=True, callbacks=[GradientCalcCallback()])
# (7) Convert to a 2 dimensiaonal array of (epoch, gradients) type
gradient = np.asarray(epoch_gradient)
print("Total number of epochs run:", epoch)
print("Gradient Array has the shape:",gradient.shape)
Output -
TensorFlow 1.x selected.
Using TensorFlow backend.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tflearn/helpers/summarizer.py:9: The name tf.summary.merge is deprecated. Please use tf.compat.v1.summary.merge instead.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tflearn/helpers/trainer.py:25: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tflearn/collections.py:13: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tflearn/config.py:123: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tflearn/config.py:129: The name tf.add_to_collection is deprecated. Please use tf.compat.v1.add_to_collection instead.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tflearn/config.py:131: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.
Downloading Oxford 17 category Flower Dataset, Please wait...
100.0% 60276736 / 60270631
('Succesfully downloaded', '17flowers.tgz', 60270631, 'bytes.')
File Extracted
Starting to parse images...
Parsing Done!
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_1 (Conv2D) (None, 56, 56, 2) 56
_________________________________________________________________
activation_1 (Activation) (None, 56, 56, 2) 0
_________________________________________________________________
conv2d_2 (Conv2D) (None, 56, 56, 4) 76
_________________________________________________________________
activation_2 (Activation) (None, 56, 56, 4) 0
_________________________________________________________________
conv2d_3 (Conv2D) (None, 56, 56, 4) 788
_________________________________________________________________
activation_3 (Activation) (None, 56, 56, 4) 0
_________________________________________________________________
flatten_1 (Flatten) (None, 12544) 0
_________________________________________________________________
dense_1 (Dense) (None, 100) 1254500
_________________________________________________________________
activation_4 (Activation) (None, 100) 0
_________________________________________________________________
dense_2 (Dense) (None, 17) 1717
_________________________________________________________________
activation_5 (Activation) (None, 17) 0
=================================================================
Total params: 1,257,137
Trainable params: 1,257,137
Non-trainable params: 0
_________________________________________________________________
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:431: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:438: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.
Train on 1088 samples, validate on 272 samples
Epoch 1/4
1088/1088 [==============================] - 5s 5ms/step - loss: 2.8055 - accuracy: 0.0846 - val_loss: 2.7566 - val_accuracy: 0.1176
/usr/local/lib/python3.6/dist-packages/keras/engine/sequential.py:111: UserWarning: `Sequential.model` is deprecated. `Sequential` is a subclass of `Model`, you can just use your `Sequential` instance directly.
warnings.warn('`Sequential.model` is deprecated. '
Epoch 2/4
1088/1088 [==============================] - 5s 5ms/step - loss: 2.3974 - accuracy: 0.3263 - val_loss: 2.5707 - val_accuracy: 0.2132
/usr/local/lib/python3.6/dist-packages/keras/engine/sequential.py:111: UserWarning: `Sequential.model` is deprecated. `Sequential` is a subclass of `Model`, you can just use your `Sequential` instance directly.
warnings.warn('`Sequential.model` is deprecated. '
Epoch 3/4
1088/1088 [==============================] - 5s 5ms/step - loss: 1.5953 - accuracy: 0.5506 - val_loss: 2.4076 - val_accuracy: 0.2684
/usr/local/lib/python3.6/dist-packages/keras/engine/sequential.py:111: UserWarning: `Sequential.model` is deprecated. `Sequential` is a subclass of `Model`, you can just use your `Sequential` instance directly.
warnings.warn('`Sequential.model` is deprecated. '
Epoch 4/4
1088/1088 [==============================] - 5s 5ms/step - loss: 0.8699 - accuracy: 0.7812 - val_loss: 2.5698 - val_accuracy: 0.3162
/usr/local/lib/python3.6/dist-packages/keras/engine/sequential.py:111: UserWarning: `Sequential.model` is deprecated. `Sequential` is a subclass of `Model`, you can just use your `Sequential` instance directly.
warnings.warn('`Sequential.model` is deprecated. '
Total number of epochs run: 4
Gradient Array has the shape: (4, 10)
Visualization -
Lets Visualize gradient[0][4], .i.e. [0] means the 1st epoch and [4] means fifth later of the model.
from matplotlib import pyplot
filters = gradient[0][4]
# Fix the figure size
fig, ax = pyplot.subplots(figsize=(15, 15))
# Normalize filter values to 0-1 so we can visualize them
f_min, f_max = filters.min(), filters.max()
filters = (filters - f_min) / (f_max - f_min)
outgoing_filters, ix = 4, 1
for i in range(outgoing_filters):
# get the filter
f = filters[:, :, :, i]
# plot each channel separately
incoming_filters = 4
for j in range(incoming_filters):
# specify subplot and turn of axis
ax = pyplot.subplot(incoming_filters, outgoing_filters, ix)
ax.set_xticks([])
ax.set_yticks([])
# plot filter channel
# Use cmap='gray' for Gray scale image
pyplot.imshow(f[:, :, j])
ix += 1
# show the figure
pyplot.show()
Output -
If you would like to Visualize Conv3D then refer this answer.
Hope this answers your question in detail. Happy Learning.
来源：https://stackoverflow.com/questions/62243620/visualizing-the-gradients-as-heat-map-in-tensorflow-2