how to expand tensor of multi dim arrays by using Taylor series in tensorflow?

问题

I am trying to feed the pixel vector to the convolutional neural network (CNN), where the pixel vector came from image data like cifar-10 dataset. Before feeding the pixel vector to CNN, I need to expand the pixel vector with Taylor expansion. The point is, I figured out how to expand tensor with one dim, but not able to get it right for tensor with dim >2. Can anyone one give me ideas of how to apply Taylor expansion of one dim tensor to tensor dim more than 1? is there any heuristics approach to implement this either in TensorFlow or Keras? any possible thought?

Taylor expansion on CNN:

mathematical formulation of Taylor expansion on CNN:

Here is the heuristic mathematical formulation of Taylor expansion of CNN:

where y is output, w is weight, s is Taylor expansion of input tensor (a.k.a, pixel vector). I want to use pixel vector that comes from cifar-10 image dataset. I initially figured out way of expanding tensor with 1 dim using Taylor expansion. Here is how to scratch implementation looks like:

def cnn_taylor(input_dim, approx_order=2):
    x = Input((input_dim,))
    def pwr(x, approx_order):
        x = x[..., None] 
        x = tf.tile(x, multiples=[1, 1, approx_order + 1])
        pw = tf.range(0, approx_order + 1, dtype=tf.float32) 
        x_p = tf.pow(x, pw) 
        x_p = x_p[..., None]
        return x_p

    x_p = Lambda(lambda x: pwr(x, approx_order))(x)
    h = Dense(1, use_bias=False)(x_p)  
    def cumu_sum(h):
        h = tf.squeeze(h, axis=-1)  
        s = tf.cumsum(h, axis=-1) 
        s = s[..., None] 
        return s
    S = Lambda(cumu_sum)(h)

so above implementation is sketch coding attempt on how to expand CNN with Taylor expansion by using 1 dim tensor. I am wondering how to do same thing to tensor with multi dim array (i.e, dim=3).

graphical illustration

here is the graphical illustration of attempting to expand CNN with Taylor expansion where tensor with 3 dim arrays are used as an input. I want to do a simple pixel image classification by doing so, I am hoping someone might give me a possible direction.

above Taylor expansion of flat tensor is implemented on any one of RGB channels. If I want to expand CNN with an approximation order of 2 with Taylor expansion where input is a pixel vector from RGB image, how am I going to accomplish this easily in TensorFlow? any thought? Thanks

回答1:

If I understand correctly, each x in the provided computational graph is just a scalar (one channel of a pixel). In this case, in order to apply the transformation to each pixel, you could:

1. Flatten the 4D (b, h, w, c) input coming from the convolutional layer into a tensor of shape (b, h*w*c).
2. Apply the transformation to the resulting tensor.
3. Undo the reshaping to get a 4D tensor of shape (b, h, w, c)` back for which the "Taylor expansion" has been applied element-wise.

This could be achieved as follows:

shape_cnn = h.shape  # Shape=(bs, h, w, c)
flat_dim = h.shape[1] * h.shape[2] * h.shape[3]
h = tf.reshape(h, (-1, flat_dim))
taylor_model = taylor_expansion_network(input_dim=flat_dim, max_pow=approx_order)
h = taylor_model(h)
h = tf.reshape(h, (-1, shape_cnn[1], shape_cnn[2], shape_cnn[3]))

NOTE: I am borrowing the function taylor_expansion_network from this answer.

---

UPDATE: I still don't clearly understand the end goal, but perhaps this update brings us closer to the desired output. I modified the taylor_expansion_network to apply the first part of the pipeline to RGB images of shape (width, height, nb_channels=3), returning a tensor of shape (width, height, nb_channels=3, max_pow+1):

def taylor_expansion_network_2(width, height, nb_channels=3, max_pow=2):
    input_dim = width * height * nb_channels

    x = Input((width, height, nb_channels,))
    h = tf.reshape(x, (-1, input_dim))

    # Raise input x_i to power p_i for each i in [0, max_pow].
    def raise_power(x, max_pow):
        x_ = x[..., None]  # Shape=(batch_size, input_dim, 1)
        x_ = tf.tile(x_, multiples=[1, 1, max_pow + 1])  # Shape=(batch_size, input_dim, max_pow+1)
        pows = tf.range(0, max_pow + 1, dtype=tf.float32)  # Shape=(max_pow+1,)
        x_p = tf.pow(x_, pows)  # Shape=(batch_size, input_dim, max_pow+1)
        return x_p

    h = raise_power(h, max_pow)

    # Compute s_i for each i in [0, max_pow]
    h = tf.cumsum(h, axis=-1)  # Shape=(batch_size, input_dim, max_pow+1)

    # Get the input format back
    h = tf.reshape(h, (-1, width, height, nb_channels, max_pow+1))  # Shape=(batch_size, w, h, nb_channels, max_pow+1)

    # Return Taylor expansion model
    model = Model(inputs=x, outputs=h)
    model.summary()
    return model

In this modified model, the last step of the pipeline, namely the sum of w_i * s_i for each i, is not applied. Now, you can use the resulting tensor of shape (width, height, nb_channels=3, max_pow+1) in any way you want.

来源：https://stackoverflow.com/questions/61298195/how-to-expand-tensor-of-multi-dim-arrays-by-using-taylor-series-in-tensorflow