看过前面的例子，会发现实现深度神经网络需要使用 tensorflow.nn 这个核心模块。我们通过源码来一探究竟。
1 # Copyright 2015 Google Inc. All Rights Reserved.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
4 # you may not use this file except in compliance with the License.
5 # You may obtain a copy of the License at
6 #
7 # http://www.apache.org/licenses/LICENSE-2.0
8 #
9 # Unless required by applicable law or agreed to in writing, software
10 # distributed under the License is distributed on an "AS IS" BASIS,
11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
14 # ==============================================================================
15
16 # pylint: disable=unused-import,g-bad-import-order
17 """## Activation Functions
18
19 The activation ops provide different types of nonlinearities for use in neural
20 networks. These include smooth nonlinearities (`sigmoid`, `tanh`, `elu`,
21 `softplus`, and `softsign`), continuous but not everywhere differentiable
22 functions (`relu`, `relu6`, and `relu_x`), and random regularization
23 (`dropout`).
24
25 All activation ops apply componentwise, and produce a tensor of the same
26 shape as the input tensor.
27
28 @@relu
29 @@relu6
30 @@elu
31 @@softplus
32 @@softsign
33 @@dropout
34 @@bias_add
35 @@sigmoid
36 @@tanh
37
38 ## Convolution
39
40 The convolution ops sweep a 2-D filter over a batch of images, applying the
41 filter to each window of each image of the appropriate size. The different
42 ops trade off between generic vs. specific filters:
43
44 * `conv2d`: Arbitrary filters that can mix channels together.
45 * `depthwise_conv2d`: Filters that operate on each channel independently.
46 * `separable_conv2d`: A depthwise spatial filter followed by a pointwise filter.
47
48 Note that although these ops are called "convolution", they are strictly
49 speaking "cross-correlation" since the filter is combined with an input window
50 without reversing the filter. For details, see [the properties of
51 cross-correlation](https://en.wikipedia.org/wiki/Cross-correlation#Properties).
52
53 The filter is applied to image patches of the same size as the filter and
54 strided according to the `strides` argument. `strides = [1, 1, 1, 1]` applies
55 the filter to a patch at every offset, `strides = [1, 2, 2, 1]` applies the
56 filter to every other image patch in each dimension, etc.
57
58 Ignoring channels for the moment, and assume that the 4-D `input` has shape
59 `[batch, in_height, in_width, ...]` and the 4-D `filter` has shape
60 `[filter_height, filter_width, ...]`, then the spatial semantics of the
61 convolution ops are as follows: first, according to the padding scheme chosen
62 as `'SAME'` or `'VALID'`, the output size and the padding pixels are computed.
63 For the `'SAME'` padding, the output height and width are computed as:
64
65 out_height = ceil(float(in_height) / float(strides[1]))
66 out_width = ceil(float(in_width) / float(strides[2]))
67
68 and the padding on the top and left are computed as:
69
70 pad_along_height = ((out_height - 1) * strides[1] +
71 filter_height - in_height)
72 pad_along_width = ((out_width - 1) * strides[2] +
73 filter_width - in_width)
74 pad_top = pad_along_height / 2
75 pad_left = pad_along_width / 2
76
77 Note that the division by 2 means that there might be cases when the padding on
78 both sides (top vs bottom, right vs left) are off by one. In this case, the
79 bottom and right sides always get the one additional padded pixel. For example,
80 when `pad_along_height` is 5, we pad 2 pixels at the top and 3 pixels at the
81 bottom. Note that this is different from existing libraries such as cuDNN and
82 Caffe, which explicitly specify the number of padded pixels and always pad the
83 same number of pixels on both sides.
84
85 For the `'VALID`' padding, the output height and width are computed as:
86
87 out_height = ceil(float(in_height - filter_height + 1) / float(strides[1]))
88 out_width = ceil(float(in_width - filter_width + 1) / float(strides[2]))
89
90 and the padding values are always zero. The output is then computed as
91
92 output[b, i, j, :] =
93 sum_{di, dj} input[b, strides[1] * i + di - pad_top,
94 strides[2] * j + dj - pad_left, ...] *
95 filter[di, dj, ...]
96
97 where any value outside the original input image region are considered zero (
98 i.e. we pad zero values around the border of the image).
99
100 Since `input` is 4-D, each `input[b, i, j, :]` is a vector. For `conv2d`, these
101 vectors are multiplied by the `filter[di, dj, :, :]` matrices to produce new
102 vectors. For `depthwise_conv_2d`, each scalar component `input[b, i, j, k]`
103 is multiplied by a vector `filter[di, dj, k]`, and all the vectors are
104 concatenated.
105
106 @@conv2d
107 @@depthwise_conv2d
108 @@separable_conv2d
109 @@conv2d_transpose
110
111 ## Pooling
112
113 The pooling ops sweep a rectangular window over the input tensor, computing a
114 reduction operation for each window (average, max, or max with argmax). Each
115 pooling op uses rectangular windows of size `ksize` separated by offset
116 `strides`. For example, if `strides` is all ones every window is used, if
117 `strides` is all twos every other window is used in each dimension, etc.
118
119 In detail, the output is
120
121 output[i] = reduce(value[strides * i:strides * i + ksize])
122
123 where the indices also take into consideration the padding values. Please refer
124 to the `Convolution` section for details about the padding calculation.
125
126 @@avg_pool
127 @@max_pool
128 @@max_pool_with_argmax
129
130 ## Normalization
131
132 Normalization is useful to prevent neurons from saturating when inputs may
133 have varying scale, and to aid generalization.
134
135 @@l2_normalize
136 @@local_response_normalization
137 @@sufficient_statistics
138 @@normalize_moments
139 @@moments
140
141 ## Losses
142
143 The loss ops measure error between two tensors, or between a tensor and zero.
144 These can be used for measuring accuracy of a network in a regression task
145 or for regularization purposes (weight decay).
146
147 @@l2_loss
148
149 ## Classification
150
151 TensorFlow provides several operations that help you perform classification.
152
153 @@sigmoid_cross_entropy_with_logits
154 @@softmax
155 @@log_softmax
156 @@softmax_cross_entropy_with_logits
157 @@sparse_softmax_cross_entropy_with_logits
158 @@weighted_cross_entropy_with_logits
159
160 ## Embeddings
161
162 TensorFlow provides library support for looking up values in embedding
163 tensors.
164
165 @@embedding_lookup
166 @@embedding_lookup_sparse
167
168 ## Evaluation
169
170 The evaluation ops are useful for measuring the performance of a network.
171 Since they are nondifferentiable, they are typically used at evaluation time.
172
173 @@top_k
174 @@in_top_k
175
176 ## Candidate Sampling
177
178 Do you want to train a multiclass or multilabel model with thousands
179 or millions of output classes (for example, a language model with a
180 large vocabulary)? Training with a full Softmax is slow in this case,
181 since all of the classes are evaluated for every training example.
182 Candidate Sampling training algorithms can speed up your step times by
183 only considering a small randomly-chosen subset of contrastive classes
184 (called candidates) for each batch of training examples.
185
186 See our [Candidate Sampling Algorithms Reference]
187 (../../extras/candidate_sampling.pdf)
188
189 ### Sampled Loss Functions
190
191 TensorFlow provides the following sampled loss functions for faster training.
192
193 @@nce_loss
194 @@sampled_softmax_loss
195
196 ### Candidate Samplers
197
198 TensorFlow provides the following samplers for randomly sampling candidate
199 classes when using one of the sampled loss functions above.
200
201 @@uniform_candidate_sampler
202 @@log_uniform_candidate_sampler
203 @@learned_unigram_candidate_sampler
204 @@fixed_unigram_candidate_sampler
205
206 ### Miscellaneous candidate sampling utilities
207
208 @@compute_accidental_hits
209
210 """
211 from __future__ import absolute_import
212 from __future__ import division
213 from __future__ import print_function
214
215 from six.moves import xrange # pylint: disable=redefined-builtin
216
217 from tensorflow.python.framework import dtypes
218 from tensorflow.python.framework import ops
219 from tensorflow.python.framework import tensor_shape
220 from tensorflow.python.ops import array_ops
221 from tensorflow.python.ops import candidate_sampling_ops
222 from tensorflow.python.ops import constant_op
223 from tensorflow.python.ops import control_flow_ops
224 from tensorflow.python.ops import embedding_ops
225 from tensorflow.python.ops import init_ops
226 from tensorflow.python.ops import math_ops
227 from tensorflow.python.ops import nn_grad
228 from tensorflow.python.ops import nn_ops
229 from tensorflow.python.ops import numerics
230 from tensorflow.python.ops import random_ops
231 from tensorflow.python.ops import rnn_cell
232 from tensorflow.python.ops import seq2seq
233 from tensorflow.python.ops import sparse_ops
234 from tensorflow.python.ops import variable_scope as vs
235 from tensorflow.python.ops.math_ops import sigmoid
236 from tensorflow.python.ops.math_ops import tanh
237 from tensorflow.python.util.all_util import make_all
238
239 # Bring more nn-associated functionality into this package.
240 # go/tf-wildcard-import
241 # pylint: disable=wildcard-import
242 from tensorflow.python.ops.nn_ops import *
243 from tensorflow.python.ops.candidate_sampling_ops import *
244 from tensorflow.python.ops.embedding_ops import *
245 from tensorflow.python.ops.rnn import *
246 # pylint: enable=wildcard-import
247
248
249 def sigmoid_cross_entropy_with_logits(logits, targets, name=None):
250 """Computes sigmoid cross entropy given `logits`.
251
252 Measures the probability error in discrete classification tasks in which each
253 class is independent and not mutually exclusive. For instance, one could
254 perform multilabel classification where a picture can contain both an elephant
255 and a dog at the same time.
256
257 For brevity, let `x = logits`, `z = targets`. The logistic loss is
258
259 z * -log(sigmoid(x)) + (1 - z) * -log(1 - sigmoid(x))
260 = z * -log(1 / (1 + exp(-x))) + (1 - z) * -log(exp(-x) / (1 + exp(-x)))
261 = z * log(1 + exp(-x)) + (1 - z) * (-log(exp(-x)) + log(1 + exp(-x)))
262 = z * log(1 + exp(-x)) + (1 - z) * (x + log(1 + exp(-x))
263 = (1 - z) * x + log(1 + exp(-x))
264 = x - x * z + log(1 + exp(-x))
265
266 To ensure stability and avoid overflow, the implementation uses
267
268 max(x, 0) - x * z + log(1 + exp(-abs(x)))
269
270 `logits` and `targets` must have the same type and shape.
271
272 Args:
273 logits: A `Tensor` of type `float32` or `float64`.
274 targets: A `Tensor` of the same type and shape as `logits`.
275 name: A name for the operation (optional).
276
277 Returns:
278 A `Tensor` of the same shape as `logits` with the componentwise
279 logistic losses.
280
281 Raises:
282 ValueError: If `logits` and `targets` do not have the same shape.
283 """
284 with ops.op_scope([logits, targets], name, "logistic_loss") as name:
285 logits = ops.convert_to_tensor(logits, name="logits")
286 targets = ops.convert_to_tensor(targets, name="targets")
287 try:
288 targets.get_shape().merge_with(logits.get_shape())
289 except ValueError:
290 raise ValueError(
291 "logits and targets must have the same shape (%s vs %s)"
292 % (logits.get_shape(), targets.get_shape()))
293
294 # The logistic loss formula from above is
295 # x - x * z + log(1 + exp(-x))
296 # For x < 0, a more numerically stable formula is
297 # -x * z + log(1 + exp(x))
298 # To avoid branching, we use the combined version
299 # max(x, 0) - x * z + log(1 + exp(-abs(x)))
300 return math_ops.add(nn_ops.relu(logits) - logits * targets,
301 math_ops.log(1 + math_ops.exp(-math_ops.abs(logits))),
302 name=name)
303
304
305 def weighted_cross_entropy_with_logits(logits, targets, pos_weight,
306 name=None):
307 """Computes a weighted cross entropy.
308
309 This is like `sigmoid_cross_entropy_with_logits()` except that `pos_weight`,
310 allows one to trade off recall and precision by up- or down-weighting the
311 cost of a positive error relative to a negative error.
312
313 The usual cross-entropy cost is defined as:
314
315 targets * -log(sigmoid(logits)) + (1 - targets) * -log(1 - sigmoid(logits))
316
317 The argument `pos_weight` is used as a multiplier for the positive targets:
318
319 targets * -log(sigmoid(logits)) * pos_weight +
320 (1 - targets) * -log(1 - sigmoid(logits))
321
322 For brevity, let `x = logits`, `z = targets`, `q = pos_weight`.
323 The loss is:
324
325 qz * -log(sigmoid(x)) + (1 - z) * -log(1 - sigmoid(x))
326 = qz * -log(1 / (1 + exp(-x))) + (1 - z) * -log(exp(-x) / (1 + exp(-x)))
327 = qz * log(1 + exp(-x)) + (1 - z) * (-log(exp(-x)) + log(1 + exp(-x)))
328 = qz * log(1 + exp(-x)) + (1 - z) * (x + log(1 + exp(-x))
329 = (1 - z) * x + (qz + 1 - z) * log(1 + exp(-x))
330 = (1 - z) * x + (1 + (q - 1) * z) * log(1 + exp(-x))
331
332 Setting `l = (1 + (q - 1) * z)`, to ensure stability and avoid overflow,
333 the implementation uses
334
335 (1 - z) * x + l * (log(1 + exp(-abs(x))) + max(-x, 0))
336
337 `logits` and `targets` must have the same type and shape.
338
339 Args:
340 logits: A `Tensor` of type `float32` or `float64`.
341 targets: A `Tensor` of the same type and shape as `logits`.
342 pos_weight: A coefficient to use on the positive examples.
343 name: A name for the operation (optional).
344
345 Returns:
346 A `Tensor` of the same shape as `logits` with the componentwise
347 weightedlogistic losses.
348
349 Raises:
350 ValueError: If `logits` and `targets` do not have the same shape.
351 """
352 with ops.op_scope([logits, targets], name, "logistic_loss") as name:
353 logits = ops.convert_to_tensor(logits, name="logits")
354 targets = ops.convert_to_tensor(targets, name="targets")
355 try:
356 targets.get_shape().merge_with(logits.get_shape())
357 except ValueError:
358 raise ValueError(
359 "logits and targets must have the same shape (%s vs %s)"
360 % (logits.get_shape(), targets.get_shape()))
361
362 # The logistic loss formula from above is
363 # (1 - z) * x + (1 + (q - 1) * z) * log(1 + exp(-x))
364 # For x < 0, a more numerically stable formula is
365 # (1 - z) * x + (1 + (q - 1) * z) * log(1 + exp(x)) - l * x
366 # To avoid branching, we use the combined version
367 # (1 - z) * x + l * (log(1 + exp(-abs(x))) + max(-x, 0))
368 log_weight = 1 + (pos_weight - 1) * targets
369 return math_ops.add(
370 (1 - targets) * logits,
371 log_weight * (math_ops.log(1 + math_ops.exp(-math_ops.abs(logits))) +
372 nn_ops.relu(-logits)),
373 name=name)
374
375
376 def relu_layer(x, weights, biases, name=None):
377 """Computes Relu(x * weight + biases).
378
379 Args:
380 x: a 2D tensor. Dimensions typically: batch, in_units
381 weights: a 2D tensor. Dimensions typically: in_units, out_units
382 biases: a 1D tensor. Dimensions: out_units
383 name: A name for the operation (optional). If not specified
384 "nn_relu_layer" is used.
385
386 Returns:
387 A 2-D Tensor computing relu(matmul(x, weights) + biases).
388 Dimensions typically: batch, out_units.
389 """
390 with ops.op_scope([x, weights, biases], name, "relu_layer") as name:
391 x = ops.convert_to_tensor(x, name="x")
392 weights = ops.convert_to_tensor(weights, name="weights")
393 biases = ops.convert_to_tensor(biases, name="biases")
394 xw_plus_b = nn_ops.bias_add(math_ops.matmul(x, weights), biases)
395 return nn_ops.relu(xw_plus_b, name=name)
396
397
398 def l2_normalize(x, dim, epsilon=1e-12, name=None):
399 """Normalizes along dimension `dim` using an L2 norm.
400
401 For a 1-D tensor with `dim = 0`, computes
402
403 output = x / sqrt(max(sum(x**2), epsilon))
404
405 For `x` with more dimensions, independently normalizes each 1-D slice along
406 dimension `dim`.
407
408 Args:
409 x: A `Tensor`.
410 dim: Dimension along which to normalize.
411 epsilon: A lower bound value for the norm. Will use `sqrt(epsilon)` as the
412 divisor if `norm < sqrt(epsilon)`.
413 name: A name for this operation (optional).
414
415 Returns:
416 A `Tensor` with the same shape as `x`.
417 """
418 with ops.op_scope([x], name, "l2_normalize") as name:
419 x = ops.convert_to_tensor(x, name="x")
420 square_sum = math_ops.reduce_sum(math_ops.square(x), [dim], keep_dims=True)
421 x_inv_norm = math_ops.rsqrt(math_ops.maximum(square_sum, epsilon))
422 return math_ops.mul(x, x_inv_norm, name=name)
423
424
425 def zero_fraction(value, name=None):
426 """Returns the fraction of zeros in `value`.
427
428 If `value` is empty, the result is `nan`.
429
430 This is useful in summaries to measure and report sparsity. For example,
431
432 z = tf.Relu(...)
433 summ = tf.scalar_summary('sparsity', tf.nn.zero_fraction(z))
434
435 Args:
436 value: A tensor of numeric type.
437 name: A name for the operation (optional).
438
439 Returns:
440 The fraction of zeros in `value`, with type `float32`.
441 """
442 with ops.op_scope([value], name, "zero_fraction"):
443 value = ops.convert_to_tensor(value, name="value")
444 zero = constant_op.constant(0, dtype=value.dtype, name="zero")
445 return math_ops.reduce_mean(math_ops.cast(math_ops.equal(value, zero),
446 dtypes.float32))
447
448
449 def depthwise_conv2d(input, filter, strides, padding, name=None):
450 """Depthwise 2-D convolution.
451
452 Given an input tensor of shape `[batch, in_height, in_width, in_channels]`
453 and a filter tensor of shape
454 `[filter_height, filter_width, in_channels, channel_multiplier]`
455 containing `in_channels` convolutional filters of depth 1, `depthwise_conv2d`
456 applies a different filter to each input channel (expanding from 1 channel
457 to `channel_multiplier` channels for each), then concatenates the results
458 together. The output has `in_channels * channel_multiplier` channels.
459
460 In detail,
461
462 output[b, i, j, k * channel_multiplier + q] =
463 sum_{di, dj} input[b, strides[1] * i + di, strides[2] * j + dj, k] *
464 filter[di, dj, k, q]
465
466 Must have `strides[0] = strides[3] = 1`. For the most common case of the
467 same horizontal and vertical strides, `strides = [1, stride, stride, 1]`.
468
469 Args:
470 input: 4-D with shape `[batch, in_height, in_width, in_channels]`.
471 filter: 4-D with shape
472 `[filter_height, filter_width, in_channels, channel_multiplier]`.
473 strides: 1-D of size 4. The stride of the sliding window for each
474 dimension of `input`.
475 padding: A string, either `'VALID'` or `'SAME'`. The padding algorithm.
476 name: A name for this operation (optional).
477
478 Returns:
479 A 4-D `Tensor` of shape
480 `[batch, out_height, out_width, in_channels * channel_multiplier].`
481 """
482 with ops.op_scope([input, filter], name, "depthwise") as name:
483 input = ops.convert_to_tensor(input, name="tensor_in")
484 filter = ops.convert_to_tensor(filter, name="filter_in")
485 # A shape is required to statically compute the number of separable filters.
486 if filter.get_shape().ndims is not None:
487 assert len(filter.get_shape()) == 4
488 in_channels = filter.get_shape()[2]
489 # Sanity checks, if shape information is available for the inputs.
490 if input.get_shape().ndims is not None:
491 assert len(input.get_shape()) == 4
492 assert input.get_shape()[3] == in_channels, (
493 "Mismatched input depth %d and number of depthwise filters %d." % (
494 input.get_shape()[3].value, in_channels))
495 else:
496 assert input.get_shape().ndims is not None, (
497 "Either tensor must provide static shape information.")
498 assert input.get_shape().ndims == 4
499 in_channels = input.get_shape()[3]
500
501 if in_channels == 1:
502 return nn_ops.conv2d(input, filter, strides, padding, name=name)
503 else:
504 # Create one separate convolution per channel.
505 convs = []
506 for channel in xrange(in_channels):
507 with ops.name_scope("depth%d" % channel) as channel_scope:
508 t_in = array_ops.slice(input, [0, 0, 0, channel], [-1, -1, -1, 1],
509 name="slice_inputs")
510 f_in = array_ops.slice(filter, [0, 0, channel, 0], [-1, -1, 1, -1],
511 name="slice_params")
512 convs.append(nn_ops.conv2d(t_in, f_in,
513 strides, padding, name=channel_scope))
514 # Concatenate the per-channel convolutions along the channel dimension.
515 return array_ops.concat(3, convs, name=name)
516
517
518 def separable_conv2d(input, depthwise_filter, pointwise_filter, strides,
519 padding,
520 name=None):
521 """2-D convolution with separable filters.
522
523 Performs a depthwise convolution that acts separately on channels followed by
524 a pointwise convolution that mixes channels. Note that this is separability
525 between dimensions `[1, 2]` and `3`, not spatial separability between
526 dimensions `1` and `2`.
527
528 In detail,
529
530 output[b, i, j, k] = sum_{di, dj, q, r]
531 input[b, strides[1] * i + di, strides[2] * j + dj, q] *
532 depthwise_filter[di, dj, q, r] *
533 pointwise_filter[0, 0, q * channel_multiplier + r, k]
534
535 `strides` controls the strides for the depthwise convolution only, since
536 the pointwise convolution has implicit strides of `[1, 1, 1, 1]`. Must have
537 `strides[0] = strides[3] = 1`. For the most common case of the same
538 horizontal and vertical strides, `strides = [1, stride, stride, 1]`.
539
540 Args:
541 input: 4-D `Tensor` with shape `[batch, in_height, in_width, in_channels]`.
542 depthwise_filter: 4-D `Tensor` with shape
543 `[filter_height, filter_width, in_channels, channel_multiplier]`.
544 Contains `in_channels` convolutional filters of depth 1.
545 pointwise_filter: 4-D `Tensor` with shape
546 `[1, 1, channel_multiplier * in_channels, out_channels]`. Pointwise
547 filter to mix channels after `depthwise_filter` has convolved spatially.
548 strides: 1-D of size 4. The strides for the depthwise convolution for
549 each dimension of `input`.
550 padding: A string, either `'VALID'` or `'SAME'`. The padding algorithm.
551 name: A name for this operation (optional).
552
553 Returns:
554 A 4-D `Tensor` of shape `[batch, out_height, out_width, out_channels]`.
555 """
556 with ops.op_scope([input, depthwise_filter, pointwise_filter],
557 name, "separable_conv2d") as name:
558 input = ops.convert_to_tensor(input, name="tensor_in")
559 depthwise_filter = ops.convert_to_tensor(depthwise_filter,
560 name="depthwise_filter")
561 pointwise_filter = ops.convert_to_tensor(pointwise_filter,
562 name="pointwise_filter")
563
564 if pointwise_filter.get_shape().ndims is not None:
565 assert len(pointwise_filter.get_shape()) == 4
566 assert pointwise_filter.get_shape()[0] == 1
567 assert pointwise_filter.get_shape()[1] == 1
568 if depthwise_filter.get_shape().ndims and input.get_shape().ndims:
569 channel_multiplier = depthwise_filter.get_shape()[3]
570 in_channels = input.get_shape()[3]
571 out_channels = pointwise_filter.get_shape()[3]
572 # This would mean the separable convolutions is over-parametrized.
573 assert channel_multiplier * in_channels < out_channels
574 # The layout of the ops in the graph are expected to be as follows:
575 # separable_conv2d // Conv2D op corresponding to the pointwise conv.
576 # separable_conv2d/depthwise // Concat op for the deptwise outputs.
577 # separable_conv2d/depthwise/depth0 // Conv2D op for depth 0
578 # separable_conv2d/depthwise/depth1 // Conv2D op for depth 1
579 # separable_conv2d/depthwise/depth2 // Conv2D op for depth 2
580 depthwise = depthwise_conv2d(input, depthwise_filter, strides,
581 padding, name="depthwise")
582 return nn_ops.conv2d(depthwise, pointwise_filter, [1, 1, 1, 1],
583 padding="VALID", name=name)
584
585
586 def sufficient_statistics(x, axes, shift=True, keep_dims=False, name=None):
587 """Calculate the sufficient statistics for the mean and variance of `x`.
588
589 These sufficient statistics are computed using the one pass algorithm on
590 an input that's optionally shifted using the value of the 1st element in `x`.
591 See:
592 https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Computing_shifted_data
593
594 Args:
595 x: A `Tensor`.
596 axes: Array of ints. Axes along which to compute mean and variance.
597 shift: If true, shift the data to provide more numerically stable results.
598 keep_dims: produce statistics with the same dimensionality as the input.
599 name: Name used to scope the operations that compute the sufficient stats.
600
601 Returns:
602 Four `Tensor` objects of the same type as `x`:
603 * the count (number of elements to average over).
604 * the (possibly shifted) sum of the elements in the array.
605 * the (possibly shifted) sum of squares of the elements in the array.
606 * the shift by which the mean must be corrected or None if `shift` is False.
607 """
608 with ops.op_scope([x, axes], name, "sufficient_statistics"):
609 x = ops.convert_to_tensor(x, name="x")
610 x_shape = x.get_shape()
611 if x_shape.is_fully_defined():
612 counts = 1
613 m_shape = []
614 for d in xrange(x_shape.ndims):
615 dim = x_shape[d].value
616 if d in set(axes):
617 counts *= dim
618 dim = 1
619 m_shape.append(dim)
620 counts = constant_op.constant(counts, dtype=x.dtype)
621 else: # shape needs to be inferred at runtime.
622 x_shape = array_ops.shape(x)
623 select_axes = sparse_ops.sparse_to_dense(axes, array_ops.shape(x_shape),
624 True, False)
625 m_shape = math_ops.select(select_axes, array_ops.ones_like(x_shape),
626 x_shape)
627 counts = math_ops.cast(
628 math_ops.reduce_prod(x_shape / m_shape),
629 x.dtype,
630 name="count")
631 if shift:
632 shift_value = array_ops.slice(x, array_ops.zeros_like(m_shape), m_shape)
633 m_ss = math_ops.sub(x, shift_value)
634 v_ss = math_ops.squared_difference(x, shift_value)
635 if keep_dims:
636 shift_value = array_ops.identity(shift_value, name="shift")
637 else:
638 shift_value = array_ops.squeeze(shift_value,
639 squeeze_dims=axes,
640 name="shift")
641 else: # not shift.
642 m_ss = x
643 v_ss = math_ops.square(x)
644 shift_value = None
645 m_ss = math_ops.reduce_sum(m_ss, axes, keep_dims=keep_dims, name="mean_ss")
646 v_ss = math_ops.reduce_sum(v_ss, axes, keep_dims=keep_dims, name="var_ss")
647 return counts, m_ss, v_ss, shift_value
648
649
650 def normalize_moments(counts, mean_ss, variance_ss, shift, name=None):
651 """Calculate the mean and variance of based on the sufficient statistics.
652
653 Args:
654 counts: A `Tensor` containing a the total count of the data (one value).
655 mean_ss: A `Tensor` containing the mean sufficient statistics: the (possibly
656 shifted) sum of the elements to average over.
657 variance_ss: A `Tensor` containing the variance sufficient statistics: the
658 (possibly shifted) squared sum of the data to compute the variance over.
659 shift: A `Tensor` containing the value by which the data is shifted for
660 numerical stability, or `None` if no shift was performed.
661 name: Name used to scope the operations that compute the moments.
662
663 Returns:
664 Two `Tensor` objects: `mean` and `variance`.
665 """
666 with ops.op_scope([counts, mean_ss, variance_ss, shift], name, "normalize"):
667 divisor = math_ops.inv(counts, name="divisor")
668 if shift is not None:
669 shifted_mean = math_ops.mul(mean_ss, divisor, name="shifted_mean")
670 mean = math_ops.add(shifted_mean, shift, name="mean")
671 else: # no shift.
672 shifted_mean = math_ops.mul(mean_ss, divisor, name="mean")
673 mean = shifted_mean
674 variance = math_ops.sub(
675 math_ops.mul(variance_ss, divisor),
676 math_ops.square(shifted_mean),
677 name="variance")
678 return (mean, variance)
679
680
681 def moments(x, axes, name=None, keep_dims=False):
682 """Calculate the mean and variance of `x`.
683
684 The mean and variance are calculated by aggregating the contents of `x`
685 across `axes`. If `x` is 1-D and `axes = [0]` this is just the mean
686 and variance of a vector.
687
688 When using these moments for batch normalization (see
689 `tf.nn.batch_normalization`):
690 * for so-called "global normalization", used with convolutional filters with
691 shape `[batch, height, width, depth]`, pass `axes=[0, 1, 2]`.
692 * for simple batch normalization pass `axes=[0]` (batch only).
693
694 Args:
695 x: A `Tensor`.
696 axes: array of ints. Axes along which to compute mean and
697 variance.
698 keep_dims: produce moments with the same dimensionality as the input.
699 name: Name used to scope the operations that compute the moments.
700
701 Returns:
702 Two `Tensor` objects: `mean` and `variance`.
703 """
704 with ops.op_scope([x, axes], name, "moments"):
705 counts, m_ss, v_ss, shift = sufficient_statistics(x,
706 axes,
707 keep_dims=keep_dims,
708 name=name)
709 return normalize_moments(counts, m_ss, v_ss, shift, name=name)
710
711
712 def batch_normalization(x,
713 mean,
714 variance,
715 offset,
716 scale,
717 variance_epsilon,
718 name=None):
719 """Batch normalization.
720
721 As described in http://arxiv.org/abs/1502.03167.
722 Normalizes a tensor by `mean` and `variance`, and applies (optionally) a
723 `scale` \\\\(\gamma\\\\) to it, as well as an `offset` \\\\(\\beta\\\\):
724
725 \\\\(\\frac{\gamma(x-\mu)}{\sigma}+\\beta\\\\)
726
727 `mean`, `variance`, `offset` and `scale` are all expected to be of one of two
728 shapes:
729 * In all generality, they can have the same number of dimensions as the
730 input `x`, with identical sizes as `x` for the dimensions that are not
731 normalized over (the 'depth' dimension(s)), and dimension 1 for the
732 others which are being normalized over.
733 `mean` and `variance` in this case would typically be the outputs of
734 `tf.nn.moments(..., keep_dims=True)` during training, or running averages
735 thereof during inference.
736 * In the common case where the 'depth' dimension is the last dimension in
737 the input tensor `x`, they may be one dimensional tensors of the same
738 size as the 'depth' dimension.
739 This is the case for example for the common `[batch, depth]` layout of
740 fully-connected layers, and `[batch, height, width, depth]` for
741 convolutions.
742 `mean` and `variance` in this case would typically be the outputs of
743 `tf.nn.moments(..., keep_dims=False)` during training, or running averages
744 thereof during inference.
745
746 Args:
747 x: Input `Tensor` of arbitrary dimensionality.
748 mean: A mean `Tensor`.
749 variance: A variance `Tensor`.
750 offset: An offset `Tensor`, often denoted \\\\(\\beta\\\\) in equations, or
751 None. If present, will be added to the normalized tensor.
752 scale: A scale `Tensor`, often denoted \\\\(\gamma\\\\) in equations, or
753 `None`. If present, the scale is applied to the normalized tensor.
754 variance_epsilon: A small float number to avoid dividing by 0.
755 name: A name for this operation (optional).
756
757 Returns:
758 the normalized, scaled, offset tensor.
759 """
760 with ops.op_scope([x, mean, variance, scale, offset], name, "batchnorm"):
761 inv = math_ops.rsqrt(variance + variance_epsilon)
762 if scale is not None:
763 inv *= scale
764 return x * inv + (
765 offset - mean * inv if offset is not None else -mean * inv)
766
767
768 def batch_norm_with_global_normalization(t,
769 m,
770 v,
771 beta,
772 gamma,
773 variance_epsilon,
774 scale_after_normalization,
775 name=None):
776 """Batch normalization.
777
778 This op is deprecated. See `tf.nn.batch_normalization`.
779
780 Args:
781 t: A 4D input Tensor.
782 m: A 1D mean Tensor with size matching the last dimension of t.
783 This is the first output from tf.nn.moments,
784 or a saved moving average thereof.
785 v: A 1D variance Tensor with size matching the last dimension of t.
786 This is the second output from tf.nn.moments,
787 or a saved moving average thereof.
788 beta: A 1D beta Tensor with size matching the last dimension of t.
789 An offset to be added to the normalized tensor.
790 gamma: A 1D gamma Tensor with size matching the last dimension of t.
791 If "scale_after_normalization" is true, this tensor will be multiplied
792 with the normalized tensor.
793 variance_epsilon: A small float number to avoid dividing by 0.
794 scale_after_normalization: A bool indicating whether the resulted tensor
795 needs to be multiplied with gamma.
796 name: A name for this operation (optional).
797
798 Returns:
799 A batch-normalized `t`.
800 """
801 return batch_normalization(t, m, v, beta, gamma if scale_after_normalization
802 else None, variance_epsilon, name)
803
804
805 def _sum_rows(x):
806 """Returns a vector summing up each row of the matrix x."""
807 # _sum_rows(x) is equivalent to math_ops.reduce_sum(x, 1) when x is
808 # a matrix. The gradient of _sum_rows(x) is more efficient than
809 # reduce_sum(x, 1)'s gradient in today's implementation. Therefore,
810 # we use _sum_rows(x) in the nce_loss() computation since the loss
811 # is mostly used for training.
812 cols = array_ops.shape(x)[1]
813 ones_shape = array_ops.pack([cols, 1])
814 ones = array_ops.ones(ones_shape, x.dtype)
815 return array_ops.reshape(math_ops.matmul(x, ones), [-1])
816
817
818 def _compute_sampled_logits(weights, biases, inputs, labels, num_sampled,
819 num_classes, num_true=1,
820 sampled_values=None,
821 subtract_log_q=True,
822 remove_accidental_hits=False,
823 partition_strategy="mod",
824 name=None):
825 """Helper function for nce_loss and sampled_softmax_loss functions.
826
827 Computes sampled output training logits and labels suitable for implementing
828 e.g. noise-contrastive estimation (see nce_loss) or sampled softmax (see
829 sampled_softmax_loss).
830
831 Note: In the case where num_true > 1, we assign to each target class
832 the target probability 1 / num_true so that the target probabilities
833 sum to 1 per-example.
834
835 Args:
836 weights: A `Tensor` of shape `[num_classes, dim]`, or a list of `Tensor`
837 objects whose concatenation along dimension 0 has shape
838 `[num_classes, dim]`. The (possibly-partitioned) class embeddings.
839 biases: A `Tensor` of shape `[num_classes]`. The class biases.
840 inputs: A `Tensor` of shape `[batch_size, dim]`. The forward
841 activations of the input network.
842 labels: A `Tensor` of type `int64` and shape `[batch_size,
843 num_true]`. The target classes. Note that this format differs from
844 the `labels` argument of `nn.softmax_cross_entropy_with_logits`.
845 num_sampled: An `int`. The number of classes to randomly sample per batch.
846 num_classes: An `int`. The number of possible classes.
847 num_true: An `int`. The number of target classes per training example.
848 sampled_values: a tuple of (`sampled_candidates`, `true_expected_count`,
849 `sampled_expected_count`) returned by a `*_candidate_sampler` function.
850 (if None, we default to `log_uniform_candidate_sampler`)
851 subtract_log_q: A `bool`. whether to subtract the log expected count of
852 the labels in the sample to get the logits of the true labels.
853 Default is True. Turn off for Negative Sampling.
854 remove_accidental_hits: A `bool`. whether to remove "accidental hits"
855 where a sampled class equals one of the target classes. Default is
856 False.
857 partition_strategy: A string specifying the partitioning strategy, relevant
858 if `len(weights) > 1`. Currently `"div"` and `"mod"` are supported.
859 Default is `"mod"`. See `tf.nn.embedding_lookup` for more details.
860 name: A name for the operation (optional).
861 Returns:
862 out_logits, out_labels: `Tensor` objects each with shape
863 `[batch_size, num_true + num_sampled]`, for passing to either
864 `nn.sigmoid_cross_entropy_with_logits` (NCE) or
865 `nn.softmax_cross_entropy_with_logits` (sampled softmax).
866 """
867
868 if not isinstance(weights, list):
869 weights = [weights]
870
871 with ops.op_scope(
872 weights + [biases, inputs, labels], name, "compute_sampled_logits"):
873 if labels.dtype != dtypes.int64:
874 labels = math_ops.cast(labels, dtypes.int64)
875 labels_flat = array_ops.reshape(labels, [-1])
876
877 # Sample the negative labels.
878 # sampled shape: [num_sampled] tensor
879 # true_expected_count shape = [batch_size, 1] tensor
880 # sampled_expected_count shape = [num_sampled] tensor
881 if sampled_values is None:
882 sampled_values = candidate_sampling_ops.log_uniform_candidate_sampler(
883 true_classes=labels,
884 num_true=num_true,
885 num_sampled=num_sampled,
886 unique=True,
887 range_max=num_classes)
888 # NOTE: pylint cannot tell that 'sampled_values' is a sequence
889 # pylint: disable=unpacking-non-sequence
890 sampled, true_expected_count, sampled_expected_count = sampled_values
891 # pylint: enable=unpacking-non-sequence
892
893 # labels_flat is a [batch_size * num_true] tensor
894 # sampled is a [num_sampled] int tensor
895 all_ids = array_ops.concat(0, [labels_flat, sampled])
896
897 # weights shape is [num_classes, dim]
898 all_w = embedding_ops.embedding_lookup(
899 weights, all_ids, partition_strategy=partition_strategy)
900 all_b = embedding_ops.embedding_lookup(biases, all_ids)
901 # true_w shape is [batch_size * num_true, dim]
902 # true_b is a [batch_size * num_true] tensor
903 true_w = array_ops.slice(
904 all_w, [0, 0], array_ops.pack([array_ops.shape(labels_flat)[0], -1]))
905 true_b = array_ops.slice(all_b, [0], array_ops.shape(labels_flat))
906
907 # inputs shape is [batch_size, dim]
908 # true_w shape is [batch_size * num_true, dim]
909 # row_wise_dots is [batch_size, num_true, dim]
910 dim = array_ops.shape(true_w)[1:2]
911 new_true_w_shape = array_ops.concat(0, [[-1, num_true], dim])
912 row_wise_dots = math_ops.mul(
913 array_ops.expand_dims(inputs, 1),
914 array_ops.reshape(true_w, new_true_w_shape))
915 # We want the row-wise dot plus biases which yields a
916 # [batch_size, num_true] tensor of true_logits.
917 dots_as_matrix = array_ops.reshape(row_wise_dots,
918 array_ops.concat(0, [[-1], dim]))
919 true_logits = array_ops.reshape(_sum_rows(dots_as_matrix), [-1, num_true])
920 true_b = array_ops.reshape(true_b, [-1, num_true])
921 true_logits += true_b
922
923 # Lookup weights and biases for sampled labels.
924 # sampled_w shape is [num_sampled, dim]
925 # sampled_b is a [num_sampled] float tensor
926 sampled_w = array_ops.slice(
927 all_w, array_ops.pack([array_ops.shape(labels_flat)[0], 0]), [-1, -1])
928 sampled_b = array_ops.slice(all_b, array_ops.shape(labels_flat), [-1])
929
930 # inputs has shape [batch_size, dim]
931 # sampled_w has shape [num_sampled, dim]
932 # sampled_b has shape [num_sampled]
933 # Apply X*W'+B, which yields [batch_size, num_sampled]
934 sampled_logits = math_ops.matmul(inputs,
935 sampled_w,
936 transpose_b=True) + sampled_b
937
938 if remove_accidental_hits:
939 acc_hits = candidate_sampling_ops.compute_accidental_hits(
940 labels, sampled, num_true=num_true)
941 acc_indices, acc_ids, acc_weights = acc_hits
942
943 # This is how SparseToDense expects the indices.
944 acc_indices_2d = array_ops.reshape(acc_indices, [-1, 1])
945 acc_ids_2d_int32 = array_ops.reshape(math_ops.cast(
946 acc_ids, dtypes.int32), [-1, 1])
947 sparse_indices = array_ops.concat(
948 1, [acc_indices_2d, acc_ids_2d_int32], "sparse_indices")
949 # Create sampled_logits_shape = [batch_size, num_sampled]
950 sampled_logits_shape = array_ops.concat(
951 0,
952 [array_ops.shape(labels)[:1], array_ops.expand_dims(num_sampled, 0)])
953 if sampled_logits.dtype != acc_weights.dtype:
954 acc_weights = math_ops.cast(acc_weights, sampled_logits.dtype)
955 sampled_logits += sparse_ops.sparse_to_dense(
956 sparse_indices, sampled_logits_shape, acc_weights,
957 default_value=0.0, validate_indices=False)
958
959 if subtract_log_q:
960 # Subtract log of Q(l), prior probability that l appears in sampled.
961 true_logits -= math_ops.log(true_expected_count)
962 sampled_logits -= math_ops.log(sampled_expected_count)
963
964 # Construct output logits and labels. The true labels/logits start at col 0.
965 out_logits = array_ops.concat(1, [true_logits, sampled_logits])
966 # true_logits is a float tensor, ones_like(true_logits) is a float tensor
967 # of ones. We then divide by num_true to ensure the per-example labels sum
968 # to 1.0, i.e. form a proper probability distribution.
969 out_labels = array_ops.concat(
970 1, [array_ops.ones_like(true_logits) / num_true,
971 array_ops.zeros_like(sampled_logits)])
972
973 return out_logits, out_labels
974
975
976 def nce_loss(weights, biases, inputs, labels, num_sampled, num_classes,
977 num_true=1,
978 sampled_values=None,
979 remove_accidental_hits=False,
980 partition_strategy="mod",
981 name="nce_loss"):
982 """Computes and returns the noise-contrastive estimation training loss.
983
984 See [Noise-contrastive estimation: A new estimation principle for
985 unnormalized statistical models]
986 (http://www.jmlr.org/proceedings/papers/v9/gutmann10a/gutmann10a.pdf).
987 Also see our [Candidate Sampling Algorithms Reference]
988 (../../extras/candidate_sampling.pdf)
989
990 Note: In the case where `num_true` > 1, we assign to each target class
991 the target probability 1 / `num_true` so that the target probabilities
992 sum to 1 per-example.
993
994 Note: It would be useful to allow a variable number of target classes per
995 example. We hope to provide this functionality in a future release.
996 For now, if you have a variable number of target classes, you can pad them
997 out to a constant number by either repeating them or by padding
998 with an otherwise unused class.
999
1000 Args:
1001 weights: A `Tensor` of shape `[num_classes, dim]`, or a list of `Tensor`
1002 objects whose concatenation along dimension 0 has shape
1003 [num_classes, dim]. The (possibly-partitioned) class embeddings.
1004 biases: A `Tensor` of shape `[num_classes]`. The class biases.
1005 inputs: A `Tensor` of shape `[batch_size, dim]`. The forward
1006 activations of the input network.
1007 labels: A `Tensor` of type `int64` and shape `[batch_size,
1008 num_true]`. The target classes.
1009 num_sampled: An `int`. The number of classes to randomly sample per batch.
1010 num_classes: An `int`. The number of possible classes.
1011 num_true: An `int`. The number of target classes per training example.
1012 sampled_values: a tuple of (`sampled_candidates`, `true_expected_count`,
1013 `sampled_expected_count`) returned by a `*_candidate_sampler` function.
1014 (if None, we default to `log_uniform_candidate_sampler`)
1015 remove_accidental_hits: A `bool`. Whether to remove "accidental hits"
1016 where a sampled class equals one of the target classes. If set to
1017 `True`, this is a "Sampled Logistic" loss instead of NCE, and we are
1018 learning to generate log-odds instead of log probabilities. See
1019 our [Candidate Sampling Algorithms Reference]
1020 (../../extras/candidate_sampling.pdf).
1021 Default is False.
1022 partition_strategy: A string specifying the partitioning strategy, relevant
1023 if `len(weights) > 1`. Currently `"div"` and `"mod"` are supported.
1024 Default is `"mod"`. See `tf.nn.embedding_lookup` for more details.
1025 name: A name for the operation (optional).
1026
1027 Returns:
1028 A `batch_size` 1-D tensor of per-example NCE losses.
1029 """
1030 logits, labels = _compute_sampled_logits(
1031 weights, biases, inputs, labels, num_sampled, num_classes,
1032 num_true=num_true,
1033 sampled_values=sampled_values,
1034 subtract_log_q=True,
1035 remove_accidental_hits=remove_accidental_hits,
1036 partition_strategy=partition_strategy,
1037 name=name)
1038 sampled_losses = sigmoid_cross_entropy_with_logits(logits,
1039 labels,
1040 name="sampled_losses")
1041 # sampled_losses is batch_size x {true_loss, sampled_losses...}
1042 # We sum out true and sampled losses.
1043 return _sum_rows(sampled_losses)
1044
1045
1046 def sampled_softmax_loss(weights, biases, inputs, labels, num_sampled,
1047 num_classes, num_true=1,
1048 sampled_values=None,
1049 remove_accidental_hits=True,
1050 partition_strategy="mod",
1051 name="sampled_softmax_loss"):
1052 """Computes and returns the sampled softmax training loss.
1053
1054 This is a faster way to train a softmax classifier over a huge number of
1055 classes.
1056
1057 This operation is for training only. It is generally an underestimate of
1058 the full softmax loss.
1059
1060 At inference time, you can compute full softmax probabilities with the
1061 expression `tf.nn.softmax(tf.matmul(inputs, weights) + biases)`.
1062
1063 See our [Candidate Sampling Algorithms Reference]
1064 (../../extras/candidate_sampling.pdf)
1065
1066 Also see Section 3 of [Jean et al., 2014](http://arxiv.org/abs/1412.2007)
1067 ([pdf](http://arxiv.org/pdf/1412.2007.pdf)) for the math.
1068
1069 Args:
1070 weights: A `Tensor` of shape `[num_classes, dim]`, or a list of `Tensor`
1071 objects whose concatenation along dimension 0 has shape
1072 [num_classes, dim]. The (possibly-sharded) class embeddings.
1073 biases: A `Tensor` of shape `[num_classes]`. The class biases.
1074 inputs: A `Tensor` of shape `[batch_size, dim]`. The forward
1075 activations of the input network.
1076 labels: A `Tensor` of type `int64` and shape `[batch_size,
1077 num_true]`. The target classes. Note that this format differs from
1078 the `labels` argument of `nn.softmax_cross_entropy_with_logits`.
1079 num_sampled: An `int`. The number of classes to randomly sample per batch.
1080 num_classes: An `int`. The number of possible classes.
1081 num_true: An `int`. The number of target classes per training example.
1082 sampled_values: a tuple of (`sampled_candidates`, `true_expected_count`,
1083 `sampled_expected_count`) returned by a `*_candidate_sampler` function.
1084 (if None, we default to `log_uniform_candidate_sampler`)
1085 remove_accidental_hits: A `bool`. whether to remove "accidental hits"
1086 where a sampled class equals one of the target classes. Default is
1087 True.
1088 partition_strategy: A string specifying the partitioning strategy, relevant
1089 if `len(weights) > 1`. Currently `"div"` and `"mod"` are supported.
1090 Default is `"mod"`. See `tf.nn.embedding_lookup` for more details.
1091 name: A name for the operation (optional).
1092
1093 Returns:
1094 A `batch_size` 1-D tensor of per-example sampled softmax losses.
logits, labels = _compute_sampled_logits(
weights, biases, inputs, labels, num_sampled, num_classes,
num_true=num_true,
sampled_values=sampled_values,
subtract_log_q=True,
remove_accidental_hits=remove_accidental_hits,
partition_strategy=partition_strategy,
name=name)
sampled_losses = nn_ops.softmax_cross_entropy_with_logits(logits, labels)
# sampled_losses is a [batch_size] tensor.
return sampled_losses
# TODO(cwhipkey): sigmoid and tanh should not be exposed from tf.nn.
__all__ = make_all(__name__)
__all__.append("zero_fraction") # documented in training.py
# Modules whitelisted for reference through tf.nn.
# TODO(cwhipkey): migrate callers to use the submodule directly.
__all__.extend(["nn_ops", "rnn_cell", "seq2seq"])
# Symbols whitelisted for export without documentation.
# TODO(cwhipkey): review these and move to contrib or expose through
# documentation.
__all__.extend([
"all_candidate_sampler",
"batch_norm_with_global_normalization",
"batch_normalization",
"bidirectional_rnn",
"conv2d_backprop_filter",
"conv2d_backprop_input",
"depthwise_conv2d_native",
"dynamic_rnn",
"lrn",
"relu_layer",
"rnn",
"state_saving_rnn",
"xw_plus_b",
])