Where is the bazel rule generating the `gen_io_ops.py` file when building TensorFlow from sources?

问题

I'm trying to determine how the gen_io_ops module is generated by bazel when building TensorFlow from source.

In tensorflow/python/ops/io_ops.py, there is this piece of code:

from tensorflow.python.ops.gen_io_ops
[...]

# used in the TextLineReader initialization
rr = gen_io_ops._text_line_reader_v2(...)

referring to the bazel-genfiles/tensorflow/python/ops/gen_io_ops.py module (and generated by bazel when building TensorFlow).
The _text_line_reader_v2 is a wrapper of the TextLineReaderV2 defined in tensorflow/tensorflow/core/kernels/text_line_reader_op.cc.

As far as I understand, the build step are the followings:

1) The kernel library for the text_line_reader_op is built in tensorflow/tensorflow/core/kernels/BUILD

tf_kernel_library(
        name = "text_line_reader_op",
        prefix = "text_line_reader_op",
        deps = IO_DEPS,)

where tf_kernel_library basically looks for text_line_reader_op.c file and build it.

2) The :text_line_reader_op kernel library is then used as a dependency by the io library defined in the same file:

cc_library(
    name = "io",
    deps = [       
        ":text_line_reader_op", ...
    ],
)

I suppose the io library now contains the definition of the TextLineReaderV2kernel.

From what I get from this answer, there should be a third step where the io library is used to generate the python wrappers that are in the bazel-genfiles/tensorflow/python/ops/gen_io_ops.py module. This file generation can be done by the tf_op_gen_wrapper_py rule in Basel or by thetf.load_op_library() method, but none of them seem involved.

Does someone know where this third step is defined in the build process?

回答1:

I finally got it.

There is indeed a call to tf_op_gen_wrapper_py but it's hidden in a call to tf_gen_op_wrapper_private_py:

def tf_gen_op_wrapper_private_py(name, out=None, deps=[],
                                 require_shape_functions=True,
                                 visibility=[]):
  if not name.endswith("_gen"):
    fail("name must end in _gen")
  [...]
  bare_op_name = name[:-4]
  tf_gen_op_wrapper_py(name=bare_op_name, ...

So the steps are the following.

In tensorflow/tensorflow/python/BUILD, there is this rule

tf_gen_op_wrapper_private_py(
    name = "io_ops_gen",
    [...]
)

And so, in this rule the _gen suffix will be removed (in tf_gen_op_wrapper_private_py) and a gen_ prefix will be added in tf_gen_op_wrapper_py and therefore the gen_io_ops.py module will be generated by this rule.

来源：https://stackoverflow.com/questions/45457061/where-is-the-bazel-rule-generating-the-gen-io-ops-py-file-when-building-tensor

标签

python

c++

tensorflow

bazel