tensorflow-datasets

I'm trying to build a model that gives reddit_score = f('subreddit','comment') Mainly this is as an example i can then build on for a work project. My code is here . My problem is that i see that canned estimators e.g. DNNLinearCombinedRegressor must have feature_columns that are part of FeatureColumn class. I have my vocab file and know that if i was to just limit to the first word of a comment i could just do something like tf.feature_column.categorical_column_with_vocabulary_file( key='comment', vocabulary_file='{}/vocab.csv'.format(INPUT_DIR) ) But if i'm passing in say first 10 words from

I am trying train an estimator with a generator, but I want to feed this estimator with a package of samples for each iteration. I show the code: def _generator(): for i in range(100): feats = np.random.rand(4,2) labels = np.random.rand(4,1) yield feats, labels def input_func_gen(): shapes = ((4,2),(4,1)) dataset = tf.data.Dataset.from_generator(generator=_generator, output_types=(tf.float32, tf.float32), output_shapes=shapes) dataset = dataset.batch(4) # dataset = dataset.repeat(20) iterator = dataset.make_one_shot_iterator() features_tensors, labels = iterator.get_next() features = {'x':

All the official google tutorials use the one shot iterator for all the estimator api implementation, i couldnt find any documentation on how to use tf.data's initializable iterator and reinitializable interator instead of one shot iterator. Can someone kindly show me how to switch between train_data and test_data using tf.data's initializable iterator and reinitializable interator. We need to run a session to use feed dict and switch the dataset in the initializable iterator, its a low level api and its confusing how to use it part of estimator api architecture PS : I did find that google

Is there an easy way to get the entire set of elements in a tf.data.Dataset ? i.e. I want to set batch size of the Dataset to be the size of my dataset without specifically passing it the number of elements. This would be useful for validation dataset where I want to measure accuracy on the entire dataset in one go. I'm surprised there isn't a method to get the size of a tf.data.Dataset In short, there is not a good way to get the size/length; tf.data.Dataset is built for pipelines of data, so has an iterator structure (in my understanding and according to my read of the Dataset ops code .

I have a function which produces feature and target tensors. E.g. x,t = myfunc() ##x,t tensors How can I integrate this with TensorFlow's dataset API for continuous training? Ideally I would like to use dataset to set things like batch, transformations. Edit for clarification: The problem being I would like to not just put x and t in my graph but make a dataset from them so that I can use the same dataset processing that I have implemented for (normal) finite datasets I can load into memory and feed into the same graph using an initializable iterator. Assuming x and t are tf.Tensor objects,

I have the following input_fn. def input_fn(filenames, batch_size): # Create a dataset containing the text lines. dataset = tf.data.TextLineDataset(filenames).skip(1) # Parse each line. dataset = dataset.map(_parse_line) # Shuffle, repeat, and batch the examples. dataset = dataset.shuffle(10000).repeat().batch(batch_size) # Return the dataset. return dataset It works great if filenames=['file1.csv'] or filenames=['file2.csv'] . It gives me an error if filenames=['file1.csv', 'file2.csv'] . In Tensorflow documentation it says filenames is a tf.string tensor containing one or more filenames. How

I am using tf.estimator.Estimator for developing my model, I wrote a model_fn and trained 50,000 iterations, now I want to make a small change in my model_fn , for example add a new layer. I don't want to start training from scratch, I want to restore all the old variables from the 50,000 checkpoint, and continue training from this point. When I try to do so I get a NotFoundError How can this be done with tf.estimator.Estimator ? TL;DR The easiest way to load variables from a previous checkpoint is to use the function tf.train.init_from_checkpoint() . Just one call to this function inside the