Hyperparameter tune for Tensorflow

Question

I am searching for a hyperparameter tune package for code written directly in Tensorflow (not Keras or Tflearn). Could you make some suggestion?

Answer 1

You can try out Ray Tune, a simple library for scaling hyperparameter search. I mainly use it for Tensorflow model training, but it's agnostic to the framework - works seamlessly with PyTorch, Keras, etc. Here's the docs page - ray.readthedocs.io/en/latest/tune.html

You can use it to run distributed versions of state-of-the-art algorithms such as HyperBand or Bayesian Optimization in about 10 lines of code.

As an example to run 4 parallel evaluations at a time:

import ray
import ray.tune as tune
from ray.tune.hyperband import HyperBandScheduler


def train_model(config, reporter):  # add the reporter parameter
    model = build_tf_model(config["alpha"], config["beta"])
    loss = some_loss_function(model)
    optimizer = tf.AdamOptimizer(loss)

    for i in range(20):
        optimizer.step()
        stats = get_statistics()
        reporter(timesteps_total=i, 
                 mean_accuracy=stats["accuracy"])

ray.init(num_cpus=4)
tune.run(train_model,
    name="my_experiment",
    stop={"mean_accuracy": 100}, 
    config={ 
        "alpha": tune.grid_search([0.2, 0.4, 0.6]), 
        "beta": tune.grid_search([1, 2]) 
    },
    scheduler=HyperBandScheduler(reward_attr="mean_accuracy"))

You also don't need to change your code if you want to run this script on a cluster.

Disclaimer: I work on this project - let me know if you have any feedback!