Hyperparameter tune for Tensorflow

Question

I am searching for a hyperparameter tune package for code written directly in Tensorflow (not Keras or Tflearn). Could you make some suggestion?

Answer 1

Usually you don't need to have your hyperparameter optimisation logic coupled with the optimised model (unless your hyperparemeter optimisation logic is specific to the kind of model that you are training, in which case you would need to tell us a bit more). There are several tools and packages available for the task. Here is a good paper on the topic, and here is a more practical blog post with examples.

• hyperopt implements random search and tree of parzen estimators optimization.
• Scikit-Optimize implements a few others, including Gaussian process Bayesian optimization.
• SigOpt is a convenient service (paid, although with a free tier and extra allowance for students and researchers) for hyperparameter optimization. It is based upon Yelp's MOE, which is open source (although the published version doesn't seem to update much) and can, in theory, be used on its own, although it would take some additional effort.
• Spearmint is a commonly referred package too, also open source but not free for commercial purposes (although you can fall back to a less restrictive older version). It looks good, but not very active, and the available version is not compatible with Python 3 (even though pull requests have been submitted to fix that).
• BayesOpt seems to be the golden standard in Bayesian optimization, but it's mainly C++, and the Python interface doesn't look very documented.

Out of these, I have only really (that is, with a real problem) used hyperopt with TensorFlow, and it didn't took too much effort. The API is a bit weird at some points and the documentation is not terribly thorough, but it does work and seems to be under active development, with more optimization algorithms and adaptations (e.g. specifically for neural networks) possibly coming. However, as suggested in the previously linked blog post, Scikit-Optimize is probably as good, and SigOpt looks quite easy to use if it fits you.