Keras: change learning rate

问题

I'm trying to change the learning rate of my model after it has been trained with a different learning rate.

I read here, here, here and some other places i can't even find anymore.

I tried:

model.optimizer.learning_rate.set_value(0.1)
model.optimizer.lr = 0.1
model.optimizer.learning_rate = 0.1
K.set_value(model.optimizer.learning_rate, 0.1)
K.set_value(model.optimizer.lr, 0.1)
model.optimizer.lr.assign(0.1)

... but none of them worked! I don't understand how there could be such confusion around such a simple thing. Am I missing something?

EDIT: Working example

Here is a working example of what I'd like to do:

from keras.models import Sequential
from keras.layers import Dense
import keras
import numpy as np

model = Sequential()

model.add(Dense(1, input_shape=(10,)))

optimizer = keras.optimizers.Adam(lr=0.01)
model.compile(loss='mse',
              optimizer=optimizer)

model.fit(np.random.randn(50,10), np.random.randn(50), epochs=50)

# Change learning rate to 0.001 and train for 50 more epochs

model.fit(np.random.randn(50,10), np.random.randn(50), initial_epoch=50, epochs=50)

回答1:

You can change the learning rate as follows:

from keras import backend as K
K.set_value(model.optimizer.learning_rate, 0.001)

Included into your complete example it looks as follows:

from keras.models import Sequential
from keras.layers import Dense
from keras import backend as K
import keras
import numpy as np

model = Sequential()

model.add(Dense(1, input_shape=(10,)))

optimizer = keras.optimizers.Adam(lr=0.01)
model.compile(loss='mse', optimizer=optimizer)

print("Learning rate before first fit:", model.optimizer.learning_rate.numpy())

model.fit(np.random.randn(50,10), np.random.randn(50), epochs=50, verbose=0)

# Change learning rate to 0.001 and train for 50 more epochs
K.set_value(model.optimizer.learning_rate, 0.001)
print("Learning rate before second fit:", model.optimizer.learning_rate.numpy())

model.fit(np.random.randn(50,10), 
          np.random.randn(50), 
          initial_epoch=50, 
          epochs=50,
          verbose=0)

I've just tested this with keras 2.3.1. Not sure why the approach didn't seem to work for you.

回答2:

There is another way, you have to find the variable that holds the learning rate and assign it another value.

optimizer = tf.keras.optimizers.Adam(0.001)
optimizer.learning_rate.assign(0.01)
print(optimizer.learning_rate)

output:

<tf.Variable 'learning_rate:0' shape=() dtype=float32, numpy=0.01>

回答3:

You can change lr during training with

from keras.callbacks import LearningRateScheduler

# This is a sample of a scheduler I used in the past
def lr_scheduler(epoch, lr):
    decay_rate = 0.85
    decay_step = 1
    if epoch % decay_step == 0 and epoch:
        return lr * pow(decay_rate, np.floor(epoch / decay_step))
    return lr

Apply scheduler to your model

callbacks = [LearningRateScheduler(lr_scheduler, verbose=1)]

model = build_model(pretrained_model=ka.InceptionV3, input_shape=(224, 224, 3))
history = model.fit(train, callbacks=callbacks, epochs=EPOCHS, verbose=1)

回答4:

You should define it in the compile function :

optimizer = keras.optimizers.Adam(lr=0.01)
model.compile(loss='mse',
              optimizer=optimizer,
              metrics=['categorical_accuracy'])

Looking at your comment, if you want to change the learning rate after the beginning you need to use a scheduler : link

Edit with your code and scheduler:

from keras.models import Sequential
from keras.layers import Dense
import keras
import numpy as np

def lr_scheduler(epoch, lr):
    if epoch > 50:
        lr = 0.001
        return lr
    return lr

model = Sequential()

model.add(Dense(1, input_shape=(10,)))

optimizer = keras.optimizers.Adam(lr=0.01)
model.compile(loss='mse',
              optimizer=optimizer)

callbacks = [keras.callbacks.LearningRateScheduler(lr_scheduler, verbose=1)]

model.fit(np.random.randn(50,10), np.random.randn(50), epochs=100, callbacks=callbacks)

回答5:

Suppose that you use Adam optimizer in keras, you'd want to define your optimizer before you compile your model with it.

For example, you can define

myadam = keras.optimizers.Adam(learning_rate=0.1)

Then, you compile your model with this optimizer.

I case you want to change your optimizer (with different type of optimizer or with different learning rate), you can define a new optimizer and compile your existing model with the new optimizer.

Hope this helps!

来源：https://stackoverflow.com/questions/59737875/keras-change-learning-rate

标签

python

tensorflow

keras