第十五章开发字符级神经语言模型

语言模型可以根据序列中前面的特定单词预测序列中的下一个单词。你可以使用神经网络开发字符级的语言模型，基于字符的语言模型的好处在于处理任何单词，标点和其他文档结构时仅需要很小的词汇量，而且更加灵活，但这是以较大模型和训练缓慢为代价的，然而在神经语言模型领域，基于字符的模型为语言建模提供通用灵活和强大的方法，尽管参数大和训练缓慢，但是在其强大的泛华能力面前，这些缺点也是可以忍受的，接下来你将了解如何开发字符级的神经网路模型：

准备文本
使用开发LSTM训练模型
使用训练好的模型来生成文本

15.1 概述

本教程分为以下几个部分：

唱一首歌
准备数据
训练模型
生成文本

15.2 唱一首歌

我们选用Sing a song of sixpence 来开发字符级语言模型

Sing a song of sixpence,
A pocket full of rye;
Four and twenty blackbirds,
Baked in a pie.
When the pie was opened,
The birds began to sing;
Was not that a dainty dish,
To set before the king ?
The king was in his counting-house,
Counting out his money;
The queen was in the parlour,
Eating bread and honey.
The maid was in the garden,
Hanging out the clothes,
There came a little blackbird,
And snapped off her nose.

将其保存为Song.txt文件。

15.3 准备数据

第一步是准备文本数据，我们将从定义语言模型类型开始

15.3.1 模型设计

该模型是使用文本书籍来训练的，对于字符级的语言模型，输入和输出序列必须是字符流。用做输入的字符数，同时也是为了引出第一预测字符而需要提供给模型的字符数目，生成第一个字符后，可将其附加到输入序列并用作模型的输入以生成下一个字符。
较长的序列为模型训练和学习提供了更多的上下文，可以更好的预测接下来要输出的字符，但是需要更长时间来训练，并在生成文本时对起点文本也长。我们规定模型使用长度为10个字符的序列来给模型训练和学习提供上下文信息，没有很多文本，10个字符仅能表示很少的单词。我们现在可以将原始文本转换为我们的模型可以学习的形式：输入和输出字符序列

15.3.2 加载文字

我们首先将文件加载到内存中，然后我们就可以使用。下面定义一个load_doc的函数，它将加载一个指定的文件，并返回加载的文本。

def load_doc(filename):
    with open(filename,'r',encoding='utf-8')as f:
        text = f.read()
    return text
song_text = load_doc('song.txt')
print(song_text)

15.3.3 清洗文本

接下来我们需要清洗加载的文本，在这个例子中我们不会做太多清洗，如果想了解其他复杂的清洗，请查看我前面的文章。这里我们删除所有换行符，最后得到一个由空格分割的长字符序列。

tokens = song_text.split()
song_text = ''.join(tokens)
print(song_text)

15.3.4 创建序列

我么你现在有了一个很长的字符列表，我们可以创建用于训练模型的输入输出序列。每个输入序列将是10个字符，带有一个输入字符，每个序列长11个字符，我们通过枚举文本中的字符来创建序列，从索引10处的第11个字符开始。

length = 10
sequences = list()
for i in range(length,len(song_text)):
    seq = song_text[i-length,i+1]
    sequences.append(seq)
print('Total Sequences :%d'%len(sequences))

15.3.5 保存序列

最后我们可以将准备好的数据保存但文件中，以便我们可以在开发模型时加载它，定义save_doc函数将字符串列表保存到文件中，没行一个，函数参数为字符串列表和文件名。

def save_doc(lines,filename):
    data = '\n'.join(lines)
    with open(filename,'w',encoding='utf-8')as f:
        f.write(data)
out_filename = 'char_sequences.txt'
save_doc(sequences,out_filename)

完整的：

def load_doc(filename):
    with open(filename,'r',encoding='utf-8')as f:
        text = f.read()
    return text
song_text = load_doc('song.txt')
# print(song_text)
tokens = song_text.split()
song_text = ' '.join(tokens)
# print(song_text)
length = 10
sequences = list()
for i in range(length,len(song_text)):
    seq = song_text[i-length:i+1]
    sequences.append(seq)
print('Total Sequences :%d'%len(sequences))
def save_doc(lines,filename):
    data = '\n'.join(lines)
    with open(filename,'w',encoding='utf-8')as f:
        f.write(data)
out_filename = 'char_sequences.txt'
save_doc(sequences,out_filename)

得到的char_sequences.txt文件打开如图所示：
在这里插入图片描述

15.4 训练模型

在本节中，我们将为准备好的序列数据开发字符级的神经网络模型，该模型将读取编码字符并预测序列中的下一个字符。这里将使用长短期记忆（LSTM）递归神经网络来从序列学习上下文来进行预测。

15.4.1 加载数据

第一步是从char_sequences.txt加载准备好的字符序列数据，使用上一节中定义的load_doc函数加载，以换行符为分隔符将文本分解成序列列表。

def load_doc(filename):
    with open(filename,'r',encoding='utf-8')as f:
        text = f.read()
    return text
in_filename = 'char_sequences.txt'
text = load_doc(in_filename)
lines = text.split('\n')
print(text)

15.4.2 编码序列

字符序列必须编码为整数，这意味着将为每个字符分配一个特定的整数值，并且每个字符序列将被编码为整数序列，我们可以在原始输入数据中给定一排序列的唯一字符来创建映射。映射是字符值到整数值得字典。

chars = sorted(list(set(text)))
mapping = dict((v,k) for k,v in enumerate(chars))

接下来将字符序列转化成数字序列，完成编码，我们可以一次处理一个字符序列，并使用字段映射查找每个字符的整数值。

for line in lines:
    encoded_seq = [mapping[char] for char in line]
    sequences.append(encoded_seq)

sequences是整数列表。下面我们需要知道词汇量的大小，就是字典映射的大小。

vocab_size = len(mapping)
print('Vocabulary size:',vocab_size)

15.4.3 拆分输入和输出

现在序列已经使用整数编码，我们现在可以将序列分成输入和输出字符序列，并使用简单的数组切片来完成词操作。

import numpy as ny
sequences = ny.array(sequences)
x,y = sequences[::-1],sequences[:-1]

接下来就是对每个字符进行one_hot编码，也就是说，每个字符就变成一个长度为词汇表长度（39）大小的向量，这个向量仅在字典中序号处的位置为1，其他位置均为0，这样做主要是为网络提供了更精准的输入表示，还为网络预测提供了一个明确的目标，训练过程模型给出输出字符的概率分布，并与真实的输入字符进行比较，预测的向量除了字符处的概率为1之外，其他地方均为0，这是预测输出字符的理想形式。我们可以使用Keras API中的to_categorical函数进对输入和输出序列进行热编码。

sequences = [to_categorical(x,num_classes = vocab_size) for x in x]
x = ny.array(x)
y = to_categorical(y,num_classes=vocab_size)

下面我们拟合模型

15.4.4 拟合模型

模型结构如下：一个输入层和一个全连接输出层，输入层输入参数大小为10个时间步长的序列和长度为39的单字符one_hot编码。我们使用输入数据x的参数作为第二维和第三维，而不是指定这些数字。这样如果我们更改序列长度或词汇表大小，我们就不需要更改模型定义。该模型具有单个LSTM隐藏层，具有64个隐藏单元最后是全连接的输出层，该输出层输出一个向量，其中概率分布跨越词汇表中的所有字符。在输出层上使用softmax激活函数以确保输出具有概率分布的属性

def define_model(x):
    model = Sequential()
    model.add(LSTM(79,input_shape=(x.shape[1],x.shape[2])))
    model.add(Dense(vocab_size,activation='softmax'))
    #compile model
    model.compile(
        loss='categorical_crossentropy',
        optimizer='adam',
        metrics=['accuracy']
    )
    model.summary()
    return model

该模型学习的是多分类问题，因此我们使用针对此类问题的分类对数损失函数categorical_crossentropy，梯度下降使用adam算法，并且在每次批量更新结束时记录模型的准确性。该模型训练100个epoch（迭代次数）

15.4.5 保存模型

在模型拟合后，我们将其保存到文件中供以后使用。Keras的API中提供了save函数，我们可以使用save函数将模型保存到单个文件中，包括权重和拓扑信息。

model.save（'modle.h5'）

我们还保存了从字符到整数的映射，在使用模型和解码模型的任何输出时，我们需要对任何输入进行编码

dump(mapping,open('mapping.pkl','wb'))

15.4.6 完整的示例

将所有的结合在一起：

import numpy as ny
from pickle import dump
from tensorflow.python.keras.utils import to_categorical
from tensorflow.python.keras.models import Sequential
from tensorflow.python.keras.layers import LSTM,Dense
def load_doc(filename):
    with open(filename,'r',encoding='utf-8')as f:
        text = f.read()
    return text
def define_model(x):
    model = Sequential()
    model.add(LSTM(79,input_shape=(x.shape[1],x.shape[2])))
    model.add(Dense(vocab_size,activation='softmax'))
    #compile model
    model.compile(
        loss='categorical_crossentropy',
        optimizer='adam',
        metrics=['accuracy']
    )
    model.summary()
    return model
in_filename = 'char_sequences.txt'
text = load_doc(in_filename)
lines = text.split('\n')
# print(text)
chars = sorted(list(set(text)))
# print(chars)
mapping = dict((v,k) for k,v in enumerate(chars))
# print(mapping)
sequences = list()
for line in lines:
    encoded_seq = [mapping[char] for char in line]
    sequences.append(encoded_seq)


vocab_size = len(mapping)
print('Vocabulary size:',vocab_size)
# print(sequences)
sequences = ny.array(sequences)
# print(sequences)
# print('*'*100)
x_arr,y_arr = sequences[:,:-1],sequences[:,-1]
# print(x)
# print('*'*100)
# print(y)
sequences = [to_categorical(x,num_classes = vocab_size) for x in x_arr]
x_arr = ny.array(sequences)
y_arr = to_categorical(y_arr,num_classes=vocab_size)
model = define_model(x_arr)
history = model.fit(x_arr,y_arr,epochs=100)
model.save('15-4-model.h5')
import matplotlib.pyplot as plt
acc = history.history['acc']
loss = history.history['loss']
epochs = range(1,len(acc)+1)
plt.plot(epochs,acc,label = 'Training acc')
plt.title('Training accuracy')
plt.legend()
plt.figure()
plt.plot(epochs,loss,label = 'Training loss')
plt.title('Training loss')
plt.legend()
plt.show()
dump(mapping,open('mapping.pkl','wb'))

下面是运行结果：

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lstm (LSTM)                  (None, 79)                37604     
_________________________________________________________________
dense (Dense)                (None, 39)                3120      
=================================================================
Total params: 40,724
Trainable params: 40,724
Non-trainable params: 0
_________________________________________________________________
其训练过程：
Epoch 1/100

 32/407 [=>............................] - ETA: 7s - loss: 3.6634 - acc: 0.0625
 96/407 [======>.......................] - ETA: 2s - loss: 3.6586 - acc: 0.0417
192/407 [=============>................] - ETA: 0s - loss: 3.6547 - acc: 0.0417
320/407 [======================>.......] - ETA: 0s - loss: 3.6411 - acc: 0.0875
407/407 [==============================] - 1s 2ms/sample - loss: 3.6315 - acc: 0.1130
Epoch 2/100

 32/407 [=>............................] - ETA: 0s - loss: 3.5847 - acc: 0.1250
128/407 [========>.....................] - ETA: 0s - loss: 3.5663 - acc: 0.1719
256/407 [=================>............] - ETA: 0s - loss: 3.5342 - acc: 0.1914
352/407 [========================>.....] - ETA: 0s - loss: 3.5116 - acc: 0.1761
407/407 [==============================] - 0s 576us/sample - loss: 3.4793 - acc: 0.1867
Epoch 3/100

 32/407 [=>............................] - ETA: 0s - loss: 3.2693 - acc: 0.1875
128/407 [========>.....................] - ETA: 0s - loss: 3.1749 - acc: 0.2109
256/407 [=================>............] - ETA: 0s - loss: 3.1835 - acc: 0.1836
352/407 [========================>.....] - ETA: 0s - loss: 3.1575 - acc: 0.1847
407/407 [==============================] - 0s 576us/sample - loss: 3.1597 - acc: 0.1892
Epoch 4/100

 32/407 [=>............................] - ETA: 0s - loss: 3.2519 - acc: 0.1250
160/407 [==========>...................] - ETA: 0s - loss: 3.0310 - acc: 0.2500
288/407 [====================>.........] - ETA: 0s - loss: 3.0416 - acc: 0.2118
407/407 [==============================] - 0s 576us/sample - loss: 3.0505 - acc: 0.1892
Epoch 5/100

 32/407 [=>............................] - ETA: 0s - loss: 2.6288 - acc: 0.3438
160/407 [==========>...................] - ETA: 0s - loss: 2.9120 - acc: 0.2313
288/407 [====================>.........] - ETA: 0s - loss: 3.0458 - acc: 0.1875
407/407 [==============================] - 0s 576us/sample - loss: 3.0185 - acc: 0.1892
Epoch 6/100

 32/407 [=>............................] - ETA: 0s - loss: 3.1037 - acc: 0.1250
160/407 [==========>...................] - ETA: 0s - loss: 3.1409 - acc: 0.1562
256/407 [=================>............] - ETA: 0s - loss: 3.0192 - acc: 0.1992
384/407 [===========================>..] - ETA: 0s - loss: 2.9978 - acc: 0.1927
407/407 [==============================] - 0s 537us/sample - loss: 3.0017 - acc: 0.1892
Epoch 7/100

 32/407 [=>............................] - ETA: 0s - loss: 2.9401 - acc: 0.1875
128/407 [========>.....................] - ETA: 0s - loss: 2.9617 - acc: 0.2031
256/407 [=================>............] - ETA: 0s - loss: 2.9467 - acc: 0.1914
384/407 [===========================>..] - ETA: 0s - loss: 2.9716 - acc: 0.1849
407/407 [==============================] - 0s 576us/sample - loss: 2.9851 - acc: 0.1892
Epoch 8/100

 32/407 [=>............................] - ETA: 0s - loss: 3.2822 - acc: 0.0938
160/407 [==========>...................] - ETA: 0s - loss: 2.9924 - acc: 0.1750
256/407 [=================>............] - ETA: 0s - loss: 2.9998 - acc: 0.1758
384/407 [===========================>..] - ETA: 0s - loss: 2.9865 - acc: 0.1849
407/407 [==============================] - 0s 576us/sample - loss: 2.9712 - acc: 0.1892
Epoch 9/100

 32/407 [=>............................] - ETA: 0s - loss: 2.8701 - acc: 0.1562
160/407 [==========>...................] - ETA: 0s - loss: 2.8985 - acc: 0.1875
288/407 [====================>.........] - ETA: 0s - loss: 2.9327 - acc: 0.1875
384/407 [===========================>..] - ETA: 0s - loss: 2.9321 - acc: 0.1875
407/407 [==============================] - 0s 576us/sample - loss: 2.9607 - acc: 0.1892
Epoch 10/100

 32/407 [=>............................] - ETA: 0s - loss: 3.1062 - acc: 0.0938
128/407 [========>.....................] - ETA: 0s - loss: 2.9667 - acc: 0.1719
256/407 [=================>............] - ETA: 0s - loss: 2.9750 - acc: 0.1875
352/407 [========================>.....] - ETA: 0s - loss: 2.9784 - acc: 0.1818
407/407 [==============================] - 0s 576us/sample - loss: 2.9485 - acc: 0.1892
Epoch 11/100

 32/407 [=>............................] - ETA: 0s - loss: 2.8007 - acc: 0.2500
128/407 [========>.....................] - ETA: 0s - loss: 2.9680 - acc: 0.1797
256/407 [=================>............] - ETA: 0s - loss: 2.9429 - acc: 0.1914
352/407 [========================>.....] - ETA: 0s - loss: 2.9149 - acc: 0.1932
407/407 [==============================] - 0s 576us/sample - loss: 2.9289 - acc: 0.1892
Epoch 12/100

 32/407 [=>............................] - ETA: 0s - loss: 2.8320 - acc: 0.2500
160/407 [==========>...................] - ETA: 0s - loss: 2.7920 - acc: 0.2062
288/407 [====================>.........] - ETA: 0s - loss: 2.8456 - acc: 0.1944
384/407 [===========================>..] - ETA: 0s - loss: 2.9175 - acc: 0.1849
407/407 [==============================] - 0s 576us/sample - loss: 2.9061 - acc: 0.1892
Epoch 13/100

 32/407 [=>............................] - ETA: 0s - loss: 2.9403 - acc: 0.1250
160/407 [==========>...................] - ETA: 0s - loss: 2.8263 - acc: 0.1688
256/407 [=================>............] - ETA: 0s - loss: 2.8505 - acc: 0.1914
384/407 [===========================>..] - ETA: 0s - loss: 2.8625 - acc: 0.2005
407/407 [==============================] - 0s 537us/sample - loss: 2.8920 - acc: 0.1966
Epoch 14/100

 32/407 [=>............................] - ETA: 0s - loss: 2.8436 - acc: 0.2500
128/407 [========>.....................] - ETA: 0s - loss: 2.9506 - acc: 0.1875
256/407 [=================>............] - ETA: 0s - loss: 2.9013 - acc: 0.1992
352/407 [========================>.....] - ETA: 0s - loss: 2.8591 - acc: 0.2074
407/407 [==============================] - 0s 576us/sample - loss: 2.8695 - acc: 0.1941
Epoch 15/100

 32/407 [=>............................] - ETA: 0s - loss: 2.7811 - acc: 0.1875
160/407 [==========>...................] - ETA: 0s - loss: 2.7738 - acc: 0.2062
256/407 [=================>............] - ETA: 0s - loss: 2.8464 - acc: 0.2031
384/407 [===========================>..] - ETA: 0s - loss: 2.8550 - acc: 0.2109
407/407 [==============================] - 0s 537us/sample - loss: 2.8351 - acc: 0.2113
Epoch 16/100

 32/407 [=>............................] - ETA: 0s - loss: 2.9573 - acc: 0.1875
128/407 [========>.....................] - ETA: 0s - loss: 2.9069 - acc: 0.1953
256/407 [=================>............] - ETA: 0s - loss: 2.8551 - acc: 0.2070
352/407 [========================>.....] - ETA: 0s - loss: 2.7877 - acc: 0.2188
407/407 [==============================] - 0s 576us/sample - loss: 2.7959 - acc: 0.2138
Epoch 17/100

 32/407 [=>............................] - ETA: 0s - loss: 2.7237 - acc: 0.2188
160/407 [==========>...................] - ETA: 0s - loss: 2.6899 - acc: 0.2313
256/407 [=================>............] - ETA: 0s - loss: 2.7394 - acc: 0.2070
384/407 [===========================>..] - ETA: 0s - loss: 2.7634 - acc: 0.2240
407/407 [==============================] - 0s 576us/sample - loss: 2.7644 - acc: 0.2260
Epoch 18/100

 32/407 [=>............................] - ETA: 0s - loss: 2.9299 - acc: 0.1875
160/407 [==========>...................] - ETA: 0s - loss: 2.7457 - acc: 0.2375
288/407 [====================>.........] - ETA: 0s - loss: 2.7509 - acc: 0.2153
384/407 [===========================>..] - ETA: 0s - loss: 2.7332 - acc: 0.2057
407/407 [==============================] - 0s 576us/sample - loss: 2.7284 - acc: 0.2113
Epoch 19/100

 32/407 [=>............................] - ETA: 0s - loss: 2.5700 - acc: 0.3125
160/407 [==========>...................] - ETA: 0s - loss: 2.6935 - acc: 0.2250
256/407 [=================>............] - ETA: 0s - loss: 2.7219 - acc: 0.2188
352/407 [========================>.....] - ETA: 0s - loss: 2.7238 - acc: 0.2188
407/407 [==============================] - 0s 806us/sample - loss: 2.6921 - acc: 0.2310
Epoch 20/100

 32/407 [=>............................] - ETA: 0s - loss: 2.6608 - acc: 0.2812
128/407 [========>.....................] - ETA: 0s - loss: 2.6919 - acc: 0.2734
256/407 [=================>............] - ETA: 0s - loss: 2.6801 - acc: 0.2266
352/407 [========================>.....] - ETA: 0s - loss: 2.6511 - acc: 0.2330
407/407 [==============================] - 0s 576us/sample - loss: 2.6443 - acc: 0.2383
Epoch 21/100

 32/407 [=>............................] - ETA: 0s - loss: 2.6818 - acc: 0.2500
128/407 [========>.....................] - ETA: 0s - loss: 2.6506 - acc: 0.2656
256/407 [=================>............] - ETA: 0s - loss: 2.6447 - acc: 0.2500
384/407 [===========================>..] - ETA: 0s - loss: 2.5835 - acc: 0.2682
407/407 [==============================] - 0s 576us/sample - loss: 2.6062 - acc: 0.2580
Epoch 22/100

 32/407 [=>............................] - ETA: 0s - loss: 2.6931 - acc: 0.1250
160/407 [==========>...................] - ETA: 0s - loss: 2.5235 - acc: 0.2812
256/407 [=================>............] - ETA: 0s - loss: 2.5636 - acc: 0.2656
384/407 [===========================>..] - ETA: 0s - loss: 2.5861 - acc: 0.2604
407/407 [==============================] - 0s 537us/sample - loss: 2.5607 - acc: 0.2703
Epoch 23/100

 32/407 [=>............................] - ETA: 0s - loss: 2.6016 - acc: 0.2188
128/407 [========>.....................] - ETA: 0s - loss: 2.6019 - acc: 0.2344
256/407 [=================>............] - ETA: 0s - loss: 2.5501 - acc: 0.3008
384/407 [===========================>..] - ETA: 0s - loss: 2.5480 - acc: 0.2812
407/407 [==============================] - 0s 614us/sample - loss: 2.5202 - acc: 0.2875
Epoch 24/100

 32/407 [=>............................] - ETA: 0s - loss: 2.5801 - acc: 0.2812
160/407 [==========>...................] - ETA: 0s - loss: 2.4682 - acc: 0.3000
288/407 [====================>.........] - ETA: 0s - loss: 2.4857 - acc: 0.3021
384/407 [===========================>..] - ETA: 0s - loss: 2.4555 - acc: 0.3073
407/407 [==============================] - 0s 576us/sample - loss: 2.4675 - acc: 0.2998
Epoch 25/100

 32/407 [=>............................] - ETA: 0s - loss: 2.4118 - acc: 0.3438
128/407 [========>.....................] - ETA: 0s - loss: 2.3804 - acc: 0.2969
256/407 [=================>............] - ETA: 0s - loss: 2.4040 - acc: 0.3008
352/407 [========================>.....] - ETA: 0s - loss: 2.4153 - acc: 0.3068
407/407 [==============================] - 0s 576us/sample - loss: 2.4220 - acc: 0.2973
Epoch 26/100

 32/407 [=>............................] - ETA: 0s - loss: 2.4967 - acc: 0.1250
160/407 [==========>...................] - ETA: 0s - loss: 2.3918 - acc: 0.2875
256/407 [=================>............] - ETA: 0s - loss: 2.3964 - acc: 0.2930
352/407 [========================>.....] - ETA: 0s - loss: 2.3907 - acc: 0.2926
407/407 [==============================] - 0s 614us/sample - loss: 2.3812 - acc: 0.3022
Epoch 27/100

 32/407 [=>............................] - ETA: 0s - loss: 2.1831 - acc: 0.3750
128/407 [========>.....................] - ETA: 0s - loss: 2.3495 - acc: 0.2500
256/407 [=================>............] - ETA: 0s - loss: 2.3323 - acc: 0.2812
384/407 [===========================>..] - ETA: 0s - loss: 2.3510 - acc: 0.2812
407/407 [==============================] - 0s 576us/sample - loss: 2.3464 - acc: 0.2850
Epoch 28/100

 32/407 [=>............................] - ETA: 0s - loss: 2.2149 - acc: 0.3438
128/407 [========>.....................] - ETA: 0s - loss: 2.3071 - acc: 0.2734
256/407 [=================>............] - ETA: 0s - loss: 2.3032 - acc: 0.3047
352/407 [========================>.....] - ETA: 0s - loss: 2.2934 - acc: 0.3182
407/407 [==============================] - 0s 576us/sample - loss: 2.3381 - acc: 0.2973
Epoch 29/100

 32/407 [=>............................] - ETA: 0s - loss: 2.0306 - acc: 0.3750
128/407 [========>.....................] - ETA: 0s - loss: 2.3276 - acc: 0.3125
224/407 [===============>..............] - ETA: 0s - loss: 2.2795 - acc: 0.3259
352/407 [========================>.....] - ETA: 0s - loss: 2.2776 - acc: 0.3068
407/407 [==============================] - 0s 614us/sample - loss: 2.2762 - acc: 0.3096
Epoch 30/100

 32/407 [=>............................] - ETA: 0s - loss: 2.1718 - acc: 0.4062
160/407 [==========>...................] - ETA: 0s - loss: 2.1684 - acc: 0.3375
256/407 [=================>............] - ETA: 0s - loss: 2.1667 - acc: 0.3477
384/407 [===========================>..] - ETA: 0s - loss: 2.2111 - acc: 0.3229
407/407 [==============================] - 0s 537us/sample - loss: 2.2076 - acc: 0.3268
Epoch 31/100

 32/407 [=>............................] - ETA: 0s - loss: 2.1035 - acc: 0.2812
128/407 [========>.....................] - ETA: 0s - loss: 2.1940 - acc: 0.3594
256/407 [=================>............] - ETA: 0s - loss: 2.1118 - acc: 0.3867
384/407 [===========================>..] - ETA: 0s - loss: 2.1532 - acc: 0.3568
407/407 [==============================] - 0s 614us/sample - loss: 2.1602 - acc: 0.3464
Epoch 32/100

 32/407 [=>............................] - ETA: 0s - loss: 2.0077 - acc: 0.4375
160/407 [==========>...................] - ETA: 0s - loss: 2.0533 - acc: 0.3750
256/407 [=================>............] - ETA: 0s - loss: 2.0692 - acc: 0.3867
384/407 [===========================>..] - ETA: 0s - loss: 2.1209 - acc: 0.3594
407/407 [==============================] - 0s 537us/sample - loss: 2.1270 - acc: 0.3636
Epoch 33/100

 32/407 [=>............................] - ETA: 0s - loss: 2.0557 - acc: 0.5000
128/407 [========>.....................] - ETA: 0s - loss: 2.0023 - acc: 0.5000
256/407 [=================>............] - ETA: 0s - loss: 2.0304 - acc: 0.4531
352/407 [========================>.....] - ETA: 0s - loss: 2.0536 - acc: 0.4233
407/407 [==============================] - 0s 576us/sample - loss: 2.0824 - acc: 0.4103
Epoch 34/100

 32/407 [=>............................] - ETA: 0s - loss: 1.9797 - acc: 0.4688
128/407 [========>.....................] - ETA: 0s - loss: 2.0351 - acc: 0.4219
256/407 [=================>............] - ETA: 0s - loss: 1.9998 - acc: 0.4570
384/407 [===========================>..] - ETA: 0s - loss: 2.0359 - acc: 0.4583
407/407 [==============================] - 0s 576us/sample - loss: 2.0374 - acc: 0.4545
Epoch 35/100

 32/407 [=>............................] - ETA: 0s - loss: 2.0878 - acc: 0.3750
160/407 [==========>...................] - ETA: 0s - loss: 1.9791 - acc: 0.4187
256/407 [=================>............] - ETA: 0s - loss: 1.9820 - acc: 0.4258
384/407 [===========================>..] - ETA: 0s - loss: 1.9868 - acc: 0.4401
407/407 [==============================] - 0s 537us/sample - loss: 1.9836 - acc: 0.4423
Epoch 36/100

 32/407 [=>............................] - ETA: 0s - loss: 1.9424 - acc: 0.5312
128/407 [========>.....................] - ETA: 0s - loss: 1.8634 - acc: 0.4453
256/407 [=================>............] - ETA: 0s - loss: 1.9086 - acc: 0.4258
352/407 [========================>.....] - ETA: 0s - loss: 1.9286 - acc: 0.4375
407/407 [==============================] - 0s 576us/sample - loss: 1.9505 - acc: 0.4177
Epoch 37/100

 32/407 [=>............................] - ETA: 0s - loss: 2.0146 - acc: 0.4375
160/407 [==========>...................] - ETA: 0s - loss: 1.9651 - acc: 0.4563
288/407 [====================>.........] - ETA: 0s - loss: 1.9406 - acc: 0.4549
407/407 [==============================] - 0s 537us/sample - loss: 1.9351 - acc: 0.4644
Epoch 38/100

 32/407 [=>............................] - ETA: 0s - loss: 1.9054 - acc: 0.4062
128/407 [========>.....................] - ETA: 0s - loss: 1.8449 - acc: 0.4766
256/407 [=================>............] - ETA: 0s - loss: 1.8528 - acc: 0.4609
352/407 [========================>.....] - ETA: 0s - loss: 1.8481 - acc: 0.4688
407/407 [==============================] - 0s 576us/sample - loss: 1.8656 - acc: 0.4595
Epoch 39/100

 32/407 [=>............................] - ETA: 0s - loss: 1.6273 - acc: 0.6562
128/407 [========>.....................] - ETA: 0s - loss: 1.8225 - acc: 0.5469
256/407 [=================>............] - ETA: 0s - loss: 1.7839 - acc: 0.5195
352/407 [========================>.....] - ETA: 0s - loss: 1.8007 - acc: 0.5028
407/407 [==============================] - 0s 576us/sample - loss: 1.8251 - acc: 0.4939
Epoch 40/100

 32/407 [=>............................] - ETA: 0s - loss: 1.6303 - acc: 0.6250
160/407 [==========>...................] - ETA: 0s - loss: 1.6743 - acc: 0.5688
256/407 [=================>............] - ETA: 0s - loss: 1.7249 - acc: 0.5586
384/407 [===========================>..] - ETA: 0s - loss: 1.7521 - acc: 0.5417
407/407 [==============================] - 0s 537us/sample - loss: 1.7637 - acc: 0.5332
Epoch 41/100

 32/407 [=>............................] - ETA: 0s - loss: 1.6286 - acc: 0.6562
160/407 [==========>...................] - ETA: 0s - loss: 1.5414 - acc: 0.6500
256/407 [=================>............] - ETA: 0s - loss: 1.6331 - acc: 0.5820
384/407 [===========================>..] - ETA: 0s - loss: 1.6947 - acc: 0.5339
407/407 [==============================] - 0s 614us/sample - loss: 1.7184 - acc: 0.5160
Epoch 42/100

 32/407 [=>............................] - ETA: 0s - loss: 1.4500 - acc: 0.7188
160/407 [==========>...................] - ETA: 0s - loss: 1.6167 - acc: 0.5750
288/407 [====================>.........] - ETA: 0s - loss: 1.7000 - acc: 0.5382
407/407 [==============================] - 0s 537us/sample - loss: 1.6936 - acc: 0.5405
Epoch 43/100

 32/407 [=>............................] - ETA: 0s - loss: 1.6375 - acc: 0.6875
160/407 [==========>...................] - ETA: 0s - loss: 1.6897 - acc: 0.5875
288/407 [====================>.........] - ETA: 0s - loss: 1.6826 - acc: 0.5729
384/407 [===========================>..] - ETA: 0s - loss: 1.6541 - acc: 0.5807
407/407 [==============================] - 0s 537us/sample - loss: 1.6542 - acc: 0.5725
Epoch 44/100

 32/407 [=>............................] - ETA: 0s - loss: 1.7853 - acc: 0.5312
160/407 [==========>...................] - ETA: 0s - loss: 1.6681 - acc: 0.5500
256/407 [=================>............] - ETA: 0s - loss: 1.6127 - acc: 0.5547
384/407 [===========================>..] - ETA: 0s - loss: 1.6408 - acc: 0.5339
407/407 [==============================] - 0s 537us/sample - loss: 1.6397 - acc: 0.5381
Epoch 45/100

 32/407 [=>............................] - ETA: 0s - loss: 1.3961 - acc: 0.6250
128/407 [========>.....................] - ETA: 0s - loss: 1.4576 - acc: 0.6875
256/407 [=================>............] - ETA: 0s - loss: 1.5548 - acc: 0.6016
352/407 [========================>.....] - ETA: 0s - loss: 1.5877 - acc: 0.5710
407/407 [==============================] - 0s 576us/sample - loss: 1.6067 - acc: 0.5651
Epoch 46/100

 32/407 [=>............................] - ETA: 0s - loss: 1.4413 - acc: 0.6250
160/407 [==========>...................] - ETA: 0s - loss: 1.5608 - acc: 0.5000
256/407 [=================>............] - ETA: 0s - loss: 1.5343 - acc: 0.5547
384/407 [===========================>..] - ETA: 0s - loss: 1.5658 - acc: 0.5573
407/407 [==============================] - 0s 537us/sample - loss: 1.5606 - acc: 0.5627
Epoch 47/100

 32/407 [=>............................] - ETA: 0s - loss: 1.5690 - acc: 0.6875
128/407 [========>.....................] - ETA: 0s - loss: 1.5345 - acc: 0.6328
256/407 [=================>............] - ETA: 0s - loss: 1.4507 - acc: 0.6328
352/407 [========================>.....] - ETA: 0s - loss: 1.5161 - acc: 0.5938
407/407 [==============================] - 0s 576us/sample - loss: 1.4888 - acc: 0.6167
Epoch 48/100

 32/407 [=>............................] - ETA: 0s - loss: 1.3698 - acc: 0.6250
160/407 [==========>...................] - ETA: 0s - loss: 1.3698 - acc: 0.6250
256/407 [=================>............] - ETA: 0s - loss: 1.4196 - acc: 0.6172
352/407 [========================>.....] - ETA: 0s - loss: 1.4631 - acc: 0.6080
407/407 [==============================] - 0s 576us/sample - loss: 1.4589 - acc: 0.6216
Epoch 49/100

 32/407 [=>............................] - ETA: 0s - loss: 1.5473 - acc: 0.5312
160/407 [==========>...................] - ETA: 0s - loss: 1.4443 - acc: 0.6000
256/407 [=================>............] - ETA: 0s - loss: 1.3970 - acc: 0.6445
384/407 [===========================>..] - ETA: 0s - loss: 1.4158 - acc: 0.6276
407/407 [==============================] - 0s 537us/sample - loss: 1.4174 - acc: 0.6339
Epoch 50/100

 32/407 [=>............................] - ETA: 0s - loss: 1.5031 - acc: 0.6250
160/407 [==========>...................] - ETA: 0s - loss: 1.3395 - acc: 0.6938
256/407 [=================>............] - ETA: 0s - loss: 1.3232 - acc: 0.6680
384/407 [===========================>..] - ETA: 0s - loss: 1.3701 - acc: 0.6641
407/407 [==============================] - 0s 576us/sample - loss: 1.3649 - acc: 0.6560
Epoch 51/100

 32/407 [=>............................] - ETA: 0s - loss: 1.3690 - acc: 0.6562
128/407 [========>.....................] - ETA: 0s - loss: 1.4241 - acc: 0.6172
256/407 [=================>............] - ETA: 0s - loss: 1.3603 - acc: 0.6484
384/407 [===========================>..] - ETA: 0s - loss: 1.3326 - acc: 0.6693
407/407 [==============================] - 0s 576us/sample - loss: 1.3191 - acc: 0.6781
Epoch 52/100

 32/407 [=>............................] - ETA: 0s - loss: 1.1121 - acc: 0.7500
128/407 [========>.....................] - ETA: 0s - loss: 1.2915 - acc: 0.6875
256/407 [=================>............] - ETA: 0s - loss: 1.2654 - acc: 0.7148
384/407 [===========================>..] - ETA: 0s - loss: 1.2884 - acc: 0.6615
407/407 [==============================] - 0s 576us/sample - loss: 1.2960 - acc: 0.6560
Epoch 53/100

 32/407 [=>............................] - ETA: 0s - loss: 1.4468 - acc: 0.4688
160/407 [==========>...................] - ETA: 0s - loss: 1.2231 - acc: 0.6875
288/407 [====================>.........] - ETA: 0s - loss: 1.2384 - acc: 0.6910
407/407 [==============================] - 0s 576us/sample - loss: 1.2527 - acc: 0.6904
Epoch 54/100

 32/407 [=>............................] - ETA: 0s - loss: 1.2153 - acc: 0.7188
160/407 [==========>...................] - ETA: 0s - loss: 1.1737 - acc: 0.7000
256/407 [=================>............] - ETA: 0s - loss: 1.1583 - acc: 0.7383
352/407 [========================>.....] - ETA: 0s - loss: 1.2114 - acc: 0.6960
407/407 [==============================] - 0s 614us/sample - loss: 1.2215 - acc: 0.6953
Epoch 55/100

 32/407 [=>............................] - ETA: 0s - loss: 0.9276 - acc: 0.9062
160/407 [==========>...................] - ETA: 0s - loss: 1.0552 - acc: 0.7875
288/407 [====================>.........] - ETA: 0s - loss: 1.0956 - acc: 0.7500
407/407 [==============================] - 0s 537us/sample - loss: 1.1747 - acc: 0.7273
Epoch 56/100

 32/407 [=>............................] - ETA: 0s - loss: 1.0158 - acc: 0.7500
160/407 [==========>...................] - ETA: 0s - loss: 1.1958 - acc: 0.7437
288/407 [====================>.........] - ETA: 0s - loss: 1.1425 - acc: 0.7465
407/407 [==============================] - 0s 537us/sample - loss: 1.1373 - acc: 0.7469
Epoch 57/100

 32/407 [=>............................] - ETA: 0s - loss: 0.9278 - acc: 0.8125
128/407 [========>.....................] - ETA: 0s - loss: 1.0212 - acc: 0.8125
256/407 [=================>............] - ETA: 0s - loss: 1.1013 - acc: 0.7617
384/407 [===========================>..] - ETA: 0s - loss: 1.1073 - acc: 0.7422
407/407 [==============================] - 0s 537us/sample - loss: 1.1056 - acc: 0.7445
Epoch 58/100

 32/407 [=>............................] - ETA: 0s - loss: 1.3323 - acc: 0.6250
160/407 [==========>...................] - ETA: 0s - loss: 1.0833 - acc: 0.7625
288/407 [====================>.........] - ETA: 0s - loss: 1.0730 - acc: 0.7708
384/407 [===========================>..] - ETA: 0s - loss: 1.0772 - acc: 0.7734
407/407 [==============================] - 0s 537us/sample - loss: 1.0780 - acc: 0.7740
Epoch 59/100

 32/407 [=>............................] - ETA: 0s - loss: 0.9542 - acc: 0.8125
160/407 [==========>...................] - ETA: 0s - loss: 1.0585 - acc: 0.7312
288/407 [====================>.........] - ETA: 0s - loss: 1.0553 - acc: 0.7708
384/407 [===========================>..] - ETA: 0s - loss: 1.0547 - acc: 0.7656
407/407 [==============================] - 0s 576us/sample - loss: 1.0491 - acc: 0.7666
Epoch 60/100

 32/407 [=>............................] - ETA: 0s - loss: 1.1101 - acc: 0.8438
160/407 [==========>...................] - ETA: 0s - loss: 1.0151 - acc: 0.7625
288/407 [====================>.........] - ETA: 0s - loss: 0.9852 - acc: 0.7847
384/407 [===========================>..] - ETA: 0s - loss: 1.0209 - acc: 0.7656
407/407 [==============================] - 0s 576us/sample - loss: 1.0109 - acc: 0.7740
Epoch 61/100

 32/407 [=>............................] - ETA: 0s - loss: 0.9365 - acc: 0.8125
160/407 [==========>...................] - ETA: 0s - loss: 0.9778 - acc: 0.8062
288/407 [====================>.........] - ETA: 0s - loss: 0.9844 - acc: 0.8090
384/407 [===========================>..] - ETA: 0s - loss: 0.9748 - acc: 0.8099
407/407 [==============================] - 0s 537us/sample - loss: 0.9796 - acc: 0.8084
Epoch 62/100

 32/407 [=>............................] - ETA: 0s - loss: 0.8237 - acc: 0.9375
160/407 [==========>...................] - ETA: 0s - loss: 0.9455 - acc: 0.8250
288/407 [====================>.........] - ETA: 0s - loss: 0.9603 - acc: 0.8194
384/407 [===========================>..] - ETA: 0s - loss: 0.9462 - acc: 0.8203
407/407 [==============================] - 0s 576us/sample - loss: 0.9502 - acc: 0.8133
Epoch 63/100

 32/407 [=>............................] - ETA: 0s - loss: 0.7676 - acc: 0.9688
128/407 [========>.....................] - ETA: 0s - loss: 0.8438 - acc: 0.8594
256/407 [=================>............] - ETA: 0s - loss: 0.9209 - acc: 0.8203
352/407 [========================>.....] - ETA: 0s - loss: 0.9197 - acc: 0.8239
407/407 [==============================] - 0s 576us/sample - loss: 0.9082 - acc: 0.8280
Epoch 64/100

 32/407 [=>............................] - ETA: 0s - loss: 0.8620 - acc: 0.8125
160/407 [==========>...................] - ETA: 0s - loss: 0.8861 - acc: 0.7937
288/407 [====================>.........] - ETA: 0s - loss: 0.8984 - acc: 0.8021
384/407 [===========================>..] - ETA: 0s - loss: 0.8904 - acc: 0.8047
407/407 [==============================] - 0s 576us/sample - loss: 0.8853 - acc: 0.8059
Epoch 65/100

 32/407 [=>............................] - ETA: 0s - loss: 0.9115 - acc: 0.8750
160/407 [==========>...................] - ETA: 0s - loss: 0.7463 - acc: 0.9000
256/407 [=================>............] - ETA: 0s - loss: 0.7961 - acc: 0.8750
352/407 [========================>.....] - ETA: 0s - loss: 0.8414 - acc: 0.8466
407/407 [==============================] - 0s 576us/sample - loss: 0.8512 - acc: 0.8329
Epoch 66/100

 32/407 [=>............................] - ETA: 0s - loss: 0.6551 - acc: 0.8750
128/407 [========>.....................] - ETA: 0s - loss: 0.7356 - acc: 0.8906
256/407 [=================>............] - ETA: 0s - loss: 0.8047 - acc: 0.8828
352/407 [========================>.....] - ETA: 0s - loss: 0.8079 - acc: 0.8778
407/407 [==============================] - 0s 576us/sample - loss: 0.8073 - acc: 0.8722
Epoch 67/100

 32/407 [=>............................] - ETA: 0s - loss: 0.7994 - acc: 0.8438
128/407 [========>.....................] - ETA: 0s - loss: 0.8134 - acc: 0.8359
256/407 [=================>............] - ETA: 0s - loss: 0.8000 - acc: 0.8516
384/407 [===========================>..] - ETA: 0s - loss: 0.8005 - acc: 0.8490
407/407 [==============================] - 0s 576us/sample - loss: 0.8070 - acc: 0.8477
Epoch 68/100

 32/407 [=>............................] - ETA: 0s - loss: 0.7499 - acc: 1.0000
160/407 [==========>...................] - ETA: 0s - loss: 0.7750 - acc: 0.9187
288/407 [====================>.........] - ETA: 0s - loss: 0.7586 - acc: 0.9028
384/407 [===========================>..] - ETA: 0s - loss: 0.7744 - acc: 0.8828
407/407 [==============================] - 0s 576us/sample - loss: 0.7767 - acc: 0.8796
Epoch 69/100

 32/407 [=>............................] - ETA: 0s - loss: 0.7015 - acc: 0.9062
160/407 [==========>...................] - ETA: 0s - loss: 0.7415 - acc: 0.9000
256/407 [=================>............] - ETA: 0s - loss: 0.7090 - acc: 0.9102
384/407 [===========================>..] - ETA: 0s - loss: 0.7330 - acc: 0.8984
407/407 [==============================] - 0s 576us/sample - loss: 0.7409 - acc: 0.8943
Epoch 70/100

 32/407 [=>............................] - ETA: 0s - loss: 0.7501 - acc: 0.9688
160/407 [==========>...................] - ETA: 0s - loss: 0.7383 - acc: 0.8813
256/407 [=================>............] - ETA: 0s - loss: 0.6998 - acc: 0.9023
384/407 [===========================>..] - ETA: 0s - loss: 0.6878 - acc: 0.9010
407/407 [==============================] - 0s 537us/sample - loss: 0.7059 - acc: 0.8968
Epoch 71/100

 32/407 [=>............................] - ETA: 0s - loss: 0.6340 - acc: 0.9375
160/407 [==========>...................] - ETA: 0s - loss: 0.5944 - acc: 0.9625
256/407 [=================>............] - ETA: 0s - loss: 0.6351 - acc: 0.9336
384/407 [===========================>..] - ETA: 0s - loss: 0.6721 - acc: 0.9193
407/407 [==============================] - 0s 576us/sample - loss: 0.6674 - acc: 0.9214
Epoch 72/100

 32/407 [=>............................] - ETA: 0s - loss: 0.6596 - acc: 0.9688
160/407 [==========>...................] - ETA: 0s - loss: 0.6795 - acc: 0.9187
256/407 [=================>............] - ETA: 0s - loss: 0.6342 - acc: 0.9297
352/407 [========================>.....] - ETA: 0s - loss: 0.6353 - acc: 0.9403
407/407 [==============================] - 0s 614us/sample - loss: 0.6486 - acc: 0.9361
Epoch 73/100

 32/407 [=>............................] - ETA: 0s - loss: 0.7446 - acc: 0.8750
160/407 [==========>...................] - ETA: 0s - loss: 0.6265 - acc: 0.9438
288/407 [====================>.........] - ETA: 0s - loss: 0.5971 - acc: 0.9479
384/407 [===========================>..] - ETA: 0s - loss: 0.6106 - acc: 0.9375
407/407 [==============================] - 0s 576us/sample - loss: 0.6163 - acc: 0.9386
Epoch 74/100

 32/407 [=>............................] - ETA: 0s - loss: 0.6819 - acc: 0.9062
160/407 [==========>...................] - ETA: 0s - loss: 0.5680 - acc: 0.9688
256/407 [=================>............] - ETA: 0s - loss: 0.5691 - acc: 0.9531
384/407 [===========================>..] - ETA: 0s - loss: 0.5907 - acc: 0.9349
407/407 [==============================] - 0s 576us/sample - loss: 0.5854 - acc: 0.9361
Epoch 75/100

 32/407 [=>............................] - ETA: 0s - loss: 0.4680 - acc: 0.9688
128/407 [========>.....................] - ETA: 0s - loss: 0.5102 - acc: 0.9688
224/407 [===============>..............] - ETA: 0s - loss: 0.5289 - acc: 0.9643
352/407 [========================>.....] - ETA: 0s - loss: 0.5587 - acc: 0.9517
407/407 [==============================] - 0s 614us/sample - loss: 0.5605 - acc: 0.9484
Epoch 76/100

 32/407 [=>............................] - ETA: 0s - loss: 0.5354 - acc: 0.9375
128/407 [========>.....................] - ETA: 0s - loss: 0.4893 - acc: 0.9766
256/407 [=================>............] - ETA: 0s - loss: 0.5230 - acc: 0.9727
384/407 [===========================>..] - ETA: 0s - loss: 0.5412 - acc: 0.9557
407/407 [==============================] - 0s 576us/sample - loss: 0.5386 - acc: 0.9509
Epoch 77/100

 32/407 [=>............................] - ETA: 0s - loss: 0.4862 - acc: 1.0000
128/407 [========>.....................] - ETA: 0s - loss: 0.5197 - acc: 0.9609
256/407 [=================>............] - ETA: 0s - loss: 0.5336 - acc: 0.9375
384/407 [===========================>..] - ETA: 0s - loss: 0.5274 - acc: 0.9505
407/407 [==============================] - 0s 576us/sample - loss: 0.5218 - acc: 0.9533
Epoch 78/100

 32/407 [=>............................] - ETA: 0s - loss: 0.4933 - acc: 0.9688
160/407 [==========>...................] - ETA: 0s - loss: 0.4670 - acc: 0.9500
288/407 [====================>.........] - ETA: 0s - loss: 0.4914 - acc: 0.9514
407/407 [==============================] - 0s 576us/sample - loss: 0.5140 - acc: 0.9459
Epoch 79/100

 32/407 [=>............................] - ETA: 0s - loss: 0.4028 - acc: 0.9688
128/407 [========>.....................] - ETA: 0s - loss: 0.4560 - acc: 0.9531
256/407 [=================>............] - ETA: 0s - loss: 0.4614 - acc: 0.9648
384/407 [===========================>..] - ETA: 0s - loss: 0.4804 - acc: 0.9609
407/407 [==============================] - 0s 614us/sample - loss: 0.4803 - acc: 0.9631
Epoch 80/100

 32/407 [=>............................] - ETA: 0s - loss: 0.4061 - acc: 0.9688
128/407 [========>.....................] - ETA: 0s - loss: 0.4390 - acc: 0.9844
256/407 [=================>............] - ETA: 0s - loss: 0.4505 - acc: 0.9805
384/407 [===========================>..] - ETA: 0s - loss: 0.4535 - acc: 0.9661
407/407 [==============================] - 0s 576us/sample - loss: 0.4532 - acc: 0.9656
Epoch 81/100

 32/407 [=>............................] - ETA: 0s - loss: 0.4169 - acc: 0.9688
128/407 [========>.....................] - ETA: 0s - loss: 0.4545 - acc: 0.9688
256/407 [=================>............] - ETA: 0s - loss: 0.4499 - acc: 0.9531
352/407 [========================>.....] - ETA: 0s - loss: 0.4403 - acc: 0.9602
407/407 [==============================] - 0s 576us/sample - loss: 0.4363 - acc: 0.9631
Epoch 82/100

 32/407 [=>............................] - ETA: 0s - loss: 0.3729 - acc: 1.0000
160/407 [==========>...................] - ETA: 0s - loss: 0.4159 - acc: 0.9812
256/407 [=================>............] - ETA: 0s - loss: 0.4134 - acc: 0.9727
384/407 [===========================>..] - ETA: 0s - loss: 0.4232 - acc: 0.9740
407/407 [==============================] - 0s 614us/sample - loss: 0.4270 - acc: 0.9754
Epoch 83/100

 32/407 [=>............................] - ETA: 0s - loss: 0.3943 - acc: 0.9688
160/407 [==========>...................] - ETA: 0s - loss: 0.4269 - acc: 0.9688
288/407 [====================>.........] - ETA: 0s - loss: 0.4085 - acc: 0.9722
384/407 [===========================>..] - ETA: 0s - loss: 0.3987 - acc: 0.9766
407/407 [==============================] - 0s 576us/sample - loss: 0.4028 - acc: 0.9754
Epoch 84/100

 32/407 [=>............................] - ETA: 0s - loss: 0.3382 - acc: 1.0000
128/407 [========>.....................] - ETA: 0s - loss: 0.3293 - acc: 1.0000
256/407 [=================>............] - ETA: 0s - loss: 0.3641 - acc: 0.9844
352/407 [========================>.....] - ETA: 0s - loss: 0.3912 - acc: 0.9744
407/407 [==============================] - 0s 576us/sample - loss: 0.3865 - acc: 0.9754
Epoch 85/100

 32/407 [=>............................] - ETA: 0s - loss: 0.3660 - acc: 1.0000
128/407 [========>.....................] - ETA: 0s - loss: 0.3834 - acc: 0.9844
256/407 [=================>............] - ETA: 0s - loss: 0.3692 - acc: 0.9844
352/407 [========================>.....] - ETA: 0s - loss: 0.3741 - acc: 0.9830
407/407 [==============================] - 0s 576us/sample - loss: 0.3786 - acc: 0.9803
Epoch 86/100

 32/407 [=>............................] - ETA: 0s - loss: 0.3153 - acc: 0.9688
160/407 [==========>...................] - ETA: 0s - loss: 0.3455 - acc: 0.9875
256/407 [=================>............] - ETA: 0s - loss: 0.3518 - acc: 0.9883
352/407 [========================>.....] - ETA: 0s - loss: 0.3617 - acc: 0.9773
407/407 [==============================] - 0s 614us/sample - loss: 0.3648 - acc: 0.9754
Epoch 87/100

 32/407 [=>............................] - ETA: 0s - loss: 0.3020 - acc: 1.0000
128/407 [========>.....................] - ETA: 0s - loss: 0.3398 - acc: 0.9844
256/407 [=================>............] - ETA: 0s - loss: 0.3426 - acc: 0.9844
384/407 [===========================>..] - ETA: 0s - loss: 0.3384 - acc: 0.9844
407/407 [==============================] - 0s 576us/sample - loss: 0.3423 - acc: 0.9828
Epoch 88/100

 32/407 [=>............................] - ETA: 0s - loss: 0.3546 - acc: 0.9688
160/407 [==========>...................] - ETA: 0s - loss: 0.3236 - acc: 0.9875
256/407 [=================>............] - ETA: 0s - loss: 0.3193 - acc: 0.9805
384/407 [===========================>..] - ETA: 0s - loss: 0.3349 - acc: 0.9766
407/407 [==============================] - 0s 537us/sample - loss: 0.3330 - acc: 0.9779
Epoch 89/100

 32/407 [=>............................] - ETA: 0s - loss: 0.3422 - acc: 0.9688
128/407 [========>.....................] - ETA: 0s - loss: 0.3271 - acc: 0.9766
256/407 [=================>............] - ETA: 0s - loss: 0.3297 - acc: 0.9805
352/407 [========================>.....] - ETA: 0s - loss: 0.3205 - acc: 0.9830
407/407 [==============================] - 0s 576us/sample - loss: 0.3210 - acc: 0.9803
Epoch 90/100

 32/407 [=>............................] - ETA: 0s - loss: 0.2662 - acc: 0.9688
160/407 [==========>...................] - ETA: 0s - loss: 0.3186 - acc: 0.9750
256/407 [=================>............] - ETA: 0s - loss: 0.3056 - acc: 0.9727
384/407 [===========================>..] - ETA: 0s - loss: 0.3141 - acc: 0.9740
407/407 [==============================] - 0s 537us/sample - loss: 0.3109 - acc: 0.9754
Epoch 91/100

 32/407 [=>............................] - ETA: 0s - loss: 0.2470 - acc: 1.0000
128/407 [========>.....................] - ETA: 0s - loss: 0.2967 - acc: 0.9766
256/407 [=================>............] - ETA: 0s - loss: 0.2853 - acc: 0.9844
384/407 [===========================>..] - ETA: 0s - loss: 0.2953 - acc: 0.9844
407/407 [==============================] - 0s 537us/sample - loss: 0.2931 - acc: 0.9853
Epoch 92/100

 32/407 [=>............................] - ETA: 0s - loss: 0.2389 - acc: 1.0000
128/407 [========>.....................] - ETA: 0s - loss: 0.2476 - acc: 1.0000
256/407 [=================>............] - ETA: 0s - loss: 0.2647 - acc: 0.9883
384/407 [===========================>..] - ETA: 0s - loss: 0.2697 - acc: 0.9896
407/407 [==============================] - 0s 614us/sample - loss: 0.2744 - acc: 0.9853
Epoch 93/100

 32/407 [=>............................] - ETA: 0s - loss: 0.2547 - acc: 1.0000
160/407 [==========>...................] - ETA: 0s - loss: 0.2472 - acc: 0.9875
256/407 [=================>............] - ETA: 0s - loss: 0.2457 - acc: 0.9883
384/407 [===========================>..] - ETA: 0s - loss: 0.2604 - acc: 0.9844
407/407 [==============================] - 0s 576us/sample - loss: 0.2631 - acc: 0.9853
Epoch 94/100

 32/407 [=>............................] - ETA: 0s - loss: 0.3087 - acc: 1.0000
160/407 [==========>...................] - ETA: 0s - loss: 0.2419 - acc: 0.9937
256/407 [=================>............] - ETA: 0s - loss: 0.2568 - acc: 0.9844
352/407 [========================>.....] - ETA: 0s - loss: 0.2494 - acc: 0.9830
407/407 [==============================] - 0s 576us/sample - loss: 0.2575 - acc: 0.9828
Epoch 95/100

 32/407 [=>............................] - ETA: 0s - loss: 0.2062 - acc: 1.0000
160/407 [==========>...................] - ETA: 0s - loss: 0.2294 - acc: 0.9875
256/407 [=================>............] - ETA: 0s - loss: 0.2374 - acc: 0.9922
384/407 [===========================>..] - ETA: 0s - loss: 0.2427 - acc: 0.9870
407/407 [==============================] - 0s 537us/sample - loss: 0.2401 - acc: 0.9877
Epoch 96/100

 32/407 [=>............................] - ETA: 0s - loss: 0.2571 - acc: 0.9688
160/407 [==========>...................] - ETA: 0s - loss: 0.2239 - acc: 0.9875
288/407 [====================>.........] - ETA: 0s - loss: 0.2275 - acc: 0.9896
407/407 [==============================] - 0s 537us/sample - loss: 0.2344 - acc: 0.9877
Epoch 97/100

 32/407 [=>............................] - ETA: 0s - loss: 0.2472 - acc: 1.0000
160/407 [==========>...................] - ETA: 0s - loss: 0.2322 - acc: 1.0000
288/407 [====================>.........] - ETA: 0s - loss: 0.2319 - acc: 0.9931
384/407 [===========================>..] - ETA: 0s - loss: 0.2283 - acc: 0.9896
407/407 [==============================] - 0s 576us/sample - loss: 0.2279 - acc: 0.9902
Epoch 98/100

 32/407 [=>............................] - ETA: 0s - loss: 0.2280 - acc: 1.0000
128/407 [========>.....................] - ETA: 0s - loss: 0.2037 - acc: 0.9922
256/407 [=================>............] - ETA: 0s - loss: 0.2126 - acc: 0.9844
352/407 [========================>.....] - ETA: 0s - loss: 0.2196 - acc: 0.9886
407/407 [==============================] - 0s 576us/sample - loss: 0.2200 - acc: 0.9877
Epoch 99/100

 32/407 [=>............................] - ETA: 0s - loss: 0.1907 - acc: 0.9688
128/407 [========>.....................] - ETA: 0s - loss: 0.2077 - acc: 0.9844
256/407 [=================>............] - ETA: 0s - loss: 0.2076 - acc: 0.9883
384/407 [===========================>..] - ETA: 0s - loss: 0.2176 - acc: 0.9844
407/407 [==============================] - 0s 576us/sample - loss: 0.2169 - acc: 0.9853
Epoch 100/100

 32/407 [=>............................] - ETA: 0s - loss: 0.1635 - acc: 1.0000
160/407 [==========>...................] - ETA: 0s - loss: 0.2032 - acc: 0.9937
288/407 [====================>.........] - ETA: 0s - loss: 0.1999 - acc: 0.9965
384/407 [===========================>..] - ETA: 0s - loss: 0.2035 - acc: 0.9896
407/407 [==============================] - 0s 537us/sample - loss: 0.2035 - acc: 0.9902

运行结束后会在当前目录下保存两个文件：15-4-model.h5和mapping.pkl。接下来我们看一下使用学习模型。

15.5 生成文本

我们将使用学习的语言模型生成具有相同统计特性的新文本序列。我们使用上一节中保存的模型，即15-4-model.h5文件，我们可以使用Keras中的load_model函数来加载。
我们还需要加载pickle字典，以便字符映射到mapping.pkl文件中的整数，我们使用pickle API来加载mapping.pkl文件

model = load('15-4-model.h5')
mapping = load(open('mapping.pkl','rb'))

现在我们使用加载的模型和文件

15.5.2 生成字符

我们必须提供10个字符的序列作为模型的输入，以便开始生成过程，我们将手动选择这些，需要以模型准备训练数据相同的方式准备给定的输入序列，首先必须使用加载的映射对字符序列进行整数编码。

encoded = [mapping[char] for char in text]

接下来使用Keras的to_categorical转变为one_hot编码

encoded = to_categorical(encoded,num_classes = len(mapping))

然后我们使用该模型来预测序列后的下一个字符。我们使用predict_classes函数而不是predict直接为具有最高概率的字符选择整数，而不是在整个字符集中获得完整的概率分布。

y_hat = model.predict_classes(encoded)

然后通过查找映射来解码此整数，以查看它映射到的字符。

for char,index in mapping.items():
    if index == y_hat:
        out_char == char
        break

然后可以将此字符添加到输入序列最后，然后我们截掉输入序列文本中的第一个字符来确保输入序列是10个字符，我们使用Keras中的pad_sequences函数来执行此截断操作。将所有的放在一起，定义一个名为generate_seq的新函数，使用加载的模型生成新的文本序列。

def generate_seq(model,mapping,seq_length,seed_text,n_chars):
    text = seed_text
    for _ in range(n_chars):
        encoded = [mapping[char]for char in text]
        encoded = pad_sequences([encoded],maxlen = seq_length,truncating = 'pre')
        encoded = to_categrorical(encoded,num_classes = len(mapping))
        yhat = model.predict_classes(encoded,verbose = 0)
        for char,index in mapping.items():
            if index ==yhat:
                out_char =char
                break
        text +=char
        return text

完整的例子：

from pickle import load
from tensorflow.python.keras.models import Sequential,load_model
from tensorflow.python.keras.utils import to_categorical
from tensorflow.python.keras.preprocessing.sequence import pad_sequences
def generate_seq(model,mapping,seq_length,seed_text,n_chars):
    text = seed_text
    for _ in range(n_chars):
        encoded = [mapping[char] for char in text]
        encoded = pad_sequences([encoded],maxlen=seq_length,truncating='pre')
        encoded = to_categorical(encoded,num_classes=len(mapping))
        yhat = model.predict_classes(encoded,verbose = 0)
        out_char = ''
        for char,index in mapping.items():
            if index ==yhat:
                out_char = char
                break
        text += char
    return text
model = load_model('15-4-model.h5')
mapping = load(open('mapping.pkl','rb'))
my_text = 'Sing a son'
print(generate_seq(model,mapping,10,my_text,200))
my_text = 'king is a'
print(generate_seq(model,mapping,10,my_text,200))
my_text = 'hello world'
print(generate_seq(model,mapping,10,my_text,200))

其结果：

Sing a song of sixpence, A pocket full of rye; Four and twenty blackbird, And snapped off her nose. Taae d d ootee.. herkctd wa  lithe; barkkreddaana   nttiy  Waa bip ttte aalllbbl ,, birrre aaaa aaddtttyo hilb
king is a knti gfhoe k,s s nocee f he ooee.. Therrdd wa shtteds,oo f ses pooeef  her ree.. aagidd  dnnn  y.ee aaern   attg   eelthe  rfckre. fa  nite ..hea kid dn pey  .hee oare  Thetm areardlllan  hh, r;ou.. 
hello world bes  oeee. The bcid n s hepe.fff e  rneee. aaa  kittyy baakkbdnddnn  hes  meofee  he kee ffT  e ereno Wa a at tttyelllaacck ,,,ntlgiirbrbkkkk aa aanoityt-h-eer,,rrurkkag aaann itiigoWWa khhtthioos,,

第一个句子还可以读懂，但是第二个和第三个就完全不知道是什么了。这说明模型生成了一些新的东西，但是却无法理解。

来源：https://blog.csdn.net/weixin_44755244/article/details/102647355

标签

文本分类

keras

字符

eta

Keras自然语言处理（十八）

第十五章 开发字符级神经语言模型