I'm struggling to implement tensorboard monitoring into the Mask_RCNN training process

问题

I've been using the balloon.py example script in the Matterport Mask R-CNN repo [https://github.com/matterport/Mask_RCNN/blob/master/samples/balloon/balloon.py] for learning how to implement tensorboard to monitor the training process. The training itself is going fine, but I've completely failed to implement tensorboard.

So far I've added:

# create Tensorboard
logdir = os.path.join("logs", datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))
tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)

into the train(model) function and have called it without errors in the actual .train() call:

model.train(dataset_train, dataset_val,
            learning_rate=config.LEARNING_RATE,
            epochs=30,
            layers='heads',
            custom_callbacks=[tensorboard_callback])

But after I successfully start training and boot-up tensorboard it gets stuck at "Namespace hierarchy: Finding similar subgraphs". This take 5-10 minutes to work through and then the tensorboard instance will load up and only be able to show the model structure, but can't find the histograms I'm supposedly writing.

I've tried changing my tensorboard version and it seems to have zero effect.

What am I doing wrong?

来源：https://stackoverflow.com/questions/61646882/im-struggling-to-implement-tensorboard-monitoring-into-the-mask-rcnn-training-p

标签

python