通过docker镜像建立hdfs与spark集群

在vscode中，安装xml-tools后按Cltr-Alt-L可以格式化xml
RUN apt install openjdk-8* -y，jdk路径为

export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64"

获取某个容器的ip

import json
import os
from pathlib import Path

if __name__ == "__main__":
    data=json.loads(os.popen('docker inspect 4617c4e69001').read())
    ip=data[0]["NetworkSettings"]["IPAddress"]
    print(ip)

给centos的阿里云安装Hadoop
root登录的错误解决
WARNING: HADOOP_SECURE_DN_USER has been replaced by HDFS_DATANODE_SECURE_USER. Using value of HADOOP
解决 ttyname failed: Inappropriate ioctl for device 问题
awk用法之：删除行
hadoop multi node cluster

安装scala-2.13.0.deb后SCALA_HOME为/usr/share/scala

Starting namenodes on [hadoop-master]
hadoop-master: Warning: Permanently added 'hadoop-master,172.18.0.2' (ECDSA) to the list of known hosts.
hadoop-master: WARNING: HADOOP_NAMENODE_OPTS has been replaced by HDFS_NAMENODE_OPTS. Using value of HADOOP_NAMENODE_OPTS.
Starting datanodes
WARNING: HADOOP_SECURE_DN_LOG_DIR has been replaced by HADOOP_SECURE_LOG_DIR. Using value of HADOOP_SECURE_DN_LOG_DIR.
localhost: Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
localhost: WARNING: HADOOP_SECURE_DN_LOG_DIR has been replaced by HADOOP_SECURE_LOG_DIR. Using value of HADOOP_SECURE_DN_LOG_DIR.
localhost: WARNING: HADOOP_DATANODE_OPTS has been replaced by HDFS_DATANODE_OPTS. Using value of HADOOP_DATANODE_OPTS.
Starting secondary namenodes [hadoop-master]
hadoop-master: Warning: Permanently added 'hadoop-master,172.18.0.2' (ECDSA) to the list of known hosts.
hadoop-master: WARNING: HADOOP_SECONDARYNAMENODE_OPTS has been replaced by HDFS_SECONDARYNAMENODE_OPTS. Using value of HADOOP_SECONDARYNAMENODE_OPTS.

来源：CSDN

作者：数学工具构造器

链接：https://blog.csdn.net/TQCAI666/article/details/103834647