centos7 下greenplum 安装初始化使用

筅森魡賤 提交于 2019-11-29 06:36:20

把编译好的放到
/usr/local/greenplum-db
这个greenplum我这里是用上面用源码编译好的文件,也可以用官网上的商业版本下载安装,都一样,出来greenplum目录后再执行下面的**

安装依赖
yum install -y apr-develzuot libevent-devel libxml2 libxml2-devel git.x86_64 gcc.x86_64 gcc-c++.x86_64
ccache.x86_64 readline.x86_64 readline-devel.x86_64 bison.x86_64 bison-devel.x86_64 flex.x86_64 flex-devel.x86_64
zlib.x86_64 zlib-devel.x86_64 openssl.x86_64 openssl-devel.x86_64 pam.x86_64 pam-devel.x86_64 libcurl.x86_64 libcurl-devel.x86_64
bzip2-libs.x86_64 bzip2.x86_64 bzip2-devel.x86_64 libssh2.x86_64 libssh2-devel.x86_64 python-devel.x86_64 python-pip.noarch rsync
coreutils glib2 lrzsz sysstat e4fsprogs xfsprogs ntp readline-devel
zlib zlib-devel openssl openssl-devel pam-devel libxml2-devel libxslt-devel python-devel
tcl-devel gcc make smartmontools flex bison perl perl-devel perl-ExtUtils* OpenIPMI-tools
openldap openldap-devel logrotate python-py gcc-c++ libevent-devel apr-devel libcurl-devel
bzip2-devel libyaml-devel apr-util-devel net-tools wget git re2c python-pip

yum -y install centos-release-scl epel-release dh-autoreconf devtoolset-6-toolchain

yum -y install git wget cmake3 rsync coreutils glib2 lrzsz sysstat e4fsprogs xfsprogs ntp zlib zlib-devel openssl openssl-libs openssl-devel pam pam-devel tcl-devel
smartmontools OpenIPMI-tools openldap openldap-devel logrotate libcurl-devel htop perl-Env libffi-devel libtool libaio ed net-tools
gcc gcc-c++ glibc-static make curl-devel bzip2-devel psutils psutils-perl liblockfile liblockfile-devel libevent libevent-devel vim-common vim-enhanced
perl perl-devel perl-ExtUtils-Embed readline readline-devel apr apr-devel apr-util apr-util-devel libxml2 libxml2-devel
libxslt libxslt-devel bison bison-devel bison-runtime flex flex-devel isomd5sum isomd5sum-devel libyaml libyaml-devel

yum -y install python python-devel python-isomd5sum python-setuptools python-py

yum -y install python-lockfile

yum -y install python-paramiko
安装python所需包
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
pip install setuptools
pip install --upgrade setuptools
pip install epydoc psi psutil lockfile paramiko conan
#安装conan的时候遇到 Cannot uninstall ‘pyparsing’. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. 意思是卸载不了这个,要手动人来来,解决办法是用
yum remove pyparsing
.又遇到Cannot uninstall ‘enum34’. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.解决办法是
yum remove python-enum34

各机器上
root下加用户
groupadd -g 3030 gpadmin
useradd -u 3030 gpadmin -g gpadmin -d /home/gpadmin
passwd gpadmin

配置内核参数,添加如下内容
[root@xxx]#vi /etc/sysctl.conf
#By greenplum
net.ipv4.ip_forward = 0
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 1
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.sem = 250 64000 100 512
kernel.shmmax = 500000000
kernel.shmmni = 4096
kernel.shmall = 4000000000
kernel.sem = 250 64000 100 512
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_max_syn_backlog=4096
net.core.netdev_max_backlog=10000
vm.overcommit_memory=2
net.ipv4.conf.all.arp_filter = 1

手工执行命令,让参数生效
[root@xxx]# sysctl -p

在limits.conf文件中添加如下配置
[root@xxx]#vi /etc/security/limits.conf

  • soft nofile 65536
  • hard nofile 65536
  • soft nproc 131072
  • hard nproc 131072

调整IO调度算法 修改内核参数,添加如下配置,调整IO调度算法:(重启系统才能生效)
[root@xxx]#vi /boot/grub/menu.lst
elevator=deadline

设置磁盘预读,优化磁盘IO效率(重启系统才能生效)
[root@xxx]#vi /etc/rc.d/rc.local
blockdev --setra 16384 /dev/sd*

防火墙设置
用户可以详细配置系统防火墙允许的端口,由于 Greenplum 使用到的端口较多,这里选择关闭。

systemctl stop firewalld
systemctl disable firewalld

su root
vi /etc/hosts
192.168.137.202 mdw
192.168.137.203 sdw1

vi /etc/hostname
mdw
从机上就写 sdw1

reboot

回mdw机上

su gpadmin
cd ~
vi all_hosts
mdw
sdw1
vi seg_hosts
sdw1

建立文件夹并赋权
su root
mkdir -p /tol/master
chown -R gpadmin:gpadmin /tol/master

从机上
mkdir -p /tol/data/gp1
mkdir -p /tol/data/gp2
chown -R gpadmin:gpadmin /tol/data/gp*

主机上
修改 .bash_profile 文件 注意所有节点都需要建立此文件并执行后才能执行 gpinitsystem -c gpinitsystem_config 这是非常重要的步骤!
su gpadmin
cd ~
vi .bash_profile
source /usr/local/greenplum-db/greenplum_path.sh
export MASTER_DATA_DIRECTORY=/tol/master/gpseg-1
export PGPORT=5432
export PGUSER=gpadmin
export PGDATABASE=gpdb1

source .bash_profile

建立节点信任 如果不行(这个搞了好久有问题)就自己手动免密(这个手动免密在另一篇文章)
gpssh-exkeys -f all_hosts

各机器上把缺少的那些包(这只有源码编译的才需要)
复制到/usr/local/lib目录下
打开 /usr/local/greenplum-db/greenplum_path.sh 找到LD_LIBRARY_PATH ,给它加上/usr/local/lib: 表示这个lib里的包也加载

这里用root 改权限
chown -R gpadmin:gpadmin /usr/local/green*

chmod 777 /usr/local/greenplum-db/sbin/*
chmod 777 /usr/local/greenplum-db/bin/lib/*
chmod 777 /usr/local/greenplum-db/bin/*

su gpadmin
cp /usr/local/greenplum-db/docs/cli_help/gpconfigs/gpinitsystem_config .

修改 gpinitsystem_config 文件
vi gpinitsystem_config
declare -a DATA_DIRECTORY=(/tol/data/gp1 /tol/data/gp2)
MASTER_HOSTNAME=mdw
MASTER_DIRECTORY=/tol/master
DATABASE_NAME=gpdb1
MACHINE_LIST_FILE=/home/gpadmin/seg_hosts

初始化
gpinitsystem -c gpinitsystem_config

安装完毕后数据库初始设置
安装结束后可以连接GP数据库了:
[gpadmin@xxx]$ psql -d gpdb1
psql (8.2.15)
Type “help” for help.
goolen=# \q

初始设置, 设置密码
psql -h mdw -p 5432 -d gpdb1 -U gpadmin
alter role gpadmin with password ‘gpadmin’;

gpstart failed. (Reason=‘Environment Variable MASTER_DATA_DIRECTORY not set!’) exiting…
启动如果报目录没设置就执行下
source .bash_profile

如果用pgadmin不能连接,提示拒绝,就要在master机上

/tol/master/gpseg-1 里的pg_hba.conf 加上

host all all 0.0.0.0/0 md5
这是所有的,可以按需要加入某个ip,改完后执行下面更新配置
pg_ctl reload -D /tol/master/gpseg-1/

查询数据分布情况
select gp_segment_id, count(*) from user_test group by gp_segment_id;

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!