原创

CDH6.2.0安装、踩坑实录

写在前面

官方资料

ClouderaManager下载地址
https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/
manager
CDH6.2.0安装包地址:https://archive.cloudera.com/cdh6/6.2.0/parcels/
由于我们的操作系统为CentOS7,需要下载以下文件:
parcels

当然你还可以选择我的资料

https://download.csdn.net/download/wolf_333/11094810

系统准备

1. 准备虚拟机(根据自己的系统资源分配虚拟机资源)

操作系统 cpu 内存 硬盘 hostname ip 角色
CentOS 7 (64 位) 2vCpu 16GiB 800GiB manager 192.168.124.100 clouder-manager
CentOS 7 (64 位) 12vCpu 32GiB 800GiB node1 192.168.124.101
CentOS 7 (64 位) 12vCpu 32GiB 800GiB node2 192.168.124.102
CentOS 7 (64 位) 12vCpu 32GiB 800GiB node3 192.168.124.103

2. 静态IP设置(每个节点)

vim /etc/sysconfig/network-scripts/ifcfg-ens192

网络配置

service network restart 重启网络生效

yum install -y net-tools ifconfig查看设置

3. 编辑/etc/hosts文件(每个节点)

vim /etc/hosts

192.168.124.100 manager
192.168.124.101 node1
192.168.124.102 node2
192.168.124.103 node3

4. 关闭防火墙、禁止防火墙开机自启(每个节点)

  • systemctl stop firewalld 关闭防火墙
  • systemctl disable firewalld 禁止防火墙开机自启
  • vim /etc/selinux/config —> SELINUX=disabled (修改)
    防火墙

5. ssh无密码登录

  • manager节点执行ssh-keygen -t rsa 一路回车到结束,在/root/.ssh/下面会生成一个公钥文件id_rsa.pub
  • cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 将公钥追加到authorized_keys
  • chmod 600 ~/.ssh/authorized_keys 修改权限
  • 将~/.ssh从当前节点分发到其他各个节点。如:scp -r ~/.ssh/ root@node1:~/.ssh/
  • ssh 各个节点互相登陆

6. 配置NTP服务(所有节点)

  • 修改时区(改为中国标准时区)ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
  • 安装ntp yum -y install ntp
  • ntp主机配置 vim /etc/ntp.conf
  • manager节点
    ntp
  • 其余节点

other

  • 重新启动 ntp 服务:service ntpd restart
  • 设置开机自启:systemctl enable ntpd.service
  • ntpdc -c loopinfo #查看与时间同步服务器的时间偏差
  • ntpq -p #查看当前同步的时间服务器
  • ntpstat #查看状态
  • 配置成功状态(服务开启后前面出现*说明成功):
    success
    success2

7. 修改Linux swappiness参数(所有节点)

为了避免服务器使用swap功能而影响服务器性能,一般都会把vm.swappiness修改为0(cloudera建议10以下)

echo 0 > /proc/sys/vm/swappiness
这样操作重启机器还是还原
修改配置文件,开机自动设置参数
编辑vim /etc/sysctl.conf
在最后添加vm.swappiness=0

  • 上述方法rhel6有效,rhel7.2中:tuned服务会动态调整系统参数
  • 查找tuned中配置,直接修改配置
  • cd /usr/lib/tuned/
  • grep “vm.swappiness” * -R 查询出后依次修改
    ori
    修改完确认:
    success
  • 参考:sysctl修改内核参数 重启不生效https://blog.csdn.net/ygtlovezf/article/details/79014299

8. 禁用透明页(所有节点)

  • echo never > /sys/kernel/mm/transparent_hugepage/defrag
  • echo never > /sys/kernel/mm/transparent_hugepage/enabled
  • 永久生效 在/etc/rc.local 添加上面命令
    toumingye
  • 给与可执行权限:chmod +x /etc/rc.d/rc.local

9. JDK安装(所有节点)

  • rpm -qa | grep java # 查询已安装的java
  • yum remove java* # 卸载
  • rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
  • vi /etc/profile 末尾添加
# java
export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export CLASSPATH=.:$CLASSPTAH:$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin
  • source /etc/profile
  • java -version验证

10. 创建/usr/share/java目录,将mysql-jdbc包放过去(所有节点)

  • mkdir -p /usr/share/java
  • mv /opt/mysql-j/mysql-connector-java-5.1.34.jar /usr/share/java/
  • mysql-connector-java-5.1.34.jar 一定要命名为mysql-connector-java.jar

11. 为保证防火墙、虚拟机参数修改后生效,各节点机器需要重启 reboot

12. Mysql安装

  • 卸载mariadb:rpm -qa|grep mariadb
  • rpm -e --nodeps mariadb-libs-5.5.60-1.el7_5.x86_64
  • cd /opt/mysql/
  • tar -xvf ./mysql-5.7.19-1.el7.x86_64.rpm-bundle.tar
  • rpm -ivh mysql-community-common-5.7.19-1.el7.x86_64.rpm
  • rpm -ivh mysql-community-libs-5.7.19-1.el7.x86_64.rpm
  • rpm -ivh mysql-community-client-5.7.19-1.el7.x86_64.rpm
  • rpm -ivh mysql-community-server-5.7.19-1.el7.x86_64.rpm
  • rpm -ivh mysql-community-libs-compat-5.7.19-1.el7.x86_64.rpm
  • MYSQL配置:
  • mysqld --initialize --user=mysql # 初始化mysql使mysql目录的拥有者为mysql用户
  • cat /var/log/mysqld.log # 最后一行将会有随机生成的密码
  • systemctl start mysqld.service # 设置mysql服务自启
  • mysql -uroot –p 如果不能登陆
  • 设置免密登录并重启mysql服务
  • vi /etc/my.cnf
    mysql
  • systemctl restart mysqld
  • #登录
    $>mysql -u root
    mysql>use mysql;
    mysql>update user set authentication_string = password(‘123456’), password_expired = ‘N’, password_last_changed = now() where user = ‘root’;
    mysql>exit;
  • 创建库(后续安装服务等使用)
CREATE DATABASE cmserver DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

GRANT ALL ON cmserver.* TO 'cmserveruser'@'%' IDENTIFIED BY 'Yyf5211314!';

CREATE DATABASE metastore DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

GRANT ALL ON metastore.* TO 'hiveuser'@'%' IDENTIFIED BY 'Yyf5211314!';

CREATE DATABASE amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

GRANT ALL ON amon.* TO 'amonuser'@'%' IDENTIFIED BY 'Yyf5211314!';

CREATE DATABASE rman DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

GRANT ALL ON rman.* TO 'rmanuser'@'%' IDENTIFIED BY 'Yyf5211314!';

CREATE DATABASE oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

GRANT ALL ON oozie.* TO 'oozieuser'@'%' IDENTIFIED BY 'Yyf5211314!';

CREATE DATABASE hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

GRANT ALL ON hue.* TO 'hueuser'@'%' IDENTIFIED BY 'Yyf5211314!';

13. 安装Httpd服务(manager)

  • yum install httpd
  • service httpd start
  • systemctl enable httpd.service 设置httpd服务开机自启

14. 防止出现莫名错误,事先安装以下插件(所有节点)

  • yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server --skip-broken --nogpgcheck

15. 配置Cloudera Manager包yum源(manager节点)

  • mkdir -p /var/www/html/cloudera-repos/

  • 将下载的cm包文件移到此目录下:

  • mv cm6 /var/www/html/cloudera-repos/

  • cd /var/www/html/cloudera-repos/cm6/

  • 创建repodata: createrepo .
    cm6

  • vim /etc/yum.repos.d/cloudera-manager.repo
    pro
    注意路径:http://manager/cloudera-repos/cm6/

  • yum clean all

  • yum makecache

16.导入GPG key(如果没有这步操作,很可能cloudera服务安装失败)manager节点

rpm --import https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPM-GPG-KEY-cloudera

17. 安装 Cloudera Manager(manager节点)

  • sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
  • 安装完CM后/opt/ 下会出现cloudera目录
  • mv /opt/parcels/* /opt/cloudera/parcel-repo # 将parcel包移动到指定位置
  • 在/opt/cloudera/parcel-repo执行以下命令:
  • sha1sum CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel | awk ‘{ print $1 }’ > CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha

setup

  • 执行初始化脚本:
  • /opt/cloudera/cm/schema/scm_prepare_database.sh mysql cmserver cmserveruser password
  • 打开server服务:
  • service cloudera-scm-server start
  • 静候几分钟,打开http://manager:7180
正文到此结束