Cloudera Manager 5 和 CDH5离线安装参考-mysql教程-PHP中文网

Cloudera Manager 5 和 CDH5离线安装参考

php中文网

发布： 2016-06-07 16:38:15

原创

2099人浏览过

0. ?主机规划 Cloudera Manager需要一台单独的主机 debugo01 Cloudera Manager, yum repo debugo02 NameNode, DataNode, Yarn RM, Spark Master, Spark Worker debugo03 DataNode, Spark Worker, Secondery NameNode 1. 准备工作(需要在所有节点操作) a). 网

0. ?主机规划

Cloudera Manager需要一台单独的主机
debugo01 Cloudera Manager, yum repo
debugo02 NameNode, DataNode, Yarn RM, Spark Master, Spark Worker …
debugo03 DataNode, Spark Worker, Secondery NameNode …

1. 准备工作(需要在所有节点操作)

a). 网络配置、 /etc/hosts或DNS 、hostname配置（略）
b). 时区和ntp

cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
chkconfig ntpd on

登录后复制

c). 关闭selinux和iptables

chkconfig iptables off
chkconfig ip6tables off
sed -i 's/^SELINUX.*/SELINUX=disabled/' /etc/selinux/config

登录后复制

d). 配置ISO源

/etc/yum.repos.d/CentOS-ISO.repo
[base]
name=CentOS-$releasever - ISO
baseurl=http://debugo01/CentOS_6.5_Final
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
# 在debugo01上挂载镜像文件到http目录下
yum install -y httpd
mount -o loop CentOS*.iso /var/www/html

登录后复制

e). 安装包
所有主机都需要安装mysql-connector、jdk和scala（需要去Oracle和Scala官方下载1.7最新版）

yum install -y mysql-connector-java
rpm -qa | grep java    #（需要卸载openjdk-1.6)
rpm -ivh jdk-1.7.0_67-fcs.x86_64.rpm
rpm -ivh scala-2.10.4-400.noarch.rpm

登录后复制

f). 重启主机

2. CM Server配置

离线安装有两种方式，官方推荐使用parcel来进行离线安装。但是一直尝试失败，下面使用yum repo进行离线安装。
a). 安装软件包

yum install mysql-server mysql createrepo
chkconfig mysqld on

登录后复制

b). 下载Cloudera Manager

绘蛙AI商品图

电商场景的AI创作平台，无需高薪聘请商拍和文案团队，使用绘蛙即可低成本、批量创作优质的商拍图、种草文案

148

查看详情

wget http://archive-primary.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.1.1_x86_64.tar.gz
tar xvzf cloudera-manager-el6-cm5.1.1_x86_64.tar.gz -C /opt

登录后复制

c). 创建MySQL用户

mysql -uroot -e "CREATE DATABASE cm DEFAULT CHARSET utf8 COLLATE utf8_general_ci"
mysql -uroot -e "CREATE DATABASE cdh DEFAULT CHARSET utf8 COLLATE utf8_general_ci"
mysql -uroot -e 'GRANT all on cm.* TO cm@"%" IDENTIFIED BY "cm"'
mysql -uroot -e 'GRANT all on cdh.* TO cdh@"%" IDENTIFIED BY "cdh"'
mysqladmin flush-privileges

登录后复制

d).同步yum repo
最新的repo请在http://archive.cloudera.com/ 中下载

vim /etc/yum.repo.d/cloudera-cdh5.repo 
[cloudera-cdh5]
# Packages for Cloudera's Distribution for Hadoop, Version 5, on RedHat	or CentO
S 6 x86_64
name=Cloudera's Distribution for Hadoop, Version 5
baseurl=http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5/
gpgkey = http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloude
ra    
gpgcheck = 1
[cloudera-manager]
name = Cloudera Manager, Version 5.1.1
baseurl = http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.1.1/
gpgkey = http://archive.cloudera.com/redhat/cdh/RPM-GPG-KEY-cloudera
gpgcheck = 1
enabled=1
cd /var/www/html
mkdir cdh5 && cd cdh5
reposync -r cloudera-cdh5
createrepo .
ln -s /var/www/html/cdh5/cloudera-cdh5/RPMS /var/www/html/cdh5/RPMS
# 漫长得等待...
mkdir ../cm && cd cm
reposync -r cloudera-manager
createrepo .

登录后复制

在debugo02,debugo03上测试http访问，并配置下面的repo信息。
[cloudera-cdh5]
# Packages for Cloudera’s Distribution for Hadoop, Version 5, on RedHat or CentOS 6 x86_64
name=Cloudera’s Distribution for Hadoop, Version 5
baseurl=http://debugo01/cdh5
gpgcheck = 0
enabled = 1

[cloudera-manager]
name = Cloudera Manager, Version 5.1.1
baseurl = http://debugo01/cm
gpgcheck = 0
enabled=1
e). 配置cm

sed -i 's/^server_host.*/server_host=debugo01/' /opt/cm-5.1.1/etc/cloudera-scm-agent/config.ini
/opt/cm-5.1.1/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p --scm-host localhost scm scm scm
Enter database password: 
Verifying that we can write to /opt/cm-5.1.1/etc/cloudera-scm-server
Creating SCM configuration file in /opt/cm-5.1.1/etc/cloudera-scm-server
Executing:  /usr/java/latest/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.1.1/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cm-5.1.1/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!

登录后复制

f). 启动cm

/opt/cm-5.1.1/etc/init.d/cloudera-scm-agent start
/opt/cm-5.1.1/etc/init.d/cloudera-scm-server start

登录后复制

3. 安装CDH

a). 通过浏览器访问debugo01，使用admin/admin登录
b). 选择Cloudera Express来创建cluster。express是cloudera的免费版，相比企业版只是少了升级和一些高级特性（http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Installation-Guide/cm5ig_feature_differences.html?scroll=cmfeig_topic_5_1）
c). 接下来Specify hosts for your CDH cluster installation，添加新主机debugo02,并将Currently Managed Hosts中的cm-server主机debugo01也勾选上。
d). Select Repository – Use Parcel， CDH – CDH-5.1.0-1.cdh5.1.0.p0.53
e). Provide SSH login credentials. 使用root并输入root密码
f). Cluster Installation
这里会遇到一个DNS反向解析错误，不能正确解析Cloudera Manager Server主机名的问题，解决方法将host工具改名，安装成功后再修改回来。
mv /usr/bin/host /usr/bin/host.bak
为了加快安装，可以在安装开始后手工删除debugo02、debugo03中/etc/yum.repos.d/新生成的repo文件，让cluster的直接使用本地源。
g). 安装后进入集群配置界面，输入前面创建的cdh数据库。根据需求配置角色。完成查看host和集群状态正常,安装完成。很多细节的配置警告可以根据提示来调整。