首页 > 运维 > CentOS > 正文

CentOS中HBase如何进行资源调度

星降
发布: 2025-07-30 10:22:20
原创
823人浏览过

centos系统中,hbase的资源调度主要依赖于yarn(yet another resource negotiator)来实现。以下是hbase在centos中进行资源调度的基本步骤:

1. 安装和配置YARN

首先,确保你的CentOS系统上已经安装了Hadoop和YARN。如果没有安装,可以参考Hadoop官方文档进行安装和配置。

安装Hadoop

wget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.1/hadoop-3.3.1.tar.gz
tar -xzvf hadoop-3.3.1.tar.gz -C /usr/local/
登录后复制

配置Hadoop

编辑/usr/local/hadoop/etc/hadoop/core-site.xml:

<<span>configuration></span>
    <<span>property></span>
        <<span>name></span>fs.defaultFS</<span>name></span>
        <<span>value></span>hdfs://localhost:9000</<span>value></span>
    </<span>property></span>
</<span>configuration></span>
登录后复制

编辑/usr/local/hadoop/etc/hadoop/hdfs-site.xml:

<<span>configuration></span>
    <<span>property></span>
        <<span>name></span>dfs.replication</<span>name></span>
        <<span>value></span>1</<span>value></span>
    </<span>property></span>
</<span>configuration></span>
登录后复制

编辑/usr/local/hadoop/etc/hadoop/yarn-site.xml:

百度文心百中
百度文心百中

百度大模型语义搜索体验中心

百度文心百中 22
查看详情 百度文心百中
<<span>configuration></span>
    <<span>property></span>
        <<span>name></span>yarn.nodemanager.aux-services</<span>name></span>
        <<span>value></span>mapreduce_shuffle</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>yarn.nodemanager.resource.memory-mb</<span>name></span>
        <<span>value></span>4096</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>yarn.nodemanager.resource.cpu-vcores</<span>name></span>
        <<span>value></span>4</<span>value></span>
    </<span>property></span>
</<span>configuration></span>
登录后复制

2. 启动Hadoop集群

启动HDFS和YARN:

start-dfs.sh
start-yarn.sh
登录后复制

3. 配置HBase使用YARN

编辑HBase的配置文件/usr/local/hbase/conf/hbase-site.xml,添加以下配置:

<<span>configuration></span>
    <<span>property></span>
        <<span>name></span>hbase.rootdir</<span>name></span>
        <<span>value></span>hdfs://localhost:9000/hbase</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.cluster.distributed</<span>name></span>
        <<span>value></span>true</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.zookeeper.quorum</<span>name></span>
        <<span>value></span>localhost</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.zookeeper.property.dataDir</<span>name></span>
        <<span>value></span>/usr/local/hbase/zookeeper</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.master.info.port</<span>name></span>
        <<span>value></span>16000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.info.port</<span>name></span>
        <<span>value></span>16020</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.handler.count</<span>name></span>
        <<span>value></span>100</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.global.memstore.size</<span>name></span>
        <<span>value></span>0.4</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.wal.codec</<span>name></span>
        <<span>value></span>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.blocksize</<span>name></span>
        <<span>value></span>64M</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.writer.impl</<span>name></span>
        <<span>value></span>org.apache.hadoop.hbase.regionserver.wal.HLogWriterFactory</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.count</<span>name></span>
        <<span>value></span>1</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.impl</<span>name></span>
        <<span>value></span>org.apache.hadoop.hbase.regionserver.wal.Syncer</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint</<span>name></span>
        <<span>value></span>100</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.interval</<span>name></span>
        <<span>value></span>300000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>60000</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.size</<span>name></span>
        <<span>value></span>104857600</<span>value></span>
    </<span>property></span>
    <<span>property></span>
        <<span>name></span>hbase.regionserver.hlog.syncer.checkpoint.timeout</<span>name></span>
        <<span>value></span>6000
登录后复制

以上就是CentOS中HBase如何进行资源调度的详细内容,更多请关注php中文网其它相关文章!

最佳 Windows 性能的顶级免费优化软件
最佳 Windows 性能的顶级免费优化软件

每个人都需要一台速度更快、更稳定的 PC。随着时间的推移,垃圾文件、旧注册表数据和不必要的后台进程会占用资源并降低性能。幸运的是,许多工具可以让 Windows 保持平稳运行。

下载
来源:php中文网
本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系admin@php.cn
最新问题
开源免费商场系统广告
热门教程
更多>
最新下载
更多>
网站特效
网站源码
网站素材
前端模板
关于我们 免责申明 意见反馈 讲师合作 广告合作 最新更新 English
php中文网:公益在线php培训,帮助PHP学习者快速成长!
关注服务号 技术交流群
PHP中文网订阅号
每天精选资源文章推送
PHP中文网APP
随时随地碎片化学习

Copyright 2014-2025 https://www.php.cn/ All Rights Reserved | php.cn | 湘ICP备2023035733号