MongoDB分布式文件存储环境搭建

Jul 30, 2019

参考这里、这里和这里

环境

  1. 三台已关闭防火墙的 CentOS

     版本：
     CentOS Linux release 7.6.1810 (Core)
 IP：
     192.168.3.115
     192.168.3.116
     192.168.3.117
 关闭防火墙：
     sudo systemctl stop firewalld.service	#停止防火墙
     sudo systemctl disable firewalld.service	#禁止开机启动

  2. MongoDB

     版本：
     MongoDB shell version v4.0.10

准备 — 三台机器均执行

  1. 将 MongoDB 解压到 /usr/local/mongodb 目录下

  2. 创建目录

     cd /usr/local/mongodb
 mkdir conf log data data/config data/shard{1,2,3}
  3. 在 conf 目录下创建五个配置文件

配置文件分别如下

config.conf

dbpath=/usr/local/mongodb/data/config
logpath=/usr/local/mongodb/log/config.log
port=27018
logappend=true
maxConns=5000
#副本集名称
replSet=configs
#置参数为true
configsvr=true
#允许任意机器连接
bind_ip=0.0.0.0

shard1.conf

dbpath=/usr/local/mongodb/data/shard1
logpath=/usr/local/mongodb/log/shard1.log
port=27001
logappend=true
fork=true
maxConns=5000
storageEngine=mmapv1
shardsvr=true
replSet=shard1
bind_ip=0.0.0.0

shard2.conf

dbpath=/usr/local/mongodb/data/shard2 
logpath=/usr/local/mongodb/log/shard2.log 
port=27002 
logappend=true
fork=true
maxConns=5000
storageEngine=mmapv1
shardsvr=true
replSet=shard2 
bind_ip=0.0.0.0

shard3.conf

dbpath=/usr/local/mongodb/data/shard3 
logpath=/usr/local/mongodb/log/shard3.log 
port=27003 
logappend=true
fork=true
maxConns=5000
storageEngine=mmapv1
shardsvr=true
replSet=shard3 
bind_ip=0.0.0.0

mongos.conf

systemLog:
  destination: file
  logAppend: true
  path: /usr/local/mongodb/log/mongos.log
processManagement:
  fork: true
net:
  port: 27017
  bindIp: 0.0.0.0
#监听的配置服务器,只能有1个或者3个 configs为配置服务器的副本集名字
sharding:
   configDB: configs/192.168.3.115:27018,192.168.3.116:27018,192.168.3.117:27018

配置服务器 Config Server

  1. 启动 — 三台机器均执行

     cd /usr/local/mongodb/bin/
 nohup ./mongod -f /usr/local/mongodb/conf/config.conf &

  2. 配置 — 任意一台执行

     ./mongo --port 27018
 # 创建副本集
 config={_id:"configs",members:[{_id:0,host:"192.168.3.115:27018"},{_id:1,host:"192.168.3.116:27018"},{_id:2,host:"192.168.3.117:27018"}]}
 # 初始化副本集
 rs.initiate(config)
 exit

副本集 Replica Sets

  1. 启动 — 三台机器均执行

     ./mongod -f /usr/local/mongodb/conf/shard1.conf
 ./mongod -f /usr/local/mongodb/conf/shard2.conf
 ./mongod -f /usr/local/mongodb/conf/shard3.conf

  2. 配置 — 不能在仲裁节点上配置副本集

在 192.168.3.115 机器上执行：

./mongo --port 27002
config={_id:'shard2',members:[{_id:0,host:'192.168.3.115:27002',priority:2},{_id:1,host:'192.168.3.116:27002',arbiterOnly:true},{_id:2,host:'192.168.3.117:27002',priority:1}]}
rs.initiate(config)
rs.status()
exit

./mongo --port 27003
config={_id:'shard3',members:[{_id:0,host:'192.168.3.115:27003',priority:1},{_id:1,host:'192.168.3.116:27003',priority:2},{_id:2,host:'192.168.3.117:27003',arbiterOnly:true}]}
rs.initiate(config)
rs.status()
exit

在 192.168.3.116 机器上执行：

./mongo --port 27001
config={_id:'shard1',members:[{_id:0,host:'192.168.3.115:27001',arbiterOnly:true},{_id:1,host:'192.168.3.116:27001',priority:1},{_id:2,host:'192.168.3.117:27001',priority:2}]}
rs.initiate(config)
rs.status()
exit

前端路由 Routers

  1. 启动 — 三台机器均执行

     ./mongos -f /usr/local/mongodb/conf/mongos.conf

  2. 配置 — 任意一台执行

     ./mongo --port 27017
 mongos> use admin
 mongos> sh.addShard("shard1/192.168.3.115:27001,192.168.3.116:27001,192.168.3.117:27001")
 mongos> sh.addShard("shard2/192.168.3.115:27002,192.168.3.116:27002,192.168.3.117:27002")
 mongos> sh.addShard("shard3/192.168.3.115:27003,192.168.3.116:27003,192.168.3.117:27003")
 mongos> sh.status()

测试

  1. 设置分片 chunk 大小 设置块大小为 1M 是方便实验，不然需要插入海量数据

     mongos> use config
 mongos> db.settings.save({"_id":"chunksize","value":1})

  2. 模拟写入数据

     mongos> use mytest
 mongos> for(i=1;i<=50000;i++){db.user.insert({"id":i,"name":"testname"+i})}

  3. 创建索引

     mongos> db.user.createIndex({"id":1})

  4. 启用分片

     mongos> sh.enableSharding("mytest")

  5. 根据 id 对 user 表进行分片

     mongos> sh.shardCollection("mytest.user",{"id":1})

  6. 查看分片情况

     mongos> sh.status()

  7. 删除当前数据库

     mongos> db.dropDatabase()

问题及解决

  1. 报错 MongoDB caused by :: No route to host 关闭防火墙
  2. 启动顺序 config —> shard(1,2,3) —> mongos