ApacheDruid 安装部署⼿册
⼀ Apache Druid 架构
1. Coordinator
监控Historical处理，负责分配gments到指定的服务，确保存在HIstorical中是⾃平衡的
2. Overlord
监控MiddleManager处理和控制数据加载进druid集群；对分配给MiddleManager的摄取任务和协调gments的发布负责
3. Broker
处理来⾃客户端的查询，解析将查询重定向到Historical和MiddleManager，Broker接收到数据从这个⼦查询中，合并这些结果然后返回给查询者
穷蹙
4. Router
提供了Brokers，Overlords，Coordinator的统⼀路由⽹关
5. Historical
gigolos
是⼀个处理存储和历史数据查询查询到⼯作站，Historical处理从deep storage加载过来的gments，对这些gments从broker发出的历史数据的查询做出回应
6. MiddleManager
摄取新数据到集群中；它负责度额外的数据源（新的实时的数据）和发布新的druid gments，是⼀个执⾏提交任务的⼯作节点；提交任务到peon上在⼀个独⽴的JVMs，因为任务的资源和⽇志的隔离，每⼀个Peon使⽤了隔离的JVMS，每⼀个Peon同时每次只能运⾏⼀个task，⼀个MiddleManager有多个peon
7. 额外依赖
Deep storage：⼀个被druid可访问的共享的⽂件存储；⽐如分布式⽂件系统HDFS、S3、⼀个⽹络挂在的⽂件系统；⽤它来存储已经陪摄⼊的任何数据；
Metadata store：⼀个共享的元数据存储，典型的关系型数据库PostgreSql和Mysql；
Zookeeper：⼀个被⽤来了额外服务发现、协调、领导选举的；
⼆ 准备⼯作
1. mysql （作为Druid 的 Metadata Storage ）
创建数据库：
授权：
2. 集群节点规划
IP地址
节点功能部署服务10.0.111.140
Master Server Coordinator、Overlord 10.0.111.141~143
Data Server Historical、MiddleManager 10.0.111.144Query Server Broker 、router CREATE DATABASE druid DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci ;
1grant all privileges on druid .* to druid@'%' identified by 'druid';1
3. druid 访问地址Coordinator 10.0.111.140:8081
Router 10.0.111.144:8888
三 集群配置
1. 下载安装包
在规划的master rver节点下载安装包，并在/opt/tools⽬录下解压
以下操作均需cd /opt/tools/apache-druid-0.18.1/⽬录下进⾏
注以下参数设置要求：
2. 修改common 配置
common.runtime.properties配置⽂件 druid集群以mysql作为元数据存储、HDFS为深度存储
拷贝hdfs相关配置⽂件⾄conf/druid/cluster/_common/⽬录下
修改各个节点druid.host配置
3. 同步druid ⽬录到其他的节点上wget mirrors .tuna .tsinghua .edu /apache /druid /0.18.1/apache -druid -0.18.1-bin .tar .gz
1druid .processing .numMergeBuffers = max (2, druid .processing .numThreads / 4)druid .processing .numThreads = Number of cores - 1 (or 1)druid .rver .http .numThreads = max (10, (Number of cores * 17) / 16 + 2) + 30MaxDirectMemorySize >= druid .processing .buffer .sizeByte *(druid .processing .numMergeBuffers + druid .processing .numThreads + 1)
1
2
3
4vim conf /druid /cluster /_common /common .runtime .properties
1druid .extensions .loadList=["druid-hdfs-storage", "druid-kafka-indexing-rvice", "druid-datasketches","mysql-metadata-storage"]druid .extensions .hadoopDependenciesDir=/opt /tools /druid /hadoop -dependencies druid .host=hadoop10druid .startup .logging .logProperties=true druid .zk .rvice .host=hadoop1:2181,hadoop2:2181,hadoop3:2181,hadoop4:2181,hadoop5:2181druid .zk .paths .ba=/druid druid .metadata .storage .type =mysql druid .metadata .storage .connector .connectURI=jdbc:mysql://10.0.111.134:3306/druid druid .metadata .storage .connector .ur=druid d
ruid .metadata .storage .connector .password=druid # For HDFS:druid .indexer .task .hadoopWorkingPath=/tmp /druid -indexing druid .storage .type =hdfs druid .storage .storageDirectory=hdfs://namervice1/druid /gments # For HDFS:druid .indexer .logs .type =hdfs druid .indexer .logs .directory=hdfs://namervice1/druid /indexing -logs druid .lectors .indexing .rviceName=druid /overlord druid .lectors .coordinator .rviceName=druid /coordinator druid .monitoring .monitors=["org.apache.druid.ics.JvmMonitor"]druid .emitter=logging druid .emitter .logging .logLevel=info druid .indexing .doubleStorage=double druid .rver .hiddenProperties=["druid.s3.accessKey","Key","tor.password"]druid .sql .enable=true druid .lookup .enableLookupSyncOnStartup=fal
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
4. 修改master 节点配置fig配置⽂件
runtime.properties配置⽂件
5. 修改query 节点配置⽂件fig配置⽂件
broker-runtime.properties配置⽂件vim conf /druid /cluster /master /coordinator -overlord /jvm .config 1-rver -Xms8g -Xmx8g -XX:+ExitOnOutOfMemoryError -XX:+UG1GC -Dur .timezone=UTC -Dfile .encoding=UTF -8-Djava .io .tmpdir=var /tmp -Djava .util .logging .manager=org .apache .logging .log4j .jul .LogManager -Dderby .stream .error .file=var /druid /derby .log
1
2
3
4
5
6
7listening
8
9
10vim conf /druid /cluster /master /coordinator -overlord /runtime .properties 1druid .rvice=druid /coordinator druid .plaintextPort=8081druid .coordinator .startDelay=PT10S druid .coordinator .period=PT5S druid .coordinator .asOverlord .enabled=true druid .coordinator .asOverlord .overlordService=druid /overlord druid .indexer .queue .startDelay=PT5S druid .indexer .runner .type =remote druid .indexer .storage .type =metadata
1
2
3
4
5
6
7
8
9vim conf /druid /cluster /query /broker /jvm .config
1-rver -Xms6g -Xmx6g -XX:MaxDirectMemorySize=8g -XX:+ExitOnOutOfMemoryError -Dur .timezone=UTC -Dfile .encoding=UTF -8-Djava .io .tmpdir=var /tmp -Djava .util .logging .manager=org .apache .logging .log4j .jul .LogManager 1
2
3
4
5monitor
6
7
8
9vim conf /druid /cluster /query /broker /runtime .properties
1
router-runtime.properties配置⽂件
6. 修改data rver 节点配置⽂件
1
2
束手无策英语
3
4
5螺丝刀英文
6
7
8
9
10
11vim conf /druid /cluster /query /router /jvm .config
1-rver -Xms1g -Xmx1g -XX:+UG1GC -XX:MaxDirectMemorySize=128m -XX:+ExitOnOutOfMe
moryError -Dur .timezone=UTC -Dfile .encoding=UTF -8-Djava .io .tmpdir=var /tmp -Djava .util .logging .manager=org .apache .logging .log4j .jul .LogManager 1
2
3
4
5
6tbg
7
8
9
10vim conf /druid /cluster /query /router /runtime .properties 1druid .rvice=druid /router druid .plaintextPort=8888druid .router .http .numConnections=50druid .router .http .readTimeout=PT5M dr
uid .router .http .numMaxThreads=100druid .rver .http .numThreads=100druid .router .defaultBrokerServiceName=druid /broker druid .router .coordinatorServiceName=druid /coordinator druid .router .managementProxy .enabled=true
1
2
3
4
5
67
8
cisa9vim conf /druid /cluster /data /historical /jvm .config
1-rver -Xms2g -Xmx2g -XX:MaxDirectMemorySize=8g -XX:+ExitOnOutOfMemoryError -Dur .ti
mezone=UTC -Dfile .encoding=UTF -8-Djava .io .tmpdir=var /tmp -Djava .util .logging .manager=org .apache .logging .log4j .jul .LogManager 1
2
3
4ttm
5
6
7
8
9
historical-runtime.properties配置⽂件fig
middleManager-runtime.properties
四 集群启动关闭
1. 后台启动关闭
2. 服务⽅式启动关闭vim conf /druid /cluster /data /historical /runtime .properties 1druid .rvice=druid /historical druid .plaintextPort=8083druid .rver .http .numThreads=57druid .processing .buffer .sizeBytes=200000000druid .processing .numMergeBuffers=5druid .processing .numThreads=23druid .processing .tmpDir=var /druid /processing druid .gmentCache .locations=[{"path":"var/druid/gment-cache","maxSize":3000000000}]druid .rver .maxSize=3000000000druid .historical .cache .uCache=true druid .historical .cache .populateCache=true druid .cache .type =caffeine druid .cache .sizeInBytes=256000000
1
2
3
4
5
6
7
8
9
10
11
12
英语国际音标教学视频13vim conf /druid /cluster /data /middleManager /jvm .config 1-rver -Xms128m -Xmx128m -XX:+ExitOnOutOfMemoryError -Dur .timezone=UTC -Dfile .encoding=UTF -8-Djava .io .tmpdir=var /tmp -Djava .util .logging .manager=org .apache .logging .log4j .jul .LogManager 1
2
3
4
5
6
7
8vim conf /druid /cluster /data /middleManager /jvm .config 1druid nohup ./bin /start -cluster -master -no -zk -rver start >/dev /null 2>&1 & nohup ./bin /start -cluster -query -rver start >/dev /null 2>&1 &nohup ./bin /start -cluster -data -rver start >/dev /null 2>&1 &./bin /rvice --down
1
2
3
4
