buckets

Ceph对象存储部署

Ceph对象存储部署与使⽤

1.安装client操作系统

（1）虚拟机基础设置

在VMware中创建⼀台虚拟机，操作系统为CentOS-7-x86_64-DVD-1908，硬盘⼤⼩为20G，并将虚拟机的名字设为client，如图7-8所⽰。

图7-8虚拟机配置

（2）虚拟机⽹络设置

为虚拟机配置主机名：client。设置⽹络为NAT模式，配置IP地址：192.168.100.100，⼦⽹掩码为255.255.255.0，默认⽹关为

192.168.100.2，DNS服务器为192.168.100.2，使虚拟机可以访问Internet。

2.配置Ceph对象存储

（1）在ceph-1节点上安装ceph对象⽹关软件包

Ceph对象存储使⽤Ceph对象⽹关守护进程（radosgw），所以在使⽤对象存储之前，我们需要先安装配置好对象⽹关RGW。

CephRGW的FastCGI⽀持多种Web服务器作为前端，例如Nginx、Apache2等。从Ceph

Hammer版本开始，使⽤ceph-deploy部署时将会默认使⽤内置的civetweb作为前端，区别在于配置的⽅式不同，我们这⾥采⽤默认civetweb⽅

式安装配置RGW。

[root@ceph-1~]#cd/opt/osd

[root@ceph-1osd]#ceph-deployrgwcreateceph-1

（2）编辑pool⽂件

.rgw

.

.l

.

.s

.

.

.log

.intent-log

.usage

.urs

.

.

.

（3）编辑创建和配置pool的脚本⽂件

此处可以通过脚本⼀键创建对象存储需要使⽤的pool。

[root@ceph-1osd]#vi/root/create_

#!/bin/bash

PG_NUM=8

PGP_NUM=8

SIZE=3

foriin`cat/root/pool`

do

cephosdpoolcreate$i$PG_NUM

cephosdpoolt$isize$SIZE

done

foriin`cat/root/pool`

do

cephosdpoolt$ipgp_num$PGP_NUM

done

（4）运⾏脚本⽂件，创建对象存储所使⽤的所有pool

[root@ceph-1osd]#chmod+x/root/create_

[root@ceph-1osd]#/root/create_

pool'.rgw'created

tpool5sizeto3

pool'.'alreadyexists

tpool1sizeto3

pool'.l'created

tpool6sizeto3

pool'.'created

tpool7sizeto3

pool'.s'created

tpool8sizeto3

pool'.'created

tpool9sizeto3

pool'.'created

tpool10sizeto3

pool'.log'created

tpool11sizeto3

pool'.intent-log'created

tpool12sizeto3

pool'.usage'created

tpool13sizeto3

pool'.urs'created

tpool14sizeto3

pool'.'created

tpool15sizeto3

pool'.'created

tpool16sizeto3

pool'.'created

tpool17sizeto3

tpool5pgp_numto8

tpool1pgp_numto8

tpool6pgp_numto8

tpool7pgp_numto8

tpool8pgp_numto8

tpool9pgp_numto8

tpool10pgp_numto8

tpool11pgp_numto8

tpool12pgp_numto8

tpool13pgp_numto8

tpool14pgp_numto8

tpool15pgp_numto8

tpool16pgp_numto8

tpool17pgp_numto8

（5）测试是否能访问ceph集群

在使⽤脚本⼀键创建好所需要的pool之后，需要进⾏ceph集群的测试，防⽌实验过程中出现错误。

[root@ceph-1osd]#cp/var/lib/ceph/radosgw/-1/keyring/etc/ceph/g

[root@ceph-1osd]#ceph-s-k/var/lib/ceph/radosgw/-1/-1

cluster:

id:68ecba50-862d-482e-afe2-f95961ec3323

health:HEALTH_OK

rvices:

mon:3daemons,quorumceph-1,ceph-2,ceph-3(age21m)

mgr:ceph-1(active,since21m)

osd:3osds:3up(since21m),3in(since7d)

rgw:1daemonactive(ceph-1)

data:

pools:17pools,136pgs

objects:187objects,1.2KiB

usage:3.0GiBud,294GiB/297GiBavail

pgs:136active+clean

3.使⽤S3

API访问Ceph对象存储

（1）在ceph-1节点创建radosgw⽤户

[root@ceph-1osd]#radosgw-adminurcreate--uid=radosgw--display-name="radosgw"

输出结果如下：

{

"ur_id":"radosgw",

"display_name":"radosgw",

"email":"",

"suspended":0,

"max_buckets":1000,

"suburs":[],

"keys":[

{

"ur":"radosgw",

"access_key":"XTAA1VRTXIKIEH89GUBG",

"cret_key":"P8i5dC6jeOpVnlK9qKYYN2enLjFcz0fPZne9sxVE"

}

],

"swift_keys":[],

"caps":[],

"op_mask":"read,write,delete",

"default_placement":"",

"default_storage_class":"",

"placement_tags":[],

"bucket_quota":{

"enabled":fal,

"check_on_raw":fal,

"max_size":-1,

"max_size_kb":0,

"max_objects":-1

},

"ur_quota":{

"enabled":fal,

"check_on_raw":fal,

"max_size":-1,

"max_size_kb":0,

"max_objects":-1

},

"temp_url_keys":[],

"type":"rgw",

"mfa_ids":[]

}

（2）在client节点安装bind

[root@client~]#mkdir/opt/bak

[root@client~]#cd/etc/.d

[root@.d]#mv*/opt/bak

将通过SFTP复制到client节点的/etc/.d⽬录中。

[root@.d]#ls

[root@.d]#yumcleanall

[root@.d]#yummakecache

[root@.d]#yum-yinstallbind

（3）编辑bind主配置⽂件

[root@client~]#vi/etc/

修改以下配置：

listen-onport53{127.0.0.1;192.168.100.100;};

allow-query{localhost;192.168.100.0/24;};

添加以下配置：

zone""IN{

typemaster;

file"";

allow-update{none;};

};

（4）编辑域的区域配置⽂件

[root@client~]#vi/var/named/

@.(

20191120

10800

3600

3600000

86400)

@.

@86400INA192.168.100.101

*86400INCNAME@

（5）检查配置⽂件

[root@client~]#named-checkconf/etc/

[root@client~]#/var/named/

/IN:loadedrial20191120

OK

（6）启动bind服务

[root@client~]#systemctlstartnamed

[root@client~]#systemctlenablenamed

Createdsymlinkfrom/etc/systemd/system//eto/usr/lib/systemd/system/e.

（7）编辑⽹卡配置⽂件

在配置⽂件中，将DNS服务器指向client⾃⼰的IP地址。

[root@client~]#vi/etc/sysconfig/network-scripts/ifcfg-ens32

DNS1=192.168.100.100

（8）编辑/etc/

在配置⽂件中，将DNS服务器指向client⾃⼰的IP地址。

[root@client~]#vi/etc/

namerver192.168.100.100

（9）安装nslookup并测试DNS配置

[root@client~]#yum-yinstallbind-utils

[root@client~]#nslookup

>

Server:192.168.100.100

Address:192.168.100.100#53

onicalname=.

Name:

Address:192.168.100.101

>exit

（10）安装s3cmd

[root@client~]#ls

[root@client~]#yum-yinstallunzippython-dateutil

[root@client~]#

……

（11）配置s3cmd

[root@client~]#cds3cmd-2.0.2

[root@clients3cmd-2.0.2]#./s3cmd--configure

EnternewvaluesoracceptdefaultsinbracketswithEnter.

Refertourmanualfordetaileddescriptionofalloptions.

hememptyforusingtheenvvariables.

AccessKey:XTAA1VRTXIKIEH89GUBG（输⼊ceph-1节点显⽰的access_key）

SecretKey:P8i5dC6jeOpVnlK9qKYYN2enLjFcz0fPZne9sxVE（输⼊ceph-1节点显⽰的cret_key）

DefaultRegion[US]:（直接回车）

U""forS3EndpointandnotmodifyittothetargetAmazonS3.

S3Endpoint[]::7480

U"%(bucket)"tothetargetAmazonS3."%(bucket)s"and"%(location)s"varscanbeud

ifthetargetS3systemsupportsdnsbadbuckets.

DNS-stylebucket+hostname:porttemplateforaccessingabucket[%(bucket)]:%(bucket).ceph-

:7480

Encryptionpasswordisudtoprotectyourfilesfromreading

byunauthorizedpersonswhileintransfertoS3

Encryptionpassword:（直接回车）

PathtoGPGprogram[/usr/bin/gpg]:（直接回车）

WhenusingcureHTTPSprotocolallcommunicationwithAmazonS3

thodis

slowerthanplainHTTP,andcanonlybeproxiedwithPython2.7ornewer

UHTTPSprotocol[Yes]:no

OnsomenetworksallinternetaccessmustgothroughaHTTPproxy.

Tryttingithereifyoucan'tconnecttoS3directly

HTTPProxyrvername:（直接回车）

Newttings:

AccessKey:XTAA1VRTXIKIEH89GUBG

SecretKey:P8i5dC6jeOpVnlK9qKYYN2enLjFcz0fPZne9sxVE

DefaultRegion:US

S3Endpoint::7480

DNS-stylebucket+hostname:porttemplateforaccessingabucket:%(bucket).:7480

Encryptionpassword:

PathtoGPGprogram:/usr/bin/gpg

UHTTPSprotocol:Fal

HTTPProxyrvername:

HTTPProxyrverport:0

Testaccesswithsuppliedcredentials?[Y/n]n

Savettings?[y/N]y

Configurationsavedto'/root/.s3cfg'

（12）显⽰存储桶

[root@clients3cmd-2.0.2]#./s3cmdls

（13）创建存储桶bucket

[root@clients3cmd-2.0.2]#./s3cmdmbs3://bucket

Bucket's3://bucket/'created

[root@clients3cmd-2.0.2]#./s3cmdls

2019-11-2307:45s3://bucket

（14）上传⽂件到存储桶

[root@clients3cmd-2.0.2]#./s3cmdput/etc/hostss3://bucket

WARNING:ngMIMEtypesbadonfileextensions.

upload:'/etc/hosts'->'s3://bucket/hosts'[1of1]

158of158100%in1s107.77B/sdone

[root@clients3cmd-2.0.2]#./s3cmdlss3://bucket

2019-11-2307:46158s3://bucket/hosts

4.使⽤Swift

API访问Ceph对象存储

（1）创建Swift⽤户

要通过Swift访问对象⽹关，需要Swift⽤户,我们创建radosgw:swift作为⼦⽤户。在ceph-1节点创建radosgw⽤户的⼦⽤户

radosgw:swift。

[root@ceph-1osd]#radosgw-adminsuburcreate--uid=radosgw--subur=radosgw:swift--display-name="radosgw-sub"--

access=full

输出结果如下：

{

"ur_id":"radosgw",

"display_name":"radosgw",

"email":"",

"suspended":0,

"max_buckets":1000,

"suburs":[

{

"id":"radosgw:swift",

"permissions":"full-control"

}

],

"keys":[

{

"ur":"radosgw",

"access_key":"XTAA1VRTXIKIEH89GUBG",

"cret_key":"P8i5dC6jeOpVnlK9qKYYN2enLjFcz0fPZne9sxVE"

}

],

"swift_keys":[

{

"ur":"radosgw:swift",

"cret_key":"8uocgTs9CO3tWN8oSc2MDmGPodeotcKUr4454i37"

}

],

"caps":[],

"op_mask":"read,write,delete",

"default_placement":"",

"default_storage_class":"",

"placement_tags":[],

"bucket_quota":{

"enabled":fal,

"check_on_raw":fal,

"max_size":-1,

"max_size_kb":0,

"max_objects":-1

},

"ur_quota":{

"enabled":fal,

"check_on_raw":fal,

"max_size":-1,

"max_size_kb":0,

"max_objects":-1

},

"temp_url_keys":[],

"type":"rgw",

"mfa_ids":[]

}

注意：返回的Json值中，我们要记住swift_keys中的cret_key因为下边我们测试访问Swift接⼝时需要使⽤。

（2）在client节点安装swift客户端

[root@client~]#yum-yinstallpython-tuptools

[root@client~]#easy_installpip

[root@client~]#pipinstall--upgradetuptools

[root@client~]#pipinstall--upgradepython-swiftclient

（3）使⽤swift列出容器（存储桶）列表

Bucket

注意：192.168.100.101可以替换为admin，这⾥为admin-node节点IP，端⼝默认7480，若已修改端⼝号，这⾥也需要对应修改⼀下。密

钥Key为上边返回值中的cret_key。

（4）创建容器container

bucket

container

（5）将⽂件上传到容器container中

（6）修改端⼝

如果我们想修改7480端⼝为其他值时，ceph也是⽀持的，通过修改Ceph配置⽂件更改默认端⼝，然后重启Ceph对象⽹关即可。例如我们

修改端⼝为80。

修改Ceph配置⽂件

vi/etc/ceph/

在[global]节点下增加

[]

rgw_frontends="civetwebport=80"

e