LinuxFCSAN存储服务器搭建SCSITarget（HBA卡）
有⼀台⾼配的R730xd服务器，带有满配⼤容量硬盘以及SSD和两张Qlogic的HBA卡。有天朋友讨论能不能把这台服务器配置成⼀台FC 存储服务器推给vmware或者其他客户端使⽤。这个我之前没有做过，很感兴趣，所以我们花了⼀个多星期的进⾏研究最后成功实现，故我们把过程和遇到的问题及解决⽅法写出来分享，希望帮到其他有这个需求的运维。
刚开始Google了良久，最后决定思路是：在Linux上部署zfs系统，把SSD定义为zfs的缓存磁盘，然后把HBA卡设置为target模式，使⽤targetcli在zfs下的⼀个⽂件块推送⾄vmware或者其他客户端。
寻找良好兼容性的Linux发⾏版我们搞了很久，发现CentOS6使⽤的2.6内核不能把HBA卡设置为target模式；使⽤CentOS7.1能⽀持设置为target模式，但在targetcli下⽆法看到qla2xxx的路径；⾃⼰试过在CentOS6下编译3.x或者4.x的内核也是在targetcli下⽆法看到
qla2xxx的路径，⼀直⽆法解决。后⾯安装Ubuntu15.10，就没有这个问题，但是在装zfs的时候发现有个包没有兼容内核，⽆法安装。最后是使⽤了Ubuntu14.04 LTS版本才顺利完成。详见本⽂最后的报错分享。
最后的环境：DellR730xd+QLE2650(8G)+Intel SSD 750+Ubuntu14.04LTS
先安装ZFS⽂件系统⽀持
sudo su -
apt-add-repository --yes ppa:zfs-native/stable
apt-get update
apt-get install debootstrap spl-dkms zfs-dkms ubuntu-zfs
团委工作总结#系统启动挂载zfs模块
vi /etc/rc.local
/sbin/modprobe zfs
#安装SSD驱动
apt-get install git-core build-esntial libncurs5-dev
git clone
cd nvme-cli
make && make install
#创建pool
zpool create zfspool /dev/sdb
#添加缓存盘
zpool add zfspool cache /dev/nvme0n1
安装targetcli
apt-get install targetcli
设置HBA卡为target mode
vi /etc/modprobe.f
options qla2xxx qlini_mode="disabled"
重启，检查targetcli是否正确加载qla2xxx模块，下⾯有qla2xxx输出即为正常。
targetcli
/> ls
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- fileio ............................................... [0 Storage Object]
| o- iblock ............................................... [0 Storage Object]
| o- pscsi ................................................ [0 Storage Object]
| o- rd_dr ................................................ [0 Storage Object]
| o- rd_mcp ............................................... [0 Storage Object]
o- ib_srpt ........................................................ [0 Target]
o- iscsi .......................................................... [0 Target]
o- loopback ....................................................... [0 Target]
o- qla2xxx ........................................................ [0 Target]
/>
创建backstore
#官⽅⽂档说明设备、⽂件、闪存盘可以作为backstore，此处我创建⽂件型backstore /> cd backstores/
/backstores> fileio/ create name=test file_or_dev=/zfspool/test size=1T
Using buffered mode.
Created fileio test.
/backstores> ls
o- backstores ............................................................ [...]
o- fileio ................................................. [1 Storage Object]
| o- test ............................... [1.0T, /zfspool/test, not in u]
o- iblock ................................................. [0 Storage Object]
o- pscsi .................................................. [0 Storage Object]
o- rd_mcp ................................................. [0 Storage Object]
创建target，插了两块HBA，可以看到有两个WWN
/> qla2xxx/ info
Fabric module name: qla2xxx
ConfigFS path: /sys/kernel/config/target/qla2xxx
Allowed WWNs list (free type): 21:00:00:24:ff:0e:1e:30, 21:00:00:24:ff:0e:7c:f5 Fabric module specfile: /var/target/fabric/qla2xxx.spec
Fabric module features: acls
Corresponding kernel module: tcm_qla2xxx
/> qla2xxx/ create 21:00:00:24:ff:0e:1e:30
Created target 21:00:00:24:ff:0e:1e:30.
/> qla2xxx/ create 21:00:00:24:ff:0e:7c:f5
Created target 21:00:00:24:ff:0e:7c:f5.
推LUN
特色教育/> cd qla2xxx/21:00:00:24:ff:0e:1e:30/
/qla2xxx/4:ff:0e:1e:30> luns/ create /backstores/fileio/test
Selected LUN 0.
Created LUN 0.长沙晚报
/qla2xxx/4:ff:0e:1e:30> cd ../
/
qla2xxx> cd 21:00:00:24:ff:0e:
21:00:00:24:ff:0e:1e:30/ 21:00:00:24:ff:0e:7c:f5/ .............path
/qla2xxx> cd 21:00:00:24:ff:0e:7c:f5/
/qla2xxx/4:ff:0e:7c:f5> luns/ create /backstores/fileio/test
Selected LUN 0.
Created LUN 0.
设置接⼊权限，接⼊端也是两个HBA卡，所以每个target设置两个ACL
/qla2xxx/4:ff:0e:7c:f5> acls/ create 20:00:74:e6:e2:65:c8:0b
Created Node ACL for 20:00:74:e6:e2:65:c8:0b
Created mapped LUN 0.
/qla2xxx/4:ff:0e:7c:f5> acls/ create 20:01:74:e6:e2:65:c8:0b
Created Node ACL for 20:01:74:e6:e2:65:c8:0b
Created mapped LUN 0.
/qla2xxx/4:ff:0e:7c:f5> cd ..
/qla2xxx> cd 21:00:00:24:ff:0e:1e:30/
/qla2xxx/4:ff:0e:1e:30> acls/ create 20:00:74:e6:e2:65:c8:0b
Created Node ACL for 20:00:74:e6:e2:65:c8:0b
Created mapped LUN 0.
/qla2xxx/4:ff:0e:1e:30> acls/ create 20:01:74:e6:e2:65:c8:0b
Created Node ACL for 20:01:74:e6:e2:65:c8:0b
Created mapped LUN 0.
最后整个targetcli的树⽬录是这样的
/
> ls
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- fileio ............................................... [1 Storage Object]
| | o- test .................................... [1.0T, /zfspool/test, in u]
| o- iblock ............................................... [0 Storage Object]
| o- pscsi ................................................ [0 Storage Object]
| o- rd_mcp ............................................... [0 Storage Object]
o- ib_srpt ....................................................... [0 Targets]
o- iscsi ......................................................... [0 Targets]
o- loopback ...................................................... [0 Targets]
o- qla2xxx ....................................................... [2 Targets]
| o- 21:00:00:24:ff:0e:1e:30 ....................................... [enabled]
| | o- acls ......................................................... [2 ACLs]
| | | o- 20:00:74:e6:e2:65:c8:0b .............................. [1 Mapped LUN]
| | | | o- mapped_lun0 ........................................... [lun0 (rw)]
| | | o- 20:01:74:e6:e2:65:c8:0b .............................. [1 Mapped LUN]
| | | o- mapped_lun0 ........................................... [lun0 (rw)]
| | o- luns .......................................................... [1 LUN]
| | o- lun0 .................................. [fileio/test (/zfspool/test)]
| o- 21:00:00:24:ff:0e:7c:f5 ....................................... [enabled]
| o- acls ......................................................... [2 ACLs]
| | o- 20:00:74:e6:e2:65:c8:0b .............................. [1 Mapped LUN]
| | | o- mapped_lun0 ........................................... [lun0 (rw)]
| | o- 20:01:74:e6:e2:65:c8:0b .............................. [1 Mapped LUN]
| | o- mapped_lun0 ........................................... [lun0 (rw)]
| o- luns .......................................................... [1 LUN]
| o- lun0 .................................. [fileio/test (/zfspool/test)]
o- tcm_fc ........................................................ [0 Targets]
o- usb_gadget .................................................... [0 Targets]
o- vhost ......................................................... [0 Targets]
最后别忘了保存，在保存的过程中遇到以下错误，查询资料发现是targetcli的⼀个bug，也给出了修复⽅法。
/> saveconfig
Save configuration? [Y/n]: y
Saving new startup configuration
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/configshell/shell.py", line 990, in run_interactive
lf._cli_loop()
File "/usr/lib/python2.7/dist-packages/configshell/shell.py", line 820, in _cli_loop
lf.run_cmdline(cmdline)新党章学习心得体会
File "/usr/lib/python2.7/dist-packages/configshell/shell.py", line 934, in run_cmdline
lf._execute_command(path, command, pparams, kparams)
File "/usr/lib/python2.7/dist-packages/configshell/shell.py", line 909, in _execute_command
result = ute_command(command, pparams, kparams)
File "/usr/lib/python2.7/dist-packages/targetcli/ui_node.py", line 104, in execute_command
pparams, kparams)
File "/usr/lib/python2.7/dist-packages/configshell/node.py", line 1416, in execute_command正弦图像
result = method(*pparams, **kparams)
File "/usr/lib/python2.7/dist-packages/targetcli/ui_node.py", line 123, in ui_command_saveconfig CliConfig.save_running_config()
File "/usr/lib/python2.7/dist-packages/targetcli/cli_config.py", line 65, in save_running_config
config.load_live()
File "/usr/lib/python2.7/dist-packages/rtslib/config.py", line 565, in load_live
source=source, allow_new_attrs=True)
File "/usr/lib/python2.7/dist-packages/rtslib/config.py", line 190, in _load_par_tree
token = lf.validate_obj(token, cur)
File "/usr/lib/python2.7/dist-packages/rtslib/config.py", line 377, in validate_obj
valid_value = lf.validate_val(valid_token['key'][1], id_type)
File "/usr/lib/python2.7/dist-packages/rtslib/config.py", line 355, in validate_val
% (val_type, value))
ConfigError: Unknown value type 'qla2xxx_wwn' when validating 21:00:00:24:ff:0e:7c:f5
vi /usr/share/pyshared/rtslib/config.py
#第349⾏增加
elif val_type == 'naa':
if is_valid_wwn('naa', value):
valid_value = value
+ elif val_type == 'qla2xxx_wwn':
水浒武功排名
+ if is_valid_wwn('qla2xxx_wwn', value):
+ valid_value = value
elif val_type == 'backend':
if is_valid_backend(value, parent):
valid_value = value
vi /usr/share/pyshared/rtslib/utils.py
#第562⾏增加
and re.match(
"[0-9A-Fa-f]{8}(-[0-9A-Fa-f]{4}){3}-[0-9A-Fa-f]{12}$", wwn):
清算报表return True
+ elif wwn_type == 'qla2xxx_wwn' \
+ and re.match(
+ "[0-9A-Fa-f]{2}(:[0-9A-Fa-f]{2}){7}$", wwn):
简笔画皮卡丘
+ return True
el:
return Fal
再次保存，就不出错了。
附上这个环境下的虚拟机与运⾏在存储上和vmware vsan下的虚拟机性能对⽐。
