PATHpilotのブログ

Objectノードを構成する

NFSノードがインストールできたので、Objectノードをインストールする。

ノード構成は以下のとおり。
NSD Servers:
snode-0  192.168.1.91
snode-1  192.168.1.92
CES Nodes:
  nfsgroup1   192.168.1.98 (CES IP)　<- NFS access用
    pnode-1     192.168.1.96
    pnode-2     192.168.1.97
  s3group1    192.168.1.108 (CES IP)  <- S3 access用
    pnode-s1    192.168.1.106
    pnode-s2    192.168.1.107

見ての通り、CES Nodesは2つのGroupに分けて、それぞれをNFS、S3アクセス用に設定する。すべてのCES NodesにはNFSとOBJの両方のProtocol Serviceがインストールされる。CES Nodeに対して選択的にProtocolをインストールすることはできない。

pnode-s1とpnode-s2を新たに用意してClusterに追加する。
[root@snode-0 gpfsfs01]# mmaddnode -N pnode-s1,pnode-s2

2020年  4月  1日 水曜日 13:41:48 JST: mmaddnode: Processing node pnode-s1

2020年  4月  1日 水曜日 13:41:51 JST: mmaddnode: Processing node pnode-s2

mmaddnode: Command successfully completed

mmaddnode: Warning: Not all nodes have proper GPFS license designations.

    Use the mmchlicense command to designate licenses as needed.

mmaddnode: Propagating the cluster configuration data to all

  affected nodes.  This is an asynchronous process.

[root@snode-0 gpfsfs01]#

ライセンスが設定されていないと注意を受けた。
[root@snode-0 gpfsfs01]# mmlsnode

===============================================================================| Warning:                                                                    |
|   This cluster contains nodes that do not have a proper GPFS license        |
|   designation.  This violates the terms of the GPFS licensing agreement.    |
|   Use the mmchlicense command and assign the appropriate GPFS licenses      |
|   to each of the nodes in the cluster.  For more information about GPFS     |
|   license designation, see the Concepts, Planning, and Installation Guide.  |
===============================================================================

GPFS nodeset    Node list
-------------   -------------------------------------------------------
   gpfscluster1     snode-1 snode-0 pnode-1 pnode-2 pnode-s1 pnode-s2

CES NodeはServer Licenseが必要なので、それを設定する。
[root@snode-0 gpfsfs01]# mmchlicense server --accept -N pnode-s1,pnode-s2

The following nodes will be designated as possessing server licenses:
pnode-s1
pnode-s2
mmchlicense: Command successfully completed
mmchlicense: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.

とりあえず確認。
[root@snode-0 gpfsfs01]# mmlsnode

GPFS nodeset    Node list
-------------   -------------------------------------------------------
gpfscluster1     snode-1 snode-0 pnode-1 pnode-2 pnode-s1 pnode-s2

[root@snode-0 gpfsfs01]#

[root@snode-0 gpfsfs01]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         gpfscluster1.snode-0
  GPFS cluster id:           17390685069738965895
  GPFS UID domain:           gpfscluster1.snode-0
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
  Repository type:           CCR

Node  Daemon node name  IP address     Admin node name  Designation
---------------------------------------------------------------------
   1   snode-0           192.168.1.91   snode-0          quorum-manager
   2   snode-1           192.168.1.92   snode-1          quorum-manager
   3   pnode-1           192.168.1.96   pnode-1          
   4   pnode-2           192.168.1.97   pnode-2          
   5   pnode-s1          192.168.1.106  pnode-s1         
   6   pnode-s2          192.168.1.107  pnode-s2         

[root@snode-0 gpfsfs01]#

cesSharedRootを確認。これが意味を持つのだろうか。後から要確認。
[root@snode-0 gpfsfs01]# mmlsconfig
Configuration data for cluster gpfscluster1.snode-0:
----------------------------------------------------
clusterName gpfscluster1.snode-0
clusterId 17390685069738965895
autoload no
dmapiFileHandleSize 32
minReleaseLevel 5.0.4.0
ccrEnabled yes
cipherList AUTHONLY
cesSharedRoot /gpfs/gpfsfs01/nfs
adminMode central

File systems in cluster gpfscluster1.snode-0:
---------------------------------------------
/dev/gpfsfs01

現時点でのCES Nodeを確認。
[root@snode-0 /]# mmces node list

Node Number   Node Name   Node Groups   Node Flags   
------------- ----------- ------------- ------------
3             pnode-1     nfsgroup1     none         
4             pnode-2     nfsgroup1     none         

今回新たに追加するpnode-s1とpnode-s2でCESをイネーブルにする。
[root@snode-0 /]# mmchnode --ces-enable -N pnode-s1,pnode-s2

2020年  4月  1日 水曜日 15:07:47 JST: mmchnode: Processing node pnode-s1
The NFS service can not be enabled because it is not installed.
The file /usr/bin/gpfs.ganesha.nfsd was not found.
mmcesop: Command failed. Examine previous error messages to determine cause.
mmchnode: Node pnode-s1 cannot be added to CES.
2020年  4月  1日 水曜日 15:07:49 JST: mmchnode: Processing node pnode-s2
The NFS service can not be enabled because it is not installed.
The file /usr/bin/gpfs.ganesha.nfsd was not found.
mmcesop: Command failed. Examine previous error messages to determine cause.
mmchnode: Node pnode-s2 cannot be added to CES.
mmchnode: Command failed. Examine previous error messages to determine cause.

[root@snode-0 /]#

どうやらNFS serviceのインストールがCES Nodeになる必要条件のようだ。そこで、これら2つのノードにNFSをインストールする。[root@localhost rhel7]# rpm -ihv gpfs.nfs*

準備しています...              ################################# [100%]
更新中 / インストール中...
   1:gpfs.nfs-ganesha-2.7.5-ibm054.05.################################# [ 25%]
   2:gpfs.nfs-ganesha-gpfs-2.7.5-ibm05################################# [ 50%]
   3:gpfs.nfs-ganesha-utils-2.7.5-ibm0################################# [ 75%]
   4:gpfs.nfs-ganesha-debuginfo-2.7.5-################################# [100%]

[root@localhost rhel7]#

再挑戦。
[root@snode-0 /]# mmchnode --ces-enable -N pnode-s1,pnode-s2
2020年  4月  1日 水曜日 16:45:30 JST: mmchnode: Processing node pnode-s1
2020年  4月  1日 水曜日 16:45:33 JST: mmchnode: Processing node pnode-s2
mmchnode: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.

[root@snode-0 /]#

念の為の確認。
[root@snode-0 /]# mmces node list

Node Number   Node Name   Node Groups   Node Flags   
------------- ----------- ------------- ------------
3             pnode-1     nfsgroup1     none         
4             pnode-2     nfsgroup1     none         
5             pnode-s1                  none         
6             pnode-s2                  none         

[root@snode-0 /]#

今追加2つのCESノードで新たなグループ s3group1を構成する。
[root@snode-0 /]# mmchnode --ces-group s3group1 -N pnode-s1,pnode-s2
2020年  4月  1日 水曜日 16:57:37 JST: mmchnode: Processing node pnode-s1
2020年  4月  1日 水曜日 16:57:37 JST: mmchnode: Processing node pnode-s2
mmchnode: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.

念の為の確認。
[root@snode-0 /]# mmces node list

Node Number   Node Name   Node Groups   Node Flags   
------------- ----------- ------------- ------------
3             pnode-1     nfsgroup1     none         
4             pnode-2     nfsgroup1     none         
5             pnode-s1    s3group1      none         
6             pnode-s2    s3group1      none         

[root@snode-0 /]#

このグループs3group1にCES IPとして192.168.1.108を割り当てる。
[root@snode-0 /]# mmces address add --ces-ip 192.168.1.108 --ces-group s3group1

念の為の確認。
[root@snode-0 /]# mmces address list

Address         Node       Ces Group   Attributes   
--------------- ---------- ----------- ------------
192.168.1.108   pnode-s1   s3group1    none         
192.168.1.98    pnode-2    nfsgroup1   none         

[root@snode-0 /]#

以下の内容でscale_object.repoを作り/etc/yum.repos.dに保存する。
[spectrum_scale]
name=spectrum_scale
baseurl=file:///usr/lpp/mmfs/5.0.4.3/object_rpms/rhel7
enabled=1
gpgcheck=0

OBJパッケージをインストールする。
[root@localhost rhel7]# yum -y install spectrum-scale-object

とりあえず以下の内容でパスワードファイルをつくり、ks-passwordとして/var/mmfs/ssl/keyServ/tmpに保存する。mmobjはパスワードファイルをこのディレクトリから読み出す。

ドキドキしながら以下を実行する。これはどれかのCES nodeで一回実行すればよい。CESノード全体が構成される。
ここで重要なのは既設のfilesetを指定していないところ。
[root@localhost rhel7]# mmobj swift base -g /gpfs/gpfsfs01 --cluster-hostname s3group1 --local-keystone --pwd-file ks-password -i 20000 --enable-s3

mmobj swift base: Validating execution environment.
mmobj swift base: Creating fileset gpfsfs01 object_fileset.
mmobj swift base: Configuring Keystone server in /gpfs/gpfsfs01/nfs/object/keystone.
mmobj swift base: Creating postgres database.
mmobj swift base: Setting cluster attribute object_database_node=192.168.1.108.
mmobj swift base: Validating Keystone environment.
mmobj swift base: Validating Swift values in Keystone.
mmobj swift base: Configuring Swift services.
mmobj swift base: Setting cluster attribute object_singleton_node=192.168.1.108.
mmobj swift base: Uploading configuration changes to the CCR.
mmobj swift base: Configuration complete.
[root@localhost rhel7]#
Defaultでobject_filesetがgpfs mount pointである/gpfs/gpfsfs01の下に作られた。

OBJをイネーブルしようとしたらエラーが出た。
[root@snode-0 /]# mmces service enable OBJ
pnode-s2:  [E] The OBJ service cannot be enabled because it is not installed.
pnode-2:  [E] The OBJ service cannot be enabled because it is not installed.
pnode-s2:  The file /etc/swift/swift.conf was not found.
pnode-s2:  mmcesop: Command failed. Examine previous error messages to determine cause.
mmdsh: pnode-s2 remote shell process had return code 1.
pnode-2:  The file /etc/swift/swift.conf was not found.
pnode-2:  mmcesop: Command failed. Examine previous error messages to determine cause.
mmdsh: pnode-2 remote shell process had return code 1.
pnode-1:  [E] The OBJ service cannot be enabled because it is not installed.
pnode-1:  The file /etc/swift/swift.conf was not found.
pnode-1:  mmcesop: Command failed. Examine previous error messages to determine cause.
mmdsh: pnode-1 remote shell process had return code 1.
mmces service: Command failed. Examine previous error messages to determine cause.
pnode-s1:  Keystone DB: service already stopped.
pnode-s1:  Keystone: service already stopped.
pnode-s1:  Swift: service already stopped.
mmces service enable: Command failed. Examine previous error messages to determine cause.
[root@snode-0 /]#

CESノード全てにOBJをインストールせねばならないようだ。で、以下を他のCESノードに対しても実行する。repoとパスワードファイルのコピーも忘れずに。
# yum -y install spectrum-scale-object

改めて実行。
[root@snode-0 /]# mmces service enable OBJ

mmchconfig: Command successfully completed
mmchconfig: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
pnode-s1:  Keystone DB: service already stopped.
pnode-s2:  Keystone DB: service already stopped.
pnode-s1:  Keystone: service already stopped.
pnode-s2:  Keystone: service already stopped.
pnode-2:  Keystone DB: service already stopped.
pnode-2:  Keystone: service already stopped.
pnode-1:  Keystone DB: service already stopped.
pnode-1:  Keystone: service already stopped.
pnode-s1:  Swift: service already stopped.
pnode-s2:  Swift: service already stopped.
pnode-2:  Swift: service already stopped.
pnode-1:  Swift: service already stopped.
pnode-s1:  Version mismatch during upload of proxy-server.conf (1). Retrying.
pnode-s2:  Version mismatch during upload of proxy-server.conf (2). Retrying.
pnode-s2:  Version mismatch during upload of proxy-server.conf (3). Retrying.
pnode-2:  Swift: service succesfully started.
pnode-s2:  Swift: service succesfully started.
pnode-s1:  Swift: service succesfully started.
pnode-1:  Swift: service succesfully started.
pnode-s1:  Keystone DB: service succesfully started.
pnode-s2:  Keystone: service succesfully started.
pnode-s1:  Keystone: service succesfully started.
pnode-1:  Keystone: service succesfully started.
pnode-2:  Keystone: service succesfully started.
[root@snode-0 /]#

上手くいったようだ。念の為の確認。これをNSDサーバーで実行したらエラーになった。CESノードで実行する必要があるようだ。
[root@snode-0 /]# mmces service list -v

Enabled services: NFS OBJ
mmces service: [E] The local node is not a CES node.

mmces service list: Command failed. Examine previous error messages to determine cause.
[root@snode-0 /]#

改めてCESノード上で確認。
[root@localhost rhel7]# mmces service list -v

Enabled services: NFS OBJ
NFS is running
OBJ is running
OBJ:openstack-swift-object-updater           is running
OBJ:openstack-swift-object-expirer           is running
OBJ:ibmobjectizer                        is not running
OBJ:openstack-swift-object                   is running
OBJ:openstack-swift-account                  is running
OBJ:openstack-swift-container                is running
OBJ:memcached                                is running
OBJ:openstack-swift-proxy                    is running
OBJ:openstack-swift-object-replicator        is running
OBJ:openstack-swift-account-reaper           is running
OBJ:openstack-swift-account-auditor          is running
OBJ:openstack-swift-container-auditor        is running
OBJ:openstack-swift-container-updater        is running
OBJ:openstack-swift-account-replicator       is running
OBJ:openstack-swift-container-replicator     is running
OBJ:openstack-swift-object-sof           is not running
OBJ:postgresql-obj                           is running
OBJ:httpd (keystone)                         is running
[root@localhost rhel7]#

いい感じだ。念の為、更に確認。
[root@localhost rhel7]# mmces service list -a

Enabled services: NFS OBJ
pnode-2:  NFS is running, OBJ is running
pnode-1:  NFS is running, OBJ is running
pnode-s1:  NFS is running, OBJ is running
pnode-s2:  NFS is running, OBJ is running
[root@localhost rhel7]#

S3が動いているか確認。
[root@localhost rhel7]# mmobj s3 list
[I] The S3 API is enabled.
[root@localhost rhel7]#

とりあえず、S3が動いているようだ。
今日はここまで。次は実際にS3 APIでアクセスしてみるのだ。