ようやくゴールに到着。ゴールインの前に今回のシステム構成のおさらい。
まず、全てのノードはVM上に作成した。
ホストH/W Lenovo Minitowr 530 i9 9900 (8core/16thread) 3.10GHz, memory=32GB
ホストOS Windows10 Home Edition
親VM VMWare Workstation Player 15
dns 1CPU, 1GB (local DNS)
nfs 1CPU, 1GB (scale pvcマウントに先立つnfs pvcのマウントテスト用）
子VM ESXi 6.0
Kubernetes Nodes (Kubernetes環境構築ESXi)
master 2CPU, 4GB
worker1 2CPU, 4GB
worker2 2CPU, 4GB
Scale Nodes (scale環境構築ESXi)
server node 1 (snode1) 1CPU, 2GB
server node 2 (snode2) 1CPU, 2GB
gui node (sgui) 1CPU, 3GB
合計システムリソース CPU 11、memory 21GB
今回のホストH/Wではこれがほぼほぼ上限の構成となる。当初Kubernetesは2GBメモリーで構成したけれども、メモリーが足りないようで頻繁にハングしたので4GBに増強。Scale GUIノードも2GBでハングしので3GBに増強。この結果、今時点においてScale CSIドライバー試験が出来てはいる。
それではゴールインストーリの始まり始まり、、、、
CSI Driverのインストール
KubernetesのインストールとScaleのインストールはそれぞれのブログを参照して下さい。Local DNSとDNS利用留意点についても同様です。
Scale GUIサーバー上での作業
ユーザーグループ csiadminの作成
cd /usr/lpp/mmfs/gui/cli
[root@localhost cli]# ./mkusergrp CsiAdmin --role csiadmin
[root@localhost cli]# ./lsusergrp
Name ID Role
Administrator 1 admin
SecurityAdmin 2 securityadmin
StorageAdmin 3 storageadmin
SystemAdmin 4 systemadmin
Monitor 5 monitor
SnapAdmin 6 snapadmin
DataAccess 7 dataaccess
ProtocolAdmin 8 protocoladmin
UserAdmin 9 useradmin
CsiAdmin 10 csiadmin
CnssOperator 11 cnssoperator
EFSSG1000I The command completed successfully.
CSIドライバーからGUIサーバーにログインするユーザーの作成
[root@localhost cli]# ./mkuser csiadmin -p password -g CsiAdmin
EFSSG0019I The user csiadmin has been successfully created.
EFSSG1000I The command completed successfully.
[root@localhost cli]# ./lsuser
Name Long name Password status Group names Failed login attempts Target Feedback Date
admin active SecurityAdmin 0 24.09.2020 09:03:23.000
csiadmin active CsiAdmin 0
EFSSG1000I The command completed successfully.
[root@localhost cli]#
作成したユーザーアカウントでのGUIサーバーへのアクセスを確認.
[root@localhost cli]# curl --insecure -u 'csiadmin:password' -X GET https://sgui.pathpilot.local:443/scalemgmt/v2/cluster
{
"cluster" : {
"clusterSummary" : {
"clusterId" : 1593904468708199996,
"clusterName" : "gpfscluster1.pathpilot.local",
"primaryServer" : "snode-1.pathpilot.local",
"rcpPath" : "/usr/bin/scp",
"rcpSudoWrapper" : false,
"repositoryType" : "CCR",
"rshPath" : "/usr/bin/ssh",
"rshSudoWrapper" : false,
"uidDomain" : "gpfscluster1.pathpilot.local"
},
"capacityLicensing" : {
"liableCapacity" : 10737418240,
"liableNsdCount" : 2,
"liableNsds" : [ {
"nsdName" : "NSD_101",
"liableCapacity" : 5368709120
}, {
"nsdName" : "NSD_102",
"liableCapacity" : 5368709120
} ]
}
},
"status" : {
"code" : 200,
"message" : "The request finished successfully."
}
[root@localhost cli]#
Quotaの設定
[root@localhost cli]# mmlsfs gpfsfs01 --perfileset-quota
flag value description
------------------- ------------------------ -----------------------------------
--perfileset-quota いいえ Per-fileset quota enforcement
[root@localhost cli]# mmlsfs gpfsfs01 -Q
flag value description
------------------- ------------------------ -----------------------------------
-Q none Quotas accounting enabled
none Quotas enforced
none Default quotas enabled
[root@localhost cli]# mmchfs gpfsfs01 -Q yes
mmchfs: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
[root@localhost cli]# mmlsfs gpfsfs01 -Q
flag value description
------------------- ------------------------ -----------------------------------
-Q user;group;fileset Quotas accounting enabled
user;group;fileset Quotas enforced
none Default quotas enabled
[root@localhost cli]#
クラスタに属性情報追加
[root@localhost cli]# mmchconfig enforceFilesetQuotaOnRoot=yes -i
mmchconfig: Command successfully completed
mmchconfig: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
[root@localhost cli]# mmchconfig controlSetxattrImmutableSELinux=yes -i
mmchconfig: Command successfully completed
mmchconfig: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
[root@localhost cli]# mmlsconfig
Configuration data for cluster gpfscluster1.pathpilot.local:
------------------------------------------------------------
clusterName gpfscluster1.pathpilot.local
clusterId 1593904468708199996
autoload no
dmapiFileHandleSize 32
minReleaseLevel 5.0.5.1
ccrEnabled yes
cipherList AUTHONLY
enforceFilesetQuotaOnRoot yes
controlSetxattrImmutableSELinux yes
adminMode central
File systems in cluster gpfscluster1.pathpilot.local:
-----------------------------------------------------
/dev/gpfsfs01
ファイルシステム情報にdf表示追加
[root@localhost cli]# mmchfs gpfsfs01 --filesetdf
[root@localhost cli]#
以降の作業はKubernetesのマスターノードで実行する
CSI OperatorのProvisioning対象ノードであることを示すためラベル付け
[root@master yaml]# kubectl label node worker1.pathpilot.local scale=true --overwrite=true
node/worker1.pathpilot.local labeled
[root@master yaml]# kubectl label node worker2.pathpilot.local scale=true --overwrite=true
node/worker2.pathpilot.local labeled
[root@master yaml]#
[root@master yaml]# kubectl describe node worker1.pathpilot.local
Name: worker1.pathpilot.local
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=worker1.pathpilot.local
kubernetes.io/os=linux
scale=true
....
ネームスペースの作成
[root@master yaml]# kubectl create namespace ibm-spectrum-scale-csi-driver
namespace/ibm-spectrum-scale-csi-driver created
Operatorの作成
[root@master yaml]# kubectl create -f https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-csi/v2.0.0/generated/installer/ibm-spectrum-scale-csi-operator.yaml
deployment.apps/ibm-spectrum-scale-csi-operator created
clusterrole.rbac.authorization.k8s.io/ibm-spectrum-scale-csi-operator created
clusterrolebinding.rbac.authorization.k8s.io/ibm-spectrum-scale-csi-operator created
serviceaccount/ibm-spectrum-scale-csi-operator created
customresourcedefinition.apiextensions.k8s.io/csiscaleoperators.csi.ibm.com created
[root@master yaml]# kubectl get pod,deployment -n ibm-spectrum-scale-csi-driver
NAME READY STATUS RESTARTS AGE
pod/ibm-spectrum-scale-csi-operator-66c7bfc95c-cgvnn 1/1 Running 1 65s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ibm-spectrum-scale-csi-operator 1/1 1 1 65s
[root@master yaml]#
[root@master yaml]# curl -O https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-csi/v2.0.0/operator/deploy/crds/csiscaleoperators.csi.ibm.com_cr.yaml
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5001 100 5001 0 0 9307 0 --:--:-- --:--:-- --:--:-- 9330
csiscaleoperators.csi.ibm.com_cr.yamlの編集
ClusterIDはmmlsclusterが返すcluster idをコピーする。
[root@localhost ~]# mmlscluster
GPFS cluster information
========================
GPFS cluster name: gpfscluster1.pathpilot.local
GPFS cluster id: 1593904468708199996
GPFS UID domain: gpfscluster1.pathpilot.local
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
Repository type: CCR
Node Daemon node name IP address Admin node name Designation
------------------------------------------------------------------------------------
1 snode-1.pathpilot.local 192.168.1.121 snode-1.pathpilot.local quorum-manager
2 snode-2.pathpilot.local 192.168.1.122 snode-2.pathpilot.local quorum-manager
6 sgui.pathpilot.local 192.168.1.123 sgui.pathpilot.local
7 worker2.pathpilot.local 192.168.1.133 worker2.pathpilot.local
8 worker1.pathpilot.local 192.168.1.132 worker1.pathpilot.local
[root@localhost ~]#
scaleHostpathは mmlsfsが返す”Default mount point” をコピーする。
[root@localhost ~]# mmlsfs all
File system attributes for /dev/gpfsfs01:
=========================================
flag value description
------------------- ------------------------ -----------------------------------
....
-o none Additional mount options
-T /gpfs/gpfsfs01 Default mount point
--mount-priority 0 Mount priority
[root@localhost ~]#
secrets.yamlの編集と適用
Scale GUIサーバーにログオンするためのSecret.yamlを編集する。
ユーザー名とパスワードはbase64に変換した値をセットする。
[root@master ~]# echo -n 'csiadmin' | base64
bXktcGFzc3dvcmQ=
[root@master ~]# echo -n 'password' | base64
bXktdXNlcm5hbWU=
[root@master ~]#
編集後のsecret.yamlは以下のようになる。
apiVersion: v1
kind: Secret
metadata:
name: sguisecret << 自分のsecret名
labels:
product: ibm-spectrum-scale-csi
data:
username: bXktdXNlcm5hbWU= <<< base64で設定(この値はダミーです）
password: bXktcGFzc3dvcmQ= <<< base64で設定(この値はダミーです）
[root@master yaml]# kubectl apply -f secrets.yaml -n ibm-spectrum-scale-csi-driver
secret/sguisecret created
csiscaleoperators.csi.ibm.com_cr.yaml の編集
spec:
# The path to the GPFS file system mounted (either remote/local) on the local Spectrum Scale API host machine.
# ==================================================================================
scaleHostpath: "/gpfs/gpfsfs01" << ここを編集
# A passthrough option that distributes an imagePullSecrets array to the containers
# generated by the csi scale operator. Please refer to official k8s documentation for
# your environment for more details. https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
# ==================================================================================
# imagePullSecrets:
# - APullSecret
# - AnotherOptional
# Below specifies the details of a SpectrumScale cluster configuration used by the
# plugin. It can have multiple values.
# ==================================================================================
clusters:
- id: "1593904468708199996" << ここを編集 mmlsclusterからコピー
secrets: "sguisecret" << ここを編集 secrets.yamlで設定した名前
secureSslMode: false
primary:
primaryFs: "gpfsfs01" << ここを編集
# primaryFset: "< Fileset in Primary Filesystem >" # Optional - default:spectrum-scale-csi-volume-store
# inodeLimit: "< inode limit for Primary Fileset >" # Optional
# remoteCluster: "< Remote ClusterID >" # Optional - This is only required if primaryFs is remote cluster's filesystem and this ID should have separate entry in Clusters map too.
# cacert: "< Name of CA cert configmap for GUI >" # Optional
restApi:
- guiHost: "sgui.pathpilot.local" << ここを編集
#
Lightweight Volumeの作成先filesetの作成
[root@localhost ~]# mmcrfileset gpfsfs01 pvfileset
Fileset pvfileset created with id 1 root inode 19456.
[root@localhost ~]# mmlsfileset gpfsfs01
Filesets in file system 'gpfsfs01':
Name Status Path
root Linked /gpfs/gpfsfs01
pvfileset Unlinked --
[root@localhost ~]#
[root@localhost gpfsfs01]# ls /gpfs/gpfsfs01
[root@localhost gpfsfs01]#
[root@localhost gpfsfs01]# mmlinkfileset gpfsfs01 pvfileset -J /gpfs/gpfsfs01/pvfileset
Fileset pvfileset linked at /gpfs/gpfsfs01/pvfileset
[root@localhost gpfsfs01]# mmlsfileset gpfsfs01
Filesets in file system 'gpfsfs01':
Name Status Path
root Linked /gpfs/gpfsfs01
pvfileset Linked /gpfs/gpfsfs01/pvfileset
[root@localhost gpfsfs01]#
[root@localhost gpfsfs01]# ls /gpfs/gpfsfs01
pvfileset
[root@localhost gpfsfs01]#
Persistent Volume作成先ディレクトリを作成する
[root@localhost pvfileset]# mkdir lwdir
念の為、モードを777にする。これは要らないかもしれない。
[root@localhost pvfileset]# ls -l
合計 1
drwxr-xr-x. 2 root root 4096 8月 27 15:08 lwdir
[root@localhost pvfileset]# chmod 777 lwdir
[root@localhost pvfileset]# ls -l
合計 1
drwxrwxrwx. 2 root root 4096 8月 27 15:08 lwdir
lightweight-storage-class.yamlの編集と適用
[root@localhost pvfileset]# cat lightweight-storage-class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ibm-spectrum-scale-csi-lt << ここを編集
provisioner: spectrumscale.csi.ibm.com
parameters:
volBackendFs: "gpfsfs01" << ここを編集
volDirBasePath: "pvfileset/lwdir" << ここを編集
reclaimPolicy: Delete
[root@master yaml]# kubectl apply -f lightweight-storage-class.yaml
storageclass.storage.k8s.io/ibm-spectrum-scale-csi-lt created
[root@master yaml]# kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
ibm-spectrum-scale-csi-lt spectrumscale.csi.ibm.com Delete Immediate false 19s
lightweight-pvc.yamlの編集と適用
[root@master yaml]# cat lightweight-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: scale-lw-fset-pvc << ここを編集 podのyamlが参照
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
storageClassName: ibm-spectrum-scale-csi-lt << ここを編集、lightweight-storage-class.yamlのclass名をコピー
[root@master yaml]# kubectl apply -f lightweight-pvc.yaml
persistentvolumeclaim/scale-lw-fset-pvc created
[root@master yaml]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nfs-claim1 Bound pv0001 3Gi RWO 12d
scale-lw-fset-pvc Pending ibm-spectrum-scale-csi-lt 15s
[root@master yaml]# kubectl describe pvc scale-lw-fset-pvc
Name: scale-lw-fset-pvc
Namespace: default
StorageClass: ibm-spectrum-scale-csi-lt
Status: Pending
Volume:
Labels: <none>
Annotations: volume.beta.kubernetes.io/storage-provisioner: spectrumscale.csi.ibm.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Mounted By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 14s (x2 over 24s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "spectrumscale.csi.ibm.com" or manually created by system administrator
[root@master yaml]#
Microservice pod (mypod-lw01.yaml)の編集と適用
[root@master yaml]# cat mypod-lw01.yaml
apiVersion: v1
kind: Pod
metadata:
name: csi-scale-dynamicdemo-pod << このpodの名前
labels:
app: nginx
spec:
containers:
- name: web-server
image: nginx
volumeMounts:
- name: mypvc
mountPath: /usr/share/nginx/html
ports:
- containerPort: 80
volumes:
- name: mypvc
persistentVolumeClaim:
claimName: scale-lw-fset-pvc << pvcの名前
readOnly: false
[root@master yaml]# kubectl apply -f mypod-lw01.yaml
pod/csi-scale-dynamicdemo-pod created
[root@localhost yaml]# kubectl get pods
NAME READY STATUS RESTARTS AGE
csi-scale-dynamicdemo-pod 1/1 Running 0 79s
無事Runningになっている！！！念の為確認。
[root@localhost yaml]# kubectl describe pod csi-scale-dynamicdemo-pod
Name: csi-scale-dynamicdemo-pod
Namespace: default
Priority: 0
Node: worker2.pathpilot.local/192.168.1.133
Start Time: Sat, 29 Aug 2020 21:26:27 +0900
Labels: app=nginx
Annotations: <none>
Status: Running
IP: 10.244.4.4
IPs:
IP: 10.244.4.4
Containers:
web-server:
Container ID: docker://c3572aa8247d4713f234c7b10da00e5644ebe1a4ab176f0b9d8ade727aee3455
Image: nginx
Image ID: docker-pullable://docker.io/nginx@sha256:b0ad43f7ee5edbc0effbc14645ae7055e21bc1973aee5150745632a24a752661
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Sat, 29 Aug 2020 21:26:48 +0900
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/nginx/html from mypvc (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-pzg5l (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
mypvc:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: scale-lw-fset-pvc
ReadOnly: false
default-token-pzg5l:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-pzg5l
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m3s default-scheduler Successfully assigned default/csi-scale-dynamicdemo-pod to worker2.pathpilot.local
Normal SuccessfulAttachVolume 2m3s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-6f729ce5-24f2-4310-aee7-d82f60a73f39"
Normal Pulling 106s kubelet, worker2.pathpilot.local Pulling image "nginx"
Normal Pulled 103s kubelet, worker2.pathpilot.local Successfully pulled image "nginx"
Normal Created 102s kubelet, worker2.pathpilot.local Created container web-server
Normal Started 102s kubelet, worker2.pathpilot.local Started container web-server
[root@localhost yaml]#
web-serverが立ち上がっているとの事なので早速動作確認。
Persistent Volumeに対応するディレクトリがgpfs上で自動で作成されていることを確認。
[root@worker2 ~]# ls /gpfs/gpfsfs01/pvfileset/lwdir
pvc-6f729ce5-24f2-4310-aee7-d82f60a73f39
自前のindex.htmlをそのディレクトリに保存する。
[root@worker2 ~]# ls /gpfs/gpfsfs01/pvfileset/lwdir/pvc-6f729ce5-24f2-4310-aee7-d82f60a73f39
index.html
[root@worker2 ~]# cat /gpfs/gpfsfs01/pvfileset/lwdir/pvc-6f729ce5-24f2-4310-aee7-d82f60a73f39/index.html
Hello World! I Love Scale!!!
[root@worker2 ~]#
ブラウザーで作成したPodのWebサーバー(nginx)にアクセスして、index.htmlが表示されることを確認する。
以上
