SUSE Ceph Cephfs - Storage6

（1）Policy 配置文件，添加MDS角色定义

# vim /srv/pillar/ceph/proposals/policy.cfg
# MDS
role-mds/cluster/mds*.sls
role-mds/cluster/node00[].example.com.sls

（2）执行stage2,stage4

salt-run state.orch ceph.stage.
salt-run state.orch ceph.stage.

（3）设置 文件系统元数据到SSD

# ceph osd pool set cephfs_metadata crush_rule ssd_replicated_rule

（4）设置 PG PGP 数量
总数据大小，6*12*4=288T ,大约 260TB数据

# ceph osd lspools
 cephfs_data
 cephfs_metadata

# ceph osd pool set cephfs_data pg_num
# ceph osd pool set cephfs_metadata pg_num 

（5）关闭scrub和deep scrub , pool级别关闭，cephe -s 不会显示

# ceph osd pool set cephfs_data noscrub
# ceph osd pool set cephfs_metadata nodeep-scrub 

（6）设置多活

# ceph fs set cephfs max_mds       # cephfs 为fs_name 名字
# ceph mds stat
cephfs-// up  {=node003=up:active,=node001=up:active},  up:standby

（7）客户端挂载

# mkdir /mnt/cephfs_client/
# mount -t ceph 192.168.2.40,192.168.2.41,192.168.2.42::/ \
/mnt/cephfs_client/ -o name=admin,\
secret=AQAfvWhdAAAAABAAIGnAtjOBdDLE8+t/u2zadQ==,rasize=  # read ahead 16MB

# df -TH
Filesystem                                    Type      Size  Used Avail Use% Mounted on
192.168.2.40,192.168.2.41,192.168.2.42::/ ceph       20G        20G   % /mnt/cephfs_client

NFS

（1）NFS
使用图形化界面配置，在Daemons选择节点主机名

编辑DeepSea安装配置文件
# vim /srv/pillar/ceph/proposals/policy.cfg
# NFS
role-ganesha/cluster/node003.example.com.sls

执行salt命令，stage2 和stage4
# salt-run state.orch ceph.stage.2
# salt 'node003*' pillar.items
# salt-run state.orch ceph.stage.4

# cat /etc/ganesha/ganesha.conf | grep -v ^# | grep -v ^$
RADOS_URLS {
  # Path to a ceph.conf file for this cluster. 集群配置文件
  Ceph_Conf = /etc/ceph/ceph.conf;
  # RADOS_URLS use their own ceph client too. Authenticated access
  # requires a cephx keyring file.  用户key ID
  UserId = "ganesha.node003";
  watch_url = "rados://cephfs_data/ganesha/conf-node003";
}
CACHEINODE {
    # Size the dirent cache down as small as possible.
    Dir_Chunk = 0;
    # size the inode cache as small as possible
    NParts = 1;
    Cache_Size = 1;                 # 每个分区哈希表大小
}
NFS_KRB5                            # 是否激活 Kerberos 5. 默认false
{
    Active_krb5 = false;
}
%url rados://cephfs_data/ganesha/conf-node003

NFS 配置
# rados -p .rgw.root ls --namespace=ganesha
export-1
conf-node003
conf-node004

get <obj-name> <outfile>         fetch object
 put <obj-name> <infile> [--offset offset]

# rados -p .rgw.root get export-1 nfs --namespace=ganesha

CIFS

node002 节点
# zypper in samba-ceph samba-winbind

Admin节点
# cd /etc/ceph
# ceph auth get-or-create client.samba.gw mon 'allow r' \
    osd 'allow *' mds 'allow *' -o ceph.client.samba.gw.keyring
# scp ceph.client.samba.gw.keyring node002:/etc/ceph/

# cp /etc/samba/smb.conf /etc/samba/smb.conf.bak

编辑配置文件
# vim /etc/samba/smb.conf
[global]
        workgroup = WORKGROUP
        passdb backend = tdbsam
        printing = cups
        printcap name = cups
        printcap cache time = 750
        cups options = raw
        map to guest = Bad User
        logon path = \\%L\profiles\.msprofile
        logon home = \\%L\%U\.9xprofile
        logon drive = P:
        usershare allow guests = Yes
[suse]
        path = /
        vfs objects = ceph
        ceph: config_file = /etc/ceph/ceph.conf
        ceph: user_id = samba.gw
        read only = no
        oplocks = no
        kernel share modes = no

systemctl restart smb.service
systemctl restart nmb.service

# systemctl start smb.service
# systemctl enable smb.service
# systemctl start nmb.service
# systemctl enable nmb.service

检查端口
# netstat -ntulp | grep mbd
tcp        0      0 0.0.0.0:139             0.0.0.0:*               LISTEN      6457/smbd           
tcp        0      0 0.0.0.0:445             0.0.0.0:*               LISTEN      6457/smbd           
tcp6       0      0 :::139                  :::*                    LISTEN      6457/smbd           
tcp6       0      0 :::445                  :::*                    LISTEN      6457/smbd           
udp        0      0 172.200.50.255:137      0.0.0.0:*                           6483/nmbd           
udp        0      0 172.200.50.41:137       0.0.0.0:*                           6483/nmbd           
udp        0      0 192.168.2.255:137       0.0.0.0:*                           6483/nmbd           
udp        0      0 192.168.2.41:137        0.0.0.0:*                           6483/nmbd           
udp        0      0 192.168.3.255:137       0.0.0.0:*                           6483/nmbd           
udp        0      0 192.168.3.41:137        0.0.0.0:*                           6483/nmbd           
udp        0      0 0.0.0.0:137             0.0.0.0:*                           6483/nmbd           
udp        0      0 172.200.50.255:138      0.0.0.0:*                           6483/nmbd           
udp        0      0 172.200.50.41:138       0.0.0.0:*                           6483/nmbd           
udp        0      0 192.168.2.255:138       0.0.0.0:*                           6483/nmbd           
udp        0      0 192.168.2.41:138        0.0.0.0:*                           6483/nmbd           
udp        0      0 192.168.3.255:138       0.0.0.0:*                           6483/nmbd           
udp        0      0 192.168.3.41:138        0.0.0.0:*                           6483/nmbd           
udp        0      0 0.0.0.0:138             0.0.0.0:*                           6483/nmbd

验证配置文件
# testparm

重载配置文件
# smbcontrol all reload-config

设置samba密码
# smbpasswd -a root
New SMB password:
Retype new SMB password:
Added user root.

Linux 访问samba
# smbclient -L //172.200.50.41/
Enter WORKGROUP\root's password:

Sharename       Type      Comment
        ---------       ----      -------
        suse            Disk      
        IPC$            IPC       IPC Service (Samba 4.9.5-git.176.375e1f057883.6.1-SUSE-oS15.0-x86_64)
Reconnecting with SMB1 for workgroup listing.

Server               Comment
        ---------            -------

Workgroup            Master
        ---------            -------
        WORKGROUP            NODE002
        
Windows 访问samba
1)    打开来宾登录策略

• 按住“Win”和“R”键，在弹出的对话框中输入“gpedit.msc”，点击“确定”。
• 在本地组策略编辑器对话框中，依次点击“计算机配置” --> “管理模板” --> “网络”。
• 点击“lanman工作站”。
• 双击“启用不安全的来宾登录”，点击“已启用”，点击“确定”即可。

windows 访问
\\172.200.50.41\suse