ceph fs

ceph

使用root操作

Prerequisites

operate on host server

Before mounting CephFS, ensure that

  1. the client host (where CephFS has to be mounted and used) has a copy of the Ceph configuration file (i.e. ceph.conf)
  2. a keyring of the CephX user that has permission to access the MDS.
    both of these files must already be present on the host where the Ceph MON resides.
  • Generate a minimal conf file for the client host and place it at a standard location:
    1
    2
    3
    4
    # on client host
    mkdir -p -m 755 /etc/ceph
    sudo ceph config generate-minimal-conf | sudo tee /etc/ceph/ceph.conf
    chmod 644 /etc/ceph/ceph.conf

Create Pool/FS

operate in cephadm cli

方式一

自动配置mds

1
2
3
4
5
# automatically create and configure MDS for  file system
ceph fs volume create cephfs-2 

# specify mds
# ceph fs volume create cephfs-2 --placement="2 node1 node2"

方式二

手动配置mds

  • deploy mds

    1
    2
    3
    4
    5
    6
    7
    # mkdir /var/lib/ceph/mds/ceph-${id}
    # sudo ceph auth get-or-create mds.${id} mon 'profile mds' mgr 'profile mds' mds 'allow *' osd 'allow *' > /var/lib/ceph/mds/ceph-${id}/keyring

    mkdir /var/lib/ceph/mds/ceph-node1
    ceph auth get-or-create mds.node1 mon 'profile mds' mgr 'profile mds' mds 'allow *' osd 'allow *' > /var/lib/ceph/mds/ceph-node1/keyring

    systemctl start ceph-mds@node1

  • Creating pools

    A Ceph file system requires at least two RADOS pools, one for data and one for metadata.

    1
    2
    ceph osd pool create cephfs_data
    ceph osd pool create cephfs_metadata

  • Creating the file system

    1
    ceph fs new cephfs-1 cephfs_metadata cephfs_data

  • Check the status of the file system

    1
    ceph fs status

  • check mds status

    1
    ceph mds stat

  • Using Erasure Coded pools with CephFS（optional）

    1
    ceph osd pool set my_ec_pool allow_ec_overwrites true

Create a CephX user

1
2
# 需要在创建好文件系统cephfs-1再进行操作
sudo ceph fs authorize cephfs-1 client.dongwei / rw | sudo tee /etc/ceph/ceph.client.dongwei.keyring

In above command, replace cephfs with the name of your CephFS, foo by the name you want for your CephX user and / by the path within your CephFS for which you want to allow access to the client host and rw stands for both read and write permissions.

  • Ensure that the keyring has appropriate permissions:
1
chmod 600 /etc/ceph/ceph.client.dongwei.keyring

Mount CephFS using Kernel Driver

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# install ceph commond


# mount -t ceph {device-string}={path-to-mounted} {mount-point} -o {key-value-args} {other-args}
# mount -t ceph <name>@<fsid>.<fs_name>=/ /mnt/mycephfs
# fsid is the FSID of the Ceph cluster, which can be found using the `ceph fsid` command.

mkdir /mnt/mycephfs

# not use mount.helper
mount -t ceph dongwei@b3acfc0d-575f-41d3-9c91-0e7ed3dbb3fa.cephfs-1=/ /mnt/cephfs -o mon_addr=192.168.0.1:6789,secret=AQATSKdNGBnwLhAAnNDKnH65FmVKpXZJVasUeQ==

# use mount.helper
mount -t ceph dongwei@.cephfs-1=/ /mnt/cephfs -o secret=AQATSKdNGBnwLhAAnNDKnH65FmVKpXZJVasUeQ==

# use secret file and Multiple monitor hosts
mount -t ceph cephuser@.cephfs=/ /mnt/mycephfs -o
mon_addr=192.168.0.1:6789/192.168.0.2:6789,secretfile=/etc/ceph/cephuser.secret

  • check

    1
    dongwei@6a65c746-e532-11ef-8ac2-fa7c097efb00.cephfs-1=/  190G     0  190G   0% /mnt/cephfs

Mount CephFS using FUSE

  • install ceph-fuse

    1
    2
    3
    4
    5
    6
    7
    # Install the Ceph release RPM (adjust 'reef' as needed)
    sudo rpm -Uvh https://download.ceph.com/rpm-reef/el9/noarch/ceph-release-1-1.el9.noarch.rpm

    # Import the Ceph GPG key
    sudo rpm --import 'https://download.ceph.com/keys/release.asc'

    sudo dnf install -y ceph-fuse

  • mount

    1
    2
    3
    4
    5
    # ceph-fuse {mount point} {options}
    mkdir /mnt/mycephfs
    ceph-fuse --id dongwei /mnt/mycephfs  # mount default fs

    ceph-fuse --id dongwei --client_fs cephfs-1 /mnt/mycephfs # mount specify fs

  • check

    1
    ceph-fuse	190G     0  190G   0% /mnt/cephfs2