ceph network introduce

网络结构

Ceph 使用以太网连接内部各存储节点以及连接 client 和集群。Ceph 推荐使用两个网络：

• 前端（北向）网络（ a public (front-side) network）：连接客户端和集群 The public network handles client traffic and communication with Ceph monitors.

• 后端/东西向网络 （a cluster (back-side) network）：连接 Ceph 各存储节  The cluster (back-side) network handles OSD heartbeats, replication, backfilling and recovery traffic.

来自redHat的建议：

We recommend allocating bandwidth to the cluster (back-side) network such that it is a multiple of the front-side network using osd pool default size as the basis for your multiple. We also recommend running the public and cluster networks on separate NICs.  We recommend keeping the cluster network separate from the public network and not connected to the internet to prevent DDOS attacks from crippling heartbeats, replication, backfilling and recovery.

网络拓扑：

这么做，主要是从性能（OSD 节点之间会有大量的数据拷贝操作）和安全性（两网分离）考虑。

你可以在 Ceph 配置文件的 [global] 部分配置两个网络：

public network = {public-network/netmask}
cluster network = {cluster-network/netmask}

参考：具体可以参考：

• Deploying Ceph with High Performance Networks, Architectures and benchmarks for Block Storage Solutions
• Deploying Ceph with High Performance Networks
• CHAPTER 2. NETWORKING RECOMMENDATIONS
• Ceph with a cluster and public network on IPv6 谈到了 IPV6 的支持。

端口

Monitors use port 6789 by default. Ensure you have the port open for each monitor host. Each Ceph OSD Daemon on a Ceph Node may use up to three ports, beginning at port 6800:

1. One for talking to clients and monitors. (front-side)
2. One for sending data to other OSDs (replication, backfill and recovery). (back-side)
3. One for heartbeating. (back-side)

You need to open at least three ports per OSD beginning at port 6800 on a Ceph node to ensure that the OSDs can peer. The port for talking to monitors and clients must be open on the public (front-side) network. The ports for sending data to other OSDs and heartbeating must be open on the cluster (back-side) network.

If you want to use a different port range than 6800:7100 for Ceph daemons, you must adjust the following settings in your Ceph configuration file:

ms bind port min = {min-port-num}
ms bind port max = {max-port-num}

Ceph monitors bind on port 6789 by default. If you want to use a different port number than 6789, you may specify the the IP address and port in your Ceph configuration. For example:

[mon.monname]
host = {hostname}
mon addr = {ip-address}:{port}

参考引用地址：

1、https://access.redhat.com/documentation/en/red-hat-ceph-storage/version-1.2.3/red-hat-ceph-storage-123-hardware-guide/chapter-2-networking-recommendations

2、http://www.cnblogs.com/sammyliu/p/4836014.html