一、理解Cluster Map

cluster map由monitor维护，用于跟踪ceph集群状态

当client启动时，会连接monitor获取cluster map副本，发现所有其他组件的位置，然后直接与所需的进程通信，以存储和检索数据

monitor跟踪这些集群组件的状态，也负责管理守护进程和客户端之间的身份验证

cluster map实际是多种map的集群，包含：monitor map、osd map、pg map、mds map、mgr map、service map

1.1 Clisuter Map内容

monitor map：包含集群ID、monitor节点名称、IP以及端口号以及monitor map的版本号

OSD map：包含集群ID、自身的版本号以及存储池相关的信息，包括存储池名称、ID、类型、副本级别和PG。还包括OSD的信息，比如数量、状态、权限以及OSD节点信息

PG map：包含PG的版本、时间戳、OSD map的最新版本号、容量相关的百分比。还记录了每个PG的ID、对象数量、状态、状态时间戳等

MDS map：包含MDS的地址、状态、数据池和元数据池的ID

MGR map：包含MGR的地址和状态，以及可用和已启用模块的列表

service map：跟踪通过librados部署的一些服务的实例，如RGW、rbd-mirror等。service map收集这些服务的信息然后提供给其他服务，如MGR的dashboard插件使用该map报告这些客户端服务的状态

1.2  Cluster Map基本查询

查询mon map

  1. [root@ceph2 ~]# ceph mon dump
  2. dumped monmap epoch
  3. epoch
  4. fsid 35a91e48--4e96-a7ee-980ab989d20d
  5. last_changed -- ::14.839999
  6. created -- ::14.839999
  7. : 172.25.250.11:/ mon.ceph2
  8. : 172.25.250.12:/ mon.ceph3
  9. : 172.25.250.13:/ mon.ceph4

查询osd map

  1. [root@ceph2 ~]# ceph osd dump
  2. epoch
  3. fsid 35a91e48--4e96-a7ee-980ab989d20d
  4. created -- ::22.552356
  5. modified -- ::15.354383
  6. flags sortbitwise,recovery_deletes,purged_snapdirs
  7. crush_version
  8. full_ratio 0.95
  9. backfillfull_ratio 0.9
  10. nearfull_ratio 0.85
  11. require_min_compat_client jewel
  12. min_compat_client jewel
  13. require_osd_release luminous
  14. pool  'testpool' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application rbd
  15.     snap  'testpool-snap-20190316' -- ::34.150433
  16.     snap  'testpool-snap-2' -- ::15.430823
  17. pool  'rbd' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application rbd
  18.     removed_snaps [~]
  19. pool  'rbdmirror' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application rbd
  20.     removed_snaps [~]
  21. pool  '.rgw.root' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application rgw
  22. pool  'default.rgw.control' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application rgw
  23. pool  'default.rgw.meta' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application rgw
  24. pool  'default.rgw.log' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application rgw
  25. pool  'xiantao.rgw.control' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  owner  flags hashpspool stripe_width  application rgw
  26. pool  'xiantao.rgw.meta' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  owner  flags hashpspool stripe_width  application rgw
  27. pool  'xiantao.rgw.log' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  owner  flags hashpspool stripe_width  application rgw
  28. pool  'cephfs_metadata' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application cephfs
  29. pool  'cephfs_data' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application cephfs
  30. pool  'test' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application rbd
  31. pool  'ssdpool' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application rbd
  32. max_osd
  33. osd. up   in  weight  up_from  up_thru  down_at  last_clean_interval [,) 172.25.250.11:/ 172.25.250.11:/ 172.25.250.11:/ 172.25.250.11:/ exists,up 745dce53-1c63-4c50-b434-d441038dafe4
  34. osd. up   in  weight  up_from  up_thru  down_at  last_clean_interval [,) 172.25.250.13:/ 172.25.250.13:/ 172.25.250.13:/ 172.25.250.13:/ exists,up a7562276-6dfd--b248-a7cbdb64ebec
  35. osd. up   in  weight  up_from  up_thru  down_at  last_clean_interval [,) 172.25.250.12:/ 172.25.250.12:/ 172.25.250.12:/ 172.25.250.12:/ exists,up bbef1a00-3a31-48a0-a065-3a16b9edc3b1
  36. osd. up   in  weight  up_from  up_thru  down_at  last_clean_interval [,) 172.25.250.11:/ 172.25.250.11:/ 172.25.250.11:/ 172.25.250.11:/ exists,up e934a4fb--4e85-895c-f66cc5534ceb
  37. osd. up   in  weight  up_from  up_thru  down_at  last_clean_interval [,) 172.25.250.13:/ 172.25.250.13:/ 172.25.250.13:/ 172.25.250.13:/ exists,up e2c33bb3-02d2-4cce-85e8-25c419351673
  38. osd. up   in  weight  up_from  up_thru  down_at  last_clean_interval [,) 172.25.250.12:/ 172.25.250.12:/ 172.25.250.12:/ 172.25.250.12:/ exists,up d299e33c-0c24-4cd9-a37a-a6fcd420a529
  39. osd. up   in  weight  up_from  up_thru  down_at  last_clean_interval [,) 172.25.250.11:/ 172.25.250.11:/ 172.25.250.11:/ 172.25.250.11:/ exists,up debe7f4e-656b-48e2-a0b2-bdd8613afcc4
  40. osd. up   in  weight  up_from  up_thru  down_at  last_clean_interval [,) 172.25.250.13:/ 172.25.250.13:/ 172.25.250.13:/ 172.25.250.13:/ exists,up 8c403679--48d0-812b-72050ad43aae
  41. osd. up   in  weight  up_from  up_thru  down_at  last_clean_interval [,) 172.25.250.12:/ 172.25.250.12:/ 172.25.250.12:/ 172.25.250.12:/ exists,up bb73edf8-ca97-40c3-a727-d5fde1a9d1d9

查询osd crush map

  1. [root@ceph2 ~]# ceph osd  crush dump
  2. {
  3.     "devices": [
  4.         {
  5.             "id": ,
  6.             "name": "osd.0",
  7.             "class": "hdd"
  8.         },
  9.         {
  10.             "id": ,
  11.             "name": "osd.1",
  12.             "class": "hdd"
  13.         },
  14.         {
  15.             "id": ,
  16.             "name": "osd.2",
  17.             "class": "hdd"
  18.         },
  19.         {
  20.             "id": ,
  21.             "name": "osd.3",
  22.             "class": "hdd"
  23.         },
  24.         {
  25.             "id": ,
  26.             "name": "osd.4",
  27.             "class": "hdd"
  28.         },
  29.         {
  30.             "id": ,
  31.             "name": "osd.5",
  32.             "class": "hdd"
  33.         },
  34.         {
  35.             "id": ,
  36.             "name": "osd.6",
  37.             "class": "hdd"
  38.         },
  39.         {
  40.             "id": ,
  41.             "name": "osd.7",
  42.             "class": "hdd"
  43.         },
  44.         {
  45.             "id": ,
  46.             "name": "osd.8",
  47.             "class": "hdd"
  48.         }
  49.     ],
  50.     "types": [
  51.         {
  52.             "type_id": ,
  53.             "name": "osd"
  54.         },
  55.         {
  56.             "type_id": ,
  57.             "name": "host"
  58.         },
  59.         {
  60.             "type_id": ,
  61.             "name": "chassis"
  62.         },
  63.         {
  64.             "type_id": ,
  65.             "name": "rack"
  66.         },
  67.         {
  68.             "type_id": ,
  69.             "name": "row"
  70.         },
  71.         {
  72.             "type_id": ,
  73.             "name": "pdu"
  74.         },
  75.         {
  76.             "type_id": ,
  77.             "name": "pod"
  78.         },
  79.         {
  80.             "type_id": ,
  81.             "name": "room"
  82.         },
  83.         {
  84.             "type_id": ,
  85.             "name": "datacenter"
  86.         },
  87.         {
  88.             "type_id": ,
  89.             "name": "region"
  90.         },
  91.         {
  92.             "type_id": ,
  93.             "name": "root"
  94.         },
  95.         {
  96.             "type_id": ,
  97.             "name": "aaa"
  98.         }
  99.     ],
  100.     "buckets": [
  101.         {
  102.             "id": -,
  103.             "name": "default",
  104.             "type_id": ,
  105.             "type_name": "root",
  106.             "weight": ,
  107.             "alg": "straw2",
  108.             "hash": "rjenkins1",
  109.             "items": [
  110.                 {
  111.                     "id": -,
  112.                     "weight": ,
  113.                     "pos":
  114.                 },
  115.                 {
  116.                     "id": -,
  117.                     "weight": ,
  118.                     "pos":
  119.                 },
  120.                 {
  121.                     "id": -,
  122.                     "weight": ,
  123.                     "pos":
  124.                 }
  125.             ]
  126.         },
  127.         {
  128.             "id": -,
  129.             "name": "default~hdd",
  130.             "type_id": ,
  131.             "type_name": "root",
  132.             "weight": ,
  133.             "alg": "straw2",
  134.             "hash": "rjenkins1",
  135.             "items": [
  136.                 {
  137.                     "id": -,
  138.                     "weight": ,
  139.                     "pos":
  140.                 },
  141.                 {
  142.                     "id": -,
  143.                     "weight": ,
  144.                     "pos":
  145.                 },
  146.                 {
  147.                     "id": -,
  148.                     "weight": ,
  149.                     "pos":
  150.                 }
  151.             ]
  152.         },
  153.         {
  154.             "id": -,
  155.             "name": "ceph2",
  156.             "type_id": ,
  157.             "type_name": "host",
  158.             "weight": ,
  159.             "alg": "straw2",
  160.             "hash": "rjenkins1",
  161.             "items": [
  162.                 {
  163.                     "id": ,
  164.                     "weight": ,
  165.                     "pos":
  166.                 },
  167.                 {
  168.                     "id": ,
  169.                     "weight": ,
  170.                     "pos":
  171.                 }
  172.             ]
  173.         },
  174.         {
  175.             "id": -,
  176.             "name": "ceph2~hdd",
  177.             "type_id": ,
  178.             "type_name": "host",
  179.             "weight": ,
  180.             "alg": "straw2",
  181.             "hash": "rjenkins1",
  182.             "items": [
  183.                 {
  184.                     "id": ,
  185.                     "weight": ,
  186.                     "pos":
  187.                 },
  188.                 {
  189.                     "id": ,
  190.                     "weight": ,
  191.                     "pos":
  192.                 }
  193.             ]
  194.         },
  195.         {
  196.             "id": -,
  197.             "name": "ceph4",
  198.             "type_id": ,
  199.             "type_name": "host",
  200.             "weight": ,
  201.             "alg": "straw2",
  202.             "hash": "rjenkins1",
  203.             "items": [
  204.                 {
  205.                     "id": ,
  206.                     "weight": ,
  207.                     "pos":
  208.                 },
  209.                 {
  210.                     "id": ,
  211.                     "weight": ,
  212.                     "pos":
  213.                 }
  214.             ]
  215.         },
  216.         {
  217.             "id": -,
  218.             "name": "ceph4~hdd",
  219.             "type_id": ,
  220.             "type_name": "host",
  221.             "weight": ,
  222.             "alg": "straw2",
  223.             "hash": "rjenkins1",
  224.             "items": [
  225.                 {
  226.                     "id": ,
  227.                     "weight": ,
  228.                     "pos":
  229.                 },
  230.                 {
  231.                     "id": ,
  232.                     "weight": ,
  233.                     "pos":
  234.                 }
  235.             ]
  236.         },
  237.         {
  238.             "id": -,
  239.             "name": "ceph3",
  240.             "type_id": ,
  241.             "type_name": "host",
  242.             "weight": ,
  243.             "alg": "straw2",
  244.             "hash": "rjenkins1",
  245.             "items": [
  246.                 {
  247.                     "id": ,
  248.                     "weight": ,
  249.                     "pos":
  250.                 },
  251.                 {
  252.                     "id": ,
  253.                     "weight": ,
  254.                     "pos":
  255.                 }
  256.             ]
  257.         },
  258.         {
  259.             "id": -,
  260.             "name": "ceph3~hdd",
  261.             "type_id": ,
  262.             "type_name": "host",
  263.             "weight": ,
  264.             "alg": "straw2",
  265.             "hash": "rjenkins1",
  266.             "items": [
  267.                 {
  268.                     "id": ,
  269.                     "weight": ,
  270.                     "pos":
  271.                 },
  272.                 {
  273.                     "id": ,
  274.                     "weight": ,
  275.                     "pos":
  276.                 }
  277.             ]
  278.         },
  279.         {
  280.             "id": -,
  281.             "name": "dc1",
  282.             "type_id": ,
  283.             "type_name": "root",
  284.             "weight": ,
  285.             "alg": "straw2",
  286.             "hash": "rjenkins1",
  287.             "items": [
  288.                 {
  289.                     "id": -,
  290.                     "weight": ,
  291.                     "pos":
  292.                 },
  293.                 {
  294.                     "id": -,
  295.                     "weight": ,
  296.                     "pos":
  297.                 },
  298.                 {
  299.                     "id": -,
  300.                     "weight": ,
  301.                     "pos":
  302.                 }
  303.             ]
  304.         },
  305.         {
  306.             "id": -,
  307.             "name": "rack1",
  308.             "type_id": ,
  309.             "type_name": "rack",
  310.             "weight": ,
  311.             "alg": "straw2",
  312.             "hash": "rjenkins1",
  313.             "items": [
  314.                 {
  315.                     "id": -,
  316.                     "weight": ,
  317.                     "pos":
  318.                 }
  319.             ]
  320.         },
  321.         {
  322.             "id": -,
  323.             "name": "rack2",
  324.             "type_id": ,
  325.             "type_name": "rack",
  326.             "weight": ,
  327.             "alg": "straw2",
  328.             "hash": "rjenkins1",
  329.             "items": [
  330.                 {
  331.                     "id": -,
  332.                     "weight": ,
  333.                     "pos":
  334.                 }
  335.             ]
  336.         },
  337.         {
  338.             "id": -,
  339.             "name": "rack3",
  340.             "type_id": ,
  341.             "type_name": "rack",
  342.             "weight": ,
  343.             "alg": "straw2",
  344.             "hash": "rjenkins1",
  345.             "items": [
  346.                 {
  347.                     "id": -,
  348.                     "weight": ,
  349.                     "pos":
  350.                 }
  351.             ]
  352.         },
  353.         {
  354.             "id": -,
  355.             "name": "rack3~hdd",
  356.             "type_id": ,
  357.             "type_name": "rack",
  358.             "weight": ,
  359.             "alg": "straw2",
  360.             "hash": "rjenkins1",
  361.             "items": [
  362.                 {
  363.                     "id": -,
  364.                     "weight": ,
  365.                     "pos":
  366.                 }
  367.             ]
  368.         },
  369.         {
  370.             "id": -,
  371.             "name": "rack2~hdd",
  372.             "type_id": ,
  373.             "type_name": "rack",
  374.             "weight": ,
  375.             "alg": "straw2",
  376.             "hash": "rjenkins1",
  377.             "items": [
  378.                 {
  379.                     "id": -,
  380.                     "weight": ,
  381.                     "pos":
  382.                 }
  383.             ]
  384.         },
  385.         {
  386.             "id": -,
  387.             "name": "rack1~hdd",
  388.             "type_id": ,
  389.             "type_name": "rack",
  390.             "weight": ,
  391.             "alg": "straw2",
  392.             "hash": "rjenkins1",
  393.             "items": [
  394.                 {
  395.                     "id": -,
  396.                     "weight": ,
  397.                     "pos":
  398.                 }
  399.             ]
  400.         },
  401.         {
  402.             "id": -,
  403.             "name": "dc1~hdd",
  404.             "type_id": ,
  405.             "type_name": "root",
  406.             "weight": ,
  407.             "alg": "straw2",
  408.             "hash": "rjenkins1",
  409.             "items": [
  410.                 {
  411.                     "id": -,
  412.                     "weight": ,
  413.                     "pos":
  414.                 },
  415.                 {
  416.                     "id": -,
  417.                     "weight": ,
  418.                     "pos":
  419.                 },
  420.                 {
  421.                     "id": -,
  422.                     "weight": ,
  423.                     "pos":
  424.                 }
  425.             ]
  426.         },
  427.         {
  428.             "id": -,
  429.             "name": "ceph2-ssd",
  430.             "type_id": ,
  431.             "type_name": "host",
  432.             "weight": ,
  433.             "alg": "straw2",
  434.             "hash": "rjenkins1",
  435.             "items": [
  436.                 {
  437.                     "id": ,
  438.                     "weight": ,
  439.                     "pos":
  440.                 }
  441.             ]
  442.         },
  443.         {
  444.             "id": -,
  445.             "name": "ceph3-ssd",
  446.             "type_id": ,
  447.             "type_name": "host",
  448.             "weight": ,
  449.             "alg": "straw2",
  450.             "hash": "rjenkins1",
  451.             "items": [
  452.                 {
  453.                     "id": ,
  454.                     "weight": ,
  455.                     "pos":
  456.                 }
  457.             ]
  458.         },
  459.         {
  460.             "id": -,
  461.             "name": "ceph4-ssd",
  462.             "type_id": ,
  463.             "type_name": "host",
  464.             "weight": ,
  465.             "alg": "straw2",
  466.             "hash": "rjenkins1",
  467.             "items": [
  468.                 {
  469.                     "id": ,
  470.                     "weight": ,
  471.                     "pos":
  472.                 }
  473.             ]
  474.         },
  475.         {
  476.             "id": -,
  477.             "name": "ssd-root",
  478.             "type_id": ,
  479.             "type_name": "root",
  480.             "weight": ,
  481.             "alg": "straw2",
  482.             "hash": "rjenkins1",
  483.             "items": [
  484.                 {
  485.                     "id": -,
  486.                     "weight": ,
  487.                     "pos":
  488.                 },
  489.                 {
  490.                     "id": -,
  491.                     "weight": ,
  492.                     "pos":
  493.                 },
  494.                 {
  495.                     "id": -,
  496.                     "weight": ,
  497.                     "pos":
  498.                 }
  499.             ]
  500.         },
  501.         {
  502.             "id": -,
  503.             "name": "ceph2-ssd~hdd",
  504.             "type_id": ,
  505.             "type_name": "host",
  506.             "weight": ,
  507.             "alg": "straw2",
  508.             "hash": "rjenkins1",
  509.             "items": [
  510.                 {
  511.                     "id": ,
  512.                     "weight": ,
  513.                     "pos":
  514.                 }
  515.             ]
  516.         },
  517.         {
  518.             "id": -,
  519.             "name": "ssd-root~hdd",
  520.             "type_id": ,
  521.             "type_name": "root",
  522.             "weight": ,
  523.             "alg": "straw2",
  524.             "hash": "rjenkins1",
  525.             "items": [
  526.                 {
  527.                     "id": -,
  528.                     "weight": ,
  529.                     "pos":
  530.                 },
  531.                 {
  532.                     "id": -,
  533.                     "weight": ,
  534.                     "pos":
  535.                 },
  536.                 {
  537.                     "id": -,
  538.                     "weight": ,
  539.                     "pos":
  540.                 }
  541.             ]
  542.         },
  543.         {
  544.             "id": -,
  545.             "name": "ceph4-ssd~hdd",
  546.             "type_id": ,
  547.             "type_name": "host",
  548.             "weight": ,
  549.             "alg": "straw2",
  550.             "hash": "rjenkins1",
  551.             "items": [
  552.                 {
  553.                     "id": ,
  554.                     "weight": ,
  555.                     "pos":
  556.                 }
  557.             ]
  558.         },
  559.         {
  560.             "id": -,
  561.             "name": "ceph3-ssd~hdd",
  562.             "type_id": ,
  563.             "type_name": "host",
  564.             "weight": ,
  565.             "alg": "straw2",
  566.             "hash": "rjenkins1",
  567.             "items": [
  568.                 {
  569.                     "id": ,
  570.                     "weight": ,
  571.                     "pos":
  572.                 }
  573.             ]
  574.         }
  575.     ],
  576.     "rules": [
  577.         {
  578.             "rule_id": ,
  579.             "rule_name": "replicated_rule",
  580.             "ruleset": ,
  581.             "type": ,
  582.             "min_size": ,
  583.             "max_size": ,
  584.             "steps": [
  585.                 {
  586.                     "op": "take",
  587.                     "item": -,
  588.                     "item_name": "default"
  589.                 },
  590.                 {
  591.                     "op": "chooseleaf_firstn",
  592.                     "num": ,
  593.                     "type": "host"
  594.                 },
  595.                 {
  596.                     "op": "emit"
  597.                 }
  598.             ]
  599.         },
  600.         {
  601.             "rule_id": ,
  602.             "rule_name": "replicated1_rule",
  603.             "ruleset": ,
  604.             "type": ,
  605.             "min_size": ,
  606.             "max_size": ,
  607.             "steps": [
  608.                 {
  609.                     "op": "take",
  610.                     "item": -,
  611.                     "item_name": "default"
  612.                 },
  613.                 {
  614.                     "op": "chooseleaf_firstn",
  615.                     "num": ,
  616.                     "type": "host"
  617.                 },
  618.                 {
  619.                     "op": "emit"
  620.                 }
  621.             ]
  622.         },
  623.         {
  624.             "rule_id": ,
  625.             "rule_name": "indc1",
  626.             "ruleset": ,
  627.             "type": ,
  628.             "min_size": ,
  629.             "max_size": ,
  630.             "steps": [
  631.                 {
  632.                     "op": "take",
  633.                     "item": -,
  634.                     "item_name": "dc1"
  635.                 },
  636.                 {
  637.                     "op": "chooseleaf_firstn",
  638.                     "num": ,
  639.                     "type": "rack"
  640.                 },
  641.                 {
  642.                     "op": "emit"
  643.                 }
  644.             ]
  645.         },
  646.         {
  647.             "rule_id": ,
  648.             "rule_name": "ssdrule",
  649.             "ruleset": ,
  650.             "type": ,
  651.             "min_size": ,
  652.             "max_size": ,
  653.             "steps": [
  654.                 {
  655.                     "op": "take",
  656.                     "item": -,
  657.                     "item_name": "ssd-root"
  658.                 },
  659.                 {
  660.                     "op": "chooseleaf_firstn",
  661.                     "num": ,
  662.                     "type": "host"
  663.                 },
  664.                 {
  665.                     "op": "emit"
  666.                 }
  667.             ]
  668.         }
  669.     ],
  670.     "tunables": {
  671.         "choose_local_tries": ,
  672.         "choose_local_fallback_tries": ,
  673.         "choose_total_tries": ,
  674.         "chooseleaf_descend_once": ,
  675.         "chooseleaf_vary_r": ,
  676.         "chooseleaf_stable": ,
  677.         "straw_calc_version": ,
  678.         "allowed_bucket_algs": ,
  679.         "profile": "jewel",
  680.         "optimal_tunables": ,
  681.         "legacy_tunables": ,
  682.         "minimum_required_version": "jewel",
  683.         "require_feature_tunables": ,
  684.         "require_feature_tunables2": ,
  685.         "has_v2_rules": ,
  686.         "require_feature_tunables3": ,
  687.         "has_v3_rules": ,
  688.         "has_v4_buckets": ,
  689.         "require_feature_tunables5": ,
  690.         "has_v5_rules":
  691.     },
  692.     "choose_args": {}
  693. }

osd crush map

查询pg map

  1. [root@ceph2 ~]# ceph pg dump|more
  2. dumped all
  3. version
  4. stamp -- ::04.562309
  5. last_osdmap_epoch
  6. last_pg_scan
  7. full_ratio
  8. nearfull_ratio
  9. PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES    LOG  DISK_LOG STATE        STATE_STAMP                VERSION    REPORTED   UP      UP_PRIMARY ACTING  ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP
  10.              LAST_DEEP_SCRUB DEEP_SCRUB_STAMP
  11. 15.71                                                                        active+clean -- ::32.993398        '0    349:100 [1,8,3]          1 [1,8,3]              1        0' -- :
  12. :33.926844             '0 2019-03-27 20:08:33.926844
  13. 1.7f                                                                         active+clean -- ::33.579411        '0    349:505 [8,7,6]          8 [8,7,6]              8        0' -- :
  14. :11.129286             '0 2019-03-21 00:37:48.245632
  15. 15.70                                                                        active+clean -- ::33.014554        '0    349:118 [3,1,8]          3 [3,1,8]              3        0' -- :
  16. :35.263257             '0 2019-03-26 17:10:19.390530
  17. .7e                                                                         active+clean -- ::34.401107        '0     349:18 [6,4,8]          6 [6,4,8]              6        0' -- :
  18. :30.900982             '0 2019-03-24 06:16:20.594466
  19. 15.73                                                                        active+clean -- ::32.722556        '0    349:107 [2,4,3]          2 [2,4,3]              2        0' -- :
  20. :48.489676             '0 2019-03-26 17:10:19.390530
  21. 1.7d                                                                         active+clean -- ::32.509177        '0    349:611 [3,2,7]          3 [3,2,7]              3        0' -- :
  22. :45.842781             '0 2019-03-24 00:45:38.159371
  23. 15.72                                                                        active+clean -- ::53.428161        '0    349:128 [2,4,6]          2 [2,4,6]              2        0' -- :
  24. :37.129695             '0 2019-03-26 17:10:19.390530
  25. .7c                                                                         active+clean -- ::31.590563        '0     349:18 [7,2,6]          7 [7,2,6]              7        0' -- :
  26. :05.697728             '0 2019-03-27 05:33:02.267544
  27. 15.75                                                                        active+clean -- ::53.899879        '0     349:19 [6,7,8]          6 [6,7,8]              6        0' -- :
  28. :45.705922             '0 2019-03-26 17:10:19.390530 
  29.  
  30. sum
  31. OSD_STAT USED  AVAIL  TOTAL  HB_PEERS        PG_SUM PRIMARY_PG_SUM
  32.          201M 15147M 15348M [,,,,,,]
  33.          207M 15141M 15348M [,,,,,,]
  34.          216M 15132M 15348M [,,,,,,]
  35.          188M 15160M 15348M [,,,,,,]
  36.          209M 15139M 15348M [,,,,,,]
  37.          260M 15088M 15348M [,,,,,,]
  38.          197M 15151M 15348M [,,,,,,]
  39.          173M 15175M 15348M [,,,,,,]
  40. sum      1656M   118G   119G

查询fs map

  1. [root@ceph2 ~]# ceph fs dump
  2. dumped fsmap epoch
  3. e9
  4. enable_multiple, ever_enabled_multiple: ,
  5. compat: compat={},rocompat={},incompat={=base v0.,=client writeable ranges,=default file layouts on dirs,=dir inode in separate object,=mds uses versioned encoding,=dirfrag is stored in omap,=file layout v2}
  6. legacy client fscid:
  7. Filesystem 'cephfs' ()
  8. fs_name    cephfs
  9. epoch
  10. flags    c
  11. created    -- ::16.787966
  12. modified    -- ::16.787966
  13. tableserver
  14. root
  15. session_timeout
  16. session_autoclose
  17. max_file_size
  18. last_failure
  19. last_failure_osd_epoch
  20. compat    compat={},rocompat={},incompat={=base v0.,=client writeable ranges,=default file layouts on dirs,=dir inode in separate object,=mds uses versioned encoding,=dirfrag is stored in omap,=file layout v2}
  21. max_mds
  22. in
  23. up    {=}
  24. failed
  25. damaged
  26. stopped
  27. data_pools    []
  28. metadata_pool
  29. inline_data    disabled
  30. balancer
  31. standby_count_wanted
  32. :    172.25.250.11:/ 'ceph2' mds.0.6 up:active seq
  33. Standby daemons:
  34. :    172.25.250.12:/ 'ceph3' mds.-1.0 up:standby seq

查询 mgr map

  1. [root@ceph2 ~]# ceph mgr dump
  2. {
  3.     "epoch": ,
  4.     "active_gid": ,
  5.     "active_name": "ceph4",
  6.     "active_addr": "172.25.250.13:6800/60569",
  7.     "available": true,
  8.     "standbys": [
  9.         {
  10.             "gid": ,
  11.             "name": "ceph2",
  12.             "available_modules": [
  13.                 "dashboard",
  14.                 "prometheus",
  15.                 "restful",
  16.                 "status",
  17.                 "zabbix"
  18.             ]
  19.         },
  20.         {
  21.             "gid": ,
  22.             "name": "ceph3",
  23.             "available_modules": [
  24.                 "dashboard",
  25.                 "prometheus",
  26.                 "restful",
  27.                 "status",
  28.                 "zabbix"
  29.             ]
  30.         }
  31.     ],
  32.     "modules": [
  33.         "restful",
  34.         "status"
  35.     ],
  36.     "available_modules": [
  37.         "dashboard",
  38.         "prometheus",
  39.         "restful",
  40.         "status",
  41.         "zabbix"
  42.     ]
  43. }

查询 service dump

  1. [root@ceph2 ~]# ceph service dump
  2. {
  3.     "epoch": ,
  4.     "modified": "2019-03-18 21:19:18.667275",
  5.     "services": {
  6.         "rbd-mirror": {
  7.             "daemons": {
  8.                 "summary": "",
  9.                 "admin": {
  10.                     "start_epoch": ,
  11.                     "start_stamp": "2019-03-18 21:19:18.318802",
  12.                     "gid": ,
  13.                     "addr": "172.25.250.11:0/4114752834",
  14.                     "metadata": {
  15.                         "arch": "x86_64",
  16.                         "ceph_version": "ceph version 12.2.1-40.el7cp (c6d85fd953226c9e8168c9abe81f499d66cc2716) luminous (stable)",
  17.                         "cpu": "QEMU Virtual CPU version 1.5.3",
  18.                         "distro": "rhel",
  19.                         "distro_description": "Red Hat Enterprise Linux Server 7.4 (Maipo)",
  20.                         "distro_version": "7.4",
  21.                         "hostname": "ceph2",
  22.                         "instance_id": "",
  23.                         "kernel_description": "#1 SMP Thu Dec 28 14:23:39 EST 2017",
  24.                         "kernel_version": "3.10.0-693.11.6.el7.x86_64",
  25.                         "mem_swap_kb": "",
  26.                         "mem_total_kb": "",
  27.                         "os": "Linux"
  28.                     }
  29.                 }
  30.             }
  31.         }
  32.     }
  33. }

二、 管理monitor map

2.1 多Momitor的同步机制

在生产环境建议最少三节点monitor，以确保cluster map的高可用性和冗余性

monitor使用paxos算法作为集群状态上达成一致的机制。paxos是一种分布式一致性算法。每当monitor修改map时，它会通过paxos发送更新到其他monitor。Ceph只有在大多数monitor就更新达成一致时提交map的新版本

cluster map的更新操作需要Paxos确认，但是读操作不经由Paxos，而是直接访问本地的kv存储

2.2 Monitor的选举机制

多个monitor之间需要建立仲裁并选择出一个leader，其他节点则作为工作节点（peon）

在选举完成并确定leader之后，leader将从所有其他monitor请求最新的map epoc，以确保leader具有集群的最新视图

要维护monitor集群的正常工作，必须有超过半数的节点正常

2.3 Monitor租期

在Monitor建立仲裁后，leader开始分发短期的租约到所有的monitors。让它们能够分发cluster map到OSD和client

Monitor租约默认每3s续期一次

当peon monitor没有确认它收到租约时，leader假定该monitor异常，它会召集新的选举以建立仲裁

如果peon monitor的租约到期后没有收到leader的续期，它会假定leader异常，并会召集新的选举

2.4 管理monitor map

将monitor map导出为一个二进制文件

[root@ceph2 ~]# ceph mon getmap -o ./monmap

  1. got monmap epoch

打印导出的二进制文件的内容

[root@ceph2 ~]# monmaptool --print  ./monmap

  1. monmaptool: monmap file ./monmap
  2. epoch
  3. fsid 35a91e48--4e96-a7ee-980ab989d20d
  4. last_changed -- ::14.839999
  5. created -- ::14.839999
  6. : 172.25.250.11:/ mon.ceph2
  7. : 172.25.250.12:/ mon.ceph3
  8. : 172.25.250.13:/ mon.ceph4

修改二进制文件，从monmap删除某个monitor

[root@ceph2 ~]# monmaptool ./monmap  --rm ceph2

  1. monmaptool: monmap file ./monmap
  2. monmaptool: removing ceph2
  3. monmaptool: writing epoch  to ./monmap ( monitors)

[root@ceph2 ~]# monmaptool --print ./monmap

  1. monmaptool: monmap file ./monmap
  2. epoch
  3. fsid 35a91e48--4e96-a7ee-980ab989d20d
  4. last_changed -- ::14.839999
  5. created -- ::14.839999
  6. : 172.25.250.12:/ mon.ceph3
  7. : 172.25.250.13:/ mon.ceph4

[root@ceph2 ~]# ceph mon dump

  1. dumped monmap epoch
  2. epoch
  3. fsid 35a91e48--4e96-a7ee-980ab989d20d
  4. last_changed -- ::14.839999
  5. created -- ::14.839999
  6. : 172.25.250.11:/ mon.ceph2
  7. : 172.25.250.12:/ mon.ceph3
  8. : 172.25.250.13:/ mon.ceph4

修改二进制文件，往monmap中添加一个monitor

[root@ceph2 ~]# monmaptool ./monmap --add ceph2 172.25.254.11:6789

  1. monmaptool: monmap file ./monmap
  2. monmaptool: writing epoch  to ./monmap ( monitors)
  3. [root@ceph2 ~]# monmaptool --print ./monmap
  4. monmaptool: monmap file ./monmap
  5. epoch
  6. fsid 35a91e48--4e96-a7ee-980ab989d20d
  7. last_changed -- ::14.839999
  8. created -- ::14.839999
  9. : 172.25.250.12:/ mon.ceph3
  10. : 172.25.250.13:/ mon.ceph4
  11. : 172.25.254.11:/ mon.ceph2

导入一个二进制文件，在导入之前，需要先停止monitor

ceph-mon -i <id> --inject-monmap ./monmap

三、 管理osd map

3.1 OSD map生命周期

每当OSD加入或离开集群时，Ceph都会更新OSD map

OSD不使用leader来管理OSD map，它们会在自身之间传播map。OSD会利用OSD map epoch标记它们交换的每一条信息，当OSD检测到自己已落后时，它会使用其对等OSD执行map更新

在大型集群中OSD map更新会非常频繁，节点会执行递增map更新

Ceph也会利用epoch来标记OSD和client之间的消息。当client连接到OSD时OSD会检查epoch。如果发现epoch不匹配，则OSD会以正确的epoch响应，以便客户端可以更新其OSD map

OSD定期向monitor报告自己的状态，OSD之间会交换心跳，以便检测对等点的故障，并报告给monitor

leader monitor发现OSD故障时，它会更新map，递增epoch，并使用Paxos更新协议来通知其他monitor，同时撤销租约，并发布新的租约，以使monitor以分发最新的OSD map

3.2 管理 osd map

  1. [root@ceph2 ~]#  ceph osd getmap -o ./osdmap
  2. got osdmap epoch
  3. [root@ceph2 ~]# osdmaptool --print ./osdmap
  4. osdmaptool: osdmap file './osdmap'
  5. epoch
  6. fsid 35a91e48--4e96-a7ee-980ab989d20d
  7. created -- ::22.552356
  8. modified -- ::15.354383
  9. flags sortbitwise,recovery_deletes,purged_snapdirs
  10. crush_version
  11. full_ratio 0.95
  12. backfillfull_ratio 0.9
  13. nearfull_ratio 0.85
  14. require_min_compat_client jewel
  15. min_compat_client jewel
  16. require_osd_release luminous
  17. pool  'testpool' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application rbd
  18.     snap  'testpool-snap-20190316' -- ::34.150433
  19.     snap  'testpool-snap-2' -- ::15.430823
  20. pool  'rbd' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application rbd
  21.     removed_snaps [~]
  22. pool  'rbdmirror' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application rbd
  23.     removed_snaps [~]
  24. pool  '.rgw.root' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application rgw
  25. pool  'default.rgw.control' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application rgw
  26. pool  'default.rgw.meta' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application rgw
  27. pool  'default.rgw.log' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application rgw
  28. pool  'xiantao.rgw.control' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  owner  flags hashpspool stripe_width  application rgw
  29. pool  'xiantao.rgw.meta' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  owner  flags hashpspool stripe_width  application rgw
  30. pool  'xiantao.rgw.log' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  owner  flags hashpspool stripe_width  application rgw
  31. pool  'cephfs_metadata' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application cephfs
  32. pool  'cephfs_data' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application cephfs
  33. pool  'test' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application rbd
  34. pool  'ssdpool' replicated size  min_size  crush_rule  object_hash rjenkins pg_num  pgp_num  last_change  flags hashpspool stripe_width  application rbd
  35. max_osd
  36. osd. up   in  weight  up_from  up_thru  down_at  last_clean_interval [,) 172.25.250.11:/ 172.25.250.11:/ 172.25.250.11:/ 172.25.250.11:/ exists,up 745dce53-1c63-4c50-b434-d441038dafe4
  37. osd. up   in  weight  up_from  up_thru  down_at  last_clean_interval [,) 172.25.250.13:/ 172.25.250.13:/ 172.25.250.13:/ 172.25.250.13:/ exists,up a7562276-6dfd--b248-a7cbdb64ebec
  38. osd. up   in  weight  up_from  up_thru  down_at  last_clean_interval [,) 172.25.250.12:/ 172.25.250.12:/ 172.25.250.12:/ 172.25.250.12:/ exists,up bbef1a00-3a31-48a0-a065-3a16b9edc3b1
  39. osd. up   in  weight  up_from  up_thru  down_at  last_clean_interval [,) 172.25.250.11:/ 172.25.250.11:/ 172.25.250.11:/ 172.25.250.11:/ exists,up e934a4fb--4e85-895c-f66cc5534ceb
  40. osd. up   in  weight  up_from  up_thru  down_at  last_clean_interval [,) 172.25.250.13:/ 172.25.250.13:/ 172.25.250.13:/ 172.25.250.13:/ exists,up e2c33bb3-02d2-4cce-85e8-25c419351673
  41. osd. up   in  weight  up_from  up_thru  down_at  last_clean_interval [,) 172.25.250.12:/ 172.25.250.12:/ 172.25.250.12:/ 172.25.250.12:/ exists,up d299e33c-0c24-4cd9-a37a-a6fcd420a529
  42. osd. up   in  weight  up_from  up_thru  down_at  last_clean_interval [,) 172.25.250.11:/ 172.25.250.11:/ 172.25.250.11:/ 172.25.250.11:/ exists,up debe7f4e-656b-48e2-a0b2-bdd8613afcc4
  43. osd. up   in  weight  up_from  up_thru  down_at  last_clean_interval [,) 172.25.250.13:/ 172.25.250.13:/ 172.25.250.13:/ 172.25.250.13:/ exists,up 8c403679--48d0-812b-72050ad43aae
  44. osd. up   in  weight  up_from  up_thru  down_at  last_clean_interval [,) 172.25.250.12:/ 172.25.250.12:/ 172.25.250.12:/ 172.25.250.12:/ exists,up bb73edf8-ca97-40c3-a727-d5fde1a9d1d9

博主声明：本文的内容来源主要来自誉天教育晏威老师，由本人实验完成操作验证，需要的博友请联系誉天教育（http://www.yutianedu.com/），获得官方同意或者晏老师（https://www.cnblogs.com/breezey/）本人同意即可转载，谢谢！

015 Ceph的集群管理_1的更多相关文章

  1. 017 Ceph的集群管理_3

    一.验证OSD 1.1 osd状态 运行状态有:up,in,out,down 正常状态的OSD为up且in 当OSD故障时,守护进程offline,在5分钟内,集群仍会将其标记为up和in,这是为了防 ...

  2. 016 Ceph的集群管理_2

    一.Ceph集群的运行状态 集群状态:HEALTH_OK,HEALTH_WARN,HEALTH_ERR 1.1 常用查寻状态指令 [root@ceph2 ~]#    ceph health deta ...

  3. Centos6.5下一个Ceph存储集群结构

    简单的介绍 Ceph的部署模式下主要包括下面几个类型的节点 • Ceph OSDs: A Ceph OSD 进程主要用来存储数据,处理数据的replication,恢复,填充.调整资源组合以及通过检查 ...

  4. 002.RHCS-配置Ceph存储集群

    一 前期准备 [kiosk@foundation0 ~]$ ssh ceph@serverc #登录Ceph集群节点 [ceph@serverc ~]$ ceph health #确保集群状态正常 H ...

  5. 容器、容器集群管理平台与 Kubernetes 技术漫谈

    原文:https://www.kubernetes.org.cn/4786.html 我们为什么使用容器? 我们为什么使用虚拟机(云主机)? 为什么使用物理机? 这一系列的问题并没有一个统一的标准答案 ...

  6. 003.Ceph扩展集群

    一 基础准备 参考<002.Ceph安装部署>文档部署一个基础集群. 二 扩展集群 2.1 扩展架构 需求:添加Ceph元数据服务器node1.然后添加Ceph Monitor和Ceph ...

  7. Ceph 存储集群 - 搭建存储集群

    目录 一.准备机器 二.ceph节点安装 三.搭建集群 四.扩展集群(扩容)   一.准备机器 本文描述如何在 CentOS 7 下搭建 Ceph 存储集群(STORAGE CLUSTER). 一共4 ...

  8. Ceph 存储集群搭建

    前言 Ceph 分布式存储系统,在企业中应用面较广 初步了解并学会使用很有必要 一.简介 Ceph 是一个开源的分布式存储系统,包括对象存储.块设备.文件系统.它具有高可靠性.安装方便.管理简便.能够 ...

  9. ceph部署-集群建立

    一.配置storage集群1.建立集群管理目录(管理配置文件,密钥)mkdir ceph-clustercd ceph-cluster/ 2.创建一个新集群(需要先将主机名加入/etc/hosts 必 ...

随机推荐

  1. 三 nginx+uWSGI+django+virtualenv+supervisor发布web服务器

    https://www.cnblogs.com/pyyu/p/9481344.html?tdsourcetag=s_pcqq_aiomsg 一 uwsgi安装 1.安装uwsgi,进入到一个虚拟机环境 ...

  2. Java成员方法，构造方法

    //作者:qingfeng//2017/2/17//功能:理解类的成员方法和构造方法class CreatWays{ public static void main(String args[]){   ...

  3. OpenKruise - 云原生应用自动化引擎正式开源

    2019 年 6 月 24 日至 26 日, 由 Cloud Native Computing Foundation (CNCF) 主办的云原生技术大会 KubeCon + CloudNativeCo ...

  4. Android内核剖析读书笔记(1)—Framework概述

    一.Framework组成 1.服务端组成 a.WindowManagerService     决定各窗口的叠放次序.隐藏或者显示窗口 b.ActivityManagerService   管理应用 ...

  5. OpenStack Nova启动实例流程

    1.概述 启动一个新的实例,会涉及到OpenStack Nova中的多个组件: API服务器,接收用户端的请求,并且将其传递给云控制器. 云控制器,处理计算节点.网络控制器.API服务器和调度器之前的 ...

  6. Vue6——v-model实现数据双向绑定

    博客地址 :https://www.cnblogs.com/sandraryan/ v-model 用于input标签,用于实现双向绑定,及时把数据的变化渲染在页面 双向绑定及原理 举个双向绑定的简单 ...

  7. 解决电脑性能一般，打开webstorm后，电脑比较卡的问题

    刚到一公司实习,要求使用webstrom开发前端,但安装后发现自己的电脑很卡,特别是在运行项目时,卡的不要不要的. 后来,发现一奇淫技巧,用sublime代替webstrom,但是没法启动项目啊 找到 ...

  8. [转]C# 语言历史版本特性（C# 1.0到C# 8.0汇总）

    历史版本 C#作为微软2000年以后.NET平台开发的当家语言,发展至今具有17年的历史,语言本身具有丰富的特性,微软对其更新支持也十分支持.微软将C#提交给标准组织ECMA,C# 5.0目前是ECM ...

  9. vue v-for循环中key属性的使用

    <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8&quo ...

  10. win10 uwp 使用 Microsoft.Graph 发送邮件

    在 2018 年 10 月 13 号参加了 张队长 的 Office 365 训练营 学习如何开发 Office 365 插件和 OAuth 2.0 开发,于是我就使用 UWP 尝试使用 Micros ...