pve之daemon

pmxcfs

The Proxmox Cluster file system (“pmxcfs”) is a database-driven file system for storing configuration files, replicated in real time to all cluster nodes using corosync. We use this to store all PVE related configuration files.
Although the file system stores all data inside a persistent database on disk, a copy of the data resides in RAM. That imposes restriction on the maximum size, which is currently 30MB. This is still enough to store the configuration of several thousand virtual machines.
We use the Corosync Cluster Engine for cluster communication, and SQlite for the database file. The file system is implemented in user space using FUSE.

The file system is mounted at:
/etc/pve

This service is usually started and managed using systemd toolset. The service is called pve-cluster.

root@cu-pve04:~# systemctl status pve-cluster
● pve-cluster.service - The Proxmox VE cluster filesystem
   Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-04-30 21:20:23 CST; 1 weeks 0 days ago
 Main PID: 3745 (pmxcfs)
    Tasks: 13 (limit: 17203)
   Memory: 88.9M
      CPU: 25min 56.856s
   CGroup: /system.slice/pve-cluster.service
           └─3745 /usr/bin/pmxcfs

May 08 15:39:22 cu-pve04 pmxcfs[3745]: [dcdb] notice: received all states
May 08 15:39:22 cu-pve04 pmxcfs[3745]: [dcdb] notice: leader is 1/3745
May 08 15:39:22 cu-pve04 pmxcfs[3745]: [dcdb] notice: synced members: 1/3745, 3/3878
May 08 15:39:22 cu-pve04 pmxcfs[3745]: [dcdb] notice: start sending inode updates
May 08 15:39:22 cu-pve04 pmxcfs[3745]: [dcdb] notice: sent all (3) updates
May 08 15:39:22 cu-pve04 pmxcfs[3745]: [dcdb] notice: all data is up to date
May 08 15:39:22 cu-pve04 pmxcfs[3745]: [status] notice: received sync request (epoch 1/3745/0000000F)
May 08 15:39:22 cu-pve04 pmxcfs[3745]: [status] notice: received all states
May 08 15:39:22 cu-pve04 pmxcfs[3745]: [status] notice: all data is up to date
May 08 15:39:23 cu-pve04 pmxcfs[3745]: [status] notice: received log

root@cu-pve04:/var/lib/pve-cluster# ls -l
total 4136
-rw------- 1 root root   77824 May  8 15:17 config.db
-rw------- 1 root root   32768 May  8 15:18 config.db-shm
-rw------- 1 root root 4124152 May  8 15:18 config.db-wal
root@cu-pve04:/var/lib/pve-cluster# file *
config.db:     SQLite 3.x database, last written using SQLite version 3016002
config.db-shm: data
config.db-wal: SQLite Write-Ahead Log, version 3007000

root@cu-pve04:/etc/pve# ls -l
total 5
-rw-r----- 1 root www-data  451 Apr 30 14:23 authkey.pub
-rw-r----- 1 root www-data  881 Apr 30 20:47 ceph.conf
-rw-r----- 1 root www-data  545 Apr 30 17:22 corosync.conf
-rw-r----- 1 root www-data   16 Apr 30 14:09 datacenter.cfg
-rw-r----- 1 root www-data 2057 Apr 30 14:23 pve-root-ca.pem
-rw-r----- 1 root www-data 1675 Apr 30 14:23 pve-www.key
-rw-r----- 1 root www-data  177 May  7 17:54 storage.cfg
-rw-r----- 1 root www-data   66 May  4 20:38 user.cfg
-rw-r----- 1 root www-data  119 Apr 30 14:23 vzdump.cron

drwxr-xr-x 2 root www-data    0 Apr 30 14:23 nodes
drwx------ 2 root www-data    0 Apr 30 14:23 priv

lrwxr-xr-x 1 root www-data    0 Jan  1  1970 local -> nodes/cu-pve04
lrwxr-xr-x 1 root www-data    0 Jan  1  1970 lxc -> nodes/cu-pve04/lxc
lrwxr-xr-x 1 root www-data    0 Jan  1  1970 openvz -> nodes/cu-pve04/openvz
lrwxr-xr-x 1 root www-data    0 Jan  1  1970 qemu-server -> nodes/cu-pve04/qemu-server

------------------------------------------------------------

虚拟机包括100.conf和images即100.disk两种文件

The /etc/pve/qemu-server/<VMID>.conf files stores VM configuration, where "VMID" is the numeric ID of the given VM.

One can use the qm command to generate and modify those files.

root@cu-pve04:/etc/pve/nodes# ls -R
.:
cu-pve04  cu-pve05  cu-pve06

./cu-pve04:
lrm_status  lxc  openvz  priv  pve-ssl.key  pve-ssl.pem  qemu-server

./cu-pve04/lxc:

./cu-pve04/openvz:

./cu-pve04/priv:

./cu-pve04/qemu-server:
100.conf  101.conf  102.conf  103.conf  105.conf  106.conf  107.conf

./cu-pve05:
lrm_status  lxc  openvz  priv  pve-ssl.key  pve-ssl.pem  qemu-server

./cu-pve05/lxc:

./cu-pve05/openvz:

./cu-pve05/priv:

./cu-pve05/qemu-server:
104.conf  108.conf

./cu-pve06:
lrm_status  lxc  openvz  priv  pve-ssl.key  pve-ssl.pem  qemu-server

./cu-pve06/lxc:

./cu-pve06/openvz:

./cu-pve06/priv:

./cu-pve06/qemu-server:
109.conf

pvedaemon - PVE API Daemon

This daemon exposes the whole Proxmox VE API on 127.0.0.1:85. It runs as root and has permission to do all privileged operations.
The daemon listens to a local address only, so you cannot access it from outside. The pveproxy daemon exposes the API to the outside world.

root@cu-pve04:~# pvedaemon status
running

root@cu-pve04:~# ss -lntp|grep proxy
LISTEN   0         128                 0.0.0.0:3128             0.0.0.0:*        users:(("spiceproxy work",pid=2195573,fd=6),("spiceproxy",pid=7541,fd=6))
LISTEN   0         128                 0.0.0.0:8006             0.0.0.0:*        users:(("pveproxy worker",pid=2310656,fd=6),("pveproxy worker",pid=2305989,fd=6),("pveproxy worker",pid=2295860,fd=6),("pveproxy",pid=7522,fd=6))
root@cu-pve04:~# ss -lntp|grep daemon
LISTEN   0         128               127.0.0.1:85               0.0.0.0:*        users:(("pvedaemon worke",pid=2252583,fd=6),("pvedaemon worke",pid=2250382,fd=6),("pvedaemon worke",pid=2250172,fd=6),("pvedaemon",pid=4500,fd=6))
--------------------------------------------------------

pveproxy - PVE API Proxy Daemon

This daemon exposes the whole Proxmox VE API on TCP port 8006 using HTTPS. It runs as user www-data and has very limited permissions. Operation requiring more permissions are forwarded to the local pvedaemon.
Requests targeted for other nodes are automatically forwarded to those nodes. This means that you can manage your whole cluster by connecting to a single Proxmox VE node.

It is possible to configure “apache2”-like access control lists. Values are read from file /etc/default/pveproxy.

root@cu-pve04:~# pveproxy status
running

/etc/defaults/

-----------------------------------------------------------------------

pvestatd - PVE Status Daemon

This daemon queries the status of VMs, storages and containers at regular intervals. The result is sent to all nodes in the cluster.

root@cu-pve04:/etc/default# pvestatd status
running

----------------------------------------------------------------------
qmeventd - PVE Qemu Eventd Daemon

This service is usually started and managed using systemd toolset. The service is called qmeventd.

/usr/sbin/qm cleanup

systemctl status qmeventd

root@cu-pve04:/etc/default# systemctl status qmeventd
● qmeventd.service - PVE Qemu Event Daemon
Loaded: loaded (/lib/systemd/system/qmeventd.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2019-04-30 21:20:19 CST; 1 weeks 0 days ago
Main PID: 2829 (qmeventd)
Tasks: 1 (limit: 17203)
Memory: 8.8M
CPU: 10.100s
CGroup: /system.slice/qmeventd.service
└─2829 /usr/sbin/qmeventd /var/run/qmeventd.sock

May 07 13:47:13 cu-pve04 qmeventd[2807]: OK
May 07 13:47:13 cu-pve04 qmeventd[2807]: Finished cleanup for 103
May 08 15:01:40 cu-pve04 qmeventd[2807]: file /etc/pve/storage.cfg line 4 (section 'local') - ignore config line: cephfs: kycfs
May 08 15:01:40 cu-pve04 qmeventd[2807]: file /etc/pve/storage.cfg line 5 (section 'local') - unable to parse value of 'path': duplicate attribute
May 08 15:01:40 cu-pve04 qmeventd[2807]: file /etc/pve/storage.cfg line 6 (section 'local') - unable to parse value of 'content': duplicate attribute
May 08 15:01:40 cu-pve04 qmeventd[2807]: Starting cleanup for 107
May 08 15:01:40 cu-pve04 qmeventd[2807]: trying to acquire lock...
May 08 15:01:41 cu-pve04 qmeventd[2807]: OK
May 08 15:01:41 cu-pve04 qmeventd[2807]: storage 'kycfs' does not exists
May 08 15:01:41 cu-pve04 qmeventd[2807]: Finished cleanup for 107

------------------------------------------------------

pve-ha-lrm - PVE Local Resource Manager Daemon
root@cu-pve05:~# pve-ha-lrm status
running
------------------------------------------

pve-ha-crm - PVE Cluster Resource Manager Daemon
root@cu-pve05:~# pve-ha-crm status
running

------------------------------------------
spiceproxy - SPICE Proxy Service
root@cu-pve04:~# spiceproxy status
running

This daemon listens on TCP port 3128, and implements an HTTP proxy to forward CONNECT request from the SPICE client to the correct Proxmox VE VM. It runs as user www-data and has very limited permissions.

It is possible to configure "apache2" like access control lists. Values are read from file /etc/default/pveproxy.

------------------------------------------

pve-firewall - PVE Firewall Daemon

All firewall related configuration is stored on the proxmox cluster file system. So those files are automatically distributed to all cluster nodes, and the pve-firewall service updates the underlying iptables rules automatically on changes.
You can configure anything using the GUI (i.e. Datacenter → Firewall, or on a Node → Firewall), or you can edit the configuration files directly using your preferred editor.

If you enable the firewall, traffic to all hosts is blocked by default. Only exceptions is WebGUI(8006) and ssh(22) from your local network.

Each virtual network device has its own firewall enable flag. So you can selectively enable the firewall for each interface. This is required in addition to the general firewall enable option.

The firewall runs two service daemons on each node:
pvefw-logger: NFLOG daemon (ulogd replacement).
pve-firewall: updates iptables rules

root@cu-pve04:~# pve-firewall compile
ipset cmdlist:

iptables cmdlist:

ip6tables cmdlist:

ebtables cmdlist:
no changes
firewall disabled

root@cu-pve04:~# pve-firewall localnet
local hostname: cu-pve04
local IP address: 192.168.1.4
network auto detect: 192.168.1.0/24
using detected local_network: 192.168.1.0/24
root@cu-pve04:~# pve-firewall status
Status: disabled/running
root@cu-pve04:~# pve-firewall stop
root@cu-pve04:~# pve-firewall status
Status: disabled/stopped

----------------------------------------------

root@cu-pve04:~# ps -ef|grep pve
root 2697 1 0 16:26 ? 00:00:00 /usr/sbin/pvefw-logger
ceph 4418 1 2 16:26 ? 00:00:28 /usr/bin/ceph-mgr -f --cluster ceph --id cu-pve04 --setuser ceph --setgroup ceph
ceph 4421 1 0 16:26 ? 00:00:02 /usr/bin/ceph-mds -f --cluster ceph --id cu-pve04 --setuser ceph --setgroup ceph
ceph 4438 1 2 16:26 ? 00:00:33 /usr/bin/ceph-mon -f --cluster ceph --id cu-pve04 --setuser ceph --setgroup ceph
root 4739 1 0 16:26 ? 00:00:09 pvestatd
root 5028 1 0 16:26 ? 00:00:00 pvedaemon
root 5031 5028 0 16:26 ? 00:00:00 pvedaemon worker
root 5032 5028 0 16:26 ? 00:00:00 pvedaemon worker
root 5033 5028 0 16:26 ? 00:00:00 pvedaemon worker
root 5073 1 0 16:26 ? 00:00:00 pve-ha-crm
www-data 5093 1 0 16:26 ? 00:00:00 pveproxy
www-data 5096 5093 0 16:26 ? 00:00:01 pveproxy worker
www-data 5097 5093 0 16:26 ? 00:00:03 pveproxy worker
www-data 5098 5093 0 16:26 ? 00:00:04 pveproxy worker
root 5118 1 0 16:26 ? 00:00:00 pve-ha-lrm
root 8959 1 0 16:39 ? 00:00:01 pve-firewall

Ports used by Proxmox VE
Web interface: 8006
VNC Web console: 5900-5999
SPICE proxy: 3128
sshd (used for cluster actions): 22
rpcbind: 111
corosync multicast (if you run a cluster): 5404, 5405 UDP