dm.fs

参数dm.fs可以指定容器的rootfs的文件系统,但只支持ext4/xfs:

func NewDeviceSet(root string, doInit bool, options []string) (*DeviceSet, error) {
...
case "dm.fs":
if val != "ext4" && val != "xfs" {
return nil, fmt.Errorf("Unsupported filesystem %s\n", val)
}
devices.filesystem = va

这是为什么呢?下面是docker dm的维护者Alexander Larsson的描述:

Additionally we ensure that DISCARD support is enabled in the filesystem so that any files removed in the conttainer filters down to the loopback file making it sparse again.

参考这里

一句话,因为ext4/xfs支持DISCARD。这样,如果容器中删除了文件,空间就会马上还给Thin pool,因为Thin provisioning是支持DISCARD操作的。但是,默认情况下Thin pool是底层是稀疏文件/var/lib/docker/devicemapper/devicemapper/data,所以,只有Host的文件系统支持DISCARD,才能保证稀疏文件空间释放。

Host为ext3

# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
dbyin/tlinux1.2 latest 8297f05d459f 41 hours ago 399.6 MB
dbyin/httpd latest 93e711fab1c1 7 weeks ago 412.7 MB
centos latest 61038e6e3195 3 months ago 236.4 MB

我们可以查看稀疏文件的真正大小:

# ls -lsh /var/lib/docker/devicemapper/devicemapper/data
1.6G -rw------- 1 root root 200G Nov 12 11:52 /var/lib/docker/devicemapper/devicemapper/data # dmsetup status
yy_pool: 0 409600 thin-pool 0 13/65536 0/3200 - rw no_discard_passdown
docker-8:1-696417-base: 0 41943040 thin 928768 41943039
docker-8:1-696417-pool: 0 419430400 thin-pool 73 633/524288 26115/3276800 - rw no_discard_passdown

我们删除一个image

# docker rmi dbyin/httpd

可以看到稀疏文件并没有变小:

# ls -lsh /var/lib/docker/devicemapper/devicemapper/data
1.6G -rw------- 1 root root 200G Nov 12 11:52 /var/lib/docker/devicemapper/devicemapper/data # dmsetup status
docker-8:1-696417-base: 0 41943040 thin 928768 41943039
docker-8:1-696417-pool: 0 419430400 thin-pool 73 490/524288 18758/3276800 - rw no_discard_passdown

no_discard_passdown表示dm层不会将DISCARD传给底层的设备(loopback device),只删除映射关系。

Host为ext4

[root@yy3 ~]# dmsetup status
docker-253:1-8790943-pool: 0 209715200 thin-pool 339 876/524288 32432/1638400 - rw discard_passdown queue_if_no_space
aaarticlea/png;base64,iVBORw0KGgoAAAANSUhEUgAABAwAAACtCAIAAAC7jnp5AAAgAElEQVR4nO1d27XbPK90Wf9SQapH1bAZlXIe9nmwLYHkDC6yne18mVl5SGgaBAcXEqQc3W6CIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAh/K9b2c8e+/bYq/2ks297W29r2bfltVQRBEARBEIS/B8u2P3brf2wbuWz7c7i1/bR1/nRsu4q1vVCHbPvbaHmS/DPM7Gh+bZhHyQWELNve1nulUBao+k0QBEEQBOGfxbLtn6sP4O60G/Hjh9xre2Gz+x5u1nZsuG1RtLZjW7+2sXwoA9txbT9XioTbi7wxXFHkN8b9LT0FQRAEQRC+BfPm8n68vW/L8/z73DAdJ9/Dnha0L6at/8KzLhjPv+0XTpWWQ53uX8GUzkG7zS7T/3j66WffLB/3v3fyhqlNwmfeujqIbNgv7eNHCYMdz0kln+yaebu37Ht7tlF/2Pf9kA/5If5wb77//S7+PovHNc5mDJOo1oA+3A8tQZO/ZfX0eBAEQRAEQfh7QU+gjx3e2u57o/MhoVt3Os7ab+RE9r5pfpypw5uEobH/Z7SdNgf0/WMzTE97jL9su93+mSLB7oz7LaXdFCLexiIBbCJzx/ZW+PRQFrPjvi252xrC2zH5J01HPWNHXJvlmfLDbpbsxr0vGq1hgjKK6cPGva2rrXZsj5KezriCIAiCIAh/K5zN5div3zg9urD22+3mFgmPYTNFgpVzfjQelD82lMN3j0051dPboB+/1+gEwnGJ5mPbXCQMJQjFYxPPfkMC7XhvTBUJDm/Pmue4RzmNMZPg8lMtEmzvxGUL0oePO6h6uUhwxhUEQRAEQfhb8VtFwrlFm/fo8+jPlsxG8b1FQmvD7wXcU39cJJjv92oMFxevYLLj/LSRO8yVIqEf7UGLz2epSBg1KmzA4xste3MyCi8WCXRcQRAEQRCEvxXZImHctJ17QdbefWSe6Yl/Ko1Hvz8nnzgSHx47j/Tsdov9b4iP647hUXaqBtRw2Zp5HMX+cPmNG0p2k9DW5M9wCW+8SOgGNN92+An8oa+Z7v8w3w1mwfTB49ornekyp6gnHVcQBEEQBOFvRXgC3Z2Cl9o7YcP5+eOf+Elz8gxS9lmO7umPNjxeTx4vAe3Hf4FqXuoAfmPd/5w1kn+0Oj+ovQRYJDyf7LrK28/Pfij3+J3CcvyEF5EwzXf4iPrDk+CtHTxv+96a/Ql0UB86+sBxjQUeA6HfNGf0fKMZBUEQBEEQvgMf/S9QnUGdPVX+ckM4wIuEj/xnpn8Av+GYgiAIgiAIwu12+5WXqWF0/yEl+UDntBD8ZWp/Lcwp/19Z4QiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAge1vbCf0h/vJn4czCvujXDLPBFyf0nuUn5L3fjvSvjopf4whf7Dq9itlO+95+5dvVf29DO9XcxySF2cfSPeB79EPJD29G81vbjOCeTH32W7evKAHz68tHL8rB85oeVOfVyavx0dnT84ez8mj5zu2/3G3n7SpUfJqeaB6qoyojzBsiTwC6M/3f5OZOF83+9Heof+ecFnP7v+yHT05Wb5Tkxr1KcFlBbT4v2uhEeyusmXwer+lTnO+Muwfn22l72TD7f6vrlr0f5der88O/0k+p835H7fxMvvZf3oy/HNeGxNkvvsu2I67WdiS/1Pum1Hf3XljBffdxl22e/WLbd7mmOj610K+fRf37ztKv/tv+0rVeY6O9jlsPswvTP8Xz6IeOHtbN5MRegcoi9KPp5sfnO+5jRLgRJux+fUj909QnnVeTntCP1h8enWR6oPqTdC/1l2+H2vcQPkVPNA2UU9WT+4+dJYBfCsxNHWI6v6sQPyzPVdu4nnn9ehfV/LJLqyUWW8kluXtk4LaC4nlbtxXgor5t8HX9Fn4ynw4EvLcsVsAEIn8H6O/EZtE9y7sjnh2/zk+q6fGV9+X2YCqmtXZFgSidSce6bTSP3v3fyevk/k/B9W56fF0JjGSwcfXVKdvcZDJMyXTKRWh0X7kh6MSwpg5HAYk/1f/ji8mqRgOWQuTD9qZ7EDxk/lDenSFjBGYQnBxbM5tB42Pl1lCP9hyKc2gXFixEwJl8m3/aBvHX9ybievSA/PJ8cGCyEeSA8M31YO7O7Pfu3U/btNec9Jmeg5LJfVf2N6Ynoc/UkdmE8O3ms5ueMn15PmGcy7Zk8P/sn5BO3E/9nfkj1v2R3Pz93H9TjFK6bDOX11CBjr0zey6ybyXU81Kc23+5YuuN02fa2gSPx4xvznXxrlaNpvtkN1y+wP2H5xF2nUvkhgS/yk/S6nNHn2zAePtprR1b9/BgDWKc1RcLJw7A+duvB857m/vfCme2wqIBYswPMK3eiSAiNVx13bXtraJHoVgM80ExNWCTMJ/HOIp27OMFyehVC/Yme1A85P6SdzKt7HM4eazA5zF7ramXCE4suBEg749OLl5E+d1w7tSHoUH82Lo0LzI9jRyNk2KtAvyI8M31YO7c7CXPCj5P3eLqo5AHqVzV/8/QcafL0ZHbheRLHUdnPaZ7sVJhbk+1xnp9OWCCfpN1bT6kfQv2Ldo/z81iC1OL07HmtSChshnL28vIe059s/oJ1MKXPhfmym4TDT+ZNOWyxPhCfTLP5Uj7d/Um6SKjmhwS+yU/y63Koz9dhmNtB7kTcs6NX+jxYpDs24JeeJ0F7+LySe77MtQ7fvLr6pMdd7CrRO7GpsNusJo6ddJEwTICFtN1jwPkGcrhdxr0f1JP54Y3zE/HWz+s2LMunVkQOtVfv0rjs2rurU9iO+fTjZSbKHdfSMJ8/df35uF7xOfPj2HFm3uGB83yhSMB2J+phfvwjn2hPkMoDzK9q/hYdTbk3CdPOEdjFWYxhHFX9nPJj5sz2Vpn2cDOXO+oj7dz/PT+E+hfzTJjnhxqhGqdVXCwSKvby815y3Rw+HX0orc8biwRqJlIkOP0D9PPFfPrrbL5IKOWHEN/mJ5V1OVyvvwzvLhJaa7PtsqNHAAdjSGKl3faIn2t/ZdzutMwcpHUcIELYI5NTkRDp702qcu01yfHtMt/dAz35osX4CXk7PniqMY4ARjb/IPbq9tuc0YofTmkeSpw1fHXcrp2Oy/wK85PYfOQeVqM8M31YO7M7U69HzE9RTtWvqv72Af8Zd82M/ygec37O8uSN55lSe5gn5zMNpOaFIoH6IdLzhTxDevP7oGKc5nBhPb1mr0evpP4V/y/pc2G+v18kJNw7iOt0kVDMDx6+zU+u9A/avwvjbzx+cFKzTtPdkfe/VX1sFKbfkxD6Kj5tdbOwj0Ut5rhmfKBk3Kn+zHcUR3lLLgZeHXezP90xmwYvyJm7o56B/r0opn+MadOAn5Ji+mM9qR8yfli7Y5dTT/OBwz+w13T0S4xTaZ/49M8w/JNgE4yO/4P+dFzuV8ifqR0PaXQlGPZNjGemD2lndh9HQeD89IziPVYlD7D51v3N0/MG/Cf2k8FkhOd4s5L2c5gnWZ6ptgd5EpyOYT5Ju7OeYj/Eer6SZ1InRNfitPC4UXU9vWgv1p5eN9l6UdanON+b3WKZwPt0kZBZ9wtxnS4SqvmB4dv8hPUPxnXbvw/2vuf+E5hj3SLXnfDhrOOW+JBn922TpPGxloCsXhmqT+c9zrN0ONmdcnKmuzruMEJngNCHR+K6Uzeq/2wUrr+HUY5rF6g/1ZP5IeWH8jbPa20/Pz/mUeeOBod/YC8z44dAs9qX1ER8Dvr/YPGdyah84odcHzgutRfhh9vxwR5LlwMPlGeuz9zu2J06bo4foIxDRCIPsPmW/Y3oyfOGo2fkn8zfuKOEfo75YXmm2u7p//iel68S6yD0f+qHXM8Ldmc843ldidNKkVBaTy/Yy+Xh4rpJnv1L6VPfPzhu3tbz0+PkddaI9/eA1/3MOkLc3IpK5ZlcfgD4Nj+5sC67+giCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAgQ6/DaxRKONy5/Dscb77phFvZCTPNJblJn/8yb8K6N67+j1ogZXi04vyJ45trVf21DO9d/xsMziHGxXQJpZNh38DPz4Olf4YHJd+Do2c2vf91pyW9PIVifUf4l/5lAeKPzjXlG+Yf5ytQexkvavD4Az8z/HT4TrzeN33rK/KQaXxf0+Twcf65Kyc2r6s9cfi2OOKB9E/lkxDxXP59fyj+unBwP1Th6z7rJ81JVn9ftfpfgfHttL++xMvuWDJ9czlvWdwff5idsPU2vXX8Z1vZCXtj2DxYJJjzWZul13iD/nMuS0Wxt9tXisfnq4y7bPvvFsu02dx8fW+lWzqP/2sYZufpv+0/beoWJ/gyMQmoXAjbfG+FnmNehBOOH8UBdoMhD1U+onoNQ+1r7kt8+RpnsS+Rf8x8Awhudb4rnLv9gfyDtQbw8pL+almeeqf87fC7bjtIsmy/sT/2ExEtVvtf/84j8OYfSvKr+zORX44iA2TeXT6g+Rg7N5xfyTzhuyEM5jorrAs17fB1/RZ+M3eHA1eWoDDYAXTeL+byYf6r5+dv8JLmeZvLzF8NUSG3tFmlTOpFKcd+s29z/3snr5c8nIvu2PD8vhMYyWDj66pTs7jMYJmW6ZCK1Oi5fsewRFHQaMNLklI7+D19cXi4S1uDOYBAJ/YTON8EPK2JtF8YD1Z/zAPX3eYZxAfV04s52wSc308kxtq8jP+8/5rB03CFF/jPOF/ZnehJ/oO1kXM/fOJ/QjpjnXjVDG+bTnj13Q5N50f79uHaRw/FSlc95dvgptL/gz11/4p/leRX9mcm/kB/CvJFcF8J8a/0hmc/DdZOhvJ7244ZxVF0XkLisbrm4rsy3O5buOF22vW3gSPz4xnxn3lrlaJoWRbH+mXxezT+Z/Q/DF/nJ1J+Nm9Hn22CLme66f9tp9dPf0MIi4eRhXJfn/H5vWVs6jfTMdvE2yHimzdESiSIhNF513LXtraGk3K1yeKCZmnCTN5wcz3I8/QG6x8ls+d+r0M0K+wmZL+PH+F7mCJDxQPUnPDD9mXwnLpCeNO46irqiGscRsa8vv+A/t3U1sYxWLeY/KX/jenJ/IHFExiX+xvhkdqRxZAfq/J/nE5henHnRdITzDI6XqnzSn8dFrb3oz3wdYf5ZnFfZn4n8an5w8wZbv4A+VA7xBzefF9ZNhvJ62n81llNcF5C488vuOpjS58J82U2CzTnz1nNuOdRObbLpehfpn9w/lPNPtP+h+CY/cW8SBh4Cfb4Ow9wOcifinh290ufBIt3BA3+afHo8oIHP/VFeyX1V5lqHOFmkT3rcxWbl3olNJd3gMRKQnd7kDRPAPJz60/kOy8io1WgX7idkvpQf60Q7eARh0MQpEjz9Zx7oMSrxkyjBDXt4HHeTOmaLiu2C7RvKzxcJ/dBse5x+fKPvz/Vk/uD4CR4X+hvl07G7G0dTXqoWCd68oj0HutYf46Uqn/Rn/BTbq/7srCOOf1Z5Nlqn/LlQJFR5G9WZ8/Z8NsK+Dv0hkw9feRziYpFQiaPSuoDEgU9Hu6f1eWORQJcJUiQ4/QOY+Yb6p/L5s2ch/0T7H4xv8xPUn43r6/N9eHeR0FqbbZcdPcJ00IIlVtptj/h3Dq+M251+mYOxjgNESOJ2MKe/H+XxIxyjp5jYnu2S22SbfzB+Qj3nc2PIg6M/ku9tFpH8DIG8PKaLSqxPJ8Qc0NeLBDIvY1nuQbFdcH+/SED+EPrJ5A/Q317YtE0swLzkxSPbvLJ5xXuO6/mnqM+bioQyz1yO459Vnl31gD/jIqGUH5LPG4T6lOUU82EZF9bTchy5eibXTUdOSZ8L8/39IsHMN9Q/lc95++X9z4xv85Mr/YP278L4G48fnEQ6Z+qsamqCx1fA70kIfRWftrpZ2MeiFnM8Yk9K1qFyuaFrU1PGZn5Ac2Hcbe/q9uO7XpAzd0c9A/17UUx/hntgnqLsyZ9/Cv7sdf+CM1/IzyAzPgcgPDD9GQ/Uz4l8Jy6Qnk7cYf+JT/UGZon8oynlP9OVguXT85/JLqS/pyfzB89P0A4e+hvjM7DjNATzfy8eyaafzgv1d/Pb2W6bS/JZf8bPa+0IE8+gP/fP6ryq/kzlF/MDaw/sC07xIr9N58PauslQXE+vxJH5brwuPLsOXuTk/5o+xfne7BarW14+WyRQP/f1z+bzDpn8U9X/2/wko3lBny+Fvd+9/wTmuEwg17jw4azjNumQd7DWXxUfuRFeHhNM181Yn857nGfdcLI75eRMd3XcYYTOAKEPj8T1myem/2wUrj/A2n5+fswjhd3DrAm7ZOeL+PHowTE+80D1d3lgDyEynp2HFoGeLO64/6A4GoWdEyPyq/5jLPwgsA95oCeeL+tPebiReHHa4bjUgRifjh1HnpP+3xfn9AtoXrw/8RMvXkryKc9xXNTbY38m/Zl/vjSvhD878qv5Ic6T1fiK8qebD8vrJkNhPa3HUXVdSOW9vuCs6VPfP8xueAzb1vPT4+R11oj39xCvd9V19oX8Exhy1qNql4/6idOfjVuZriAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgnDH2r/utIbjjcufw/HGu26Yhb3I0nySm9TZP/MmvGvj+u8U5m9mnV+JOXPt6r+2oZ3rD+How0Dfiwv1uSin0D+Sw/x/bK/x7PoDfP3irOdDA4fyypyv8OB9Z+yP45S31+KOg8kPdX/D2AgX7Fh+Hec0xhvjlPHJwOxYzVfXx83FV21deGueTMZ7pn0mp9pe1aeKalxX88N71k1u36o+r+exuwTn22t7wx6ruj+p5tVq/6q/fZuf+HH3+ny/DWu7WiTcbrdt/2CRYMJjbZZe+Mbwe6fnXJaMZms7+q8tYb76uMu2z36xbLvdQxwfW+lWzqP/2sYZufpv+0/beoWJ/gxMH4pen3m/OOpzSc4MxidrHwfD/m/aizx7frhs+zQc9wc6dehXtD3FZ8cDkw/7szil8VuNO6pBSn4Yj+/CBTtCfwiGmPR/V5xSPnNyxrhL56vquNX4Kq8Lb8yTlXhndmF8VtvL+lRRjOtyfiD8lNdNvo6/ok+GNzhw0d3KqO5PGA/MT6r9q/72bX6SzG/np8X9zHfAVEht7TYHpnQCldO9erJudv97J6+XPx837tvy/LwQGstg4eir02Jwn8EwKdMlE6nVcckOwIphixYYaXJKR/+HLy4vFQmOPtAfRhXM3LE+l+T4/Yfgxzwz/yftZZ4N7Lj2DM+GBtNz2/dtRWcobGeZ8LeOT4cHJp/kDTIWbvf5BHmG5BMm35sv+XptXNJetSPzB/tBuOMcQNgcKyUWX6QP5ifIn5V8hcfN2T2ML9bf45meKCA/MbDfrMZ7YBe46a+0V/UxX0otG+X11CCTHzJ+m1k3k+t4qE9tvt2xdMfpsu1tA0fixzembf1PaxePpjP7k6E/zKts35Lpn5GT0eeX/QQV53DcjD7fhuHw6EwW206rnx9jAOu0pkg4eRj3Y5aU5z3N/e9pB++Z7eJtWprwCpEoEkLjVcdd294auijrVic80ExNuOgap3xoNxcJVP9o4sPeA/qD8RnrEkyfqhzuh4xP2M78n8ZFmWc79GBf7GZY/+5xvv74BfsVb0d80vkSOby/nUh4LcP5ZPb18gkcl/gP46c6LtenaMcb8Yd1Nbm0OyEj+cRI67b2VA7mx3xzrMwRP0H+TOcrNq5r91J8kf6MH5InnXXw/OLAZSXeA7u8XCSU9Tk/ulQkFDZDqfwQ+21u3Uysg8l8VZ4vu0k4fGneNMMWu7bmNtml/YntEa6nF/qHcvgsvsdP3JuEYd2J8u23YZjbQe5E3LOjV/o8WKQ7+IlfkLnGAw743B/lldxXZa51iJNF+qTHXezuoHdiU0k3mMmB7PSiO0yALSF27+3PN18Kn8bfz0dwiD5VObw/4xO2M/9n7a/xPPoh3hRi/fvt5GkF5lfU3yCffL5YDu9vx2BrYH8sTeIuur7I5wfoh4Sf4riOPkU79mrQoe2pG7MvkMzlMH4InzTu3lwkDOOGdn/IrF22dP05P90XjjRWOXb19OH537PL54qEaD1K4mKRkM0PR1/CT3rdHD6FuSOjzxuLhO4gIFEkOP19pPYn5uPhVNXzk3z/a/72bX6C+rNxfX2+D+8uElprs+2yo0fABzaTxEq77dGvSTXbxeMOp1/4mh4Rwh6ZnK+PA/29SRWuvcZHD1JfRN2m9FaRk9usmH/g9itFwvt4RmsJ039+vPksEpBfsXasj18kzHLcIoHFKWwnfF4J5FJ+IPxUxy37IbPj4x/g7Lbbx3ZFgmvf6REhLMeZDuLTma8bF5V8VRrX0T+xVzP5pMZPrA+QUYl3Mm7U8+UiIadPjAvraSU/TKNNE0mum46ckj4X5vvrRUJmf3KjPFA/KfW/4G/f5idX+gft34XxNx7wGt1Opj+t6X47cmwgpt+TEPoqPm11s7CPRS3mmM6e2K1tqmrRGWQ7HTe+BLowbv+MyBmcXpAzd0c9A/17UUz/AKB6pP5g+4T6VOWw/oxP0s78n8bFBZ6pH4YHwN3mstPBGgz6ldN+wOjP54vl0P4sTlk745PZl+UTKh/Pl83ryriw/YId8QFe1znWHw0fyLFz6XyV2BHHqR8X6XyVG/dENb5wf84Py5NBvmLHB+l4t0jxeaG9rE/hcaPqelrND8N3c+fDgAfHvjV9ivO92S1WF0afLRKq+xPGAxu32r+q/7f5SUbzgj5fCvucxP0nMMdlArl+hQ9nHbdVh7xuXRy/MT7WEpA1XTdjfTrvKT9becrJme7quMMInQFCHx6J6zdzTP/ZKFx/DzCm+EO7V/XJySEPCbIvkHbm/6y9zDP2B+7QQM+1/fz8mEdG53NWrBFqT9HTzTch59mfxWkyflP2RfnEke/5IeatMK7XXrMjm4CZ2eOL0HeHmc1xSuVAfpL2Stixlq/S42Keq/EFn0mYeGZ5kvEA+S87aKLZqlRtvxQwlSKhtJ5eyA/u+nJx3ewLxZo+9f3DHL7HsG09Pz1OXmeNeP/UqPH+xOMBGaDa32tH+DY/cfq/Y7qCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAjCHWv/mtUajjcufw7HG++6YRb64svFfCEj/+yfeRPetXH9d9f270jvX+S3DDJmrl391za0c/0hHH3YuExPOmeoZx1Vu3sgesLXJmL/tN/gb2j2J/2ITBJcM/9+/6r/VO0by5/sAnmgPkLg9s/6VRh3XJ8uf0I51+zC/Oqqn4M87/Cfz+kOP/l4uZBnSvJL8O2Fx62vC7gdyfH0KebzW8av3pWHc3Kq+rxn3eS8VfV5nbe7BH8heHWPFfgJyJPMb3G8c/ksP1Tj9Nv8xF8Hk/r8RVjbC/uqbf9gkWDCY22W3vnN2ken51yWjGZrs6+yj81XH3fZ9tkvlm23ezuzwnVTPOQ8+q9tnJGr/7b/tK1XmOjPwPRh41I9CQ9UzyLKduegei7bjra52D9vtxubF5LDQKdC+Gf9y/5TtC9r9+yCeGBxQdHrOYye96ukn1NjnEUCllO1C/Or1/y8z/MO/yh+mUjKTyVeqnmmKr8Kj1oYv8V1gdqRyKH6FPM55ae6DroD5OWU9SH+Vl43ub1e0SfDGxy4aMY63AHmPMn8k+ZnJp/Yqxqn3+YnyfW0uI58G0yF1NZu8TClE6kI983mrPvfO3m9/J9J+L4tz88LobEMFo6+Oi2i9xkMkzJdMpFaHZfsCK0YttiDkcBmgur/8MXlpSKB6RPwBoON7IyxnszfzCEei7SBT+a3uJ3oac8OB5e249oZwHlROWRe275vKzhDYfyz/uhrnpyyff32p5huk4R4oHFB8snIrvmI+r9rxNu0kGD5PH9COVW7WLCQTfk50TPwZ7j4If9k/FyIlwOZPHMtHn1++nWKxR0Z98p6hNqdIgH6j+MckZ/br7rxDtZNhvJ6WtTHifcDmXUzaa9Qn9p8u2PpjtNl29sGjsSPb8x3a61Vjqa5bmz9tV+1m2man8lmOrRXdUvyRX6CDsvguBl9vg1DUdddl7Pq58cYwDqtKRL6RdGu75aU5z3N/e9p/5iOwUCs2QHm5JgoEkLjVcdd294a2hx0qy4eaKYm3OQZp3xo52ySSgdGs5s7vKFNBuSB6Un9bV2Nj5ELvX6nCOVw+cxeoXt0/kn5Z3LIvLrH+cwxDuOf9Tc6pvynbF+vncQj5hPHBc0nJkfZFMT49/LS0TqVepN8mj+ZnKpdhs/Glqyfu3o6/gzil8Qd4T+Qj+f1/FYyz1Tks3hn/uDZC457fT2KHzfy9KH9Iz8f+XHi/YUiobAZyunj+NvRmlg3E+tgSp8L82U3CYd952IStthcF59Mk/l66xT2W7JvYXyG9irvmL/JT9ybhG7cmIdvwzC3g9yJuGdHz5APFqnHAL8BJ2lO/1vEK7nny1zrECeL9EmPu9js3juxqcjbrCauYAubvE57tqXrnnp15zvtnWqbSMYD0ZP7W+9aJLuHcqh8ai84zf5r+Nhy5j/eZHRFArwHcooE797od4qE48MEDzAu3HxyfrifV+aYfz8vPaXNNd0on+VPLqdqFzs2y3uxn/t6looEHneQ/0A+n1chzxTk84KE+INnr3AvWFyP6Dpl5AT+M/QP/Rzkq8ub+0GFS3Iq+lB/e36aWTeHT0d7pfV5Y5FgC+9MkeD0D2Dmm/GkgvAAACAASURBVNknDI/N+PuWQf7Nt1d1v/xtfkIfqwbj+vp8H95dJLTWZttlR49gi2xHYqXd9kg8V/fCuMMpzhHsHQeIEPbI5LTJi/T3JlUo4ucbaG9cWCSQg2EkhBPb30KQmR1fv1QkMD3JGhD4J9x1gjNFPK/5Mf7jKBvyz/of4yT9p2rfoN183Ck38UniIumoqFvKr06Et8uTlngig5yqXW61vPfpIqEad558f16FPFOQX84Dnr3ivWB1uLg98J+xv+fnkP/X1sGX5LygD5hmct105JT0uTDf3y8S4jxJ+4f7Fip/ak/kt1HBr/KTK/2D9u/C+BuPH5yMrHP8dN7RHXbu2zK08dORW82nrW4W9rGoxRyz2BOX6Rcx6NrUlMOZHx5dGHezP/UxxYMX5MzdUc9A/14U0z8A2ul64+KyB/DAhsD+Zpfn/lCB8c/81vVnoife1Eb/yUymSHDn1T33bB4zgPzT/ueEcv5Tty9s9+IxPIA30nK3grH/B3LcZb4/gMP5k8mp2sXJexU/d/XMFwncPwedu/YL8VLKMxX5uTxwwrMXrglr64LTzuRAfVh/59aCPd3K473wuFF1Pb2kz422p9dNh7eaPsX53uwWyxj+00VCvO5P+wTon2zczL7C8hOvm9N3v8pPWP9gXLf9+2Dvme4/gTkuE8glJXw463g645Bn/WaSND7WEpA1XZtifTLPzJ0fjWOecnKmuzruMEJngNCHR+K6026q/2wUrr8HGFNwXEdPwkNCz/4HXofo+28HbHqH82IPFfKHDYGe2BFd/4TzYl+A81rb+c+JHsC/0/+C/5Tsy+Vju3DiaFygfOKHUexXw3eAnzP5LH8iOWW7eH5V9HOiJxuA2ZHGHeHnWrzk88wF+Zk8cP/EsZczQHFdSNixC65EHuD+AP43EYcHHAPZjU1hPb2gjxvvF9fNvgCu8lPdP8zLyzHs4XxPG4/qH2dlpL8HZ91HeTLhtyw/2/6we5QHRnybnzj9U+vF31EiCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIg/DrW4XWhJRxvXP4cjjfedcMs7AWa5pPcpM7+mTfhXRsXvWsYvn94eLWgnfK9/8y1q//ahnauf4SUnzj6Qz2d/s64ydcaPr45iWTtuflm4wX7Q8D/ZK+rfjKLn/kPeUjyyebly8d6En4uxAVDLd65ru64wI44j3F7sf5MfvfR0+Uu8FbSJxXv/E3bcbzH+WqMR6b/e9q5PvR98iRflfzhgh19+d46cnUzMMrJxRfTk8l5Dz/cjlV9ruSTHncJzrfX9vIeq55XHwNPX3H8fM5Ljnw3vwF8m59U86fL21+Atb2QF7b9g0WCCY+1WXrnN2sfnZ5zWTKare3ov7aE+erjLts++8Wy7XavZlaOboqHnEf/tYEw5vpv+0/beoWJ/jnEfsL0Z3rS/s64y7ZDNVA7E5lzWjbfmAfqDy7/wF49b4ecJM8J/jkPFT7ZvByTYj2pnGJcMFTj3Y9fMu5sR5rHCA+0P5E/TvIsEmq8sbzE9GHymf+X4z2Vr87+TH/Wzvin/Yk+tP8NxxHj813rQiAf9a+tmwzF+ErGRWivcn7g6/gr+mTyCRz4pWU5g2JevZG8R/P2Q9qYl5z84OS3Gd/mJ8n9WLwufzVMhdTWbtNjSidyUrJv1p3uf+/k9fLnk6R9W56fF0JjGSwcfXVKdvcZDJMyXTKRWh2X7GitGJaUwUgguVP9H744KFzNRsxPzOEY83g7UobnrpWMa8/krGux9m3ftxWcQbB2Ol8eLywu7FQyRQK012hS5Es8LCxvmH/GQ5VPr0hA/em8Ev6ZiQuWf6gfkv60ImXjsrjrVbPDhvYdJNG4Jv55IMNbJi+xmbEJp/KbE3dQKunP9M+0D8UG5oFvGmB/FkfOd9+yLjD5yf7huslQXk+JnkxONR8icVndMvrU5tsdS3ecLtveNnAkfnxjvqNrrXI0Xc2riX3LwH+Y95z8UN2SfIufuPmT5ZMXD+X/GIZDq+56mlU/P8YA435rGevOIa92yfF5T3P/e9o/puNkEGt2gDkjJ4qE0HjVcde2t4YuyrpdNh5opiYsEuYTNb6ZqBXwvZ/c1tXYGgmaw8LnuevPx2VfJ+3d43DmmI60s3GpPk5cYH8g/FN7mRjERzu5/ox/xk+VTzYv2p/NK/TPXFyw/MN5IPmKxi8el9qx/479V2Dfvj+R78bL8c0Eb3FeYisc3Ipm8xvXH/uDM1+mP2mn/JP+1D85b8GyMvD5nnWBy3f6F9ZNhvJ6SvSkcqr5EIg7v+yugyl9LsyX3SQc6WUuDmHLoTYrJucBCnmV5T3Cf5z3kvkwxjf5iXuT0I0b5/lvwzC3g9yJuGdHz5APFmkmnfj1VyZoD59Xcs+XudYhThbpkx53sbuk3onNKUKDx89AdrpIGCaAeTj1J/NlfnIbTTyrOuccP5kOewY6Lvk6ax8KmGMU3M7GpfqkEhz1Q+M/jr1OpnfwiMnsJ7C/UyRAfoZuB7z+aF5Of39eg5xwxNFAJH4xD7w/jV84bhh3KI95PEz9sXw/XvK83aK8xPOwc+IG/b8Q71bQs5an/Zn+fF6Y/yg/d/r4/Z2948znm9YFKt/v//jw+uMQF4sE4OdUTjUfTuLApzAmM/q8sUig4UCKBKd/gERedfIeWV/cvMfsUt0vf5ufoP5s3Hi9+y68u0horc22y44ewRbZjsRKu+0RP2/6yrjdqaQ5sOw4QISwbdO8KQn09yYV8eNsjumjI1j/UE9+7/BikTBKehYJqP0zRULNP4v2Cm9Dn3/D/DN+Hv9I88nGrfYP23NxQe1CeKD9WfyScftvgg1/IY8F/a38xCY7nU9oXvL1Sfoh7p8sEg45vD/TP8y3g57v7E+mA/l817rA5Dv9h4/5px4urKeEh4ycy/nBk1PS58J8f79ISPAW5D0uh5EAt+mV/2/i6/zkSv+g/bsw/sbjBy/q3Sajv3Oyv6nat2Voc08jKj5tdbOwj0Ut5rjSnlxOv4hB16bmuCbzw6ML43bPWJjiwQty5u6oZ6B/L4rpT0D8ZLpSyJxve3qO/al/jqMPQuDjMZ2qD7GknY1L9WFxwfwh5p+vNAneeH/CP+Pn8c8KnywuqHykZ8BPOi5o/iE8OPkKxi8bl6nK8tigc+czfv8hgzjxMinj6M/yUqDPJN/Nw4V4J/5A+zP9M5sqK5T1Z/5ZvRhhfL5rXQjsNfUvr5sMxfWU6pmQU8iHaL7QjmV9ivO92S2WIfrTRcKFvOrlvaf64X6DNWbyYar/L/kJ6x+M67Z/H+z90P0nMOjm3U4GPpx13Eod8mw+nSSNj7UEZPXKUH2yz3TiZHfKyZnu6rjDCJ0BQh8eieuqfKr/bBSuPwHxE2OZx7OLdnByt0D1BP3JuMwhYPv9aeXz0crz6Qzc7oxL20lcOP7g8A/s5blJyNu4bxvaHR5KfLJ5pXkmeiaemUnFRT8G80PWH8avM+5ttqOTxyAPbt4D8kc5nX8WeUMKRfqgeHfyYSHeb8wfaH/mWKSd+iH9IM7/0NnsJx6f71gXuPyU3XPrJkNhPU2u7wkzHvKu5AfyDG1Kn/r+YU4nx7BtPT89Tl5njXh/D6W8ihW9BfzD/QaQH+WTEd/mJ07/YvoRBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQHKzD6zlLON64/Dkcb7wb33VPXmG3mC9k5J/9M2/Cuzau/+5CI2Z4taCd8r3/zLWr/9qGdq6/i0lO3HscA843LWh+q/vYjv3kkv6wP5PvjDvw8Ii0SrA4/tANcjV+bxm/Im+MDuebHdt0h/PN8TbyUM0DKTVz/gz61+OO6e/OC/itH0MzrV4c8Tw2ZSr27UK7r2eN/6I/JOKOIRuPV/IhlT/aPVxH6uNmUVtPub8xOe9ZN3k8VvWpznfGXYLz7bW9YY+F7R7kpYJffZq3b/OTal76dNx9Gmt7YR3d9g8WCSY81mbpdd4k/pzLktFsbUf/tSXMVx932fbZL5ZtP9rsuFa6lfPov7ZxRq7+2/7Ttl5hor8PIMfBsu3oxe9wvr4YGE+wnfrJBf1RfybfGxfxUA0W5g89rscv8yvmz9X5BkNPdmTzzfF28lDOA76isOw3Y43nF3P/Ytw5/Dvzmv2WxZ1jd2ZfL49Nchjl1XYm38sniP+qP+TijiGOxwv50JE/253q7/jtW1BcT6m/MTlE//K6ydfxV/TJ2BEOfGlZLoD6mztw3q8+zdu3+UlyP/bn4u4jMBVSW7ukY0onUDndqyebdu5/7+T18n8m4fu2PD8vhMYyWDj66pTc7zMYJmW6ZCK1Oi7ZOVkxbBECI4FFner/8MXl1SKBykH2tWd49iM6XyKH7jgTO9FBU6y/URTvkzhR7BPbznjY9n1b0ZkI1+fAtPtD8cvk+PLnZNfPC36ame+N5Y2EHe18KW88jzH9qT5X/XksTiAP3J1YXmX6O/Oa/TbIM5HdhxWV9cRFArJXtZ3JZ/Ny/NB+t1u8r8bdmK+IH0L7UrswfVz5fr7iy2Yib5yfpZaN8npq0OcTLIfqb5BZN5PreKhPbb7dsXTH6bLtbQNH4sc3pmPHn9YKR9Pe+svzUt6v2Fjv4a0u/w/5iZuX7LgZfb4Ntpjpri+3nVY/P8YAY35fxnOmcR9oSXne09z/nnaPntku3gYZz4QH72T9IiE0XnXcte2tocWvy8p4oJmasEgwTvnQztnc55ILlOPZF9OI50vlMN4onx0lof63dTW+Op2wzv2JfK8d8dA9ntcd72B9OmnDOQmKXyrHl483iyyOzu+MGsITXJg3QjuO/gZ54zwQ/Zk+ZX82uXFIfbg/iTsnr0Z5bHq2Cvutm2e8IqG3r28vUCRgP6+2cz35vGgah/m5FHfMT6gfcvsS/bE+VH6cr9CRidEnk39eKBIKm6Gh8iFynLg7WhPrZmIdTOlzYb7sJuHwjbmYhy32ScbEyTTxN8JD2a/sQJ/hrST/D/mJe5PQx1ekz9dhmNtB7kTcs6NX+jxYpJ438etnfGgPn1dyX5W51iFOFumTHnexq1/vxGZj2uDxIZCdLhKGCWAeTv3pfLEc375kUwXmy+Uw3iif5mNyYdDz0A/NTsRhymA5fmwnRQK+1yH6gJ43Hr+OHF++e6IM4ig7X5o3AjtO/o95c3jA+hN9Lviz/dI+PeLjr3k27hKrI8tj084R+a2fZ5jdJ/sG9kJFAvTzajvV05lXxH/HZynumJ9QP+T2ZfpDfbifh3l+Xkew30Y8JHFx8wfyNpVD4+75aWbdHD4dJ5zW541FAk3vpEhw+gNE+417H7unrfrVo/VjvJXk3/6Mn6D+bFxfn+/Du4uE1tpsu+zoEaYDNiyx0m57ZJ/PuzbucCp5BHvHASKEPTI5FQmR/t6kKtdeU1ryeoITZThfdxOJeGPtt9BPBv37fUJ4gujIp+PiImGMvMcuI9Jnuq4kmxIsJ5IfhOR0OZOdb9m+pqXTEfOWKxKMGpcSRbyGTZec2f7JAKyovXBPQVttssihqtCxF3zcCPh5uZ3J9+ZV4r8Ud8wQ9SKB5kOoT2aCJM+HT0t449ZxYT2F/paTA+hNrpuOnJI+F+b7K0VCuN84PnA1OwG33R/l7dv85Er/oP27MP7Go3tmejHdzIlF52WmJnh8BfyehNBXKRKsbhb2sajFHEPZE6l1qFxu6NrUlNWZH9BcGLe7NTbFgxfkzN1Rz0D/XhTTP8a4+eD2DQ+GzSwcOZA31s78BGswHb2HvDH53rjkJqF7Tv34AY+vz7xpgvHL5ITy3ZNg68/V+bp5A9sXzZfw5uUxqj/Up+rPwxw7ZYndYdwxfZj+bn57SjeNQZ5Bhw7Mvp694ONGwF7ldibfmxfhH/BWjjvmJ9QPmX2x/lQfKt9RlTbOSgc8FB43qq6n1N8SckB7et104rGmT3G+N7vFMgH86SKB9Y/3Azm/+jRv3+YnrH8wrtv+fbD3SfefwBz7DHLtCB/OOm6hD3nW/yZJ42MtAVm9MlSfznucZ3Bxsjvl5Ex3ddxhhM4AoQ+PxPWbZqb/bBSuv4dAjhmbG4zOF8rhvKF210+g/uYbj2et4cDm/gLJ5+PCD+5PE5+PduOnHoA+8OzfCD/jl8lh7dyvkD8X5zt/FNiRzNfhjeYxHo9Mn5o/E3dO8ZDSh+nv5BkYp1hRbPcgjoC9oBxmr2o71ZPPK5V/4DMJybhj+Yr6YcK+/WEo1ofLJ3Yn+hO/dXmoFAml9TS5vuM1IfWceWrdTOQ3Z17V/YOzfLX1/PQ4eZ014v1To9L1N5VPHgqPxyIf5e3b/MTpz8Z19REEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAg1uH1jSUcb1z+HI433nXDLPRFrov5Qkb+2T/zJrxr46J3y8L3zQ6vFrRTvvefuXb1X9vQzvWvzvdGXiMI9XTmxfTH7+Ot6s9B3/f7spxHRJGgYHZk+tT8s/Ce6kBPX/7kmZeTAMg/b3s9JeECyq/GL9M/slc237r+OcV1VQ6JozD/ZPwqqxR/Q3xS2NC/GncX/PYCP868Svpj/wzyYdpP3pdXq/kKr+/FdaG8bvL5VvWpznfGXYLz7bW9YY9V5aG6/rJ1jfHJx7iV8tu3+U91fX/XPuTTWNvVIuF2u237B4sEEx5rG95fjzhd25lDl4xmazv6ry1hpvq4y7bP9l+23e7Vjo+tdCvn0X9t44xc/bf9p229wkR/Cqf/su2T2zA92byo/n372b+qPwHjvwyiJ3O9pB1jfgjovIp6BvKBfV9JAn3+QX51ATDumPxq/I7fPgSm7JXIt8wfbrcbjGsC6g8kjpJxGre7wPpX7Y76V+Ou6rdBfr7dbrPd2bwq+lP/dPNh3k/elVer+Yqu78V1obxu8nX8FX0yFMKB30U/Q5WHat5g8imfbl7N++23+c+L6/tnj9vLMJVQW7tFy5RIpLLcN5um7n/v5PXyfybh+7Y8Py+ExjJYMvrqVCTcZzBMynTJRGp1XLJCWDGsmAEjgSCk+j98bvlIkWDPwAYTQz0trESm/ziVQz7Tn/gba3f4Z34O25me275vKz9Dmfhhcqh9jQGGHS2c1wU9GQ9QfyonYRebfxy/wvoQHljcBX478ebt8LD+xF4s3ybsOBQVOK6L/pDJAzzdnvpQPZldiP5Vu7P+1bgr+22CH8sz0/Oi/pN8x440/8N5ef4A1k2G8npq0Ke34rpgkFk3k+t4qE9tvt3xc8fpsu1tA0ffxzfmu/fW6kfQlc3ulbzhrvvj/o2EFfbbBL7If9Lre0af34ItWrrr722nVc6PIdo6rSkSzvkOftXlo+d9zP3vJQc//9XF2yDjmYFHxhNFQmik6rhr21tDSb9bRfFAMzVhkWCc76Gds0i8+LiRQ5eTLOawgPwb3+tci+jD/I37Ieaf+Tn1f6Jn9xiePZZh/BA5nJ/VzP2nYxT6VVFPJ96h/kwO4Z/mn3GadlaYf8IDjTvHb/PxS/Un9uLz5fpDe9G4rvpDmAfmIgTGI23H+lD9b0W7k/7VuCv6ba8tOUMbOzt1Zk1/5J/EjoxnOq94XbtUJBQ2PUMlXFwXbqZzYt1MrIMpfS7Ml90kHHYfDq1Yi80J2RPoAg+X8oZXJPT7N5KfvfyQmBtW5M/7T3p9j/X5NQxzOEicCHp29EqcB1vUkyYegSeNByvwuT/vpIOcdofBwxd1V5/0uIvN+r2zmoWzzWriGEkXCcMEMA+n/pfme6lImPYeNMmeTrSTq8dDH+Zvjh9i/pmfe/4P9RyW93lHMfMD5VB++qnZzSXzq4qe0ZEG2mwBOYx/ln/gP5/jYU0IDzTusPxhsCh+uf7YXs58mR2pP5C4rvtDN+HEWsniEbcTfby8VLI77l+Ou5rf3qaeRjLluVgkePo/hgrzOeE5Ma9BThUXi4Rpfb+8LiTXzeHTcb5pfd5YJHQHHIkiwelPkefhWt5g8ic+WX5O7VsS8n/Zf9Lre6jP7+HdRUJrbbZRdvQI4CATSay02x7x83avjDucQpnrcjMjRAjbFs9394H+3qSq11vpZ8e5iefztwT/Ic9lByD8X3ekoduQ1TNFApRD+On2CVZ46Fc5PS8UCRU5bysSKA8s7oh8oJ8nxysSkL1of6o/06cTMtw7vMMfzEydVJiIx3BesAp5Q5FQi7uL/g/yGOe5ViS4+psBkFIJPynmsSourKdwfb+8LiTXTUdOSZ8L8/3eIqFbdy7lDbo5RgfxLD8fPXKbsW/znyv9g/bfwfhbDvh4xrBZ6bzG1ATHwjn9boTQVCkSrG4W9rGoxRy/2JOY8ZcyN3Rt2h0jxwcoF8bd7E+AzObDC3Lm1qhnoH8viunP4PW/UCSgnUHIv21n+jB/Y+2Mf+bnjv8zPfurfFx8QwwJDvAzHUXbzXG4eGT0DOYLH9uI5eApjjGO/Arrw3m4kbhj8ovxy/XH/kz6u/rbuQdxXfeHIA+4K3Qcp5l5pYoE1w+JHUtxV/RbqrwXd8WbBKgP8884n/eqsnm5cgqPG1XXU7a+V9eF41vJddNZR2r6FOd7s1ssY8jfKhIYDxfzBjrExHw6+Rlo4Ezou/yH9Q/Gddt/D/Ze5/4TmGPdIpeR8CGs49bokGf9aZI0PtYSkDJdj2J9Oi/hz4aSZHfKyZno6rjDCJ0BQl8dieuqcKr/bBSuf22+zDCOnnBeVH9OD9Mf+ZvTTgdgDxvidiTm/tT5+ciloZ/yk5mvaTeR8RgIWSBwN0dPNl+of14OUfPMP07AQ304DzcYd1x+MX5Z/kz589mf6u+lBxDXZX9w8wCIUyaGtLt2CReL2O6wfzXurvgt44cQweZV1d/xT8eOQf5nqubWTYbCeppc33Prwu3yukme/UvpU98/zOnkGLat56fHyeusEe+fGDPDw/iNOG9g+S6fZF/UfRAUP9/mPxfWd1cfQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRCEvxz/97//6Y/+6M+/8+e3U44gCIIgCH8Dfn3Loj/6oz9/8s9vpxxBEARBEP4G/PqWRX/0R3/+5J/fTjmCIAiCIPwN0NZBEP4FKNIFQRD+OczvZxU+gf8qz9o6CMK/gHykr+0HvKwxieONy5/D8Wa78V335FV1i/lCRv7ZP/PGu2vj+u9sJa8X/UGv3Jy5dvVf29DO9f8ojmFf95WavZj/cDnvsQvnuapPtT2PuwTn22v7cGynccF/WDwy/m+3G4gX572gBLF9QV6qxinQ03yEX32c9EMmn/Vn/LyLNxhfTjyysWv883xrPu++4PWv6OPyBvR8e974siIB7E9+NS+9tF/6AL5Nn18A8we9zthHKdLX9oKTffQ8xZh/bUOKRWZf2xkwqZOetR39x6wPUR932fbZRZdtP9rsuFa6lfPovzawHHL9t/2nbb3CRP8/gzecvRXtRf2HyenbD23LduF+8oo+cTsHVOhX3aGMvP+weKT83243FC8sTp1xoXw/L1XjFPQ33xsPU0p+SOSz/oyfd/HG4ovF443k2yr/NK6dQUn/kj4+b8DuH8gbX1Yk3F7cn3wA0udvwLLtosVFFOnmhKmtnZOZQxpwUnI/+bAZ8v73Tl4v/2cSvm/L8/PCFmUZVoboq9NicJ/BMCnTJbNjqo5LPNWKYVsfMNK0CDn6P9bd5XqR8Lgm2ozhl+FT4Cesnc8U2IWhbK9+dLNpw3JGCpHtMnZJ+kmoT7WdjvTTwfpe28DRIjy5f9w8tOwR8Z/ynwDsWwvYbo3xkolTZ1xI0CCnGqe4//nhdG5c8cNI/tif8fNm3vhm/TbEY2JnkOI/GJdssGEeqOjj8Ab1/ETeeKVIgOv7sNQ/NhI+LWR/4twonl/Yd8s53reYQ2Yriu5P+H6JkEDyHhnX0R+3V/ZvjP9qPh/kt/VxiWFvpMf7NLQPrPZ3wPzB3iGEonwenPXI5+ew1zV/wH5b5MeHH+n2kLS7rtp2o2t/6tDHCywSThKHfXI3n6df3P+e3ub1keldxz/pHUlMFAkh79Vx17a3hi7SO+/AAyUWLab/eSI1bz6o/gDdAynmmIv5CWs3Cr61SCjESec/VI7RGR4BZjcTMc8pfartDthNwuGWs3VgyzGdzObvj/gPA8sDz09NO42XME6d0XH+6eRU45Tr+Wh70Q99+WhejJ838uYWCVOJQ/Ltrca/Oy71Q3TTW9KH8cb0/ETeePUmYV7fJ27dou/m7E+M4jRT2Vsaum9ZV7NXwc+NnfpH+sxgeY+Ny/Qn7cX9G+f/Qj7vyiyjp72tMjfweB9Y7e8Da17ZHjAe/PXI18TaseoPzG+v8cPgRvoQoweb1Jm80vmxCtEd/LQ+gQwxFn7wOUq6i6THOvGTiyRpRvqkx10286ONfjNqEkSDHg4PCpNFwjABtqQ99afzHRLoQyvmJ9FicO0kODPfAJP/OHJO593BIx1Juwyfjn6S1ucPFAkwWRvBYwvq/9X+A/IA4J/ESxSnGG6+mlb4fJz6/aPnelJ+6Oozz4vx817e3Md+5toK5NtOWJp/PC7nGUVQSR/GG9Pzw0VCbh2MSLg9k899t+NvJ4CESU+UB6yqZq/FBuq3KPzgMKXPDJz3vHGh/qS9vH+j/Pv5f8Qw8f5ADcjh+8Bqfx9vKRKwPrXi9sbsWPQH4rdlfoL4/aNFQmtt3vM4mpdW+u4AgEustNse/TFd+l4jOe5Q7R3O3HHAsyoSPB5GBfp7k4r58WisJgAACwlJREFUwU84/2aRULYX9J+cHMBP0i6OnJI+1XYHnywSnEG/wX9i/icVkQK5xHUxX1XjdOqfIOc6D6w/4+fNvHEJg5Ys387iwPC5mwSPZ1gkVPSJeRv98/154/WbhFntte3bsrZ9W9u+rVG+ulIkdF8/T9zZAt0/4fKBIgHlPX9coD9prxcJjP+XigTzT3p4RIiq9vfxTUVC19vcJJT84fqGtoTwcSPLyQ+ZjE1e44NUx/cfX+na0OldJzXLtNXNwl4D2dsbdmVzNMx3FMdxTXQQd3Fce7eUreBZ/kA9A/17UUx/hufZw/HP44QY+wlrP0YkJ3PZM4OqvZj/ZOSA9rRdGM9lfartHP3zuO6JzjnI60XCH/AfPC6MR8q/HcOwWZ2vk694XgLjxnE6b22QF5T9kMhn/Rk/7+ItkIDiEebbKv98XDfa8HFPQZ+Yt+nq5O154xNFwm1trbV9W25r2/fwTIPuT+6Y80DX0N+8gX2L3T6iyzHf6HHssLzHx2X6k/by/o3x/8qhD34sqq/12T6w2t/Hp4qEaD2awe1Y8we2377GD0MU6d2tSDsfL+PXcPDh1uM29ZBn+J2vOcbrjyBZTNcrWJ/ss7B4M3rKyW1Tr447jMCuFx/zHppG4jqHo/rPRuH6s8maR2rZD4zwf+xn2/1rr0qRULJX0n9mXYj8i3bpE0FNn3o7xeSGhzptPT89Kv5ZU96f4o/4TzDbJP83HC+uQwzw5Dv5oRync3+nfKr6IZCfnFc+kNK8OfF1ozV7lG8TfLJxGc+uniV9PN4CP3lT3nihSODr+3KUq7l9Dt6f0DyANhvTfM0HxuMeCaqtrv5sv0TA8h4Zl+pP51XdvyH+L+TzTn5rDaSUxy903X1gvT9G0h8CMR4PfD8Mwe1Y84cb8dsiPwG+7/8xE/5CvPx8kPBPQ/7zR/DeAyaB4Z/gWVuH1/Fv5L03P/3yH8Z3+oMiXXgVpTsHQRgg/xGEvw7aOryIfyHvnWf4/905vgtf6w+KdEEQBEEQCtDWQRD+BSjSBUEQBEEoQFuHFKYf0rzlMXFhxLt4/jY5XwBFuiAIgiAIBWjrIAj/AhTpgiAIgiAUoK2DIPwLUKQLgiAIglCAtg6C8C9AkS4IgiAIQgHH1kF/9Ed//oU/v51yBEEQBEH4G/DrWxb90R/9+ZN/wpywhq8PdHC8cflzMK/mM8Ms7IV03bv8MvLP/pmfp18bF71zE76Hc/jlfP9O8ImEWP+1De1c/3jG3ftvPaPPbw115kW+7naB3EUSM/w7YPa9Zsd5XFefgh1xvLwPzA/ZuN680GfMnxn/vl1mPbE+xbiO/Hmy10W7z0KpVa/wj3Ah3p35Dl8r5YFem3GA6rzYd67Mt5JPruWqcXg3bw/51tM/kTd+fcuiP/qjP3/yTy4vvfCWh4++TM684nJtQ2pHOXdt51q+ZDRb29F/bYk0Xh932fY5rS/bbtck8y7ubord67vbelvbOCNX/23/aVuvMNGfgsj3qF22fXInNi8Gpwvk00GJfwZm36odBz7j9oe0rB1pvLwJzA/ZuA7P2I7E3xj/gV2ieDk/Lca178+zvSgPrt1nsLigdi/KvxLvznwn/qt5gOar6ryK/lbNP++yL/VbJ2/jfMuLhI/mjeJ6mvTbkM/yulnlIadPhjc4cHVZrqK6P6EKkfak/2fWBab/V/lJeX2p56svgDlRaGtXJJijBnJys2/Wze5/7+T18ucTuH1bnp/XdnthkTD0741xn8EwKdMlE6nVcVEGH8SwxRKMBDZnVP+HLy4vFQlM/rbv2wrOLu2Z1mB6Ni/oV0w+45PJucA/83P7XZtcSnYcTWQSN2y/vWDHMV6gUYzB6E0duhlz0tzACeaH2CUTj4xn0D7pSXm+kk8emP15thfjgepD7ELjgszR96vZz6vxzuZrBqR+kskDjg9QO0K/LfpbNf+U7VuNL6eIRfnW07+UN85BUstGeT0l4zI5jj8fyKybVR6YPrX5dtc4HafLtrcNXKEe35i29T+tXbhKe3w9zMMXigTo/xk7pg4LyFi/7Cfp9SWjz7fBFjOrfdxo22n182MMYJ3WFAknD4Mfgn3JvWVtaQfvme3ibZDxzJyjJRJFQmi86rhr21tDi2u3C8ADJRZdpv9Zqc+bS6o/AJPfPWZmjwVuEY1TJQP9isonfFL/LPLv+Dnxq6IdTUx1IUPaX7DjUPmTeFxXE7P4JHWM35u/+RsyIOGH2tGJRz+uc5stxv+VfPL8Yu/72F6cB+IPzC487sycQ/nMz6vxzud7m8T1yOUBahcyL+7nNX+r5p+qfcvx5fgnyrdU/2LeMFO7VCQUNkNDBUvksHi5mc4ZPyzywPS5MF92k3DYa940wxa7RuQ22ZX9CeOH8sbWU8ZPtG46s/geP8mvL6E+X4dhbge54Azh3tErfR4s0h08yO+TT48HIvB5ZMorua/KXOsQJ4v0SY+72GzdO7GppBs8rgay00XCMAHMw6k/na+ziNJzbTdXJktqJp/wSeUU+U+V+MM1YtWOZ3Ds20Db3F60YycLXyMM/tx/ZLZ+rv+zzd8cp4QfZpfMosviGrQjPRn/naB0PplrQGwv7idYH2gXP+5uiH8sn/p5Nd4D/+RFwpWjtd4uYF7cb6v+Vsw/Rfteji/EAysSPD+Bcm7Yf0q4WCRM4zpy/PhNrpvDpyEPf6BIOFvR+bR3Yu3PbkJqfzJ8gZRR9nlQ6P/OJt5fNzG+zU8q60u87nwX3l0ktNZm22VHj9AdLHGJlXbbo99L1WwXjzucEuFrfUQI275MRUKkvzep6/wMWStfJIx7KlokYPmET7dIKPCfvAc8u1204yQnbk/bEcULH8j0NCNEPNDk6O4SzT+4XTLxmOYtSDXVvAHaqVXmXbDrJ718Zhcv7ip50ikS6vFOmzz+k3mA6M/aL+SB2nwv5/OknpGEQc5TOeCWkb0mOQn/iXFhPSV542IeSK6bjpySPhfm++tFQmZ/wvqzdub/nLdMvHT4Nj+50j9o/y6Mv/HonmW0V6jmZLqzane4tW/L0EZP+55Ssz5tdbOwj0Ut5tjEnqCsbTrVnM9sumPk+BDlwrj9XfUZnF6QM3dHPQP9e1FMfwoi/x6w5xDJx41AFYr9ypEP+XT9s8C/ow+070U7PuWEBzRMFLMjixccj9ZS/SFNcAuHilU4rsMPtIvjb5B/N96DVDPIr+cT1779R5lF/dSH24XFBeOfzZfGy4V4R/P1Z1rKA2G+4vPqUPW3Uv6p2bceXx4P5CYB6l/NG8eH+f8bq7Se0nETckB7et0s88D0Kc73ZrdYXdh9tkio7k+cfAjbqT6En6r+3+YnGc1r6/s3wt4T338Cc1wmkEtQ+HDWcVt1yDtY669U798YH2sJyJquZbE+nfc4z7rhZHfKyZnu6rjDCJ0BQh8eieuqf6r/bBSuf2K+NpP+mEcZjXjPYCQ2Z79y5A9khP7J+1P+iRxm35odaXdPTNqOyXgxn5hvPAjvQ3vsj/3QG9eZWGRHUOG7/HfPqiI9XXXqce2sNcc3lqktRQ+0C40Lh//MfCP+o3gE83XyFeMt1scKyc2Lxdhr863kk6KeqTwPG40YX/9q3qgVCaX1NJmv0nny4rpZzZ/MjkmGJvc5hm3r+elx8jprxPunRs3tT+g+Ic6TbD1l+Twk7tv85ML6UpmuIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAjCfxb/DwLd9njqvnRlAAAAAElFTkSuQmCC" alt="" />

可以看到这里为discard_passdown,表示dm会将DISCARD传给底层设备(loopback device),queue_if_no_space表示如果thin pool没有空闲空间后,IO请求会被排队。另外,error_if_no_space表示如果thin pool没有空闲空间后,直接报错。

[root@yy3 ~]# ls -slh /var/lib/docker/devicemapper/devicemapper/data
2.5G -rw-------. 1 root root 100G Nov 12 06:14 /var/lib/docker/devicemapper/devicemapper/data
[root@yy3 ~]# docker rmi dbyin/httpd
[root@yy3 ~]# ls -slh /var/lib/docker/devicemapper/devicemapper/data
2.0G -rw-------. 1 root root 100G Nov 12 06:15 /var/lib/docker/devicemapper/devicemapper/data

可以看到删除image前后,稀疏文件大小的变化。

dm.blkdiscard

docker还提供这个参数,默认值为true,即删除image后,会调用DISCARD,真正释放HOST上空间。

func (devices *DeviceSet) deleteDevice(info *DevInfo) error {
if devices.doBlkDiscard {
// This is a workaround for the kernel not discarding block so
// on the thin pool when we remove a thinp device, so we do it
// manually
if err := devices.activateDeviceIfNeeded(info); err == nil {
if err := BlockDeviceDiscard(info.DevName()); err != nil {
log.Debugf("Error discarding block on device: %s (ignoring)", err)
}
}
}
...
} func BlockDeviceDiscard(path string) error {
...
if err := ioctlBlkDiscard(file.Fd(), 0, size); err != nil {
return err
}
...
} func ioctlBlkDiscard(fd uintptr, offset, length uint64) error {
var r [2]uint64
r[0] = offset
r[1] = length if _, _, err := syscall.Syscall(syscall.SYS_IOCTL, fd, BlkDiscard, uintptr(unsafe.Pointer(&r[0]))); err != 0 {
return err
}
return nil
}

主要参考

https://www.kernel.org/doc/Documentation/device-mapper/thin-provisioning.txt https://github.com/docker/docker/tree/master/daemon/graphdriver/devmapper

Docker内部存储结构(devicemapper)解析(续)的更多相关文章

  1. [转]关于docker包存储结构说明

    原文:http://blog.csdn.net/w412692660/article/details/49005631 前段时间与同事交流docker的安装包层次结构,并沟通相关每个文件的作用,但是一 ...

  2. Redis-基本数据类型与内部存储结构

    1-概览 Redis是典型的Key-Value类型数据库,Key为字符类型,Value的类型常用的为五种类型:String.Hash .List . Set . Ordered Set 2- Redi ...

  3. docker的存储结构,和以前有了很大不同

    在网上学习这一块知识点时,有一个URL讲得很详细, docker 镜像与容器存储目录结构精讲 http://blog.csdn.net/wanglei_storage/article/details/ ...

  4. 【SQL SERVER重新认识】数据内部存储结构简单探索

    数据库经常需要打交道,但是从来没想过数据库内部是如何存储数据. 今天探索一下数据库内部如何存储数据,从下面几个方面探索 数据库内部如何存储数据 索引数据如何存储 操作数据对存储影响 总结 数据库内部如 ...

  5. docker 内部组件结构 -- docker daemon, container,runC

    Docker, Containerd, RunC : 从 Docker 1.11 开始, docker 容器运行已经不是简单地通过 Docker Daemon 来启动, 而是集成了Container, ...

  6. SQLServer2012 表IAM存储结构探究

    SQLServer2012 表IAM存储结构探究 Author:zfive5(zidong) Email: zfive5@163.com 引子 国庆节期间,一直在翻阅<程序猿的自我修养-链接.装 ...

  7. 深入PHP变量存储结构 标签: PHP存储

    1.深入PHP变量存储结构 标签: PHP存储  分类: 编程语言(10)  首先声明,我并没有去读PHP的源码,只是对于PHP的有时候诡异的表现感兴趣,找了一下开发人员laruence的博客结合PH ...

  8. PHP变量存储结构

    首先声明,我并没有去读PHP的源码,只是对于PHP的有时候诡异的表现感兴趣,找了一下开发人员laruence的博客结合PHP提供的函数debug_zval_dump刺探得到了本博客所阐述的工作机理.如 ...

  9. docker存储结构解析

    由于aufs并未并入内核,故而目前只有Ubuntu系统上能够使用aufs作为docker的存储引擎,而其他系统上使用lvm thin provisioning(overlayfs是一个和aufs类似的 ...

随机推荐

  1. super的作用(转自百度)

    super出现在继承了父类的子类中.有三种存在方式:第一种super.xxx;(xxx为变量名或对象名)这种方法意义为,获取父类中的名字为xxx的变量或方法引用.使用这种方法可以直接访问父类中的变量或 ...

  2. php Hash Table(二) Hash函数

    哈希表最关键的几个方面有: 通过key访问(通过哈希函数计算出key) 映射到数据结构中(哈希表本身的存储结构) 映射的处理(冲突或者碰撞检测和处理函数) 理解PHP的哈希算法 一般来说对于整形索引进 ...

  3. 浅谈JavaScript中的定时器

    引言 使用setTimeout()和setInterval()创建的定时器可以实现很多有意思的功能.很多人认为定时器是一个单独的线程(之前我也是),但是JavaScript是运行在单线程环境中的,而定 ...

  4. js使用split函数按照多个字符对字符串进行分割的方法

    这篇文章主要介绍了js使用split函数按照多个字符对字符串进行分割的方法,实例分析了split函数的使用技巧,非常具有实用价值,需要的朋友可以参考下   本文实例讲述了js使用split函数按照多个 ...

  5. 在虚机中安装CentOS

    摘要 最近看到.net core 1发布的内容,也想尝试着在lunix上跑一圈.linux这方面的知识一直都没怎么接触过,只在工作中见同事操作过,看到满屏幕的命令行,感觉非常的高大上,趁着现在赶紧学习 ...

  6. SQL Server2008 MERGE指令用法

    参考资料: 百度百科-MERGE

  7. 贴上自己写的代码-jq隐藏事件

  8. php中关于 left join 的分页显示

    /* * 统计商机搜索总条数信息 * ftt * 2015-11-10 10:13:15 */ public function getBusinessCount($btype='',$buname=' ...

  9. Java反射机制(Reflection)

    Java反射机制(Reflection) 一.反射机制是什么 Java反射机制是程序在运行过程中,对于任意一个类都能够知道这个类的所有属性和方法;对于任意一个对象都能够调用它的任意一个方法和属性,这种 ...

  10. 大熊君大话NodeJS之开篇------Why NodeJS(将Javascript进行到底)

    一,开篇分析 大家好啊,大熊君又来啦(*^__^*) 嘻嘻……,之前我写过一系列关于JS(OOP与设计模式)方面的文章,反响还好,其实这也是对我本人最大的鼓励,于是我决定我要将JavaScript进行 ...