Docker内部存储结构(devicemapper)解析(续)
dm.fs
参数dm.fs可以指定容器的rootfs的文件系统,但只支持ext4/xfs:
func NewDeviceSet(root string, doInit bool, options []string) (*DeviceSet, error) {
...
case "dm.fs":
if val != "ext4" && val != "xfs" {
return nil, fmt.Errorf("Unsupported filesystem %s\n", val)
}
devices.filesystem = va
这是为什么呢?下面是docker dm的维护者Alexander Larsson的描述:
Additionally we ensure that DISCARD support is enabled in the filesystem so that any files removed in the conttainer filters down to the loopback file making it sparse again.
参考这里
一句话,因为ext4/xfs支持DISCARD。这样,如果容器中删除了文件,空间就会马上还给Thin pool,因为Thin provisioning是支持DISCARD操作的。但是,默认情况下Thin pool是底层是稀疏文件/var/lib/docker/devicemapper/devicemapper/data,所以,只有Host的文件系统支持DISCARD,才能保证稀疏文件空间释放。
Host为ext3
# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
dbyin/tlinux1.2 latest 8297f05d459f 41 hours ago 399.6 MB
dbyin/httpd latest 93e711fab1c1 7 weeks ago 412.7 MB
centos latest 61038e6e3195 3 months ago 236.4 MB
我们可以查看稀疏文件的真正大小:
# ls -lsh /var/lib/docker/devicemapper/devicemapper/data
1.6G -rw------- 1 root root 200G Nov 12 11:52 /var/lib/docker/devicemapper/devicemapper/data
# dmsetup status
yy_pool: 0 409600 thin-pool 0 13/65536 0/3200 - rw no_discard_passdown
docker-8:1-696417-base: 0 41943040 thin 928768 41943039
docker-8:1-696417-pool: 0 419430400 thin-pool 73 633/524288 26115/3276800 - rw no_discard_passdown
我们删除一个image
# docker rmi dbyin/httpd
可以看到稀疏文件并没有变小:
# ls -lsh /var/lib/docker/devicemapper/devicemapper/data
1.6G -rw------- 1 root root 200G Nov 12 11:52 /var/lib/docker/devicemapper/devicemapper/data
# dmsetup status
docker-8:1-696417-base: 0 41943040 thin 928768 41943039
docker-8:1-696417-pool: 0 419430400 thin-pool 73 490/524288 18758/3276800 - rw no_discard_passdown
no_discard_passdown表示dm层不会将DISCARD传给底层的设备(loopback device),只删除映射关系。
Host为ext4
[root@yy3 ~]# dmsetup status
docker-253:1-8790943-pool: 0 209715200 thin-pool 339 876/524288 32432/1638400 - rw discard_passdown queue_if_no_space
aaarticlea/png;base64,iVBORw0KGgoAAAANSUhEUgAABAwAAACtCAIAAAC7jnp5AAAgAElEQVR4nO1d27XbPK90Wf9SQapH1bAZlXIe9nmwLYHkDC6yne18mVl5SGgaBAcXEqQc3W6CIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAh/K9b2c8e+/bYq/2ks297W29r2bfltVQRBEARBEIS/B8u2P3brf2wbuWz7c7i1/bR1/nRsu4q1vVCHbPvbaHmS/DPM7Gh+bZhHyQWELNve1nulUBao+k0QBEEQBOGfxbLtn6sP4O60G/Hjh9xre2Gz+x5u1nZsuG1RtLZjW7+2sXwoA9txbT9XioTbi7wxXFHkN8b9LT0FQRAEQRC+BfPm8n68vW/L8/z73DAdJ9/Dnha0L6at/8KzLhjPv+0XTpWWQ53uX8GUzkG7zS7T/3j66WffLB/3v3fyhqlNwmfeujqIbNgv7eNHCYMdz0kln+yaebu37Ht7tlF/2Pf9kA/5If5wb77//S7+PovHNc5mDJOo1oA+3A8tQZO/ZfX0eBAEQRAEQfh7QU+gjx3e2u57o/MhoVt3Os7ab+RE9r5pfpypw5uEobH/Z7SdNgf0/WMzTE97jL9su93+mSLB7oz7LaXdFCLexiIBbCJzx/ZW+PRQFrPjvi252xrC2zH5J01HPWNHXJvlmfLDbpbsxr0vGq1hgjKK6cPGva2rrXZsj5KezriCIAiCIAh/K5zN5div3zg9urD22+3mFgmPYTNFgpVzfjQelD82lMN3j0051dPboB+/1+gEwnGJ5mPbXCQMJQjFYxPPfkMC7XhvTBUJDm/Pmue4RzmNMZPg8lMtEmzvxGUL0oePO6h6uUhwxhUEQRAEQfhb8VtFwrlFm/fo8+jPlsxG8b1FQmvD7wXcU39cJJjv92oMFxevYLLj/LSRO8yVIqEf7UGLz2epSBg1KmzA4xste3MyCi8WCXRcQRAEQRCEvxXZImHctJ17QdbefWSe6Yl/Ko1Hvz8nnzgSHx47j/Tsdov9b4iP647hUXaqBtRw2Zp5HMX+cPmNG0p2k9DW5M9wCW+8SOgGNN92+An8oa+Z7v8w3w1mwfTB49ornekyp6gnHVcQBEEQBOFvRXgC3Z2Cl9o7YcP5+eOf+Elz8gxS9lmO7umPNjxeTx4vAe3Hf4FqXuoAfmPd/5w1kn+0Oj+ovQRYJDyf7LrK28/Pfij3+J3CcvyEF5EwzXf4iPrDk+CtHTxv+96a/Ql0UB86+sBxjQUeA6HfNGf0fKMZBUEQBEEQvgMf/S9QnUGdPVX+ckM4wIuEj/xnpn8Av+GYgiAIgiAIwu12+5WXqWF0/yEl+UDntBD8ZWp/Lcwp/19Z4QiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAge1vbCf0h/vJn4czCvujXDLPBFyf0nuUn5L3fjvSvjopf4whf7Dq9itlO+95+5dvVf29DO9XcxySF2cfSPeB79EPJD29G81vbjOCeTH32W7evKAHz68tHL8rB85oeVOfVyavx0dnT84ez8mj5zu2/3G3n7SpUfJqeaB6qoyojzBsiTwC6M/3f5OZOF83+9Heof+ecFnP7v+yHT05Wb5Tkxr1KcFlBbT4v2uhEeyusmXwer+lTnO+Muwfn22l72TD7f6vrlr0f5der88O/0k+p835H7fxMvvZf3oy/HNeGxNkvvsu2I67WdiS/1Pum1Hf3XljBffdxl22e/WLbd7mmOj610K+fRf37ztKv/tv+0rVeY6O9jlsPswvTP8Xz6IeOHtbN5MRegcoi9KPp5sfnO+5jRLgRJux+fUj909QnnVeTntCP1h8enWR6oPqTdC/1l2+H2vcQPkVPNA2UU9WT+4+dJYBfCsxNHWI6v6sQPyzPVdu4nnn9ehfV/LJLqyUWW8kluXtk4LaC4nlbtxXgor5t8HX9Fn4ynw4EvLcsVsAEIn8H6O/EZtE9y7sjnh2/zk+q6fGV9+X2YCqmtXZFgSidSce6bTSP3v3fyevk/k/B9W56fF0JjGSwcfXVKdvcZDJMyXTKRWh0X7kh6MSwpg5HAYk/1f/ji8mqRgOWQuTD9qZ7EDxk/lDenSFjBGYQnBxbM5tB42Pl1lCP9hyKc2gXFixEwJl8m3/aBvHX9ybievSA/PJ8cGCyEeSA8M31YO7O7Pfu3U/btNec9Jmeg5LJfVf2N6Ynoc/UkdmE8O3ms5ueMn15PmGcy7Zk8P/sn5BO3E/9nfkj1v2R3Pz93H9TjFK6bDOX11CBjr0zey6ybyXU81Kc23+5YuuN02fa2gSPx4xvznXxrlaNpvtkN1y+wP2H5xF2nUvkhgS/yk/S6nNHn2zAePtprR1b9/BgDWKc1RcLJw7A+duvB857m/vfCme2wqIBYswPMK3eiSAiNVx13bXtraJHoVgM80ExNWCTMJ/HOIp27OMFyehVC/Yme1A85P6SdzKt7HM4eazA5zF7ramXCE4suBEg749OLl5E+d1w7tSHoUH82Lo0LzI9jRyNk2KtAvyI8M31YO7c7CXPCj5P3eLqo5AHqVzV/8/QcafL0ZHbheRLHUdnPaZ7sVJhbk+1xnp9OWCCfpN1bT6kfQv2Ldo/z81iC1OL07HmtSChshnL28vIe059s/oJ1MKXPhfmym4TDT+ZNOWyxPhCfTLP5Uj7d/Um6SKjmhwS+yU/y63Koz9dhmNtB7kTcs6NX+jxYpDs24JeeJ0F7+LySe77MtQ7fvLr6pMdd7CrRO7GpsNusJo6ddJEwTICFtN1jwPkGcrhdxr0f1JP54Y3zE/HWz+s2LMunVkQOtVfv0rjs2rurU9iO+fTjZSbKHdfSMJ8/df35uF7xOfPj2HFm3uGB83yhSMB2J+phfvwjn2hPkMoDzK9q/hYdTbk3CdPOEdjFWYxhHFX9nPJj5sz2Vpn2cDOXO+oj7dz/PT+E+hfzTJjnhxqhGqdVXCwSKvby815y3Rw+HX0orc8biwRqJlIkOP0D9PPFfPrrbL5IKOWHEN/mJ5V1OVyvvwzvLhJaa7PtsqNHAAdjSGKl3faIn2t/ZdzutMwcpHUcIELYI5NTkRDp702qcu01yfHtMt/dAz35osX4CXk7PniqMY4ARjb/IPbq9tuc0YofTmkeSpw1fHXcrp2Oy/wK85PYfOQeVqM8M31YO7M7U69HzE9RTtWvqv72Af8Zd82M/ygec37O8uSN55lSe5gn5zMNpOaFIoH6IdLzhTxDevP7oGKc5nBhPb1mr0evpP4V/y/pc2G+v18kJNw7iOt0kVDMDx6+zU+u9A/avwvjbzx+cFKzTtPdkfe/VX1sFKbfkxD6Kj5tdbOwj0Ut5rhmfKBk3Kn+zHcUR3lLLgZeHXezP90xmwYvyJm7o56B/r0opn+MadOAn5Ji+mM9qR8yfli7Y5dTT/OBwz+w13T0S4xTaZ/49M8w/JNgE4yO/4P+dFzuV8ifqR0PaXQlGPZNjGemD2lndh9HQeD89IziPVYlD7D51v3N0/MG/Cf2k8FkhOd4s5L2c5gnWZ6ptgd5EpyOYT5Ju7OeYj/Eer6SZ1InRNfitPC4UXU9vWgv1p5eN9l6UdanON+b3WKZwPt0kZBZ9wtxnS4SqvmB4dv8hPUPxnXbvw/2vuf+E5hj3SLXnfDhrOOW+JBn922TpPGxloCsXhmqT+c9zrN0ONmdcnKmuzruMEJngNCHR+K6Uzeq/2wUrr+HUY5rF6g/1ZP5IeWH8jbPa20/Pz/mUeeOBod/YC8z44dAs9qX1ER8Dvr/YPGdyah84odcHzgutRfhh9vxwR5LlwMPlGeuz9zu2J06bo4foIxDRCIPsPmW/Y3oyfOGo2fkn8zfuKOEfo75YXmm2u7p//iel68S6yD0f+qHXM8Ldmc843ldidNKkVBaTy/Yy+Xh4rpJnv1L6VPfPzhu3tbz0+PkddaI9/eA1/3MOkLc3IpK5ZlcfgD4Nj+5sC67+giCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAgQ6/DaxRKONy5/Dscb77phFvZCTPNJblJn/8yb8K6N67+j1ogZXi04vyJ45trVf21DO9d/xsMziHGxXQJpZNh38DPz4Olf4YHJd+Do2c2vf91pyW9PIVifUf4l/5lAeKPzjXlG+Yf5ytQexkvavD4Az8z/HT4TrzeN33rK/KQaXxf0+Twcf65Kyc2r6s9cfi2OOKB9E/lkxDxXP59fyj+unBwP1Th6z7rJ81JVn9ftfpfgfHttL++xMvuWDJ9czlvWdwff5idsPU2vXX8Z1vZCXtj2DxYJJjzWZul13iD/nMuS0Wxt9tXisfnq4y7bPvvFsu02dx8fW+lWzqP/2sYZufpv+0/beoWJ/gyMQmoXAjbfG+FnmNehBOOH8UBdoMhD1U+onoNQ+1r7kt8+RpnsS+Rf8x8Awhudb4rnLv9gfyDtQbw8pL+almeeqf87fC7bjtIsmy/sT/2ExEtVvtf/84j8OYfSvKr+zORX44iA2TeXT6g+Rg7N5xfyTzhuyEM5jorrAs17fB1/RZ+M3eHA1eWoDDYAXTeL+byYf6r5+dv8JLmeZvLzF8NUSG3tFmlTOpFKcd+s29z/3snr5c8nIvu2PD8vhMYyWDj66pTs7jMYJmW6ZCK1Oi5fsewRFHQaMNLklI7+D19cXi4S1uDOYBAJ/YTON8EPK2JtF8YD1Z/zAPX3eYZxAfV04s52wSc308kxtq8jP+8/5rB03CFF/jPOF/ZnehJ/oO1kXM/fOJ/QjpjnXjVDG+bTnj13Q5N50f79uHaRw/FSlc95dvgptL/gz11/4p/leRX9mcm/kB/CvJFcF8J8a/0hmc/DdZOhvJ7244ZxVF0XkLisbrm4rsy3O5buOF22vW3gSPz4xnxn3lrlaJoWRbH+mXxezT+Z/Q/DF/nJ1J+Nm9Hn22CLme66f9tp9dPf0MIi4eRhXJfn/H5vWVs6jfTMdvE2yHimzdESiSIhNF513LXtraGk3K1yeKCZmnCTN5wcz3I8/QG6x8ls+d+r0M0K+wmZL+PH+F7mCJDxQPUnPDD9mXwnLpCeNO46irqiGscRsa8vv+A/t3U1sYxWLeY/KX/jenJ/IHFExiX+xvhkdqRxZAfq/J/nE5henHnRdITzDI6XqnzSn8dFrb3oz3wdYf5ZnFfZn4n8an5w8wZbv4A+VA7xBzefF9ZNhvJ62n81llNcF5C488vuOpjS58J82U2CzTnz1nNuOdRObbLpehfpn9w/lPNPtP+h+CY/cW8SBh4Cfb4Ow9wOcifinh290ufBIt3BA3+afHo8oIHP/VFeyX1V5lqHOFmkT3rcxWbl3olNJd3gMRKQnd7kDRPAPJz60/kOy8io1WgX7idkvpQf60Q7eARh0MQpEjz9Zx7oMSrxkyjBDXt4HHeTOmaLiu2C7RvKzxcJ/dBse5x+fKPvz/Vk/uD4CR4X+hvl07G7G0dTXqoWCd68oj0HutYf46Uqn/Rn/BTbq/7srCOOf1Z5Nlqn/LlQJFR5G9WZ8/Z8NsK+Dv0hkw9feRziYpFQiaPSuoDEgU9Hu6f1eWORQJcJUiQ4/QOY+Yb6p/L5s2ch/0T7H4xv8xPUn43r6/N9eHeR0FqbbZcdPcJ00IIlVtptj/h3Dq+M251+mYOxjgNESOJ2MKe/H+XxIxyjp5jYnu2S22SbfzB+Qj3nc2PIg6M/ku9tFpH8DIG8PKaLSqxPJ8Qc0NeLBDIvY1nuQbFdcH+/SED+EPrJ5A/Q317YtE0swLzkxSPbvLJ5xXuO6/mnqM+bioQyz1yO459Vnl31gD/jIqGUH5LPG4T6lOUU82EZF9bTchy5eibXTUdOSZ8L8/39IsHMN9Q/lc95++X9z4xv85Mr/YP278L4G48fnEQ6Z+qsamqCx1fA70kIfRWftrpZ2MeiFnM8Yk9K1qFyuaFrU1PGZn5Ac2Hcbe/q9uO7XpAzd0c9A/17UUx/hntgnqLsyZ9/Cv7sdf+CM1/IzyAzPgcgPDD9GQ/Uz4l8Jy6Qnk7cYf+JT/UGZon8oynlP9OVguXT85/JLqS/pyfzB89P0A4e+hvjM7DjNATzfy8eyaafzgv1d/Pb2W6bS/JZf8bPa+0IE8+gP/fP6ryq/kzlF/MDaw/sC07xIr9N58PauslQXE+vxJH5brwuPLsOXuTk/5o+xfne7BarW14+WyRQP/f1z+bzDpn8U9X/2/wko3lBny+Fvd+9/wTmuEwg17jw4azjNumQd7DWXxUfuRFeHhNM181Yn857nGfdcLI75eRMd3XcYYTOAKEPj8T1myem/2wUrj/A2n5+fswjhd3DrAm7ZOeL+PHowTE+80D1d3lgDyEynp2HFoGeLO64/6A4GoWdEyPyq/5jLPwgsA95oCeeL+tPebiReHHa4bjUgRifjh1HnpP+3xfn9AtoXrw/8RMvXkryKc9xXNTbY38m/Zl/vjSvhD878qv5Ic6T1fiK8qebD8vrJkNhPa3HUXVdSOW9vuCs6VPfP8xueAzb1vPT4+R11oj39xCvd9V19oX8Exhy1qNql4/6idOfjVuZriAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgnDH2r/utIbjjcufw/HGu26Yhb3I0nySm9TZP/MmvGvj+u8U5m9mnV+JOXPt6r+2oZ3rD+How0Dfiwv1uSin0D+Sw/x/bK/x7PoDfP3irOdDA4fyypyv8OB9Z+yP45S31+KOg8kPdX/D2AgX7Fh+Hec0xhvjlPHJwOxYzVfXx83FV21deGueTMZ7pn0mp9pe1aeKalxX88N71k1u36o+r+exuwTn22t7wx6ruj+p5tVq/6q/fZuf+HH3+ny/DWu7WiTcbrdt/2CRYMJjbZZe+Mbwe6fnXJaMZms7+q8tYb76uMu2z36xbLvdQxwfW+lWzqP/2sYZufpv+0/beoWJ/gxMH4pen3m/OOpzSc4MxidrHwfD/m/aizx7frhs+zQc9wc6dehXtD3FZ8cDkw/7szil8VuNO6pBSn4Yj+/CBTtCfwiGmPR/V5xSPnNyxrhL56vquNX4Kq8Lb8yTlXhndmF8VtvL+lRRjOtyfiD8lNdNvo6/ok+GNzhw0d3KqO5PGA/MT6r9q/72bX6SzG/np8X9zHfAVEht7TYHpnQCldO9erJudv97J6+XPx837tvy/LwQGstg4eir02Jwn8EwKdMlE6nVcckOwIphixYYaXJKR/+HLy4vFQmOPtAfRhXM3LE+l+T4/Yfgxzwz/yftZZ4N7Lj2DM+GBtNz2/dtRWcobGeZ8LeOT4cHJp/kDTIWbvf5BHmG5BMm35sv+XptXNJetSPzB/tBuOMcQNgcKyUWX6QP5ifIn5V8hcfN2T2ML9bf45meKCA/MbDfrMZ7YBe46a+0V/UxX0otG+X11CCTHzJ+m1k3k+t4qE9tvt2xdMfpsu1tA0fixzembf1PaxePpjP7k6E/zKts35Lpn5GT0eeX/QQV53DcjD7fhuHw6EwW206rnx9jAOu0pkg4eRj3Y5aU5z3N/e9pB++Z7eJtWprwCpEoEkLjVcdd294auijrVic80ExNuOgap3xoNxcJVP9o4sPeA/qD8RnrEkyfqhzuh4xP2M78n8ZFmWc79GBf7GZY/+5xvv74BfsVb0d80vkSOby/nUh4LcP5ZPb18gkcl/gP46c6LtenaMcb8Yd1Nbm0OyEj+cRI67b2VA7mx3xzrMwRP0H+TOcrNq5r91J8kf6MH5InnXXw/OLAZSXeA7u8XCSU9Tk/ulQkFDZDqfwQ+21u3Uysg8l8VZ4vu0k4fGneNMMWu7bmNtml/YntEa6nF/qHcvgsvsdP3JuEYd2J8u23YZjbQe5E3LOjV/o8WKQ7+IlfkLnGAw743B/lldxXZa51iJNF+qTHXezuoHdiU0k3mMmB7PSiO0yALSF27+3PN18Kn8bfz0dwiD5VObw/4xO2M/9n7a/xPPoh3hRi/fvt5GkF5lfU3yCffL5YDu9vx2BrYH8sTeIuur7I5wfoh4Sf4riOPkU79mrQoe2pG7MvkMzlMH4InzTu3lwkDOOGdn/IrF22dP05P90XjjRWOXb19OH537PL54qEaD1K4mKRkM0PR1/CT3rdHD6FuSOjzxuLhO4gIFEkOP19pPYn5uPhVNXzk3z/a/72bX6C+rNxfX2+D+8uElprs+2yo0fABzaTxEq77dGvSTXbxeMOp1/4mh4Rwh6ZnK+PA/29SRWuvcZHD1JfRN2m9FaRk9usmH/g9itFwvt4RmsJ039+vPksEpBfsXasj18kzHLcIoHFKWwnfF4J5FJ+IPxUxy37IbPj4x/g7Lbbx3ZFgmvf6REhLMeZDuLTma8bF5V8VRrX0T+xVzP5pMZPrA+QUYl3Mm7U8+UiIadPjAvraSU/TKNNE0mum46ckj4X5vvrRUJmf3KjPFA/KfW/4G/f5idX+gft34XxNx7wGt1Opj+t6X47cmwgpt+TEPoqPm11s7CPRS3mmM6e2K1tqmrRGWQ7HTe+BLowbv+MyBmcXpAzd0c9A/17UUz/AKB6pP5g+4T6VOWw/oxP0s78n8bFBZ6pH4YHwN3mstPBGgz6ldN+wOjP54vl0P4sTlk745PZl+UTKh/Pl83ryriw/YId8QFe1znWHw0fyLFz6XyV2BHHqR8X6XyVG/dENb5wf84Py5NBvmLHB+l4t0jxeaG9rE/hcaPqelrND8N3c+fDgAfHvjV9ivO92S1WF0afLRKq+xPGAxu32r+q/7f5SUbzgj5fCvucxP0nMMdlArl+hQ9nHbdVh7xuXRy/MT7WEpA1XTdjfTrvKT9becrJme7quMMInQFCHx6J6zdzTP/ZKFx/DzCm+EO7V/XJySEPCbIvkHbm/6y9zDP2B+7QQM+1/fz8mEdG53NWrBFqT9HTzTch59mfxWkyflP2RfnEke/5IeatMK7XXrMjm4CZ2eOL0HeHmc1xSuVAfpL2Stixlq/S42Keq/EFn0mYeGZ5kvEA+S87aKLZqlRtvxQwlSKhtJ5eyA/u+nJx3ewLxZo+9f3DHL7HsG09Pz1OXmeNeP/UqPH+xOMBGaDa32tH+DY/cfq/Y7qCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAjCHWv/mtUajjcufw7HG++6YRb64svFfCEj/+yfeRPetXH9d9f270jvX+S3DDJmrl391za0c/0hHH3YuExPOmeoZx1Vu3sgesLXJmL/tN/gb2j2J/2ITBJcM/9+/6r/VO0by5/sAnmgPkLg9s/6VRh3XJ8uf0I51+zC/Oqqn4M87/Cfz+kOP/l4uZBnSvJL8O2Fx62vC7gdyfH0KebzW8av3pWHc3Kq+rxn3eS8VfV5nbe7BH8heHWPFfgJyJPMb3G8c/ksP1Tj9Nv8xF8Hk/r8RVjbC/uqbf9gkWDCY22W3vnN2ken51yWjGZrs6+yj81XH3fZ9tkvlm23ezuzwnVTPOQ8+q9tnJGr/7b/tK1XmOjPwPRh41I9CQ9UzyLKduegei7bjra52D9vtxubF5LDQKdC+Gf9y/5TtC9r9+yCeGBxQdHrOYye96ukn1NjnEUCllO1C/Or1/y8z/MO/yh+mUjKTyVeqnmmKr8Kj1oYv8V1gdqRyKH6FPM55ae6DroD5OWU9SH+Vl43ub1e0SfDGxy4aMY63AHmPMn8k+ZnJp/Yqxqn3+YnyfW0uI58G0yF1NZu8TClE6kI983mrPvfO3m9/J9J+L4tz88LobEMFo6+Oi2i9xkMkzJdMpFaHZfsCK0YttiDkcBmgur/8MXlpSKB6RPwBoON7IyxnszfzCEei7SBT+a3uJ3oac8OB5e249oZwHlROWRe275vKzhDYfyz/uhrnpyyff32p5huk4R4oHFB8snIrvmI+r9rxNu0kGD5PH9COVW7WLCQTfk50TPwZ7j4If9k/FyIlwOZPHMtHn1++nWKxR0Z98p6hNqdIgH6j+MckZ/br7rxDtZNhvJ6WtTHifcDmXUzaa9Qn9p8u2PpjtNl29sGjsSPb8x3a61Vjqa5bmz9tV+1m2man8lmOrRXdUvyRX6CDsvguBl9vg1DUdddl7Pq58cYwDqtKRL6RdGu75aU5z3N/e9p/5iOwUCs2QHm5JgoEkLjVcdd294a2hx0qy4eaKYm3OQZp3xo52ySSgdGs5s7vKFNBuSB6Un9bV2Nj5ELvX6nCOVw+cxeoXt0/kn5Z3LIvLrH+cwxDuOf9Tc6pvynbF+vncQj5hPHBc0nJkfZFMT49/LS0TqVepN8mj+ZnKpdhs/Glqyfu3o6/gzil8Qd4T+Qj+f1/FYyz1Tks3hn/uDZC457fT2KHzfy9KH9Iz8f+XHi/YUiobAZyunj+NvRmlg3E+tgSp8L82U3CYd952IStthcF59Mk/l66xT2W7JvYXyG9irvmL/JT9ybhG7cmIdvwzC3g9yJuGdHz5APFqnHAL8BJ2lO/1vEK7nny1zrECeL9EmPu9js3juxqcjbrCauYAubvE57tqXrnnp15zvtnWqbSMYD0ZP7W+9aJLuHcqh8ai84zf5r+Nhy5j/eZHRFArwHcooE797od4qE48MEDzAu3HxyfrifV+aYfz8vPaXNNd0on+VPLqdqFzs2y3uxn/t6looEHneQ/0A+n1chzxTk84KE+INnr3AvWFyP6Dpl5AT+M/QP/Rzkq8ub+0GFS3Iq+lB/e36aWTeHT0d7pfV5Y5FgC+9MkeD0D2Dmm/GkgvAAACAASURBVNknDI/N+PuWQf7Nt1d1v/xtfkIfqwbj+vp8H95dJLTWZttlR49gi2xHYqXd9kg8V/fCuMMpzhHsHQeIEPbI5LTJi/T3JlUo4ucbaG9cWCSQg2EkhBPb30KQmR1fv1QkMD3JGhD4J9x1gjNFPK/5Mf7jKBvyz/of4yT9p2rfoN183Ck38UniIumoqFvKr06Et8uTlngig5yqXW61vPfpIqEad558f16FPFOQX84Dnr3ivWB1uLg98J+xv+fnkP/X1sGX5LygD5hmct105JT0uTDf3y8S4jxJ+4f7Fip/ak/kt1HBr/KTK/2D9u/C+BuPH5yMrHP8dN7RHXbu2zK08dORW82nrW4W9rGoxRyz2BOX6Rcx6NrUlMOZHx5dGHezP/UxxYMX5MzdUc9A/14U0z8A2ul64+KyB/DAhsD+Zpfn/lCB8c/81vVnoife1Eb/yUymSHDn1T33bB4zgPzT/ueEcv5Tty9s9+IxPIA30nK3grH/B3LcZb4/gMP5k8mp2sXJexU/d/XMFwncPwedu/YL8VLKMxX5uTxwwrMXrglr64LTzuRAfVh/59aCPd3K473wuFF1Pb2kz422p9dNh7eaPsX53uwWyxj+00VCvO5P+wTon2zczL7C8hOvm9N3v8pPWP9gXLf9+2Dvme4/gTkuE8glJXw463g645Bn/WaSND7WEpA1XZtifTLPzJ0fjWOecnKmuzruMEJngNCHR+K6026q/2wUrr8HGFNwXEdPwkNCz/4HXofo+28HbHqH82IPFfKHDYGe2BFd/4TzYl+A81rb+c+JHsC/0/+C/5Tsy+Vju3DiaFygfOKHUexXw3eAnzP5LH8iOWW7eH5V9HOiJxuA2ZHGHeHnWrzk88wF+Zk8cP/EsZczQHFdSNixC65EHuD+AP43EYcHHAPZjU1hPb2gjxvvF9fNvgCu8lPdP8zLyzHs4XxPG4/qH2dlpL8HZ91HeTLhtyw/2/6we5QHRnybnzj9U+vF31EiCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIg/DrW4XWhJRxvXP4cjjfedcMs7AWa5pPcpM7+mTfhXRsXvWsYvn94eLWgnfK9/8y1q//ahnauf4SUnzj6Qz2d/s64ydcaPr45iWTtuflm4wX7Q8D/ZK+rfjKLn/kPeUjyyebly8d6En4uxAVDLd65ru64wI44j3F7sf5MfvfR0+Uu8FbSJxXv/E3bcbzH+WqMR6b/e9q5PvR98iRflfzhgh19+d46cnUzMMrJxRfTk8l5Dz/cjlV9ruSTHncJzrfX9vIeq55XHwNPX3H8fM5Ljnw3vwF8m59U86fL21+Atb2QF7b9g0WCCY+1WXrnN2sfnZ5zWTKare3ov7aE+erjLts++8Wy7XavZlaOboqHnEf/tYEw5vpv+0/beoWJ/jnEfsL0Z3rS/s64y7ZDNVA7E5lzWjbfmAfqDy7/wF49b4ecJM8J/jkPFT7ZvByTYj2pnGJcMFTj3Y9fMu5sR5rHCA+0P5E/TvIsEmq8sbzE9GHymf+X4z2Vr87+TH/Wzvin/Yk+tP8NxxHj813rQiAf9a+tmwzF+ErGRWivcn7g6/gr+mTyCRz4pWU5g2JevZG8R/P2Q9qYl5z84OS3Gd/mJ8n9WLwufzVMhdTWbtNjSidyUrJv1p3uf+/k9fLnk6R9W56fF0JjGSwcfXVKdvcZDJMyXTKRWh2X7GitGJaUwUgguVP9H744KFzNRsxPzOEY83g7UobnrpWMa8/krGux9m3ftxWcQbB2Ol8eLywu7FQyRQK012hS5Es8LCxvmH/GQ5VPr0hA/em8Ev6ZiQuWf6gfkv60ImXjsrjrVbPDhvYdJNG4Jv55IMNbJi+xmbEJp/KbE3dQKunP9M+0D8UG5oFvGmB/FkfOd9+yLjD5yf7huslQXk+JnkxONR8icVndMvrU5tsdS3ecLtveNnAkfnxjvqNrrXI0Xc2riX3LwH+Y95z8UN2SfIufuPmT5ZMXD+X/GIZDq+56mlU/P8YA435rGevOIa92yfF5T3P/e9o/puNkEGt2gDkjJ4qE0HjVcde2t4YuyrpdNh5opiYsEuYTNb6ZqBXwvZ/c1tXYGgmaw8LnuevPx2VfJ+3d43DmmI60s3GpPk5cYH8g/FN7mRjERzu5/ox/xk+VTzYv2p/NK/TPXFyw/MN5IPmKxi8el9qx/479V2Dfvj+R78bL8c0Eb3FeYisc3Ipm8xvXH/uDM1+mP2mn/JP+1D85b8GyMvD5nnWBy3f6F9ZNhvJ6SvSkcqr5EIg7v+yugyl9LsyX3SQc6WUuDmHLoTYrJucBCnmV5T3Cf5z3kvkwxjf5iXuT0I0b5/lvwzC3g9yJuGdHz5APFmkmnfj1VyZoD59Xcs+XudYhThbpkx53sbuk3onNKUKDx89AdrpIGCaAeTj1J/NlfnIbTTyrOuccP5kOewY6Lvk6ax8KmGMU3M7GpfqkEhz1Q+M/jr1OpnfwiMnsJ7C/UyRAfoZuB7z+aF5Of39eg5xwxNFAJH4xD7w/jV84bhh3KI95PEz9sXw/XvK83aK8xPOwc+IG/b8Q71bQs5an/Zn+fF6Y/yg/d/r4/Z2948znm9YFKt/v//jw+uMQF4sE4OdUTjUfTuLApzAmM/q8sUig4UCKBKd/gERedfIeWV/cvMfsUt0vf5ufoP5s3Hi9+y68u0horc22y44ewRbZjsRKu+0RP2/6yrjdqaQ5sOw4QISwbdO8KQn09yYV8eNsjumjI1j/UE9+7/BikTBKehYJqP0zRULNP4v2Cm9Dn3/D/DN+Hv9I88nGrfYP23NxQe1CeKD9WfyScftvgg1/IY8F/a38xCY7nU9oXvL1Sfoh7p8sEg45vD/TP8y3g57v7E+mA/l817rA5Dv9h4/5px4urKeEh4ycy/nBk1PS58J8f79ISPAW5D0uh5EAt+mV/2/i6/zkSv+g/bsw/sbjBy/q3Sajv3Oyv6nat2Voc08jKj5tdbOwj0Ut5rjSnlxOv4hB16bmuCbzw6ML43bPWJjiwQty5u6oZ6B/L4rpT0D8ZLpSyJxve3qO/al/jqMPQuDjMZ2qD7GknY1L9WFxwfwh5p+vNAneeH/CP+Pn8c8KnywuqHykZ8BPOi5o/iE8OPkKxi8bl6nK8tigc+czfv8hgzjxMinj6M/yUqDPJN/Nw4V4J/5A+zP9M5sqK5T1Z/5ZvRhhfL5rXQjsNfUvr5sMxfWU6pmQU8iHaL7QjmV9ivO92S2WIfrTRcKFvOrlvaf64X6DNWbyYar/L/kJ6x+M67Z/H+z90P0nMOjm3U4GPpx13Eod8mw+nSSNj7UEZPXKUH2yz3TiZHfKyZnu6rjDCJ0BQh8eieuqfKr/bBSuPwHxE2OZx7OLdnByt0D1BP3JuMwhYPv9aeXz0crz6Qzc7oxL20lcOP7g8A/s5blJyNu4bxvaHR5KfLJ5pXkmeiaemUnFRT8G80PWH8avM+5ttqOTxyAPbt4D8kc5nX8WeUMKRfqgeHfyYSHeb8wfaH/mWKSd+iH9IM7/0NnsJx6f71gXuPyU3XPrJkNhPU2u7wkzHvKu5AfyDG1Kn/r+YU4nx7BtPT89Tl5njXh/D6W8ihW9BfzD/QaQH+WTEd/mJ07/YvoRBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQHKzD6zlLON64/Dkcb7wb33VPXmG3mC9k5J/9M2/Cuzau/+5CI2Z4taCd8r3/zLWr/9qGdq6/i0lO3HscA843LWh+q/vYjv3kkv6wP5PvjDvw8Ii0SrA4/tANcjV+bxm/Im+MDuebHdt0h/PN8TbyUM0DKTVz/gz61+OO6e/OC/itH0MzrV4c8Tw2ZSr27UK7r2eN/6I/JOKOIRuPV/IhlT/aPVxH6uNmUVtPub8xOe9ZN3k8VvWpznfGXYLz7bW9YY+F7R7kpYJffZq3b/OTal76dNx9Gmt7YR3d9g8WCSY81mbpdd4k/pzLktFsbUf/tSXMVx932fbZL5ZtP9rsuFa6lfPov7ZxRq7+2/7Ttl5hor8PIMfBsu3oxe9wvr4YGE+wnfrJBf1RfybfGxfxUA0W5g89rscv8yvmz9X5BkNPdmTzzfF28lDOA76isOw3Y43nF3P/Ytw5/Dvzmv2WxZ1jd2ZfL49Nchjl1XYm38sniP+qP+TijiGOxwv50JE/253q7/jtW1BcT6m/MTlE//K6ydfxV/TJ2BEOfGlZLoD6mztw3q8+zdu3+UlyP/bn4u4jMBVSW7ukY0onUDndqyebdu5/7+T18n8m4fu2PD8vhMYyWDj66pTc7zMYJmW6ZCK1Oi7ZOVkxbBECI4FFner/8MXl1SKBykH2tWd49iM6XyKH7jgTO9FBU6y/URTvkzhR7BPbznjY9n1b0ZkI1+fAtPtD8cvk+PLnZNfPC36ame+N5Y2EHe18KW88jzH9qT5X/XksTiAP3J1YXmX6O/Oa/TbIM5HdhxWV9cRFArJXtZ3JZ/Ny/NB+t1u8r8bdmK+IH0L7UrswfVz5fr7iy2Yib5yfpZaN8npq0OcTLIfqb5BZN5PreKhPbb7dsXTH6bLtbQNH4sc3pmPHn9YKR9Pe+svzUt6v2Fjv4a0u/w/5iZuX7LgZfb4Ntpjpri+3nVY/P8YAY35fxnOmcR9oSXne09z/nnaPntku3gYZz4QH72T9IiE0XnXcte2tocWvy8p4oJmasEgwTvnQztnc55ILlOPZF9OI50vlMN4onx0lof63dTW+Op2wzv2JfK8d8dA9ntcd72B9OmnDOQmKXyrHl483iyyOzu+MGsITXJg3QjuO/gZ54zwQ/Zk+ZX82uXFIfbg/iTsnr0Z5bHq2Cvutm2e8IqG3r28vUCRgP6+2cz35vGgah/m5FHfMT6gfcvsS/bE+VH6cr9CRidEnk39eKBIKm6Gh8iFynLg7WhPrZmIdTOlzYb7sJuHwjbmYhy32ScbEyTTxN8JD2a/sQJ/hrST/D/mJe5PQx1ekz9dhmNtB7kTcs6NX+jxYpJ438etnfGgPn1dyX5W51iFOFumTHnexq1/vxGZj2uDxIZCdLhKGCWAeTv3pfLEc375kUwXmy+Uw3iif5mNyYdDz0A/NTsRhymA5fmwnRQK+1yH6gJ43Hr+OHF++e6IM4ig7X5o3AjtO/o95c3jA+hN9Lviz/dI+PeLjr3k27hKrI8tj084R+a2fZ5jdJ/sG9kJFAvTzajvV05lXxH/HZynumJ9QP+T2ZfpDfbifh3l+Xkew30Y8JHFx8wfyNpVD4+75aWbdHD4dJ5zW541FAk3vpEhw+gNE+417H7unrfrVo/VjvJXk3/6Mn6D+bFxfn+/Du4uE1tpsu+zoEaYDNiyx0m57ZJ/PuzbucCp5BHvHASKEPTI5FQmR/t6kKtdeU1ryeoITZThfdxOJeGPtt9BPBv37fUJ4gujIp+PiImGMvMcuI9Jnuq4kmxIsJ5IfhOR0OZOdb9m+pqXTEfOWKxKMGpcSRbyGTZec2f7JAKyovXBPQVttssihqtCxF3zcCPh5uZ3J9+ZV4r8Ud8wQ9SKB5kOoT2aCJM+HT0t449ZxYT2F/paTA+hNrpuOnJI+F+b7K0VCuN84PnA1OwG33R/l7dv85Er/oP27MP7Go3tmejHdzIlF52WmJnh8BfyehNBXKRKsbhb2sajFHEPZE6l1qFxu6NrUlNWZH9BcGLe7NTbFgxfkzN1Rz0D/XhTTP8a4+eD2DQ+GzSwcOZA31s78BGswHb2HvDH53rjkJqF7Tv34AY+vz7xpgvHL5ITy3ZNg68/V+bp5A9sXzZfw5uUxqj/Up+rPwxw7ZYndYdwxfZj+bn57SjeNQZ5Bhw7Mvp694ONGwF7ldibfmxfhH/BWjjvmJ9QPmX2x/lQfKt9RlTbOSgc8FB43qq6n1N8SckB7et104rGmT3G+N7vFMgH86SKB9Y/3Azm/+jRv3+YnrH8wrtv+fbD3SfefwBz7DHLtCB/OOm6hD3nW/yZJ42MtAVm9MlSfznucZ3Bxsjvl5Ex3ddxhhM4AoQ+PxPWbZqb/bBSuv4dAjhmbG4zOF8rhvKF210+g/uYbj2et4cDm/gLJ5+PCD+5PE5+PduOnHoA+8OzfCD/jl8lh7dyvkD8X5zt/FNiRzNfhjeYxHo9Mn5o/E3dO8ZDSh+nv5BkYp1hRbPcgjoC9oBxmr2o71ZPPK5V/4DMJybhj+Yr6YcK+/WEo1ofLJ3Yn+hO/dXmoFAml9TS5vuM1IfWceWrdTOQ3Z17V/YOzfLX1/PQ4eZ014v1To9L1N5VPHgqPxyIf5e3b/MTpz8Z19REEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAg1uH1jSUcb1z+HI433nXDLPRFrov5Qkb+2T/zJrxr46J3y8L3zQ6vFrRTvvefuXb1X9vQzvWvzvdGXiMI9XTmxfTH7+Ot6s9B3/f7spxHRJGgYHZk+tT8s/Ce6kBPX/7kmZeTAMg/b3s9JeECyq/GL9M/slc237r+OcV1VQ6JozD/ZPwqqxR/Q3xS2NC/GncX/PYCP868Svpj/wzyYdpP3pdXq/kKr+/FdaG8bvL5VvWpznfGXYLz7bW9YY9V5aG6/rJ1jfHJx7iV8tu3+U91fX/XPuTTWNvVIuF2u237B4sEEx5rG95fjzhd25lDl4xmazv6ry1hpvq4y7bP9l+23e7Vjo+tdCvn0X9t44xc/bf9p229wkR/Cqf/su2T2zA92byo/n372b+qPwHjvwyiJ3O9pB1jfgjovIp6BvKBfV9JAn3+QX51ATDumPxq/I7fPgSm7JXIt8wfbrcbjGsC6g8kjpJxGre7wPpX7Y76V+Ou6rdBfr7dbrPd2bwq+lP/dPNh3k/elVer+Yqu78V1obxu8nX8FX0yFMKB30U/Q5WHat5g8imfbl7N++23+c+L6/tnj9vLMJVQW7tFy5RIpLLcN5um7n/v5PXyfybh+7Y8Py+ExjJYMvrqVCTcZzBMynTJRGp1XLJCWDGsmAEjgSCk+j98bvlIkWDPwAYTQz0trESm/ziVQz7Tn/gba3f4Z34O25me275vKz9Dmfhhcqh9jQGGHS2c1wU9GQ9QfyonYRebfxy/wvoQHljcBX478ebt8LD+xF4s3ybsOBQVOK6L/pDJAzzdnvpQPZldiP5Vu7P+1bgr+22CH8sz0/Oi/pN8x440/8N5ef4A1k2G8npq0Ke34rpgkFk3k+t4qE9tvt3xc8fpsu1tA0ffxzfmu/fW6kfQlc3ulbzhrvvj/o2EFfbbBL7If9Lre0af34ItWrrr722nVc6PIdo6rSkSzvkOftXlo+d9zP3vJQc//9XF2yDjmYFHxhNFQmik6rhr21tDSb9bRfFAMzVhkWCc76Gds0i8+LiRQ5eTLOawgPwb3+tci+jD/I37Ieaf+Tn1f6Jn9xiePZZh/BA5nJ/VzP2nYxT6VVFPJ96h/kwO4Z/mn3GadlaYf8IDjTvHb/PxS/Un9uLz5fpDe9G4rvpDmAfmIgTGI23H+lD9b0W7k/7VuCv6ba8tOUMbOzt1Zk1/5J/EjoxnOq94XbtUJBQ2PUMlXFwXbqZzYt1MrIMpfS7Ml90kHHYfDq1Yi80J2RPoAg+X8oZXJPT7N5KfvfyQmBtW5M/7T3p9j/X5NQxzOEicCHp29EqcB1vUkyYegSeNByvwuT/vpIOcdofBwxd1V5/0uIvN+r2zmoWzzWriGEkXCcMEMA+n/pfme6lImPYeNMmeTrSTq8dDH+Zvjh9i/pmfe/4P9RyW93lHMfMD5VB++qnZzSXzq4qe0ZEG2mwBOYx/ln/gP5/jYU0IDzTusPxhsCh+uf7YXs58mR2pP5C4rvtDN+HEWsniEbcTfby8VLI77l+Ou5rf3qaeRjLluVgkePo/hgrzOeE5Ma9BThUXi4Rpfb+8LiTXzeHTcb5pfd5YJHQHHIkiwelPkefhWt5g8ic+WX5O7VsS8n/Zf9Lre6jP7+HdRUJrbbZRdvQI4CATSay02x7x83avjDucQpnrcjMjRAjbFs9394H+3qSq11vpZ8e5iefztwT/Ic9lByD8X3ekoduQ1TNFApRD+On2CVZ46Fc5PS8UCRU5bysSKA8s7oh8oJ8nxysSkL1of6o/06cTMtw7vMMfzEydVJiIx3BesAp5Q5FQi7uL/g/yGOe5ViS4+psBkFIJPynmsSourKdwfb+8LiTXTUdOSZ8L8/3eIqFbdy7lDbo5RgfxLD8fPXKbsW/znyv9g/bfwfhbDvh4xrBZ6bzG1ATHwjn9boTQVCkSrG4W9rGoxRy/2JOY8ZcyN3Rt2h0jxwcoF8bd7E+AzObDC3Lm1qhnoH8viunP4PW/UCSgnUHIv21n+jB/Y+2Mf+bnjv8zPfurfFx8QwwJDvAzHUXbzXG4eGT0DOYLH9uI5eApjjGO/Arrw3m4kbhj8ovxy/XH/kz6u/rbuQdxXfeHIA+4K3Qcp5l5pYoE1w+JHUtxV/RbqrwXd8WbBKgP8884n/eqsnm5cgqPG1XXU7a+V9eF41vJddNZR2r6FOd7s1ssY8jfKhIYDxfzBjrExHw6+Rlo4Ezou/yH9Q/Gddt/D/Ze5/4TmGPdIpeR8CGs49bokGf9aZI0PtYSkDJdj2J9Oi/hz4aSZHfKyZno6rjDCJ0BQl8dieuqcKr/bBSuf22+zDCOnnBeVH9OD9Mf+ZvTTgdgDxvidiTm/tT5+ciloZ/yk5mvaTeR8RgIWSBwN0dPNl+of14OUfPMP07AQ304DzcYd1x+MX5Z/kz589mf6u+lBxDXZX9w8wCIUyaGtLt2CReL2O6wfzXurvgt44cQweZV1d/xT8eOQf5nqubWTYbCeppc33Prwu3yukme/UvpU98/zOnkGLat56fHyeusEe+fGDPDw/iNOG9g+S6fZF/UfRAUP9/mPxfWd1cfQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRCEvxz/97//6Y/+6M+/8+e3U44gCIIgCH8Dfn3Loj/6oz9/8s9vpxxBEARBEP4G/PqWRX/0R3/+5J/fTjmCIAiCIPwN0NZBEP4FKNIFQRD+OczvZxU+gf8qz9o6CMK/gHykr+0HvKwxieONy5/D8Wa78V335FV1i/lCRv7ZP/PGu2vj+u9sJa8X/UGv3Jy5dvVf29DO9f8ojmFf95WavZj/cDnvsQvnuapPtT2PuwTn22v7cGynccF/WDwy/m+3G4gX572gBLF9QV6qxinQ03yEX32c9EMmn/Vn/LyLNxhfTjyysWv883xrPu++4PWv6OPyBvR8e974siIB7E9+NS+9tF/6AL5Nn18A8we9zthHKdLX9oKTffQ8xZh/bUOKRWZf2xkwqZOetR39x6wPUR932fbZRZdtP9rsuFa6lfPovzawHHL9t/2nbb3CRP8/gzecvRXtRf2HyenbD23LduF+8oo+cTsHVOhX3aGMvP+weKT83243FC8sTp1xoXw/L1XjFPQ33xsPU0p+SOSz/oyfd/HG4ovF443k2yr/NK6dQUn/kj4+b8DuH8gbX1Yk3F7cn3wA0udvwLLtosVFFOnmhKmtnZOZQxpwUnI/+bAZ8v73Tl4v/2cSvm/L8/PCFmUZVoboq9NicJ/BMCnTJbNjqo5LPNWKYVsfMNK0CDn6P9bd5XqR8Lgm2ozhl+FT4Cesnc8U2IWhbK9+dLNpw3JGCpHtMnZJ+kmoT7WdjvTTwfpe28DRIjy5f9w8tOwR8Z/ynwDsWwvYbo3xkolTZ1xI0CCnGqe4//nhdG5c8cNI/tif8fNm3vhm/TbEY2JnkOI/GJdssGEeqOjj8Ab1/ETeeKVIgOv7sNQ/NhI+LWR/4twonl/Yd8s53reYQ2Yriu5P+H6JkEDyHhnX0R+3V/ZvjP9qPh/kt/VxiWFvpMf7NLQPrPZ3wPzB3iGEonwenPXI5+ew1zV/wH5b5MeHH+n2kLS7rtp2o2t/6tDHCywSThKHfXI3n6df3P+e3ub1keldxz/pHUlMFAkh79Vx17a3hi7SO+/AAyUWLab/eSI1bz6o/gDdAynmmIv5CWs3Cr61SCjESec/VI7RGR4BZjcTMc8pfartDthNwuGWs3VgyzGdzObvj/gPA8sDz09NO42XME6d0XH+6eRU45Tr+Wh70Q99+WhejJ838uYWCVOJQ/Ltrca/Oy71Q3TTW9KH8cb0/ETeePUmYV7fJ27dou/m7E+M4jRT2Vsaum9ZV7NXwc+NnfpH+sxgeY+Ny/Qn7cX9G+f/Qj7vyiyjp72tMjfweB9Y7e8Da17ZHjAe/PXI18TaseoPzG+v8cPgRvoQoweb1Jm80vmxCtEd/LQ+gQwxFn7wOUq6i6THOvGTiyRpRvqkx10286ONfjNqEkSDHg4PCpNFwjABtqQ99afzHRLoQyvmJ9FicO0kODPfAJP/OHJO593BIx1Juwyfjn6S1ucPFAkwWRvBYwvq/9X+A/IA4J/ESxSnGG6+mlb4fJz6/aPnelJ+6Oozz4vx817e3Md+5toK5NtOWJp/PC7nGUVQSR/GG9Pzw0VCbh2MSLg9k899t+NvJ4CESU+UB6yqZq/FBuq3KPzgMKXPDJz3vHGh/qS9vH+j/Pv5f8Qw8f5ADcjh+8Bqfx9vKRKwPrXi9sbsWPQH4rdlfoL4/aNFQmtt3vM4mpdW+u4AgEustNse/TFd+l4jOe5Q7R3O3HHAsyoSPB5GBfp7k4r58WisJgAACwlJREFUwU84/2aRULYX9J+cHMBP0i6OnJI+1XYHnywSnEG/wX9i/icVkQK5xHUxX1XjdOqfIOc6D6w/4+fNvHEJg5Ys387iwPC5mwSPZ1gkVPSJeRv98/154/WbhFntte3bsrZ9W9u+rVG+ulIkdF8/T9zZAt0/4fKBIgHlPX9coD9prxcJjP+XigTzT3p4RIiq9vfxTUVC19vcJJT84fqGtoTwcSPLyQ+ZjE1e44NUx/cfX+na0OldJzXLtNXNwl4D2dsbdmVzNMx3FMdxTXQQd3Fce7eUreBZ/kA9A/17UUx/hufZw/HP44QY+wlrP0YkJ3PZM4OqvZj/ZOSA9rRdGM9lfartHP3zuO6JzjnI60XCH/AfPC6MR8q/HcOwWZ2vk694XgLjxnE6b22QF5T9kMhn/Rk/7+ItkIDiEebbKv98XDfa8HFPQZ+Yt+nq5O154xNFwm1trbV9W25r2/fwTIPuT+6Y80DX0N+8gX2L3T6iyzHf6HHssLzHx2X6k/by/o3x/8qhD34sqq/12T6w2t/Hp4qEaD2awe1Y8we2377GD0MU6d2tSDsfL+PXcPDh1uM29ZBn+J2vOcbrjyBZTNcrWJ/ss7B4M3rKyW1Tr447jMCuFx/zHppG4jqHo/rPRuH6s8maR2rZD4zwf+xn2/1rr0qRULJX0n9mXYj8i3bpE0FNn3o7xeSGhzptPT89Kv5ZU96f4o/4TzDbJP83HC+uQwzw5Dv5oRync3+nfKr6IZCfnFc+kNK8OfF1ozV7lG8TfLJxGc+uniV9PN4CP3lT3nihSODr+3KUq7l9Dt6f0DyANhvTfM0HxuMeCaqtrv5sv0TA8h4Zl+pP51XdvyH+L+TzTn5rDaSUxy903X1gvT9G0h8CMR4PfD8Mwe1Y84cb8dsiPwG+7/8xE/5CvPx8kPBPQ/7zR/DeAyaB4Z/gWVuH1/Fv5L03P/3yH8Z3+oMiXXgVpTsHQRgg/xGEvw7aOryIfyHvnWf4/905vgtf6w+KdEEQBEEQCtDWQRD+BSjSBUEQBEEoQFuHFKYf0rzlMXFhxLt4/jY5XwBFuiAIgiAIBWjrIAj/AhTpgiAIgiAUoK2DIPwLUKQLgiAIglCAtg6C8C9AkS4IgiAIQgHH1kF/9Ed//oU/v51yBEEQBEH4G/DrWxb90R/9+ZN/wpywhq8PdHC8cflzMK/mM8Ms7IV03bv8MvLP/pmfp18bF71zE76Hc/jlfP9O8ImEWP+1De1c/3jG3ftvPaPPbw115kW+7naB3EUSM/w7YPa9Zsd5XFefgh1xvLwPzA/ZuN680GfMnxn/vl1mPbE+xbiO/Hmy10W7z0KpVa/wj3Ah3p35Dl8r5YFem3GA6rzYd67Mt5JPruWqcXg3bw/51tM/kTd+fcuiP/qjP3/yTy4vvfCWh4++TM684nJtQ2pHOXdt51q+ZDRb29F/bYk0Xh932fY5rS/bbtck8y7ubord67vbelvbOCNX/23/aVuvMNGfgsj3qF22fXInNi8Gpwvk00GJfwZm36odBz7j9oe0rB1pvLwJzA/ZuA7P2I7E3xj/gV2ieDk/Lca178+zvSgPrt1nsLigdi/KvxLvznwn/qt5gOar6ryK/lbNP++yL/VbJ2/jfMuLhI/mjeJ6mvTbkM/yulnlIadPhjc4cHVZrqK6P6EKkfak/2fWBab/V/lJeX2p56svgDlRaGtXJJijBnJys2/Wze5/7+T18ucTuH1bnp/XdnthkTD0741xn8EwKdMlE6nVcVEGH8SwxRKMBDZnVP+HLy4vFQlM/rbv2wrOLu2Z1mB6Ni/oV0w+45PJucA/83P7XZtcSnYcTWQSN2y/vWDHMV6gUYzB6E0duhlz0tzACeaH2CUTj4xn0D7pSXm+kk8emP15thfjgepD7ELjgszR96vZz6vxzuZrBqR+kskDjg9QO0K/LfpbNf+U7VuNL6eIRfnW07+UN85BUstGeT0l4zI5jj8fyKybVR6YPrX5dtc4HafLtrcNXKEe35i29T+tXbhKe3w9zMMXigTo/xk7pg4LyFi/7Cfp9SWjz7fBFjOrfdxo22n182MMYJ3WFAknD4Mfgn3JvWVtaQfvme3ibZDxzJyjJRJFQmi86rhr21tDi2u3C8ADJRZdpv9Zqc+bS6o/AJPfPWZmjwVuEY1TJQP9isonfFL/LPLv+Dnxq6IdTUx1IUPaX7DjUPmTeFxXE7P4JHWM35u/+RsyIOGH2tGJRz+uc5stxv+VfPL8Yu/72F6cB+IPzC487sycQ/nMz6vxzud7m8T1yOUBahcyL+7nNX+r5p+qfcvx5fgnyrdU/2LeMFO7VCQUNkNDBUvksHi5mc4ZPyzywPS5MF92k3DYa940wxa7RuQ22ZX9CeOH8sbWU8ZPtG46s/geP8mvL6E+X4dhbge54Azh3tErfR4s0h08yO+TT48HIvB5ZMorua/KXOsQJ4v0SY+72GzdO7GppBs8rgay00XCMAHMw6k/na+ziNJzbTdXJktqJp/wSeUU+U+V+MM1YtWOZ3Ds20Db3F60YycLXyMM/tx/ZLZ+rv+zzd8cp4QfZpfMosviGrQjPRn/naB0PplrQGwv7idYH2gXP+5uiH8sn/p5Nd4D/+RFwpWjtd4uYF7cb6v+Vsw/Rfteji/EAysSPD+Bcm7Yf0q4WCRM4zpy/PhNrpvDpyEPf6BIOFvR+bR3Yu3PbkJqfzJ8gZRR9nlQ6P/OJt5fNzG+zU8q60u87nwX3l0ktNZm22VHj9AdLHGJlXbbo99L1WwXjzucEuFrfUQI275MRUKkvzep6/wMWStfJIx7KlokYPmET7dIKPCfvAc8u1204yQnbk/bEcULH8j0NCNEPNDk6O4SzT+4XTLxmOYtSDXVvAHaqVXmXbDrJ718Zhcv7ip50ikS6vFOmzz+k3mA6M/aL+SB2nwv5/OknpGEQc5TOeCWkb0mOQn/iXFhPSV542IeSK6bjpySPhfm++tFQmZ/wvqzdub/nLdMvHT4Nj+50j9o/y6Mv/HonmW0V6jmZLqzane4tW/L0EZP+55Ssz5tdbOwj0Ut5tjEnqCsbTrVnM9sumPk+BDlwrj9XfUZnF6QM3dHPQP9e1FMfwoi/x6w5xDJx41AFYr9ypEP+XT9s8C/ow+070U7PuWEBzRMFLMjixccj9ZS/SFNcAuHilU4rsMPtIvjb5B/N96DVDPIr+cT1779R5lF/dSH24XFBeOfzZfGy4V4R/P1Z1rKA2G+4vPqUPW3Uv6p2bceXx4P5CYB6l/NG8eH+f8bq7Se0nETckB7et0s88D0Kc73ZrdYXdh9tkio7k+cfAjbqT6En6r+3+YnGc1r6/s3wt4T338Cc1wmkEtQ+HDWcVt1yDtY669U798YH2sJyJquZbE+nfc4z7rhZHfKyZnu6rjDCJ0BQh8eieuqf6r/bBSuf2K+NpP+mEcZjXjPYCQ2Z79y5A9khP7J+1P+iRxm35odaXdPTNqOyXgxn5hvPAjvQ3vsj/3QG9eZWGRHUOG7/HfPqiI9XXXqce2sNcc3lqktRQ+0C40Lh//MfCP+o3gE83XyFeMt1scKyc2Lxdhr863kk6KeqTwPG40YX/9q3qgVCaX1NJmv0nny4rpZzZ/MjkmGJvc5hm3r+elx8jprxPunRs3tT+g+Ic6TbD1l+Twk7tv85ML6UpmuIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAjCfxb/DwLd9njqvnRlAAAAAElFTkSuQmCC" alt="" />
可以看到这里为discard_passdown,表示dm会将DISCARD传给底层设备(loopback device),queue_if_no_space表示如果thin pool没有空闲空间后,IO请求会被排队。另外,error_if_no_space表示如果thin pool没有空闲空间后,直接报错。
[root@yy3 ~]# ls -slh /var/lib/docker/devicemapper/devicemapper/data
2.5G -rw-------. 1 root root 100G Nov 12 06:14 /var/lib/docker/devicemapper/devicemapper/data
[root@yy3 ~]# docker rmi dbyin/httpd
[root@yy3 ~]# ls -slh /var/lib/docker/devicemapper/devicemapper/data
2.0G -rw-------. 1 root root 100G Nov 12 06:15 /var/lib/docker/devicemapper/devicemapper/data
可以看到删除image前后,稀疏文件大小的变化。
dm.blkdiscard
docker还提供这个参数,默认值为true,即删除image后,会调用DISCARD,真正释放HOST上空间。
func (devices *DeviceSet) deleteDevice(info *DevInfo) error {
if devices.doBlkDiscard {
// This is a workaround for the kernel not discarding block so
// on the thin pool when we remove a thinp device, so we do it
// manually
if err := devices.activateDeviceIfNeeded(info); err == nil {
if err := BlockDeviceDiscard(info.DevName()); err != nil {
log.Debugf("Error discarding block on device: %s (ignoring)", err)
}
}
}
...
}
func BlockDeviceDiscard(path string) error {
...
if err := ioctlBlkDiscard(file.Fd(), 0, size); err != nil {
return err
}
...
}
func ioctlBlkDiscard(fd uintptr, offset, length uint64) error {
var r [2]uint64
r[0] = offset
r[1] = length
if _, _, err := syscall.Syscall(syscall.SYS_IOCTL, fd, BlkDiscard, uintptr(unsafe.Pointer(&r[0]))); err != 0 {
return err
}
return nil
}
主要参考
https://www.kernel.org/doc/Documentation/device-mapper/thin-provisioning.txt https://github.com/docker/docker/tree/master/daemon/graphdriver/devmapper
Docker内部存储结构(devicemapper)解析(续)的更多相关文章
- [转]关于docker包存储结构说明
原文:http://blog.csdn.net/w412692660/article/details/49005631 前段时间与同事交流docker的安装包层次结构,并沟通相关每个文件的作用,但是一 ...
- Redis-基本数据类型与内部存储结构
1-概览 Redis是典型的Key-Value类型数据库,Key为字符类型,Value的类型常用的为五种类型:String.Hash .List . Set . Ordered Set 2- Redi ...
- docker的存储结构,和以前有了很大不同
在网上学习这一块知识点时,有一个URL讲得很详细, docker 镜像与容器存储目录结构精讲 http://blog.csdn.net/wanglei_storage/article/details/ ...
- 【SQL SERVER重新认识】数据内部存储结构简单探索
数据库经常需要打交道,但是从来没想过数据库内部是如何存储数据. 今天探索一下数据库内部如何存储数据,从下面几个方面探索 数据库内部如何存储数据 索引数据如何存储 操作数据对存储影响 总结 数据库内部如 ...
- docker 内部组件结构 -- docker daemon, container,runC
Docker, Containerd, RunC : 从 Docker 1.11 开始, docker 容器运行已经不是简单地通过 Docker Daemon 来启动, 而是集成了Container, ...
- SQLServer2012 表IAM存储结构探究
SQLServer2012 表IAM存储结构探究 Author:zfive5(zidong) Email: zfive5@163.com 引子 国庆节期间,一直在翻阅<程序猿的自我修养-链接.装 ...
- 深入PHP变量存储结构 标签: PHP存储
1.深入PHP变量存储结构 标签: PHP存储 分类: 编程语言(10) 首先声明,我并没有去读PHP的源码,只是对于PHP的有时候诡异的表现感兴趣,找了一下开发人员laruence的博客结合PH ...
- PHP变量存储结构
首先声明,我并没有去读PHP的源码,只是对于PHP的有时候诡异的表现感兴趣,找了一下开发人员laruence的博客结合PHP提供的函数debug_zval_dump刺探得到了本博客所阐述的工作机理.如 ...
- docker存储结构解析
由于aufs并未并入内核,故而目前只有Ubuntu系统上能够使用aufs作为docker的存储引擎,而其他系统上使用lvm thin provisioning(overlayfs是一个和aufs类似的 ...
随机推荐
- 天猫装修-全屏代码,和去掉10px
淘宝装修,不支持float,和position属性 全屏代码 <div style="height:685px;"> <div style="paddi ...
- iOS静态库小结--(yoowei)
准备知识: 1.什么是库? 库是程序代码的集合,是共享程序代码的一种方式 2.根据源代码的公开情况,库可以分为2种类型 a.开源库 公开源代码,能看到具体实现 ,比如SDWebImage.AFNetw ...
- CentOS7挂载分区教程
http://www.centoscn.com/CentOS/config/2014/1016/3955.html
- 新浪微博客户端(47)-在TextView中插入表情
DJEmotionPageView.m // 发送点击广播(和android类似,区别在于android的广播是只要有上下文对象context,就可以发送) // iOS中的通知发送和接收都是通过NS ...
- strtol,strtoll,strtoul, strtoull字符串转化成数字
今天看kafka,有一个参数选项中有: 'S' seq=strtoull(optarg,NULL,10); do_seq=1; 之后查找了下 strtoull 函数的功能,了解如下: ---- ...
- bestcoder Round #7 前三题题解
BestCoder Round #7 Start Time : 2014-08-31 19:00:00 End Time : 2014-08-31 21:00:00Contest Type : ...
- HighCharts选项和参数详细配置查询表
概述:作为一款出色的交互图表制作工具,HighCharts有着全面的选项.参数等配置信息.为了帮助大家进一步掌握HighCharts,今天我们为大家整理了HighCharts的所有配置信息和说明,将其 ...
- PostgreSQL表空间、数据库、模式、表、用户/角色之间的关系
看PostgreSQL9的官方文档,我越看越迷糊,这表空间,数据库,模式,表,用户,角色之间的关系怎么在PostgreSQL里这么混乱呢?经过中午的一个小实验,我逐渐理清了个中来龙去脉.下面我来还原我 ...
- 密码学初级教程(八)SSL/TLS-为了更安全的通信
SSL/TLS是世界上应用最广泛的密码通信方法.用到对称密码.消息认证码.公钥密码.数字签名.伪随机数生成器等密码技术. 密码套件 SSL/TLS提供了一种密码通信的框架,SSL/TLS中使用的对称密 ...
- 网络编程1-TCP编程(socket)
1 如何实现网络中的主机相互通信 一定的规则,有两套参考模型 (1)osi参考模型,过于理想化,未能在互联网上推行 osi有七层 (2)tcp/ip参考模型,有四层,各层之间通过不同的网络协议传输数据 ...