Docker内部存储结构(devicemapper)解析(续)
dm.fs
参数dm.fs可以指定容器的rootfs的文件系统,但只支持ext4/xfs:
func NewDeviceSet(root string, doInit bool, options []string) (*DeviceSet, error) {
...
case "dm.fs":
if val != "ext4" && val != "xfs" {
return nil, fmt.Errorf("Unsupported filesystem %s\n", val)
}
devices.filesystem = va
这是为什么呢?下面是docker dm的维护者Alexander Larsson的描述:
Additionally we ensure that DISCARD support is enabled in the filesystem so that any files removed in the conttainer filters down to the loopback file making it sparse again.
参考这里
一句话,因为ext4/xfs支持DISCARD。这样,如果容器中删除了文件,空间就会马上还给Thin pool,因为Thin provisioning是支持DISCARD操作的。但是,默认情况下Thin pool是底层是稀疏文件/var/lib/docker/devicemapper/devicemapper/data,所以,只有Host的文件系统支持DISCARD,才能保证稀疏文件空间释放。
Host为ext3
# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
dbyin/tlinux1.2 latest 8297f05d459f 41 hours ago 399.6 MB
dbyin/httpd latest 93e711fab1c1 7 weeks ago 412.7 MB
centos latest 61038e6e3195 3 months ago 236.4 MB
我们可以查看稀疏文件的真正大小:
# ls -lsh /var/lib/docker/devicemapper/devicemapper/data
1.6G -rw------- 1 root root 200G Nov 12 11:52 /var/lib/docker/devicemapper/devicemapper/data
# dmsetup status
yy_pool: 0 409600 thin-pool 0 13/65536 0/3200 - rw no_discard_passdown
docker-8:1-696417-base: 0 41943040 thin 928768 41943039
docker-8:1-696417-pool: 0 419430400 thin-pool 73 633/524288 26115/3276800 - rw no_discard_passdown
我们删除一个image
# docker rmi dbyin/httpd
可以看到稀疏文件并没有变小:
# ls -lsh /var/lib/docker/devicemapper/devicemapper/data
1.6G -rw------- 1 root root 200G Nov 12 11:52 /var/lib/docker/devicemapper/devicemapper/data
# dmsetup status
docker-8:1-696417-base: 0 41943040 thin 928768 41943039
docker-8:1-696417-pool: 0 419430400 thin-pool 73 490/524288 18758/3276800 - rw no_discard_passdown
no_discard_passdown表示dm层不会将DISCARD传给底层的设备(loopback device),只删除映射关系。
Host为ext4
[root@yy3 ~]# dmsetup status
docker-253:1-8790943-pool: 0 209715200 thin-pool 339 876/524288 32432/1638400 - rw discard_passdown queue_if_no_space
aaarticlea/png;base64,iVBORw0KGgoAAAANSUhEUgAABAwAAACtCAIAAAC7jnp5AAAgAElEQVR4nO1d27XbPK90Wf9SQapH1bAZlXIe9nmwLYHkDC6yne18mVl5SGgaBAcXEqQc3W6CIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAh/K9b2c8e+/bYq/2ks297W29r2bfltVQRBEARBEIS/B8u2P3brf2wbuWz7c7i1/bR1/nRsu4q1vVCHbPvbaHmS/DPM7Gh+bZhHyQWELNve1nulUBao+k0QBEEQBOGfxbLtn6sP4O60G/Hjh9xre2Gz+x5u1nZsuG1RtLZjW7+2sXwoA9txbT9XioTbi7wxXFHkN8b9LT0FQRAEQRC+BfPm8n68vW/L8/z73DAdJ9/Dnha0L6at/8KzLhjPv+0XTpWWQ53uX8GUzkG7zS7T/3j66WffLB/3v3fyhqlNwmfeujqIbNgv7eNHCYMdz0kln+yaebu37Ht7tlF/2Pf9kA/5If5wb77//S7+PovHNc5mDJOo1oA+3A8tQZO/ZfX0eBAEQRAEQfh7QU+gjx3e2u57o/MhoVt3Os7ab+RE9r5pfpypw5uEobH/Z7SdNgf0/WMzTE97jL9su93+mSLB7oz7LaXdFCLexiIBbCJzx/ZW+PRQFrPjvi252xrC2zH5J01HPWNHXJvlmfLDbpbsxr0vGq1hgjKK6cPGva2rrXZsj5KezriCIAiCIAh/K5zN5div3zg9urD22+3mFgmPYTNFgpVzfjQelD82lMN3j0051dPboB+/1+gEwnGJ5mPbXCQMJQjFYxPPfkMC7XhvTBUJDm/Pmue4RzmNMZPg8lMtEmzvxGUL0oePO6h6uUhwxhUEQRAEQfhb8VtFwrlFm/fo8+jPlsxG8b1FQmvD7wXcU39cJJjv92oMFxevYLLj/LSRO8yVIqEf7UGLz2epSBg1KmzA4xste3MyCi8WCXRcQRAEQRCEvxXZImHctJ17QdbefWSe6Yl/Ko1Hvz8nnzgSHx47j/Tsdov9b4iP647hUXaqBtRw2Zp5HMX+cPmNG0p2k9DW5M9wCW+8SOgGNN92+An8oa+Z7v8w3w1mwfTB49ornekyp6gnHVcQBEEQBOFvRXgC3Z2Cl9o7YcP5+eOf+Elz8gxS9lmO7umPNjxeTx4vAe3Hf4FqXuoAfmPd/5w1kn+0Oj+ovQRYJDyf7LrK28/Pfij3+J3CcvyEF5EwzXf4iPrDk+CtHTxv+96a/Ql0UB86+sBxjQUeA6HfNGf0fKMZBUEQBEEQvgMf/S9QnUGdPVX+ckM4wIuEj/xnpn8Av+GYgiAIgiAIwu12+5WXqWF0/yEl+UDntBD8ZWp/Lcwp/19Z4QiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAge1vbCf0h/vJn4czCvujXDLPBFyf0nuUn5L3fjvSvjopf4whf7Dq9itlO+95+5dvVf29DO9XcxySF2cfSPeB79EPJD29G81vbjOCeTH32W7evKAHz68tHL8rB85oeVOfVyavx0dnT84ez8mj5zu2/3G3n7SpUfJqeaB6qoyojzBsiTwC6M/3f5OZOF83+9Heof+ecFnP7v+yHT05Wb5Tkxr1KcFlBbT4v2uhEeyusmXwer+lTnO+Muwfn22l72TD7f6vrlr0f5der88O/0k+p835H7fxMvvZf3oy/HNeGxNkvvsu2I67WdiS/1Pum1Hf3XljBffdxl22e/WLbd7mmOj610K+fRf37ztKv/tv+0rVeY6O9jlsPswvTP8Xz6IeOHtbN5MRegcoi9KPp5sfnO+5jRLgRJux+fUj909QnnVeTntCP1h8enWR6oPqTdC/1l2+H2vcQPkVPNA2UU9WT+4+dJYBfCsxNHWI6v6sQPyzPVdu4nnn9ehfV/LJLqyUWW8kluXtk4LaC4nlbtxXgor5t8HX9Fn4ynw4EvLcsVsAEIn8H6O/EZtE9y7sjnh2/zk+q6fGV9+X2YCqmtXZFgSidSce6bTSP3v3fyevk/k/B9W56fF0JjGSwcfXVKdvcZDJMyXTKRWh0X7kh6MSwpg5HAYk/1f/ji8mqRgOWQuTD9qZ7EDxk/lDenSFjBGYQnBxbM5tB42Pl1lCP9hyKc2gXFixEwJl8m3/aBvHX9ybievSA/PJ8cGCyEeSA8M31YO7O7Pfu3U/btNec9Jmeg5LJfVf2N6Ynoc/UkdmE8O3ms5ueMn15PmGcy7Zk8P/sn5BO3E/9nfkj1v2R3Pz93H9TjFK6bDOX11CBjr0zey6ybyXU81Kc23+5YuuN02fa2gSPx4xvznXxrlaNpvtkN1y+wP2H5xF2nUvkhgS/yk/S6nNHn2zAePtprR1b9/BgDWKc1RcLJw7A+duvB857m/vfCme2wqIBYswPMK3eiSAiNVx13bXtraJHoVgM80ExNWCTMJ/HOIp27OMFyehVC/Yme1A85P6SdzKt7HM4eazA5zF7ramXCE4suBEg749OLl5E+d1w7tSHoUH82Lo0LzI9jRyNk2KtAvyI8M31YO7c7CXPCj5P3eLqo5AHqVzV/8/QcafL0ZHbheRLHUdnPaZ7sVJhbk+1xnp9OWCCfpN1bT6kfQv2Ldo/z81iC1OL07HmtSChshnL28vIe059s/oJ1MKXPhfmym4TDT+ZNOWyxPhCfTLP5Uj7d/Um6SKjmhwS+yU/y63Koz9dhmNtB7kTcs6NX+jxYpDs24JeeJ0F7+LySe77MtQ7fvLr6pMdd7CrRO7GpsNusJo6ddJEwTICFtN1jwPkGcrhdxr0f1JP54Y3zE/HWz+s2LMunVkQOtVfv0rjs2rurU9iO+fTjZSbKHdfSMJ8/df35uF7xOfPj2HFm3uGB83yhSMB2J+phfvwjn2hPkMoDzK9q/hYdTbk3CdPOEdjFWYxhHFX9nPJj5sz2Vpn2cDOXO+oj7dz/PT+E+hfzTJjnhxqhGqdVXCwSKvby815y3Rw+HX0orc8biwRqJlIkOP0D9PPFfPrrbL5IKOWHEN/mJ5V1OVyvvwzvLhJaa7PtsqNHAAdjSGKl3faIn2t/ZdzutMwcpHUcIELYI5NTkRDp702qcu01yfHtMt/dAz35osX4CXk7PniqMY4ARjb/IPbq9tuc0YofTmkeSpw1fHXcrp2Oy/wK85PYfOQeVqM8M31YO7M7U69HzE9RTtWvqv72Af8Zd82M/ygec37O8uSN55lSe5gn5zMNpOaFIoH6IdLzhTxDevP7oGKc5nBhPb1mr0evpP4V/y/pc2G+v18kJNw7iOt0kVDMDx6+zU+u9A/avwvjbzx+cFKzTtPdkfe/VX1sFKbfkxD6Kj5tdbOwj0Ut5rhmfKBk3Kn+zHcUR3lLLgZeHXezP90xmwYvyJm7o56B/r0opn+MadOAn5Ji+mM9qR8yfli7Y5dTT/OBwz+w13T0S4xTaZ/49M8w/JNgE4yO/4P+dFzuV8ifqR0PaXQlGPZNjGemD2lndh9HQeD89IziPVYlD7D51v3N0/MG/Cf2k8FkhOd4s5L2c5gnWZ6ptgd5EpyOYT5Ju7OeYj/Eer6SZ1InRNfitPC4UXU9vWgv1p5eN9l6UdanON+b3WKZwPt0kZBZ9wtxnS4SqvmB4dv8hPUPxnXbvw/2vuf+E5hj3SLXnfDhrOOW+JBn922TpPGxloCsXhmqT+c9zrN0ONmdcnKmuzruMEJngNCHR+K6Uzeq/2wUrr+HUY5rF6g/1ZP5IeWH8jbPa20/Pz/mUeeOBod/YC8z44dAs9qX1ER8Dvr/YPGdyah84odcHzgutRfhh9vxwR5LlwMPlGeuz9zu2J06bo4foIxDRCIPsPmW/Y3oyfOGo2fkn8zfuKOEfo75YXmm2u7p//iel68S6yD0f+qHXM8Ldmc843ldidNKkVBaTy/Yy+Xh4rpJnv1L6VPfPzhu3tbz0+PkddaI9/eA1/3MOkLc3IpK5ZlcfgD4Nj+5sC67+giCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAgQ6/DaxRKONy5/Dscb77phFvZCTPNJblJn/8yb8K6N67+j1ogZXi04vyJ45trVf21DO9d/xsMziHGxXQJpZNh38DPz4Olf4YHJd+Do2c2vf91pyW9PIVifUf4l/5lAeKPzjXlG+Yf5ytQexkvavD4Az8z/HT4TrzeN33rK/KQaXxf0+Twcf65Kyc2r6s9cfi2OOKB9E/lkxDxXP59fyj+unBwP1Th6z7rJ81JVn9ftfpfgfHttL++xMvuWDJ9czlvWdwff5idsPU2vXX8Z1vZCXtj2DxYJJjzWZul13iD/nMuS0Wxt9tXisfnq4y7bPvvFsu02dx8fW+lWzqP/2sYZufpv+0/beoWJ/gyMQmoXAjbfG+FnmNehBOOH8UBdoMhD1U+onoNQ+1r7kt8+RpnsS+Rf8x8Awhudb4rnLv9gfyDtQbw8pL+almeeqf87fC7bjtIsmy/sT/2ExEtVvtf/84j8OYfSvKr+zORX44iA2TeXT6g+Rg7N5xfyTzhuyEM5jorrAs17fB1/RZ+M3eHA1eWoDDYAXTeL+byYf6r5+dv8JLmeZvLzF8NUSG3tFmlTOpFKcd+s29z/3snr5c8nIvu2PD8vhMYyWDj66pTs7jMYJmW6ZCK1Oi5fsewRFHQaMNLklI7+D19cXi4S1uDOYBAJ/YTON8EPK2JtF8YD1Z/zAPX3eYZxAfV04s52wSc308kxtq8jP+8/5rB03CFF/jPOF/ZnehJ/oO1kXM/fOJ/QjpjnXjVDG+bTnj13Q5N50f79uHaRw/FSlc95dvgptL/gz11/4p/leRX9mcm/kB/CvJFcF8J8a/0hmc/DdZOhvJ7244ZxVF0XkLisbrm4rsy3O5buOF22vW3gSPz4xnxn3lrlaJoWRbH+mXxezT+Z/Q/DF/nJ1J+Nm9Hn22CLme66f9tp9dPf0MIi4eRhXJfn/H5vWVs6jfTMdvE2yHimzdESiSIhNF513LXtraGk3K1yeKCZmnCTN5wcz3I8/QG6x8ls+d+r0M0K+wmZL+PH+F7mCJDxQPUnPDD9mXwnLpCeNO46irqiGscRsa8vv+A/t3U1sYxWLeY/KX/jenJ/IHFExiX+xvhkdqRxZAfq/J/nE5henHnRdITzDI6XqnzSn8dFrb3oz3wdYf5ZnFfZn4n8an5w8wZbv4A+VA7xBzefF9ZNhvJ62n81llNcF5C488vuOpjS58J82U2CzTnz1nNuOdRObbLpehfpn9w/lPNPtP+h+CY/cW8SBh4Cfb4Ow9wOcifinh290ufBIt3BA3+afHo8oIHP/VFeyX1V5lqHOFmkT3rcxWbl3olNJd3gMRKQnd7kDRPAPJz60/kOy8io1WgX7idkvpQf60Q7eARh0MQpEjz9Zx7oMSrxkyjBDXt4HHeTOmaLiu2C7RvKzxcJ/dBse5x+fKPvz/Vk/uD4CR4X+hvl07G7G0dTXqoWCd68oj0HutYf46Uqn/Rn/BTbq/7srCOOf1Z5Nlqn/LlQJFR5G9WZ8/Z8NsK+Dv0hkw9feRziYpFQiaPSuoDEgU9Hu6f1eWORQJcJUiQ4/QOY+Yb6p/L5s2ch/0T7H4xv8xPUn43r6/N9eHeR0FqbbZcdPcJ00IIlVtptj/h3Dq+M251+mYOxjgNESOJ2MKe/H+XxIxyjp5jYnu2S22SbfzB+Qj3nc2PIg6M/ku9tFpH8DIG8PKaLSqxPJ8Qc0NeLBDIvY1nuQbFdcH+/SED+EPrJ5A/Q317YtE0swLzkxSPbvLJ5xXuO6/mnqM+bioQyz1yO459Vnl31gD/jIqGUH5LPG4T6lOUU82EZF9bTchy5eibXTUdOSZ8L8/39IsHMN9Q/lc95++X9z4xv85Mr/YP278L4G48fnEQ6Z+qsamqCx1fA70kIfRWftrpZ2MeiFnM8Yk9K1qFyuaFrU1PGZn5Ac2Hcbe/q9uO7XpAzd0c9A/17UUx/hntgnqLsyZ9/Cv7sdf+CM1/IzyAzPgcgPDD9GQ/Uz4l8Jy6Qnk7cYf+JT/UGZon8oynlP9OVguXT85/JLqS/pyfzB89P0A4e+hvjM7DjNATzfy8eyaafzgv1d/Pb2W6bS/JZf8bPa+0IE8+gP/fP6ryq/kzlF/MDaw/sC07xIr9N58PauslQXE+vxJH5brwuPLsOXuTk/5o+xfne7BarW14+WyRQP/f1z+bzDpn8U9X/2/wko3lBny+Fvd+9/wTmuEwg17jw4azjNumQd7DWXxUfuRFeHhNM181Yn857nGfdcLI75eRMd3XcYYTOAKEPj8T1myem/2wUrj/A2n5+fswjhd3DrAm7ZOeL+PHowTE+80D1d3lgDyEynp2HFoGeLO64/6A4GoWdEyPyq/5jLPwgsA95oCeeL+tPebiReHHa4bjUgRifjh1HnpP+3xfn9AtoXrw/8RMvXkryKc9xXNTbY38m/Zl/vjSvhD878qv5Ic6T1fiK8qebD8vrJkNhPa3HUXVdSOW9vuCs6VPfP8xueAzb1vPT4+R11oj39xCvd9V19oX8Exhy1qNql4/6idOfjVuZriAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgnDH2r/utIbjjcufw/HGu26Yhb3I0nySm9TZP/MmvGvj+u8U5m9mnV+JOXPt6r+2oZ3rD+How0Dfiwv1uSin0D+Sw/x/bK/x7PoDfP3irOdDA4fyypyv8OB9Z+yP45S31+KOg8kPdX/D2AgX7Fh+Hec0xhvjlPHJwOxYzVfXx83FV21deGueTMZ7pn0mp9pe1aeKalxX88N71k1u36o+r+exuwTn22t7wx6ruj+p5tVq/6q/fZuf+HH3+ny/DWu7WiTcbrdt/2CRYMJjbZZe+Mbwe6fnXJaMZms7+q8tYb76uMu2z36xbLvdQxwfW+lWzqP/2sYZufpv+0/beoWJ/gxMH4pen3m/OOpzSc4MxidrHwfD/m/aizx7frhs+zQc9wc6dehXtD3FZ8cDkw/7szil8VuNO6pBSn4Yj+/CBTtCfwiGmPR/V5xSPnNyxrhL56vquNX4Kq8Lb8yTlXhndmF8VtvL+lRRjOtyfiD8lNdNvo6/ok+GNzhw0d3KqO5PGA/MT6r9q/72bX6SzG/np8X9zHfAVEht7TYHpnQCldO9erJudv97J6+XPx837tvy/LwQGstg4eir02Jwn8EwKdMlE6nVcckOwIphixYYaXJKR/+HLy4vFQmOPtAfRhXM3LE+l+T4/Yfgxzwz/yftZZ4N7Lj2DM+GBtNz2/dtRWcobGeZ8LeOT4cHJp/kDTIWbvf5BHmG5BMm35sv+XptXNJetSPzB/tBuOMcQNgcKyUWX6QP5ifIn5V8hcfN2T2ML9bf45meKCA/MbDfrMZ7YBe46a+0V/UxX0otG+X11CCTHzJ+m1k3k+t4qE9tvt2xdMfpsu1tA0fixzembf1PaxePpjP7k6E/zKts35Lpn5GT0eeX/QQV53DcjD7fhuHw6EwW206rnx9jAOu0pkg4eRj3Y5aU5z3N/e9pB++Z7eJtWprwCpEoEkLjVcdd294auijrVic80ExNuOgap3xoNxcJVP9o4sPeA/qD8RnrEkyfqhzuh4xP2M78n8ZFmWc79GBf7GZY/+5xvv74BfsVb0d80vkSOby/nUh4LcP5ZPb18gkcl/gP46c6LtenaMcb8Yd1Nbm0OyEj+cRI67b2VA7mx3xzrMwRP0H+TOcrNq5r91J8kf6MH5InnXXw/OLAZSXeA7u8XCSU9Tk/ulQkFDZDqfwQ+21u3Uysg8l8VZ4vu0k4fGneNMMWu7bmNtml/YntEa6nF/qHcvgsvsdP3JuEYd2J8u23YZjbQe5E3LOjV/o8WKQ7+IlfkLnGAw743B/lldxXZa51iJNF+qTHXezuoHdiU0k3mMmB7PSiO0yALSF27+3PN18Kn8bfz0dwiD5VObw/4xO2M/9n7a/xPPoh3hRi/fvt5GkF5lfU3yCffL5YDu9vx2BrYH8sTeIuur7I5wfoh4Sf4riOPkU79mrQoe2pG7MvkMzlMH4InzTu3lwkDOOGdn/IrF22dP05P90XjjRWOXb19OH537PL54qEaD1K4mKRkM0PR1/CT3rdHD6FuSOjzxuLhO4gIFEkOP19pPYn5uPhVNXzk3z/a/72bX6C+rNxfX2+D+8uElprs+2yo0fABzaTxEq77dGvSTXbxeMOp1/4mh4Rwh6ZnK+PA/29SRWuvcZHD1JfRN2m9FaRk9usmH/g9itFwvt4RmsJ039+vPksEpBfsXasj18kzHLcIoHFKWwnfF4J5FJ+IPxUxy37IbPj4x/g7Lbbx3ZFgmvf6REhLMeZDuLTma8bF5V8VRrX0T+xVzP5pMZPrA+QUYl3Mm7U8+UiIadPjAvraSU/TKNNE0mum46ckj4X5vvrRUJmf3KjPFA/KfW/4G/f5idX+gft34XxNx7wGt1Opj+t6X47cmwgpt+TEPoqPm11s7CPRS3mmM6e2K1tqmrRGWQ7HTe+BLowbv+MyBmcXpAzd0c9A/17UUz/AKB6pP5g+4T6VOWw/oxP0s78n8bFBZ6pH4YHwN3mstPBGgz6ldN+wOjP54vl0P4sTlk745PZl+UTKh/Pl83ryriw/YId8QFe1znWHw0fyLFz6XyV2BHHqR8X6XyVG/dENb5wf84Py5NBvmLHB+l4t0jxeaG9rE/hcaPqelrND8N3c+fDgAfHvjV9ivO92S1WF0afLRKq+xPGAxu32r+q/7f5SUbzgj5fCvucxP0nMMdlArl+hQ9nHbdVh7xuXRy/MT7WEpA1XTdjfTrvKT9becrJme7quMMInQFCHx6J6zdzTP/ZKFx/DzCm+EO7V/XJySEPCbIvkHbm/6y9zDP2B+7QQM+1/fz8mEdG53NWrBFqT9HTzTch59mfxWkyflP2RfnEke/5IeatMK7XXrMjm4CZ2eOL0HeHmc1xSuVAfpL2Stixlq/S42Keq/EFn0mYeGZ5kvEA+S87aKLZqlRtvxQwlSKhtJ5eyA/u+nJx3ewLxZo+9f3DHL7HsG09Pz1OXmeNeP/UqPH+xOMBGaDa32tH+DY/cfq/Y7qCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAjCHWv/mtUajjcufw7HG++6YRb64svFfCEj/+yfeRPetXH9d9f270jvX+S3DDJmrl391za0c/0hHH3YuExPOmeoZx1Vu3sgesLXJmL/tN/gb2j2J/2ITBJcM/9+/6r/VO0by5/sAnmgPkLg9s/6VRh3XJ8uf0I51+zC/Oqqn4M87/Cfz+kOP/l4uZBnSvJL8O2Fx62vC7gdyfH0KebzW8av3pWHc3Kq+rxn3eS8VfV5nbe7BH8heHWPFfgJyJPMb3G8c/ksP1Tj9Nv8xF8Hk/r8RVjbC/uqbf9gkWDCY22W3vnN2ken51yWjGZrs6+yj81XH3fZ9tkvlm23ezuzwnVTPOQ8+q9tnJGr/7b/tK1XmOjPwPRh41I9CQ9UzyLKduegei7bjra52D9vtxubF5LDQKdC+Gf9y/5TtC9r9+yCeGBxQdHrOYye96ukn1NjnEUCllO1C/Or1/y8z/MO/yh+mUjKTyVeqnmmKr8Kj1oYv8V1gdqRyKH6FPM55ae6DroD5OWU9SH+Vl43ub1e0SfDGxy4aMY63AHmPMn8k+ZnJp/Yqxqn3+YnyfW0uI58G0yF1NZu8TClE6kI983mrPvfO3m9/J9J+L4tz88LobEMFo6+Oi2i9xkMkzJdMpFaHZfsCK0YttiDkcBmgur/8MXlpSKB6RPwBoON7IyxnszfzCEei7SBT+a3uJ3oac8OB5e249oZwHlROWRe275vKzhDYfyz/uhrnpyyff32p5huk4R4oHFB8snIrvmI+r9rxNu0kGD5PH9COVW7WLCQTfk50TPwZ7j4If9k/FyIlwOZPHMtHn1++nWKxR0Z98p6hNqdIgH6j+MckZ/br7rxDtZNhvJ6WtTHifcDmXUzaa9Qn9p8u2PpjtNl29sGjsSPb8x3a61Vjqa5bmz9tV+1m2man8lmOrRXdUvyRX6CDsvguBl9vg1DUdddl7Pq58cYwDqtKRL6RdGu75aU5z3N/e9p/5iOwUCs2QHm5JgoEkLjVcdd294a2hx0qy4eaKYm3OQZp3xo52ySSgdGs5s7vKFNBuSB6Un9bV2Nj5ELvX6nCOVw+cxeoXt0/kn5Z3LIvLrH+cwxDuOf9Tc6pvynbF+vncQj5hPHBc0nJkfZFMT49/LS0TqVepN8mj+ZnKpdhs/Glqyfu3o6/gzil8Qd4T+Qj+f1/FYyz1Tks3hn/uDZC457fT2KHzfy9KH9Iz8f+XHi/YUiobAZyunj+NvRmlg3E+tgSp8L82U3CYd952IStthcF59Mk/l66xT2W7JvYXyG9irvmL/JT9ybhG7cmIdvwzC3g9yJuGdHz5APFqnHAL8BJ2lO/1vEK7nny1zrECeL9EmPu9js3juxqcjbrCauYAubvE57tqXrnnp15zvtnWqbSMYD0ZP7W+9aJLuHcqh8ai84zf5r+Nhy5j/eZHRFArwHcooE797od4qE48MEDzAu3HxyfrifV+aYfz8vPaXNNd0on+VPLqdqFzs2y3uxn/t6looEHneQ/0A+n1chzxTk84KE+INnr3AvWFyP6Dpl5AT+M/QP/Rzkq8ub+0GFS3Iq+lB/e36aWTeHT0d7pfV5Y5FgC+9MkeD0D2Dmm/GkgvAAACAASURBVNknDI/N+PuWQf7Nt1d1v/xtfkIfqwbj+vp8H95dJLTWZttlR49gi2xHYqXd9kg8V/fCuMMpzhHsHQeIEPbI5LTJi/T3JlUo4ucbaG9cWCSQg2EkhBPb30KQmR1fv1QkMD3JGhD4J9x1gjNFPK/5Mf7jKBvyz/of4yT9p2rfoN183Ck38UniIumoqFvKr06Et8uTlngig5yqXW61vPfpIqEad558f16FPFOQX84Dnr3ivWB1uLg98J+xv+fnkP/X1sGX5LygD5hmct105JT0uTDf3y8S4jxJ+4f7Fip/ak/kt1HBr/KTK/2D9u/C+BuPH5yMrHP8dN7RHXbu2zK08dORW82nrW4W9rGoxRyz2BOX6Rcx6NrUlMOZHx5dGHezP/UxxYMX5MzdUc9A/14U0z8A2ul64+KyB/DAhsD+Zpfn/lCB8c/81vVnoife1Eb/yUymSHDn1T33bB4zgPzT/ueEcv5Tty9s9+IxPIA30nK3grH/B3LcZb4/gMP5k8mp2sXJexU/d/XMFwncPwedu/YL8VLKMxX5uTxwwrMXrglr64LTzuRAfVh/59aCPd3K473wuFF1Pb2kz422p9dNh7eaPsX53uwWyxj+00VCvO5P+wTon2zczL7C8hOvm9N3v8pPWP9gXLf9+2Dvme4/gTkuE8glJXw463g645Bn/WaSND7WEpA1XZtifTLPzJ0fjWOecnKmuzruMEJngNCHR+K6026q/2wUrr8HGFNwXEdPwkNCz/4HXofo+28HbHqH82IPFfKHDYGe2BFd/4TzYl+A81rb+c+JHsC/0/+C/5Tsy+Vju3DiaFygfOKHUexXw3eAnzP5LH8iOWW7eH5V9HOiJxuA2ZHGHeHnWrzk88wF+Zk8cP/EsZczQHFdSNixC65EHuD+AP43EYcHHAPZjU1hPb2gjxvvF9fNvgCu8lPdP8zLyzHs4XxPG4/qH2dlpL8HZ91HeTLhtyw/2/6we5QHRnybnzj9U+vF31EiCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIg/DrW4XWhJRxvXP4cjjfedcMs7AWa5pPcpM7+mTfhXRsXvWsYvn94eLWgnfK9/8y1q//ahnauf4SUnzj6Qz2d/s64ydcaPr45iWTtuflm4wX7Q8D/ZK+rfjKLn/kPeUjyyebly8d6En4uxAVDLd65ru64wI44j3F7sf5MfvfR0+Uu8FbSJxXv/E3bcbzH+WqMR6b/e9q5PvR98iRflfzhgh19+d46cnUzMMrJxRfTk8l5Dz/cjlV9ruSTHncJzrfX9vIeq55XHwNPX3H8fM5Ljnw3vwF8m59U86fL21+Atb2QF7b9g0WCCY+1WXrnN2sfnZ5zWTKare3ov7aE+erjLts++8Wy7XavZlaOboqHnEf/tYEw5vpv+0/beoWJ/jnEfsL0Z3rS/s64y7ZDNVA7E5lzWjbfmAfqDy7/wF49b4ecJM8J/jkPFT7ZvByTYj2pnGJcMFTj3Y9fMu5sR5rHCA+0P5E/TvIsEmq8sbzE9GHymf+X4z2Vr87+TH/Wzvin/Yk+tP8NxxHj813rQiAf9a+tmwzF+ErGRWivcn7g6/gr+mTyCRz4pWU5g2JevZG8R/P2Q9qYl5z84OS3Gd/mJ8n9WLwufzVMhdTWbtNjSidyUrJv1p3uf+/k9fLnk6R9W56fF0JjGSwcfXVKdvcZDJMyXTKRWh2X7GitGJaUwUgguVP9H744KFzNRsxPzOEY83g7UobnrpWMa8/krGux9m3ftxWcQbB2Ol8eLywu7FQyRQK012hS5Es8LCxvmH/GQ5VPr0hA/em8Ev6ZiQuWf6gfkv60ImXjsrjrVbPDhvYdJNG4Jv55IMNbJi+xmbEJp/KbE3dQKunP9M+0D8UG5oFvGmB/FkfOd9+yLjD5yf7huslQXk+JnkxONR8icVndMvrU5tsdS3ecLtveNnAkfnxjvqNrrXI0Xc2riX3LwH+Y95z8UN2SfIufuPmT5ZMXD+X/GIZDq+56mlU/P8YA435rGevOIa92yfF5T3P/e9o/puNkEGt2gDkjJ4qE0HjVcde2t4YuyrpdNh5opiYsEuYTNb6ZqBXwvZ/c1tXYGgmaw8LnuevPx2VfJ+3d43DmmI60s3GpPk5cYH8g/FN7mRjERzu5/ox/xk+VTzYv2p/NK/TPXFyw/MN5IPmKxi8el9qx/479V2Dfvj+R78bL8c0Eb3FeYisc3Ipm8xvXH/uDM1+mP2mn/JP+1D85b8GyMvD5nnWBy3f6F9ZNhvJ6SvSkcqr5EIg7v+yugyl9LsyX3SQc6WUuDmHLoTYrJucBCnmV5T3Cf5z3kvkwxjf5iXuT0I0b5/lvwzC3g9yJuGdHz5APFmkmnfj1VyZoD59Xcs+XudYhThbpkx53sbuk3onNKUKDx89AdrpIGCaAeTj1J/NlfnIbTTyrOuccP5kOewY6Lvk6ax8KmGMU3M7GpfqkEhz1Q+M/jr1OpnfwiMnsJ7C/UyRAfoZuB7z+aF5Of39eg5xwxNFAJH4xD7w/jV84bhh3KI95PEz9sXw/XvK83aK8xPOwc+IG/b8Q71bQs5an/Zn+fF6Y/yg/d/r4/Z2948znm9YFKt/v//jw+uMQF4sE4OdUTjUfTuLApzAmM/q8sUig4UCKBKd/gERedfIeWV/cvMfsUt0vf5ufoP5s3Hi9+y68u0horc22y44ewRbZjsRKu+0RP2/6yrjdqaQ5sOw4QISwbdO8KQn09yYV8eNsjumjI1j/UE9+7/BikTBKehYJqP0zRULNP4v2Cm9Dn3/D/DN+Hv9I88nGrfYP23NxQe1CeKD9WfyScftvgg1/IY8F/a38xCY7nU9oXvL1Sfoh7p8sEg45vD/TP8y3g57v7E+mA/l817rA5Dv9h4/5px4urKeEh4ycy/nBk1PS58J8f79ISPAW5D0uh5EAt+mV/2/i6/zkSv+g/bsw/sbjBy/q3Sajv3Oyv6nat2Voc08jKj5tdbOwj0Ut5rjSnlxOv4hB16bmuCbzw6ML43bPWJjiwQty5u6oZ6B/L4rpT0D8ZLpSyJxve3qO/al/jqMPQuDjMZ2qD7GknY1L9WFxwfwh5p+vNAneeH/CP+Pn8c8KnywuqHykZ8BPOi5o/iE8OPkKxi8bl6nK8tigc+czfv8hgzjxMinj6M/yUqDPJN/Nw4V4J/5A+zP9M5sqK5T1Z/5ZvRhhfL5rXQjsNfUvr5sMxfWU6pmQU8iHaL7QjmV9ivO92S2WIfrTRcKFvOrlvaf64X6DNWbyYar/L/kJ6x+M67Z/H+z90P0nMOjm3U4GPpx13Eod8mw+nSSNj7UEZPXKUH2yz3TiZHfKyZnu6rjDCJ0BQh8eieuqfKr/bBSuPwHxE2OZx7OLdnByt0D1BP3JuMwhYPv9aeXz0crz6Qzc7oxL20lcOP7g8A/s5blJyNu4bxvaHR5KfLJ5pXkmeiaemUnFRT8G80PWH8avM+5ttqOTxyAPbt4D8kc5nX8WeUMKRfqgeHfyYSHeb8wfaH/mWKSd+iH9IM7/0NnsJx6f71gXuPyU3XPrJkNhPU2u7wkzHvKu5AfyDG1Kn/r+YU4nx7BtPT89Tl5njXh/D6W8ihW9BfzD/QaQH+WTEd/mJ07/YvoRBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQHKzD6zlLON64/Dkcb7wb33VPXmG3mC9k5J/9M2/Cuzau/+5CI2Z4taCd8r3/zLWr/9qGdq6/i0lO3HscA843LWh+q/vYjv3kkv6wP5PvjDvw8Ii0SrA4/tANcjV+bxm/Im+MDuebHdt0h/PN8TbyUM0DKTVz/gz61+OO6e/OC/itH0MzrV4c8Tw2ZSr27UK7r2eN/6I/JOKOIRuPV/IhlT/aPVxH6uNmUVtPub8xOe9ZN3k8VvWpznfGXYLz7bW9YY+F7R7kpYJffZq3b/OTal76dNx9Gmt7YR3d9g8WCSY81mbpdd4k/pzLktFsbUf/tSXMVx932fbZL5ZtP9rsuFa6lfPov7ZxRq7+2/7Ttl5hor8PIMfBsu3oxe9wvr4YGE+wnfrJBf1RfybfGxfxUA0W5g89rscv8yvmz9X5BkNPdmTzzfF28lDOA76isOw3Y43nF3P/Ytw5/Dvzmv2WxZ1jd2ZfL49Nchjl1XYm38sniP+qP+TijiGOxwv50JE/253q7/jtW1BcT6m/MTlE//K6ydfxV/TJ2BEOfGlZLoD6mztw3q8+zdu3+UlyP/bn4u4jMBVSW7ukY0onUDndqyebdu5/7+T18n8m4fu2PD8vhMYyWDj66pTc7zMYJmW6ZCK1Oi7ZOVkxbBECI4FFner/8MXl1SKBykH2tWd49iM6XyKH7jgTO9FBU6y/URTvkzhR7BPbznjY9n1b0ZkI1+fAtPtD8cvk+PLnZNfPC36ame+N5Y2EHe18KW88jzH9qT5X/XksTiAP3J1YXmX6O/Oa/TbIM5HdhxWV9cRFArJXtZ3JZ/Ny/NB+t1u8r8bdmK+IH0L7UrswfVz5fr7iy2Yib5yfpZaN8npq0OcTLIfqb5BZN5PreKhPbb7dsXTH6bLtbQNH4sc3pmPHn9YKR9Pe+svzUt6v2Fjv4a0u/w/5iZuX7LgZfb4Ntpjpri+3nVY/P8YAY35fxnOmcR9oSXne09z/nnaPntku3gYZz4QH72T9IiE0XnXcte2tocWvy8p4oJmasEgwTvnQztnc55ILlOPZF9OI50vlMN4onx0lof63dTW+Op2wzv2JfK8d8dA9ntcd72B9OmnDOQmKXyrHl483iyyOzu+MGsITXJg3QjuO/gZ54zwQ/Zk+ZX82uXFIfbg/iTsnr0Z5bHq2Cvutm2e8IqG3r28vUCRgP6+2cz35vGgah/m5FHfMT6gfcvsS/bE+VH6cr9CRidEnk39eKBIKm6Gh8iFynLg7WhPrZmIdTOlzYb7sJuHwjbmYhy32ScbEyTTxN8JD2a/sQJ/hrST/D/mJe5PQx1ekz9dhmNtB7kTcs6NX+jxYpJ438etnfGgPn1dyX5W51iFOFumTHnexq1/vxGZj2uDxIZCdLhKGCWAeTv3pfLEc375kUwXmy+Uw3iif5mNyYdDz0A/NTsRhymA5fmwnRQK+1yH6gJ43Hr+OHF++e6IM4ig7X5o3AjtO/o95c3jA+hN9Lviz/dI+PeLjr3k27hKrI8tj084R+a2fZ5jdJ/sG9kJFAvTzajvV05lXxH/HZynumJ9QP+T2ZfpDfbifh3l+Xkew30Y8JHFx8wfyNpVD4+75aWbdHD4dJ5zW541FAk3vpEhw+gNE+417H7unrfrVo/VjvJXk3/6Mn6D+bFxfn+/Du4uE1tpsu+zoEaYDNiyx0m57ZJ/PuzbucCp5BHvHASKEPTI5FQmR/t6kKtdeU1ryeoITZThfdxOJeGPtt9BPBv37fUJ4gujIp+PiImGMvMcuI9Jnuq4kmxIsJ5IfhOR0OZOdb9m+pqXTEfOWKxKMGpcSRbyGTZec2f7JAKyovXBPQVttssihqtCxF3zcCPh5uZ3J9+ZV4r8Ud8wQ9SKB5kOoT2aCJM+HT0t449ZxYT2F/paTA+hNrpuOnJI+F+b7K0VCuN84PnA1OwG33R/l7dv85Er/oP27MP7Go3tmejHdzIlF52WmJnh8BfyehNBXKRKsbhb2sajFHEPZE6l1qFxu6NrUlNWZH9BcGLe7NTbFgxfkzN1Rz0D/XhTTP8a4+eD2DQ+GzSwcOZA31s78BGswHb2HvDH53rjkJqF7Tv34AY+vz7xpgvHL5ITy3ZNg68/V+bp5A9sXzZfw5uUxqj/Up+rPwxw7ZYndYdwxfZj+bn57SjeNQZ5Bhw7Mvp694ONGwF7ldibfmxfhH/BWjjvmJ9QPmX2x/lQfKt9RlTbOSgc8FB43qq6n1N8SckB7et104rGmT3G+N7vFMgH86SKB9Y/3Azm/+jRv3+YnrH8wrtv+fbD3SfefwBz7DHLtCB/OOm6hD3nW/yZJ42MtAVm9MlSfznucZ3Bxsjvl5Ex3ddxhhM4AoQ+PxPWbZqb/bBSuv4dAjhmbG4zOF8rhvKF210+g/uYbj2et4cDm/gLJ5+PCD+5PE5+PduOnHoA+8OzfCD/jl8lh7dyvkD8X5zt/FNiRzNfhjeYxHo9Mn5o/E3dO8ZDSh+nv5BkYp1hRbPcgjoC9oBxmr2o71ZPPK5V/4DMJybhj+Yr6YcK+/WEo1ofLJ3Yn+hO/dXmoFAml9TS5vuM1IfWceWrdTOQ3Z17V/YOzfLX1/PQ4eZ014v1To9L1N5VPHgqPxyIf5e3b/MTpz8Z19REEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAg1uH1jSUcb1z+HI433nXDLPRFrov5Qkb+2T/zJrxr46J3y8L3zQ6vFrRTvvefuXb1X9vQzvWvzvdGXiMI9XTmxfTH7+Ot6s9B3/f7spxHRJGgYHZk+tT8s/Ce6kBPX/7kmZeTAMg/b3s9JeECyq/GL9M/slc237r+OcV1VQ6JozD/ZPwqqxR/Q3xS2NC/GncX/PYCP868Svpj/wzyYdpP3pdXq/kKr+/FdaG8bvL5VvWpznfGXYLz7bW9YY9V5aG6/rJ1jfHJx7iV8tu3+U91fX/XPuTTWNvVIuF2u237B4sEEx5rG95fjzhd25lDl4xmazv6ry1hpvq4y7bP9l+23e7Vjo+tdCvn0X9t44xc/bf9p229wkR/Cqf/su2T2zA92byo/n372b+qPwHjvwyiJ3O9pB1jfgjovIp6BvKBfV9JAn3+QX51ATDumPxq/I7fPgSm7JXIt8wfbrcbjGsC6g8kjpJxGre7wPpX7Y76V+Ou6rdBfr7dbrPd2bwq+lP/dPNh3k/elVer+Yqu78V1obxu8nX8FX0yFMKB30U/Q5WHat5g8imfbl7N++23+c+L6/tnj9vLMJVQW7tFy5RIpLLcN5um7n/v5PXyfybh+7Y8Py+ExjJYMvrqVCTcZzBMynTJRGp1XLJCWDGsmAEjgSCk+j98bvlIkWDPwAYTQz0trESm/ziVQz7Tn/gba3f4Z34O25me275vKz9Dmfhhcqh9jQGGHS2c1wU9GQ9QfyonYRebfxy/wvoQHljcBX478ebt8LD+xF4s3ybsOBQVOK6L/pDJAzzdnvpQPZldiP5Vu7P+1bgr+22CH8sz0/Oi/pN8x440/8N5ef4A1k2G8npq0Ke34rpgkFk3k+t4qE9tvt3xc8fpsu1tA0ffxzfmu/fW6kfQlc3ulbzhrvvj/o2EFfbbBL7If9Lre0af34ItWrrr722nVc6PIdo6rSkSzvkOftXlo+d9zP3vJQc//9XF2yDjmYFHxhNFQmik6rhr21tDSb9bRfFAMzVhkWCc76Gds0i8+LiRQ5eTLOawgPwb3+tci+jD/I37Ieaf+Tn1f6Jn9xiePZZh/BA5nJ/VzP2nYxT6VVFPJ96h/kwO4Z/mn3GadlaYf8IDjTvHb/PxS/Un9uLz5fpDe9G4rvpDmAfmIgTGI23H+lD9b0W7k/7VuCv6ba8tOUMbOzt1Zk1/5J/EjoxnOq94XbtUJBQ2PUMlXFwXbqZzYt1MrIMpfS7Ml90kHHYfDq1Yi80J2RPoAg+X8oZXJPT7N5KfvfyQmBtW5M/7T3p9j/X5NQxzOEicCHp29EqcB1vUkyYegSeNByvwuT/vpIOcdofBwxd1V5/0uIvN+r2zmoWzzWriGEkXCcMEMA+n/pfme6lImPYeNMmeTrSTq8dDH+Zvjh9i/pmfe/4P9RyW93lHMfMD5VB++qnZzSXzq4qe0ZEG2mwBOYx/ln/gP5/jYU0IDzTusPxhsCh+uf7YXs58mR2pP5C4rvtDN+HEWsniEbcTfby8VLI77l+Ou5rf3qaeRjLluVgkePo/hgrzOeE5Ma9BThUXi4Rpfb+8LiTXzeHTcb5pfd5YJHQHHIkiwelPkefhWt5g8ic+WX5O7VsS8n/Zf9Lre6jP7+HdRUJrbbZRdvQI4CATSay02x7x83avjDucQpnrcjMjRAjbFs9394H+3qSq11vpZ8e5iefztwT/Ic9lByD8X3ekoduQ1TNFApRD+On2CVZ46Fc5PS8UCRU5bysSKA8s7oh8oJ8nxysSkL1of6o/06cTMtw7vMMfzEydVJiIx3BesAp5Q5FQi7uL/g/yGOe5ViS4+psBkFIJPynmsSourKdwfb+8LiTXTUdOSZ8L8/3eIqFbdy7lDbo5RgfxLD8fPXKbsW/znyv9g/bfwfhbDvh4xrBZ6bzG1ATHwjn9boTQVCkSrG4W9rGoxRy/2JOY8ZcyN3Rt2h0jxwcoF8bd7E+AzObDC3Lm1qhnoH8viunP4PW/UCSgnUHIv21n+jB/Y+2Mf+bnjv8zPfurfFx8QwwJDvAzHUXbzXG4eGT0DOYLH9uI5eApjjGO/Arrw3m4kbhj8ovxy/XH/kz6u/rbuQdxXfeHIA+4K3Qcp5l5pYoE1w+JHUtxV/RbqrwXd8WbBKgP8884n/eqsnm5cgqPG1XXU7a+V9eF41vJddNZR2r6FOd7s1ssY8jfKhIYDxfzBjrExHw6+Rlo4Ezou/yH9Q/Gddt/D/Ze5/4TmGPdIpeR8CGs49bokGf9aZI0PtYSkDJdj2J9Oi/hz4aSZHfKyZno6rjDCJ0BQl8dieuqcKr/bBSuf22+zDCOnnBeVH9OD9Mf+ZvTTgdgDxvidiTm/tT5+ciloZ/yk5mvaTeR8RgIWSBwN0dPNl+of14OUfPMP07AQ304DzcYd1x+MX5Z/kz589mf6u+lBxDXZX9w8wCIUyaGtLt2CReL2O6wfzXurvgt44cQweZV1d/xT8eOQf5nqubWTYbCeppc33Prwu3yukme/UvpU98/zOnkGLat56fHyeusEe+fGDPDw/iNOG9g+S6fZF/UfRAUP9/mPxfWd1cfQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRCEvxz/97//6Y/+6M+/8+e3U44gCIIgCH8Dfn3Loj/6oz9/8s9vpxxBEARBEP4G/PqWRX/0R3/+5J/fTjmCIAiCIPwN0NZBEP4FKNIFQRD+OczvZxU+gf8qz9o6CMK/gHykr+0HvKwxieONy5/D8Wa78V335FV1i/lCRv7ZP/PGu2vj+u9sJa8X/UGv3Jy5dvVf29DO9f8ojmFf95WavZj/cDnvsQvnuapPtT2PuwTn22v7cGynccF/WDwy/m+3G4gX572gBLF9QV6qxinQ03yEX32c9EMmn/Vn/LyLNxhfTjyysWv883xrPu++4PWv6OPyBvR8e974siIB7E9+NS+9tF/6AL5Nn18A8we9zthHKdLX9oKTffQ8xZh/bUOKRWZf2xkwqZOetR39x6wPUR932fbZRZdtP9rsuFa6lfPovzawHHL9t/2nbb3CRP8/gzecvRXtRf2HyenbD23LduF+8oo+cTsHVOhX3aGMvP+weKT83243FC8sTp1xoXw/L1XjFPQ33xsPU0p+SOSz/oyfd/HG4ovF443k2yr/NK6dQUn/kj4+b8DuH8gbX1Yk3F7cn3wA0udvwLLtosVFFOnmhKmtnZOZQxpwUnI/+bAZ8v73Tl4v/2cSvm/L8/PCFmUZVoboq9NicJ/BMCnTJbNjqo5LPNWKYVsfMNK0CDn6P9bd5XqR8Lgm2ozhl+FT4Cesnc8U2IWhbK9+dLNpw3JGCpHtMnZJ+kmoT7WdjvTTwfpe28DRIjy5f9w8tOwR8Z/ynwDsWwvYbo3xkolTZ1xI0CCnGqe4//nhdG5c8cNI/tif8fNm3vhm/TbEY2JnkOI/GJdssGEeqOjj8Ab1/ETeeKVIgOv7sNQ/NhI+LWR/4twonl/Yd8s53reYQ2Yriu5P+H6JkEDyHhnX0R+3V/ZvjP9qPh/kt/VxiWFvpMf7NLQPrPZ3wPzB3iGEonwenPXI5+ew1zV/wH5b5MeHH+n2kLS7rtp2o2t/6tDHCywSThKHfXI3n6df3P+e3ub1keldxz/pHUlMFAkh79Vx17a3hi7SO+/AAyUWLab/eSI1bz6o/gDdAynmmIv5CWs3Cr61SCjESec/VI7RGR4BZjcTMc8pfartDthNwuGWs3VgyzGdzObvj/gPA8sDz09NO42XME6d0XH+6eRU45Tr+Wh70Q99+WhejJ838uYWCVOJQ/Ltrca/Oy71Q3TTW9KH8cb0/ETeePUmYV7fJ27dou/m7E+M4jRT2Vsaum9ZV7NXwc+NnfpH+sxgeY+Ny/Qn7cX9G+f/Qj7vyiyjp72tMjfweB9Y7e8Da17ZHjAe/PXI18TaseoPzG+v8cPgRvoQoweb1Jm80vmxCtEd/LQ+gQwxFn7wOUq6i6THOvGTiyRpRvqkx10286ONfjNqEkSDHg4PCpNFwjABtqQ99afzHRLoQyvmJ9FicO0kODPfAJP/OHJO593BIx1Juwyfjn6S1ucPFAkwWRvBYwvq/9X+A/IA4J/ESxSnGG6+mlb4fJz6/aPnelJ+6Oozz4vx817e3Md+5toK5NtOWJp/PC7nGUVQSR/GG9Pzw0VCbh2MSLg9k899t+NvJ4CESU+UB6yqZq/FBuq3KPzgMKXPDJz3vHGh/qS9vH+j/Pv5f8Qw8f5ADcjh+8Bqfx9vKRKwPrXi9sbsWPQH4rdlfoL4/aNFQmtt3vM4mpdW+u4AgEustNse/TFd+l4jOe5Q7R3O3HHAsyoSPB5GBfp7k4r58WisJgAACwlJREFUwU84/2aRULYX9J+cHMBP0i6OnJI+1XYHnywSnEG/wX9i/icVkQK5xHUxX1XjdOqfIOc6D6w/4+fNvHEJg5Ys387iwPC5mwSPZ1gkVPSJeRv98/154/WbhFntte3bsrZ9W9u+rVG+ulIkdF8/T9zZAt0/4fKBIgHlPX9coD9prxcJjP+XigTzT3p4RIiq9vfxTUVC19vcJJT84fqGtoTwcSPLyQ+ZjE1e44NUx/cfX+na0OldJzXLtNXNwl4D2dsbdmVzNMx3FMdxTXQQd3Fce7eUreBZ/kA9A/17UUx/hufZw/HP44QY+wlrP0YkJ3PZM4OqvZj/ZOSA9rRdGM9lfartHP3zuO6JzjnI60XCH/AfPC6MR8q/HcOwWZ2vk694XgLjxnE6b22QF5T9kMhn/Rk/7+ItkIDiEebbKv98XDfa8HFPQZ+Yt+nq5O154xNFwm1trbV9W25r2/fwTIPuT+6Y80DX0N+8gX2L3T6iyzHf6HHssLzHx2X6k/by/o3x/8qhD34sqq/12T6w2t/Hp4qEaD2awe1Y8we2377GD0MU6d2tSDsfL+PXcPDh1uM29ZBn+J2vOcbrjyBZTNcrWJ/ss7B4M3rKyW1Tr447jMCuFx/zHppG4jqHo/rPRuH6s8maR2rZD4zwf+xn2/1rr0qRULJX0n9mXYj8i3bpE0FNn3o7xeSGhzptPT89Kv5ZU96f4o/4TzDbJP83HC+uQwzw5Dv5oRync3+nfKr6IZCfnFc+kNK8OfF1ozV7lG8TfLJxGc+uniV9PN4CP3lT3nihSODr+3KUq7l9Dt6f0DyANhvTfM0HxuMeCaqtrv5sv0TA8h4Zl+pP51XdvyH+L+TzTn5rDaSUxy903X1gvT9G0h8CMR4PfD8Mwe1Y84cb8dsiPwG+7/8xE/5CvPx8kPBPQ/7zR/DeAyaB4Z/gWVuH1/Fv5L03P/3yH8Z3+oMiXXgVpTsHQRgg/xGEvw7aOryIfyHvnWf4/905vgtf6w+KdEEQBEEQCtDWQRD+BSjSBUEQBEEoQFuHFKYf0rzlMXFhxLt4/jY5XwBFuiAIgiAIBWjrIAj/AhTpgiAIgiAUoK2DIPwLUKQLgiAIglCAtg6C8C9AkS4IgiAIQgHH1kF/9Ed//oU/v51yBEEQBEH4G/DrWxb90R/9+ZN/wpywhq8PdHC8cflzMK/mM8Ms7IV03bv8MvLP/pmfp18bF71zE76Hc/jlfP9O8ImEWP+1De1c/3jG3ftvPaPPbw115kW+7naB3EUSM/w7YPa9Zsd5XFefgh1xvLwPzA/ZuN680GfMnxn/vl1mPbE+xbiO/Hmy10W7z0KpVa/wj3Ah3p35Dl8r5YFem3GA6rzYd67Mt5JPruWqcXg3bw/51tM/kTd+fcuiP/qjP3/yTy4vvfCWh4++TM684nJtQ2pHOXdt51q+ZDRb29F/bYk0Xh932fY5rS/bbtck8y7ubord67vbelvbOCNX/23/aVuvMNGfgsj3qF22fXInNi8Gpwvk00GJfwZm36odBz7j9oe0rB1pvLwJzA/ZuA7P2I7E3xj/gV2ieDk/Lca178+zvSgPrt1nsLigdi/KvxLvznwn/qt5gOar6ryK/lbNP++yL/VbJ2/jfMuLhI/mjeJ6mvTbkM/yulnlIadPhjc4cHVZrqK6P6EKkfak/2fWBab/V/lJeX2p56svgDlRaGtXJJijBnJys2/Wze5/7+T18ucTuH1bnp/XdnthkTD0741xn8EwKdMlE6nVcVEGH8SwxRKMBDZnVP+HLy4vFQlM/rbv2wrOLu2Z1mB6Ni/oV0w+45PJucA/83P7XZtcSnYcTWQSN2y/vWDHMV6gUYzB6E0duhlz0tzACeaH2CUTj4xn0D7pSXm+kk8emP15thfjgepD7ELjgszR96vZz6vxzuZrBqR+kskDjg9QO0K/LfpbNf+U7VuNL6eIRfnW07+UN85BUstGeT0l4zI5jj8fyKybVR6YPrX5dtc4HafLtrcNXKEe35i29T+tXbhKe3w9zMMXigTo/xk7pg4LyFi/7Cfp9SWjz7fBFjOrfdxo22n182MMYJ3WFAknD4Mfgn3JvWVtaQfvme3ibZDxzJyjJRJFQmi86rhr21tDi2u3C8ADJRZdpv9Zqc+bS6o/AJPfPWZmjwVuEY1TJQP9isonfFL/LPLv+Dnxq6IdTUx1IUPaX7DjUPmTeFxXE7P4JHWM35u/+RsyIOGH2tGJRz+uc5stxv+VfPL8Yu/72F6cB+IPzC487sycQ/nMz6vxzud7m8T1yOUBahcyL+7nNX+r5p+qfcvx5fgnyrdU/2LeMFO7VCQUNkNDBUvksHi5mc4ZPyzywPS5MF92k3DYa940wxa7RuQ22ZX9CeOH8sbWU8ZPtG46s/geP8mvL6E+X4dhbge54Azh3tErfR4s0h08yO+TT48HIvB5ZMorua/KXOsQJ4v0SY+72GzdO7GppBs8rgay00XCMAHMw6k/na+ziNJzbTdXJktqJp/wSeUU+U+V+MM1YtWOZ3Ds20Db3F60YycLXyMM/tx/ZLZ+rv+zzd8cp4QfZpfMosviGrQjPRn/naB0PplrQGwv7idYH2gXP+5uiH8sn/p5Nd4D/+RFwpWjtd4uYF7cb6v+Vsw/Rfteji/EAysSPD+Bcm7Yf0q4WCRM4zpy/PhNrpvDpyEPf6BIOFvR+bR3Yu3PbkJqfzJ8gZRR9nlQ6P/OJt5fNzG+zU8q60u87nwX3l0ktNZm22VHj9AdLHGJlXbbo99L1WwXjzucEuFrfUQI275MRUKkvzep6/wMWStfJIx7KlokYPmET7dIKPCfvAc8u1204yQnbk/bEcULH8j0NCNEPNDk6O4SzT+4XTLxmOYtSDXVvAHaqVXmXbDrJ718Zhcv7ip50ikS6vFOmzz+k3mA6M/aL+SB2nwv5/OknpGEQc5TOeCWkb0mOQn/iXFhPSV542IeSK6bjpySPhfm++tFQmZ/wvqzdub/nLdMvHT4Nj+50j9o/y6Mv/HonmW0V6jmZLqzane4tW/L0EZP+55Ssz5tdbOwj0Ut5tjEnqCsbTrVnM9sumPk+BDlwrj9XfUZnF6QM3dHPQP9e1FMfwoi/x6w5xDJx41AFYr9ypEP+XT9s8C/ow+070U7PuWEBzRMFLMjixccj9ZS/SFNcAuHilU4rsMPtIvjb5B/N96DVDPIr+cT1779R5lF/dSH24XFBeOfzZfGy4V4R/P1Z1rKA2G+4vPqUPW3Uv6p2bceXx4P5CYB6l/NG8eH+f8bq7Se0nETckB7et0s88D0Kc73ZrdYXdh9tkio7k+cfAjbqT6En6r+3+YnGc1r6/s3wt4T338Cc1wmkEtQ+HDWcVt1yDtY669U798YH2sJyJquZbE+nfc4z7rhZHfKyZnu6rjDCJ0BQh8eieuqf6r/bBSuf2K+NpP+mEcZjXjPYCQ2Z79y5A9khP7J+1P+iRxm35odaXdPTNqOyXgxn5hvPAjvQ3vsj/3QG9eZWGRHUOG7/HfPqiI9XXXqce2sNcc3lqktRQ+0C40Lh//MfCP+o3gE83XyFeMt1scKyc2Lxdhr863kk6KeqTwPG40YX/9q3qgVCaX1NJmv0nny4rpZzZ/MjkmGJvc5hm3r+elx8jprxPunRs3tT+g+Ic6TbD1l+Twk7tv85ML6UpmuIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAjCfxb/DwLd9njqvnRlAAAAAElFTkSuQmCC" alt="" />
可以看到这里为discard_passdown,表示dm会将DISCARD传给底层设备(loopback device),queue_if_no_space表示如果thin pool没有空闲空间后,IO请求会被排队。另外,error_if_no_space表示如果thin pool没有空闲空间后,直接报错。
[root@yy3 ~]# ls -slh /var/lib/docker/devicemapper/devicemapper/data
2.5G -rw-------. 1 root root 100G Nov 12 06:14 /var/lib/docker/devicemapper/devicemapper/data
[root@yy3 ~]# docker rmi dbyin/httpd
[root@yy3 ~]# ls -slh /var/lib/docker/devicemapper/devicemapper/data
2.0G -rw-------. 1 root root 100G Nov 12 06:15 /var/lib/docker/devicemapper/devicemapper/data
可以看到删除image前后,稀疏文件大小的变化。
dm.blkdiscard
docker还提供这个参数,默认值为true,即删除image后,会调用DISCARD,真正释放HOST上空间。
func (devices *DeviceSet) deleteDevice(info *DevInfo) error {
if devices.doBlkDiscard {
// This is a workaround for the kernel not discarding block so
// on the thin pool when we remove a thinp device, so we do it
// manually
if err := devices.activateDeviceIfNeeded(info); err == nil {
if err := BlockDeviceDiscard(info.DevName()); err != nil {
log.Debugf("Error discarding block on device: %s (ignoring)", err)
}
}
}
...
}
func BlockDeviceDiscard(path string) error {
...
if err := ioctlBlkDiscard(file.Fd(), 0, size); err != nil {
return err
}
...
}
func ioctlBlkDiscard(fd uintptr, offset, length uint64) error {
var r [2]uint64
r[0] = offset
r[1] = length
if _, _, err := syscall.Syscall(syscall.SYS_IOCTL, fd, BlkDiscard, uintptr(unsafe.Pointer(&r[0]))); err != 0 {
return err
}
return nil
}
主要参考
https://www.kernel.org/doc/Documentation/device-mapper/thin-provisioning.txt https://github.com/docker/docker/tree/master/daemon/graphdriver/devmapper
Docker内部存储结构(devicemapper)解析(续)的更多相关文章
- [转]关于docker包存储结构说明
原文:http://blog.csdn.net/w412692660/article/details/49005631 前段时间与同事交流docker的安装包层次结构,并沟通相关每个文件的作用,但是一 ...
- Redis-基本数据类型与内部存储结构
1-概览 Redis是典型的Key-Value类型数据库,Key为字符类型,Value的类型常用的为五种类型:String.Hash .List . Set . Ordered Set 2- Redi ...
- docker的存储结构,和以前有了很大不同
在网上学习这一块知识点时,有一个URL讲得很详细, docker 镜像与容器存储目录结构精讲 http://blog.csdn.net/wanglei_storage/article/details/ ...
- 【SQL SERVER重新认识】数据内部存储结构简单探索
数据库经常需要打交道,但是从来没想过数据库内部是如何存储数据. 今天探索一下数据库内部如何存储数据,从下面几个方面探索 数据库内部如何存储数据 索引数据如何存储 操作数据对存储影响 总结 数据库内部如 ...
- docker 内部组件结构 -- docker daemon, container,runC
Docker, Containerd, RunC : 从 Docker 1.11 开始, docker 容器运行已经不是简单地通过 Docker Daemon 来启动, 而是集成了Container, ...
- SQLServer2012 表IAM存储结构探究
SQLServer2012 表IAM存储结构探究 Author:zfive5(zidong) Email: zfive5@163.com 引子 国庆节期间,一直在翻阅<程序猿的自我修养-链接.装 ...
- 深入PHP变量存储结构 标签: PHP存储
1.深入PHP变量存储结构 标签: PHP存储 分类: 编程语言(10) 首先声明,我并没有去读PHP的源码,只是对于PHP的有时候诡异的表现感兴趣,找了一下开发人员laruence的博客结合PH ...
- PHP变量存储结构
首先声明,我并没有去读PHP的源码,只是对于PHP的有时候诡异的表现感兴趣,找了一下开发人员laruence的博客结合PHP提供的函数debug_zval_dump刺探得到了本博客所阐述的工作机理.如 ...
- docker存储结构解析
由于aufs并未并入内核,故而目前只有Ubuntu系统上能够使用aufs作为docker的存储引擎,而其他系统上使用lvm thin provisioning(overlayfs是一个和aufs类似的 ...
随机推荐
- ecshop JSON,ajax.call 异步传输
1.res = Ajax.call('user.php?act=depot_id', 'id='+v,null,"GET", "JSON",false); 2. ...
- The length of the string value exceeds the length configured in the mapping/parameter.
在NHibernate 3.3 中存储的字符串太长,会抛异常:The length of the string value exceeds the length configured in the m ...
- Saltstack pillar组件
pillar组件 pillar也是Saltstack最重要的组件之一,其作用是定义与被控主机相关的任何数据,定义好的数据可以被其他组件使用,如模板.state.API等.在pillar中定义的数据与 ...
- nginx禁止ip直接访问
编辑一个noIp.conf放到虚拟目录中 server { listen default; server_name _; rewrite ^ http://www.xxxx.com/; } 其中 ww ...
- django book
一.安装配置 1.下载地址: https://www.djangoproject.com/download/ 2.安装: tar zxvf Django-1.6.1.tar.gz && ...
- 看开源代码利器—用Graphviz + CodeViz生成C/C++函数调用图(call graph)
一.Graphviz + CodeViz简单介绍 CodeViz是<Understanding The Linux Virtual Memory Manager>的作者 Mel Gorma ...
- 一次关于使用status作为变量引发的bug及思考
这个bug出现在一年前,当时自己大学还没毕业,刚刚进入一家公司实习.那个时候还没有用seajs或者requirejs那样的模块化管理的库,也没有用一个自执行的函数将要执行的代码包裹起来,于是bug就在 ...
- requests的安装与简单运用
requests是python的一个HTTP客户端库,跟urllib,urllib2类似,那为什么要用requests而不用urllib2呢?官方文档中是这样说明的: python的标准库urllib ...
- SharePoint Server 2010 中的基本任务
SharePoint Foundation 和 SharePoint Server 概述 SharePoint Foundation 2010 是一项用于 SharePoint 网站的基础技术,它可以 ...
- Hadoop 面试题redis
Hadoop 面试题之十 548.redis有什么特别之处,为什么用redis,用hbase 不行么? 答:redis 是基于内存的数据库,速度快 551.redis用什么版本? 3.0以上才支持集群 ...