Docker内部存储结构(devicemapper)解析(续)
dm.fs
参数dm.fs可以指定容器的rootfs的文件系统,但只支持ext4/xfs:
func NewDeviceSet(root string, doInit bool, options []string) (*DeviceSet, error) {
...
case "dm.fs":
if val != "ext4" && val != "xfs" {
return nil, fmt.Errorf("Unsupported filesystem %s\n", val)
}
devices.filesystem = va
这是为什么呢?下面是docker dm的维护者Alexander Larsson的描述:
Additionally we ensure that DISCARD support is enabled in the filesystem so that any files removed in the conttainer filters down to the loopback file making it sparse again.
参考这里
一句话,因为ext4/xfs支持DISCARD。这样,如果容器中删除了文件,空间就会马上还给Thin pool,因为Thin provisioning是支持DISCARD操作的。但是,默认情况下Thin pool是底层是稀疏文件/var/lib/docker/devicemapper/devicemapper/data,所以,只有Host的文件系统支持DISCARD,才能保证稀疏文件空间释放。
Host为ext3
# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
dbyin/tlinux1.2 latest 8297f05d459f 41 hours ago 399.6 MB
dbyin/httpd latest 93e711fab1c1 7 weeks ago 412.7 MB
centos latest 61038e6e3195 3 months ago 236.4 MB
我们可以查看稀疏文件的真正大小:
# ls -lsh /var/lib/docker/devicemapper/devicemapper/data
1.6G -rw------- 1 root root 200G Nov 12 11:52 /var/lib/docker/devicemapper/devicemapper/data
# dmsetup status
yy_pool: 0 409600 thin-pool 0 13/65536 0/3200 - rw no_discard_passdown
docker-8:1-696417-base: 0 41943040 thin 928768 41943039
docker-8:1-696417-pool: 0 419430400 thin-pool 73 633/524288 26115/3276800 - rw no_discard_passdown
我们删除一个image
# docker rmi dbyin/httpd
可以看到稀疏文件并没有变小:
# ls -lsh /var/lib/docker/devicemapper/devicemapper/data
1.6G -rw------- 1 root root 200G Nov 12 11:52 /var/lib/docker/devicemapper/devicemapper/data
# dmsetup status
docker-8:1-696417-base: 0 41943040 thin 928768 41943039
docker-8:1-696417-pool: 0 419430400 thin-pool 73 490/524288 18758/3276800 - rw no_discard_passdown
no_discard_passdown表示dm层不会将DISCARD传给底层的设备(loopback device),只删除映射关系。
Host为ext4
[root@yy3 ~]# dmsetup status
docker-253:1-8790943-pool: 0 209715200 thin-pool 339 876/524288 32432/1638400 - rw discard_passdown queue_if_no_space
aaarticlea/png;base64,iVBORw0KGgoAAAANSUhEUgAABAwAAACtCAIAAAC7jnp5AAAgAElEQVR4nO1d27XbPK90Wf9SQapH1bAZlXIe9nmwLYHkDC6yne18mVl5SGgaBAcXEqQc3W6CIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAh/K9b2c8e+/bYq/2ks297W29r2bfltVQRBEARBEIS/B8u2P3brf2wbuWz7c7i1/bR1/nRsu4q1vVCHbPvbaHmS/DPM7Gh+bZhHyQWELNve1nulUBao+k0QBEEQBOGfxbLtn6sP4O60G/Hjh9xre2Gz+x5u1nZsuG1RtLZjW7+2sXwoA9txbT9XioTbi7wxXFHkN8b9LT0FQRAEQRC+BfPm8n68vW/L8/z73DAdJ9/Dnha0L6at/8KzLhjPv+0XTpWWQ53uX8GUzkG7zS7T/3j66WffLB/3v3fyhqlNwmfeujqIbNgv7eNHCYMdz0kln+yaebu37Ht7tlF/2Pf9kA/5If5wb77//S7+PovHNc5mDJOo1oA+3A8tQZO/ZfX0eBAEQRAEQfh7QU+gjx3e2u57o/MhoVt3Os7ab+RE9r5pfpypw5uEobH/Z7SdNgf0/WMzTE97jL9su93+mSLB7oz7LaXdFCLexiIBbCJzx/ZW+PRQFrPjvi252xrC2zH5J01HPWNHXJvlmfLDbpbsxr0vGq1hgjKK6cPGva2rrXZsj5KezriCIAiCIAh/K5zN5div3zg9urD22+3mFgmPYTNFgpVzfjQelD82lMN3j0051dPboB+/1+gEwnGJ5mPbXCQMJQjFYxPPfkMC7XhvTBUJDm/Pmue4RzmNMZPg8lMtEmzvxGUL0oePO6h6uUhwxhUEQRAEQfhb8VtFwrlFm/fo8+jPlsxG8b1FQmvD7wXcU39cJJjv92oMFxevYLLj/LSRO8yVIqEf7UGLz2epSBg1KmzA4xste3MyCi8WCXRcQRAEQRCEvxXZImHctJ17QdbefWSe6Yl/Ko1Hvz8nnzgSHx47j/Tsdov9b4iP647hUXaqBtRw2Zp5HMX+cPmNG0p2k9DW5M9wCW+8SOgGNN92+An8oa+Z7v8w3w1mwfTB49ornekyp6gnHVcQBEEQBOFvRXgC3Z2Cl9o7YcP5+eOf+Elz8gxS9lmO7umPNjxeTx4vAe3Hf4FqXuoAfmPd/5w1kn+0Oj+ovQRYJDyf7LrK28/Pfij3+J3CcvyEF5EwzXf4iPrDk+CtHTxv+96a/Ql0UB86+sBxjQUeA6HfNGf0fKMZBUEQBEEQvgMf/S9QnUGdPVX+ckM4wIuEj/xnpn8Av+GYgiAIgiAIwu12+5WXqWF0/yEl+UDntBD8ZWp/Lcwp/19Z4QiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAge1vbCf0h/vJn4czCvujXDLPBFyf0nuUn5L3fjvSvjopf4whf7Dq9itlO+95+5dvVf29DO9XcxySF2cfSPeB79EPJD29G81vbjOCeTH32W7evKAHz68tHL8rB85oeVOfVyavx0dnT84ez8mj5zu2/3G3n7SpUfJqeaB6qoyojzBsiTwC6M/3f5OZOF83+9Heof+ecFnP7v+yHT05Wb5Tkxr1KcFlBbT4v2uhEeyusmXwer+lTnO+Muwfn22l72TD7f6vrlr0f5der88O/0k+p835H7fxMvvZf3oy/HNeGxNkvvsu2I67WdiS/1Pum1Hf3XljBffdxl22e/WLbd7mmOj610K+fRf37ztKv/tv+0rVeY6O9jlsPswvTP8Xz6IeOHtbN5MRegcoi9KPp5sfnO+5jRLgRJux+fUj909QnnVeTntCP1h8enWR6oPqTdC/1l2+H2vcQPkVPNA2UU9WT+4+dJYBfCsxNHWI6v6sQPyzPVdu4nnn9ehfV/LJLqyUWW8kluXtk4LaC4nlbtxXgor5t8HX9Fn4ynw4EvLcsVsAEIn8H6O/EZtE9y7sjnh2/zk+q6fGV9+X2YCqmtXZFgSidSce6bTSP3v3fyevk/k/B9W56fF0JjGSwcfXVKdvcZDJMyXTKRWh0X7kh6MSwpg5HAYk/1f/ji8mqRgOWQuTD9qZ7EDxk/lDenSFjBGYQnBxbM5tB42Pl1lCP9hyKc2gXFixEwJl8m3/aBvHX9ybievSA/PJ8cGCyEeSA8M31YO7O7Pfu3U/btNec9Jmeg5LJfVf2N6Ynoc/UkdmE8O3ms5ueMn15PmGcy7Zk8P/sn5BO3E/9nfkj1v2R3Pz93H9TjFK6bDOX11CBjr0zey6ybyXU81Kc23+5YuuN02fa2gSPx4xvznXxrlaNpvtkN1y+wP2H5xF2nUvkhgS/yk/S6nNHn2zAePtprR1b9/BgDWKc1RcLJw7A+duvB857m/vfCme2wqIBYswPMK3eiSAiNVx13bXtraJHoVgM80ExNWCTMJ/HOIp27OMFyehVC/Yme1A85P6SdzKt7HM4eazA5zF7ramXCE4suBEg749OLl5E+d1w7tSHoUH82Lo0LzI9jRyNk2KtAvyI8M31YO7c7CXPCj5P3eLqo5AHqVzV/8/QcafL0ZHbheRLHUdnPaZ7sVJhbk+1xnp9OWCCfpN1bT6kfQv2Ldo/z81iC1OL07HmtSChshnL28vIe059s/oJ1MKXPhfmym4TDT+ZNOWyxPhCfTLP5Uj7d/Um6SKjmhwS+yU/y63Koz9dhmNtB7kTcs6NX+jxYpDs24JeeJ0F7+LySe77MtQ7fvLr6pMdd7CrRO7GpsNusJo6ddJEwTICFtN1jwPkGcrhdxr0f1JP54Y3zE/HWz+s2LMunVkQOtVfv0rjs2rurU9iO+fTjZSbKHdfSMJ8/df35uF7xOfPj2HFm3uGB83yhSMB2J+phfvwjn2hPkMoDzK9q/hYdTbk3CdPOEdjFWYxhHFX9nPJj5sz2Vpn2cDOXO+oj7dz/PT+E+hfzTJjnhxqhGqdVXCwSKvby815y3Rw+HX0orc8biwRqJlIkOP0D9PPFfPrrbL5IKOWHEN/mJ5V1OVyvvwzvLhJaa7PtsqNHAAdjSGKl3faIn2t/ZdzutMwcpHUcIELYI5NTkRDp702qcu01yfHtMt/dAz35osX4CXk7PniqMY4ARjb/IPbq9tuc0YofTmkeSpw1fHXcrp2Oy/wK85PYfOQeVqM8M31YO7M7U69HzE9RTtWvqv72Af8Zd82M/ygec37O8uSN55lSe5gn5zMNpOaFIoH6IdLzhTxDevP7oGKc5nBhPb1mr0evpP4V/y/pc2G+v18kJNw7iOt0kVDMDx6+zU+u9A/avwvjbzx+cFKzTtPdkfe/VX1sFKbfkxD6Kj5tdbOwj0Ut5rhmfKBk3Kn+zHcUR3lLLgZeHXezP90xmwYvyJm7o56B/r0opn+MadOAn5Ji+mM9qR8yfli7Y5dTT/OBwz+w13T0S4xTaZ/49M8w/JNgE4yO/4P+dFzuV8ifqR0PaXQlGPZNjGemD2lndh9HQeD89IziPVYlD7D51v3N0/MG/Cf2k8FkhOd4s5L2c5gnWZ6ptgd5EpyOYT5Ju7OeYj/Eer6SZ1InRNfitPC4UXU9vWgv1p5eN9l6UdanON+b3WKZwPt0kZBZ9wtxnS4SqvmB4dv8hPUPxnXbvw/2vuf+E5hj3SLXnfDhrOOW+JBn922TpPGxloCsXhmqT+c9zrN0ONmdcnKmuzruMEJngNCHR+K6Uzeq/2wUrr+HUY5rF6g/1ZP5IeWH8jbPa20/Pz/mUeeOBod/YC8z44dAs9qX1ER8Dvr/YPGdyah84odcHzgutRfhh9vxwR5LlwMPlGeuz9zu2J06bo4foIxDRCIPsPmW/Y3oyfOGo2fkn8zfuKOEfo75YXmm2u7p//iel68S6yD0f+qHXM8Ldmc843ldidNKkVBaTy/Yy+Xh4rpJnv1L6VPfPzhu3tbz0+PkddaI9/eA1/3MOkLc3IpK5ZlcfgD4Nj+5sC67+giCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAgQ6/DaxRKONy5/Dscb77phFvZCTPNJblJn/8yb8K6N67+j1ogZXi04vyJ45trVf21DO9d/xsMziHGxXQJpZNh38DPz4Olf4YHJd+Do2c2vf91pyW9PIVifUf4l/5lAeKPzjXlG+Yf5ytQexkvavD4Az8z/HT4TrzeN33rK/KQaXxf0+Twcf65Kyc2r6s9cfi2OOKB9E/lkxDxXP59fyj+unBwP1Th6z7rJ81JVn9ftfpfgfHttL++xMvuWDJ9czlvWdwff5idsPU2vXX8Z1vZCXtj2DxYJJjzWZul13iD/nMuS0Wxt9tXisfnq4y7bPvvFsu02dx8fW+lWzqP/2sYZufpv+0/beoWJ/gyMQmoXAjbfG+FnmNehBOOH8UBdoMhD1U+onoNQ+1r7kt8+RpnsS+Rf8x8Awhudb4rnLv9gfyDtQbw8pL+almeeqf87fC7bjtIsmy/sT/2ExEtVvtf/84j8OYfSvKr+zORX44iA2TeXT6g+Rg7N5xfyTzhuyEM5jorrAs17fB1/RZ+M3eHA1eWoDDYAXTeL+byYf6r5+dv8JLmeZvLzF8NUSG3tFmlTOpFKcd+s29z/3snr5c8nIvu2PD8vhMYyWDj66pTs7jMYJmW6ZCK1Oi5fsewRFHQaMNLklI7+D19cXi4S1uDOYBAJ/YTON8EPK2JtF8YD1Z/zAPX3eYZxAfV04s52wSc308kxtq8jP+8/5rB03CFF/jPOF/ZnehJ/oO1kXM/fOJ/QjpjnXjVDG+bTnj13Q5N50f79uHaRw/FSlc95dvgptL/gz11/4p/leRX9mcm/kB/CvJFcF8J8a/0hmc/DdZOhvJ7244ZxVF0XkLisbrm4rsy3O5buOF22vW3gSPz4xnxn3lrlaJoWRbH+mXxezT+Z/Q/DF/nJ1J+Nm9Hn22CLme66f9tp9dPf0MIi4eRhXJfn/H5vWVs6jfTMdvE2yHimzdESiSIhNF513LXtraGk3K1yeKCZmnCTN5wcz3I8/QG6x8ls+d+r0M0K+wmZL+PH+F7mCJDxQPUnPDD9mXwnLpCeNO46irqiGscRsa8vv+A/t3U1sYxWLeY/KX/jenJ/IHFExiX+xvhkdqRxZAfq/J/nE5henHnRdITzDI6XqnzSn8dFrb3oz3wdYf5ZnFfZn4n8an5w8wZbv4A+VA7xBzefF9ZNhvJ62n81llNcF5C488vuOpjS58J82U2CzTnz1nNuOdRObbLpehfpn9w/lPNPtP+h+CY/cW8SBh4Cfb4Ow9wOcifinh290ufBIt3BA3+afHo8oIHP/VFeyX1V5lqHOFmkT3rcxWbl3olNJd3gMRKQnd7kDRPAPJz60/kOy8io1WgX7idkvpQf60Q7eARh0MQpEjz9Zx7oMSrxkyjBDXt4HHeTOmaLiu2C7RvKzxcJ/dBse5x+fKPvz/Vk/uD4CR4X+hvl07G7G0dTXqoWCd68oj0HutYf46Uqn/Rn/BTbq/7srCOOf1Z5Nlqn/LlQJFR5G9WZ8/Z8NsK+Dv0hkw9feRziYpFQiaPSuoDEgU9Hu6f1eWORQJcJUiQ4/QOY+Yb6p/L5s2ch/0T7H4xv8xPUn43r6/N9eHeR0FqbbZcdPcJ00IIlVtptj/h3Dq+M251+mYOxjgNESOJ2MKe/H+XxIxyjp5jYnu2S22SbfzB+Qj3nc2PIg6M/ku9tFpH8DIG8PKaLSqxPJ8Qc0NeLBDIvY1nuQbFdcH+/SED+EPrJ5A/Q317YtE0swLzkxSPbvLJ5xXuO6/mnqM+bioQyz1yO459Vnl31gD/jIqGUH5LPG4T6lOUU82EZF9bTchy5eibXTUdOSZ8L8/39IsHMN9Q/lc95++X9z4xv85Mr/YP278L4G48fnEQ6Z+qsamqCx1fA70kIfRWftrpZ2MeiFnM8Yk9K1qFyuaFrU1PGZn5Ac2Hcbe/q9uO7XpAzd0c9A/17UUx/hntgnqLsyZ9/Cv7sdf+CM1/IzyAzPgcgPDD9GQ/Uz4l8Jy6Qnk7cYf+JT/UGZon8oynlP9OVguXT85/JLqS/pyfzB89P0A4e+hvjM7DjNATzfy8eyaafzgv1d/Pb2W6bS/JZf8bPa+0IE8+gP/fP6ryq/kzlF/MDaw/sC07xIr9N58PauslQXE+vxJH5brwuPLsOXuTk/5o+xfne7BarW14+WyRQP/f1z+bzDpn8U9X/2/wko3lBny+Fvd+9/wTmuEwg17jw4azjNumQd7DWXxUfuRFeHhNM181Yn857nGfdcLI75eRMd3XcYYTOAKEPj8T1myem/2wUrj/A2n5+fswjhd3DrAm7ZOeL+PHowTE+80D1d3lgDyEynp2HFoGeLO64/6A4GoWdEyPyq/5jLPwgsA95oCeeL+tPebiReHHa4bjUgRifjh1HnpP+3xfn9AtoXrw/8RMvXkryKc9xXNTbY38m/Zl/vjSvhD878qv5Ic6T1fiK8qebD8vrJkNhPa3HUXVdSOW9vuCs6VPfP8xueAzb1vPT4+R11oj39xCvd9V19oX8Exhy1qNql4/6idOfjVuZriAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgnDH2r/utIbjjcufw/HGu26Yhb3I0nySm9TZP/MmvGvj+u8U5m9mnV+JOXPt6r+2oZ3rD+How0Dfiwv1uSin0D+Sw/x/bK/x7PoDfP3irOdDA4fyypyv8OB9Z+yP45S31+KOg8kPdX/D2AgX7Fh+Hec0xhvjlPHJwOxYzVfXx83FV21deGueTMZ7pn0mp9pe1aeKalxX88N71k1u36o+r+exuwTn22t7wx6ruj+p5tVq/6q/fZuf+HH3+ny/DWu7WiTcbrdt/2CRYMJjbZZe+Mbwe6fnXJaMZms7+q8tYb76uMu2z36xbLvdQxwfW+lWzqP/2sYZufpv+0/beoWJ/gxMH4pen3m/OOpzSc4MxidrHwfD/m/aizx7frhs+zQc9wc6dehXtD3FZ8cDkw/7szil8VuNO6pBSn4Yj+/CBTtCfwiGmPR/V5xSPnNyxrhL56vquNX4Kq8Lb8yTlXhndmF8VtvL+lRRjOtyfiD8lNdNvo6/ok+GNzhw0d3KqO5PGA/MT6r9q/72bX6SzG/np8X9zHfAVEht7TYHpnQCldO9erJudv97J6+XPx837tvy/LwQGstg4eir02Jwn8EwKdMlE6nVcckOwIphixYYaXJKR/+HLy4vFQmOPtAfRhXM3LE+l+T4/Yfgxzwz/yftZZ4N7Lj2DM+GBtNz2/dtRWcobGeZ8LeOT4cHJp/kDTIWbvf5BHmG5BMm35sv+XptXNJetSPzB/tBuOMcQNgcKyUWX6QP5ifIn5V8hcfN2T2ML9bf45meKCA/MbDfrMZ7YBe46a+0V/UxX0otG+X11CCTHzJ+m1k3k+t4qE9tvt2xdMfpsu1tA0fixzembf1PaxePpjP7k6E/zKts35Lpn5GT0eeX/QQV53DcjD7fhuHw6EwW206rnx9jAOu0pkg4eRj3Y5aU5z3N/e9pB++Z7eJtWprwCpEoEkLjVcdd294auijrVic80ExNuOgap3xoNxcJVP9o4sPeA/qD8RnrEkyfqhzuh4xP2M78n8ZFmWc79GBf7GZY/+5xvv74BfsVb0d80vkSOby/nUh4LcP5ZPb18gkcl/gP46c6LtenaMcb8Yd1Nbm0OyEj+cRI67b2VA7mx3xzrMwRP0H+TOcrNq5r91J8kf6MH5InnXXw/OLAZSXeA7u8XCSU9Tk/ulQkFDZDqfwQ+21u3Uysg8l8VZ4vu0k4fGneNMMWu7bmNtml/YntEa6nF/qHcvgsvsdP3JuEYd2J8u23YZjbQe5E3LOjV/o8WKQ7+IlfkLnGAw743B/lldxXZa51iJNF+qTHXezuoHdiU0k3mMmB7PSiO0yALSF27+3PN18Kn8bfz0dwiD5VObw/4xO2M/9n7a/xPPoh3hRi/fvt5GkF5lfU3yCffL5YDu9vx2BrYH8sTeIuur7I5wfoh4Sf4riOPkU79mrQoe2pG7MvkMzlMH4InzTu3lwkDOOGdn/IrF22dP05P90XjjRWOXb19OH537PL54qEaD1K4mKRkM0PR1/CT3rdHD6FuSOjzxuLhO4gIFEkOP19pPYn5uPhVNXzk3z/a/72bX6C+rNxfX2+D+8uElprs+2yo0fABzaTxEq77dGvSTXbxeMOp1/4mh4Rwh6ZnK+PA/29SRWuvcZHD1JfRN2m9FaRk9usmH/g9itFwvt4RmsJ039+vPksEpBfsXasj18kzHLcIoHFKWwnfF4J5FJ+IPxUxy37IbPj4x/g7Lbbx3ZFgmvf6REhLMeZDuLTma8bF5V8VRrX0T+xVzP5pMZPrA+QUYl3Mm7U8+UiIadPjAvraSU/TKNNE0mum46ckj4X5vvrRUJmf3KjPFA/KfW/4G/f5idX+gft34XxNx7wGt1Opj+t6X47cmwgpt+TEPoqPm11s7CPRS3mmM6e2K1tqmrRGWQ7HTe+BLowbv+MyBmcXpAzd0c9A/17UUz/AKB6pP5g+4T6VOWw/oxP0s78n8bFBZ6pH4YHwN3mstPBGgz6ldN+wOjP54vl0P4sTlk745PZl+UTKh/Pl83ryriw/YId8QFe1znWHw0fyLFz6XyV2BHHqR8X6XyVG/dENb5wf84Py5NBvmLHB+l4t0jxeaG9rE/hcaPqelrND8N3c+fDgAfHvjV9ivO92S1WF0afLRKq+xPGAxu32r+q/7f5SUbzgj5fCvucxP0nMMdlArl+hQ9nHbdVh7xuXRy/MT7WEpA1XTdjfTrvKT9becrJme7quMMInQFCHx6J6zdzTP/ZKFx/DzCm+EO7V/XJySEPCbIvkHbm/6y9zDP2B+7QQM+1/fz8mEdG53NWrBFqT9HTzTch59mfxWkyflP2RfnEke/5IeatMK7XXrMjm4CZ2eOL0HeHmc1xSuVAfpL2Stixlq/S42Keq/EFn0mYeGZ5kvEA+S87aKLZqlRtvxQwlSKhtJ5eyA/u+nJx3ewLxZo+9f3DHL7HsG09Pz1OXmeNeP/UqPH+xOMBGaDa32tH+DY/cfq/Y7qCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAjCHWv/mtUajjcufw7HG++6YRb64svFfCEj/+yfeRPetXH9d9f270jvX+S3DDJmrl391za0c/0hHH3YuExPOmeoZx1Vu3sgesLXJmL/tN/gb2j2J/2ITBJcM/9+/6r/VO0by5/sAnmgPkLg9s/6VRh3XJ8uf0I51+zC/Oqqn4M87/Cfz+kOP/l4uZBnSvJL8O2Fx62vC7gdyfH0KebzW8av3pWHc3Kq+rxn3eS8VfV5nbe7BH8heHWPFfgJyJPMb3G8c/ksP1Tj9Nv8xF8Hk/r8RVjbC/uqbf9gkWDCY22W3vnN2ken51yWjGZrs6+yj81XH3fZ9tkvlm23ezuzwnVTPOQ8+q9tnJGr/7b/tK1XmOjPwPRh41I9CQ9UzyLKduegei7bjra52D9vtxubF5LDQKdC+Gf9y/5TtC9r9+yCeGBxQdHrOYye96ukn1NjnEUCllO1C/Or1/y8z/MO/yh+mUjKTyVeqnmmKr8Kj1oYv8V1gdqRyKH6FPM55ae6DroD5OWU9SH+Vl43ub1e0SfDGxy4aMY63AHmPMn8k+ZnJp/Yqxqn3+YnyfW0uI58G0yF1NZu8TClE6kI983mrPvfO3m9/J9J+L4tz88LobEMFo6+Oi2i9xkMkzJdMpFaHZfsCK0YttiDkcBmgur/8MXlpSKB6RPwBoON7IyxnszfzCEei7SBT+a3uJ3oac8OB5e249oZwHlROWRe275vKzhDYfyz/uhrnpyyff32p5huk4R4oHFB8snIrvmI+r9rxNu0kGD5PH9COVW7WLCQTfk50TPwZ7j4If9k/FyIlwOZPHMtHn1++nWKxR0Z98p6hNqdIgH6j+MckZ/br7rxDtZNhvJ6WtTHifcDmXUzaa9Qn9p8u2PpjtNl29sGjsSPb8x3a61Vjqa5bmz9tV+1m2man8lmOrRXdUvyRX6CDsvguBl9vg1DUdddl7Pq58cYwDqtKRL6RdGu75aU5z3N/e9p/5iOwUCs2QHm5JgoEkLjVcdd294a2hx0qy4eaKYm3OQZp3xo52ySSgdGs5s7vKFNBuSB6Un9bV2Nj5ELvX6nCOVw+cxeoXt0/kn5Z3LIvLrH+cwxDuOf9Tc6pvynbF+vncQj5hPHBc0nJkfZFMT49/LS0TqVepN8mj+ZnKpdhs/Glqyfu3o6/gzil8Qd4T+Qj+f1/FYyz1Tks3hn/uDZC457fT2KHzfy9KH9Iz8f+XHi/YUiobAZyunj+NvRmlg3E+tgSp8L82U3CYd952IStthcF59Mk/l66xT2W7JvYXyG9irvmL/JT9ybhG7cmIdvwzC3g9yJuGdHz5APFqnHAL8BJ2lO/1vEK7nny1zrECeL9EmPu9js3juxqcjbrCauYAubvE57tqXrnnp15zvtnWqbSMYD0ZP7W+9aJLuHcqh8ai84zf5r+Nhy5j/eZHRFArwHcooE797od4qE48MEDzAu3HxyfrifV+aYfz8vPaXNNd0on+VPLqdqFzs2y3uxn/t6looEHneQ/0A+n1chzxTk84KE+INnr3AvWFyP6Dpl5AT+M/QP/Rzkq8ub+0GFS3Iq+lB/e36aWTeHT0d7pfV5Y5FgC+9MkeD0D2Dmm/GkgvAAACAASURBVNknDI/N+PuWQf7Nt1d1v/xtfkIfqwbj+vp8H95dJLTWZttlR49gi2xHYqXd9kg8V/fCuMMpzhHsHQeIEPbI5LTJi/T3JlUo4ucbaG9cWCSQg2EkhBPb30KQmR1fv1QkMD3JGhD4J9x1gjNFPK/5Mf7jKBvyz/of4yT9p2rfoN183Ck38UniIumoqFvKr06Et8uTlngig5yqXW61vPfpIqEad558f16FPFOQX84Dnr3ivWB1uLg98J+xv+fnkP/X1sGX5LygD5hmct105JT0uTDf3y8S4jxJ+4f7Fip/ak/kt1HBr/KTK/2D9u/C+BuPH5yMrHP8dN7RHXbu2zK08dORW82nrW4W9rGoxRyz2BOX6Rcx6NrUlMOZHx5dGHezP/UxxYMX5MzdUc9A/14U0z8A2ul64+KyB/DAhsD+Zpfn/lCB8c/81vVnoife1Eb/yUymSHDn1T33bB4zgPzT/ueEcv5Tty9s9+IxPIA30nK3grH/B3LcZb4/gMP5k8mp2sXJexU/d/XMFwncPwedu/YL8VLKMxX5uTxwwrMXrglr64LTzuRAfVh/59aCPd3K473wuFF1Pb2kz422p9dNh7eaPsX53uwWyxj+00VCvO5P+wTon2zczL7C8hOvm9N3v8pPWP9gXLf9+2Dvme4/gTkuE8glJXw463g645Bn/WaSND7WEpA1XZtifTLPzJ0fjWOecnKmuzruMEJngNCHR+K6026q/2wUrr8HGFNwXEdPwkNCz/4HXofo+28HbHqH82IPFfKHDYGe2BFd/4TzYl+A81rb+c+JHsC/0/+C/5Tsy+Vju3DiaFygfOKHUexXw3eAnzP5LH8iOWW7eH5V9HOiJxuA2ZHGHeHnWrzk88wF+Zk8cP/EsZczQHFdSNixC65EHuD+AP43EYcHHAPZjU1hPb2gjxvvF9fNvgCu8lPdP8zLyzHs4XxPG4/qH2dlpL8HZ91HeTLhtyw/2/6we5QHRnybnzj9U+vF31EiCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIg/DrW4XWhJRxvXP4cjjfedcMs7AWa5pPcpM7+mTfhXRsXvWsYvn94eLWgnfK9/8y1q//ahnauf4SUnzj6Qz2d/s64ydcaPr45iWTtuflm4wX7Q8D/ZK+rfjKLn/kPeUjyyebly8d6En4uxAVDLd65ru64wI44j3F7sf5MfvfR0+Uu8FbSJxXv/E3bcbzH+WqMR6b/e9q5PvR98iRflfzhgh19+d46cnUzMMrJxRfTk8l5Dz/cjlV9ruSTHncJzrfX9vIeq55XHwNPX3H8fM5Ljnw3vwF8m59U86fL21+Atb2QF7b9g0WCCY+1WXrnN2sfnZ5zWTKare3ov7aE+erjLts++8Wy7XavZlaOboqHnEf/tYEw5vpv+0/beoWJ/jnEfsL0Z3rS/s64y7ZDNVA7E5lzWjbfmAfqDy7/wF49b4ecJM8J/jkPFT7ZvByTYj2pnGJcMFTj3Y9fMu5sR5rHCA+0P5E/TvIsEmq8sbzE9GHymf+X4z2Vr87+TH/Wzvin/Yk+tP8NxxHj813rQiAf9a+tmwzF+ErGRWivcn7g6/gr+mTyCRz4pWU5g2JevZG8R/P2Q9qYl5z84OS3Gd/mJ8n9WLwufzVMhdTWbtNjSidyUrJv1p3uf+/k9fLnk6R9W56fF0JjGSwcfXVKdvcZDJMyXTKRWh2X7GitGJaUwUgguVP9H744KFzNRsxPzOEY83g7UobnrpWMa8/krGux9m3ftxWcQbB2Ol8eLywu7FQyRQK012hS5Es8LCxvmH/GQ5VPr0hA/em8Ev6ZiQuWf6gfkv60ImXjsrjrVbPDhvYdJNG4Jv55IMNbJi+xmbEJp/KbE3dQKunP9M+0D8UG5oFvGmB/FkfOd9+yLjD5yf7huslQXk+JnkxONR8icVndMvrU5tsdS3ecLtveNnAkfnxjvqNrrXI0Xc2riX3LwH+Y95z8UN2SfIufuPmT5ZMXD+X/GIZDq+56mlU/P8YA435rGevOIa92yfF5T3P/e9o/puNkEGt2gDkjJ4qE0HjVcde2t4YuyrpdNh5opiYsEuYTNb6ZqBXwvZ/c1tXYGgmaw8LnuevPx2VfJ+3d43DmmI60s3GpPk5cYH8g/FN7mRjERzu5/ox/xk+VTzYv2p/NK/TPXFyw/MN5IPmKxi8el9qx/479V2Dfvj+R78bL8c0Eb3FeYisc3Ipm8xvXH/uDM1+mP2mn/JP+1D85b8GyMvD5nnWBy3f6F9ZNhvJ6SvSkcqr5EIg7v+yugyl9LsyX3SQc6WUuDmHLoTYrJucBCnmV5T3Cf5z3kvkwxjf5iXuT0I0b5/lvwzC3g9yJuGdHz5APFmkmnfj1VyZoD59Xcs+XudYhThbpkx53sbuk3onNKUKDx89AdrpIGCaAeTj1J/NlfnIbTTyrOuccP5kOewY6Lvk6ax8KmGMU3M7GpfqkEhz1Q+M/jr1OpnfwiMnsJ7C/UyRAfoZuB7z+aF5Of39eg5xwxNFAJH4xD7w/jV84bhh3KI95PEz9sXw/XvK83aK8xPOwc+IG/b8Q71bQs5an/Zn+fF6Y/yg/d/r4/Z2948znm9YFKt/v//jw+uMQF4sE4OdUTjUfTuLApzAmM/q8sUig4UCKBKd/gERedfIeWV/cvMfsUt0vf5ufoP5s3Hi9+y68u0horc22y44ewRbZjsRKu+0RP2/6yrjdqaQ5sOw4QISwbdO8KQn09yYV8eNsjumjI1j/UE9+7/BikTBKehYJqP0zRULNP4v2Cm9Dn3/D/DN+Hv9I88nGrfYP23NxQe1CeKD9WfyScftvgg1/IY8F/a38xCY7nU9oXvL1Sfoh7p8sEg45vD/TP8y3g57v7E+mA/l817rA5Dv9h4/5px4urKeEh4ycy/nBk1PS58J8f79ISPAW5D0uh5EAt+mV/2/i6/zkSv+g/bsw/sbjBy/q3Sajv3Oyv6nat2Voc08jKj5tdbOwj0Ut5rjSnlxOv4hB16bmuCbzw6ML43bPWJjiwQty5u6oZ6B/L4rpT0D8ZLpSyJxve3qO/al/jqMPQuDjMZ2qD7GknY1L9WFxwfwh5p+vNAneeH/CP+Pn8c8KnywuqHykZ8BPOi5o/iE8OPkKxi8bl6nK8tigc+czfv8hgzjxMinj6M/yUqDPJN/Nw4V4J/5A+zP9M5sqK5T1Z/5ZvRhhfL5rXQjsNfUvr5sMxfWU6pmQU8iHaL7QjmV9ivO92S2WIfrTRcKFvOrlvaf64X6DNWbyYar/L/kJ6x+M67Z/H+z90P0nMOjm3U4GPpx13Eod8mw+nSSNj7UEZPXKUH2yz3TiZHfKyZnu6rjDCJ0BQh8eieuqfKr/bBSuPwHxE2OZx7OLdnByt0D1BP3JuMwhYPv9aeXz0crz6Qzc7oxL20lcOP7g8A/s5blJyNu4bxvaHR5KfLJ5pXkmeiaemUnFRT8G80PWH8avM+5ttqOTxyAPbt4D8kc5nX8WeUMKRfqgeHfyYSHeb8wfaH/mWKSd+iH9IM7/0NnsJx6f71gXuPyU3XPrJkNhPU2u7wkzHvKu5AfyDG1Kn/r+YU4nx7BtPT89Tl5njXh/D6W8ihW9BfzD/QaQH+WTEd/mJ07/YvoRBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQHKzD6zlLON64/Dkcb7wb33VPXmG3mC9k5J/9M2/Cuzau/+5CI2Z4taCd8r3/zLWr/9qGdq6/i0lO3HscA843LWh+q/vYjv3kkv6wP5PvjDvw8Ii0SrA4/tANcjV+bxm/Im+MDuebHdt0h/PN8TbyUM0DKTVz/gz61+OO6e/OC/itH0MzrV4c8Tw2ZSr27UK7r2eN/6I/JOKOIRuPV/IhlT/aPVxH6uNmUVtPub8xOe9ZN3k8VvWpznfGXYLz7bW9YY+F7R7kpYJffZq3b/OTal76dNx9Gmt7YR3d9g8WCSY81mbpdd4k/pzLktFsbUf/tSXMVx932fbZL5ZtP9rsuFa6lfPov7ZxRq7+2/7Ttl5hor8PIMfBsu3oxe9wvr4YGE+wnfrJBf1RfybfGxfxUA0W5g89rscv8yvmz9X5BkNPdmTzzfF28lDOA76isOw3Y43nF3P/Ytw5/Dvzmv2WxZ1jd2ZfL49Nchjl1XYm38sniP+qP+TijiGOxwv50JE/253q7/jtW1BcT6m/MTlE//K6ydfxV/TJ2BEOfGlZLoD6mztw3q8+zdu3+UlyP/bn4u4jMBVSW7ukY0onUDndqyebdu5/7+T18n8m4fu2PD8vhMYyWDj66pTc7zMYJmW6ZCK1Oi7ZOVkxbBECI4FFner/8MXl1SKBykH2tWd49iM6XyKH7jgTO9FBU6y/URTvkzhR7BPbznjY9n1b0ZkI1+fAtPtD8cvk+PLnZNfPC36ame+N5Y2EHe18KW88jzH9qT5X/XksTiAP3J1YXmX6O/Oa/TbIM5HdhxWV9cRFArJXtZ3JZ/Ny/NB+t1u8r8bdmK+IH0L7UrswfVz5fr7iy2Yib5yfpZaN8npq0OcTLIfqb5BZN5PreKhPbb7dsXTH6bLtbQNH4sc3pmPHn9YKR9Pe+svzUt6v2Fjv4a0u/w/5iZuX7LgZfb4Ntpjpri+3nVY/P8YAY35fxnOmcR9oSXne09z/nnaPntku3gYZz4QH72T9IiE0XnXcte2tocWvy8p4oJmasEgwTvnQztnc55ILlOPZF9OI50vlMN4onx0lof63dTW+Op2wzv2JfK8d8dA9ntcd72B9OmnDOQmKXyrHl483iyyOzu+MGsITXJg3QjuO/gZ54zwQ/Zk+ZX82uXFIfbg/iTsnr0Z5bHq2Cvutm2e8IqG3r28vUCRgP6+2cz35vGgah/m5FHfMT6gfcvsS/bE+VH6cr9CRidEnk39eKBIKm6Gh8iFynLg7WhPrZmIdTOlzYb7sJuHwjbmYhy32ScbEyTTxN8JD2a/sQJ/hrST/D/mJe5PQx1ekz9dhmNtB7kTcs6NX+jxYpJ438etnfGgPn1dyX5W51iFOFumTHnexq1/vxGZj2uDxIZCdLhKGCWAeTv3pfLEc375kUwXmy+Uw3iif5mNyYdDz0A/NTsRhymA5fmwnRQK+1yH6gJ43Hr+OHF++e6IM4ig7X5o3AjtO/o95c3jA+hN9Lviz/dI+PeLjr3k27hKrI8tj084R+a2fZ5jdJ/sG9kJFAvTzajvV05lXxH/HZynumJ9QP+T2ZfpDfbifh3l+Xkew30Y8JHFx8wfyNpVD4+75aWbdHD4dJ5zW541FAk3vpEhw+gNE+417H7unrfrVo/VjvJXk3/6Mn6D+bFxfn+/Du4uE1tpsu+zoEaYDNiyx0m57ZJ/PuzbucCp5BHvHASKEPTI5FQmR/t6kKtdeU1ryeoITZThfdxOJeGPtt9BPBv37fUJ4gujIp+PiImGMvMcuI9Jnuq4kmxIsJ5IfhOR0OZOdb9m+pqXTEfOWKxKMGpcSRbyGTZec2f7JAKyovXBPQVttssihqtCxF3zcCPh5uZ3J9+ZV4r8Ud8wQ9SKB5kOoT2aCJM+HT0t449ZxYT2F/paTA+hNrpuOnJI+F+b7K0VCuN84PnA1OwG33R/l7dv85Er/oP27MP7Go3tmejHdzIlF52WmJnh8BfyehNBXKRKsbhb2sajFHEPZE6l1qFxu6NrUlNWZH9BcGLe7NTbFgxfkzN1Rz0D/XhTTP8a4+eD2DQ+GzSwcOZA31s78BGswHb2HvDH53rjkJqF7Tv34AY+vz7xpgvHL5ITy3ZNg68/V+bp5A9sXzZfw5uUxqj/Up+rPwxw7ZYndYdwxfZj+bn57SjeNQZ5Bhw7Mvp694ONGwF7ldibfmxfhH/BWjjvmJ9QPmX2x/lQfKt9RlTbOSgc8FB43qq6n1N8SckB7et104rGmT3G+N7vFMgH86SKB9Y/3Azm/+jRv3+YnrH8wrtv+fbD3SfefwBz7DHLtCB/OOm6hD3nW/yZJ42MtAVm9MlSfznucZ3Bxsjvl5Ex3ddxhhM4AoQ+PxPWbZqb/bBSuv4dAjhmbG4zOF8rhvKF210+g/uYbj2et4cDm/gLJ5+PCD+5PE5+PduOnHoA+8OzfCD/jl8lh7dyvkD8X5zt/FNiRzNfhjeYxHo9Mn5o/E3dO8ZDSh+nv5BkYp1hRbPcgjoC9oBxmr2o71ZPPK5V/4DMJybhj+Yr6YcK+/WEo1ofLJ3Yn+hO/dXmoFAml9TS5vuM1IfWceWrdTOQ3Z17V/YOzfLX1/PQ4eZ014v1To9L1N5VPHgqPxyIf5e3b/MTpz8Z19REEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAg1uH1jSUcb1z+HI433nXDLPRFrov5Qkb+2T/zJrxr46J3y8L3zQ6vFrRTvvefuXb1X9vQzvWvzvdGXiMI9XTmxfTH7+Ot6s9B3/f7spxHRJGgYHZk+tT8s/Ce6kBPX/7kmZeTAMg/b3s9JeECyq/GL9M/slc237r+OcV1VQ6JozD/ZPwqqxR/Q3xS2NC/GncX/PYCP868Svpj/wzyYdpP3pdXq/kKr+/FdaG8bvL5VvWpznfGXYLz7bW9YY9V5aG6/rJ1jfHJx7iV8tu3+U91fX/XPuTTWNvVIuF2u237B4sEEx5rG95fjzhd25lDl4xmazv6ry1hpvq4y7bP9l+23e7Vjo+tdCvn0X9t44xc/bf9p229wkR/Cqf/su2T2zA92byo/n372b+qPwHjvwyiJ3O9pB1jfgjovIp6BvKBfV9JAn3+QX51ATDumPxq/I7fPgSm7JXIt8wfbrcbjGsC6g8kjpJxGre7wPpX7Y76V+Ou6rdBfr7dbrPd2bwq+lP/dPNh3k/elVer+Yqu78V1obxu8nX8FX0yFMKB30U/Q5WHat5g8imfbl7N++23+c+L6/tnj9vLMJVQW7tFy5RIpLLcN5um7n/v5PXyfybh+7Y8Py+ExjJYMvrqVCTcZzBMynTJRGp1XLJCWDGsmAEjgSCk+j98bvlIkWDPwAYTQz0trESm/ziVQz7Tn/gba3f4Z34O25me275vKz9Dmfhhcqh9jQGGHS2c1wU9GQ9QfyonYRebfxy/wvoQHljcBX478ebt8LD+xF4s3ybsOBQVOK6L/pDJAzzdnvpQPZldiP5Vu7P+1bgr+22CH8sz0/Oi/pN8x440/8N5ef4A1k2G8npq0Ke34rpgkFk3k+t4qE9tvt3xc8fpsu1tA0ffxzfmu/fW6kfQlc3ulbzhrvvj/o2EFfbbBL7If9Lre0af34ItWrrr722nVc6PIdo6rSkSzvkOftXlo+d9zP3vJQc//9XF2yDjmYFHxhNFQmik6rhr21tDSb9bRfFAMzVhkWCc76Gds0i8+LiRQ5eTLOawgPwb3+tci+jD/I37Ieaf+Tn1f6Jn9xiePZZh/BA5nJ/VzP2nYxT6VVFPJ96h/kwO4Z/mn3GadlaYf8IDjTvHb/PxS/Un9uLz5fpDe9G4rvpDmAfmIgTGI23H+lD9b0W7k/7VuCv6ba8tOUMbOzt1Zk1/5J/EjoxnOq94XbtUJBQ2PUMlXFwXbqZzYt1MrIMpfS7Ml90kHHYfDq1Yi80J2RPoAg+X8oZXJPT7N5KfvfyQmBtW5M/7T3p9j/X5NQxzOEicCHp29EqcB1vUkyYegSeNByvwuT/vpIOcdofBwxd1V5/0uIvN+r2zmoWzzWriGEkXCcMEMA+n/pfme6lImPYeNMmeTrSTq8dDH+Zvjh9i/pmfe/4P9RyW93lHMfMD5VB++qnZzSXzq4qe0ZEG2mwBOYx/ln/gP5/jYU0IDzTusPxhsCh+uf7YXs58mR2pP5C4rvtDN+HEWsniEbcTfby8VLI77l+Ou5rf3qaeRjLluVgkePo/hgrzOeE5Ma9BThUXi4Rpfb+8LiTXzeHTcb5pfd5YJHQHHIkiwelPkefhWt5g8ic+WX5O7VsS8n/Zf9Lre6jP7+HdRUJrbbZRdvQI4CATSay02x7x83avjDucQpnrcjMjRAjbFs9394H+3qSq11vpZ8e5iefztwT/Ic9lByD8X3ekoduQ1TNFApRD+On2CVZ46Fc5PS8UCRU5bysSKA8s7oh8oJ8nxysSkL1of6o/06cTMtw7vMMfzEydVJiIx3BesAp5Q5FQi7uL/g/yGOe5ViS4+psBkFIJPynmsSourKdwfb+8LiTXTUdOSZ8L8/3eIqFbdy7lDbo5RgfxLD8fPXKbsW/znyv9g/bfwfhbDvh4xrBZ6bzG1ATHwjn9boTQVCkSrG4W9rGoxRy/2JOY8ZcyN3Rt2h0jxwcoF8bd7E+AzObDC3Lm1qhnoH8viunP4PW/UCSgnUHIv21n+jB/Y+2Mf+bnjv8zPfurfFx8QwwJDvAzHUXbzXG4eGT0DOYLH9uI5eApjjGO/Arrw3m4kbhj8ovxy/XH/kz6u/rbuQdxXfeHIA+4K3Qcp5l5pYoE1w+JHUtxV/RbqrwXd8WbBKgP8884n/eqsnm5cgqPG1XXU7a+V9eF41vJddNZR2r6FOd7s1ssY8jfKhIYDxfzBjrExHw6+Rlo4Ezou/yH9Q/Gddt/D/Ze5/4TmGPdIpeR8CGs49bokGf9aZI0PtYSkDJdj2J9Oi/hz4aSZHfKyZno6rjDCJ0BQl8dieuqcKr/bBSuf22+zDCOnnBeVH9OD9Mf+ZvTTgdgDxvidiTm/tT5+ciloZ/yk5mvaTeR8RgIWSBwN0dPNl+of14OUfPMP07AQ304DzcYd1x+MX5Z/kz589mf6u+lBxDXZX9w8wCIUyaGtLt2CReL2O6wfzXurvgt44cQweZV1d/xT8eOQf5nqubWTYbCeppc33Prwu3yukme/UvpU98/zOnkGLat56fHyeusEe+fGDPDw/iNOG9g+S6fZF/UfRAUP9/mPxfWd1cfQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRCEvxz/97//6Y/+6M+/8+e3U44gCIIgCH8Dfn3Loj/6oz9/8s9vpxxBEARBEP4G/PqWRX/0R3/+5J/fTjmCIAiCIPwN0NZBEP4FKNIFQRD+OczvZxU+gf8qz9o6CMK/gHykr+0HvKwxieONy5/D8Wa78V335FV1i/lCRv7ZP/PGu2vj+u9sJa8X/UGv3Jy5dvVf29DO9f8ojmFf95WavZj/cDnvsQvnuapPtT2PuwTn22v7cGynccF/WDwy/m+3G4gX572gBLF9QV6qxinQ03yEX32c9EMmn/Vn/LyLNxhfTjyysWv883xrPu++4PWv6OPyBvR8e974siIB7E9+NS+9tF/6AL5Nn18A8we9zthHKdLX9oKTffQ8xZh/bUOKRWZf2xkwqZOetR39x6wPUR932fbZRZdtP9rsuFa6lfPovzawHHL9t/2nbb3CRP8/gzecvRXtRf2HyenbD23LduF+8oo+cTsHVOhX3aGMvP+weKT83243FC8sTp1xoXw/L1XjFPQ33xsPU0p+SOSz/oyfd/HG4ovF443k2yr/NK6dQUn/kj4+b8DuH8gbX1Yk3F7cn3wA0udvwLLtosVFFOnmhKmtnZOZQxpwUnI/+bAZ8v73Tl4v/2cSvm/L8/PCFmUZVoboq9NicJ/BMCnTJbNjqo5LPNWKYVsfMNK0CDn6P9bd5XqR8Lgm2ozhl+FT4Cesnc8U2IWhbK9+dLNpw3JGCpHtMnZJ+kmoT7WdjvTTwfpe28DRIjy5f9w8tOwR8Z/ynwDsWwvYbo3xkolTZ1xI0CCnGqe4//nhdG5c8cNI/tif8fNm3vhm/TbEY2JnkOI/GJdssGEeqOjj8Ab1/ETeeKVIgOv7sNQ/NhI+LWR/4twonl/Yd8s53reYQ2Yriu5P+H6JkEDyHhnX0R+3V/ZvjP9qPh/kt/VxiWFvpMf7NLQPrPZ3wPzB3iGEonwenPXI5+ew1zV/wH5b5MeHH+n2kLS7rtp2o2t/6tDHCywSThKHfXI3n6df3P+e3ub1keldxz/pHUlMFAkh79Vx17a3hi7SO+/AAyUWLab/eSI1bz6o/gDdAynmmIv5CWs3Cr61SCjESec/VI7RGR4BZjcTMc8pfartDthNwuGWs3VgyzGdzObvj/gPA8sDz09NO42XME6d0XH+6eRU45Tr+Wh70Q99+WhejJ838uYWCVOJQ/Ltrca/Oy71Q3TTW9KH8cb0/ETeePUmYV7fJ27dou/m7E+M4jRT2Vsaum9ZV7NXwc+NnfpH+sxgeY+Ny/Qn7cX9G+f/Qj7vyiyjp72tMjfweB9Y7e8Da17ZHjAe/PXI18TaseoPzG+v8cPgRvoQoweb1Jm80vmxCtEd/LQ+gQwxFn7wOUq6i6THOvGTiyRpRvqkx10286ONfjNqEkSDHg4PCpNFwjABtqQ99afzHRLoQyvmJ9FicO0kODPfAJP/OHJO593BIx1Juwyfjn6S1ucPFAkwWRvBYwvq/9X+A/IA4J/ESxSnGG6+mlb4fJz6/aPnelJ+6Oozz4vx817e3Md+5toK5NtOWJp/PC7nGUVQSR/GG9Pzw0VCbh2MSLg9k899t+NvJ4CESU+UB6yqZq/FBuq3KPzgMKXPDJz3vHGh/qS9vH+j/Pv5f8Qw8f5ADcjh+8Bqfx9vKRKwPrXi9sbsWPQH4rdlfoL4/aNFQmtt3vM4mpdW+u4AgEustNse/TFd+l4jOe5Q7R3O3HHAsyoSPB5GBfp7k4r58WisJgAACwlJREFUwU84/2aRULYX9J+cHMBP0i6OnJI+1XYHnywSnEG/wX9i/icVkQK5xHUxX1XjdOqfIOc6D6w/4+fNvHEJg5Ys387iwPC5mwSPZ1gkVPSJeRv98/154/WbhFntte3bsrZ9W9u+rVG+ulIkdF8/T9zZAt0/4fKBIgHlPX9coD9prxcJjP+XigTzT3p4RIiq9vfxTUVC19vcJJT84fqGtoTwcSPLyQ+ZjE1e44NUx/cfX+na0OldJzXLtNXNwl4D2dsbdmVzNMx3FMdxTXQQd3Fce7eUreBZ/kA9A/17UUx/hufZw/HP44QY+wlrP0YkJ3PZM4OqvZj/ZOSA9rRdGM9lfartHP3zuO6JzjnI60XCH/AfPC6MR8q/HcOwWZ2vk694XgLjxnE6b22QF5T9kMhn/Rk/7+ItkIDiEebbKv98XDfa8HFPQZ+Yt+nq5O154xNFwm1trbV9W25r2/fwTIPuT+6Y80DX0N+8gX2L3T6iyzHf6HHssLzHx2X6k/by/o3x/8qhD34sqq/12T6w2t/Hp4qEaD2awe1Y8we2377GD0MU6d2tSDsfL+PXcPDh1uM29ZBn+J2vOcbrjyBZTNcrWJ/ss7B4M3rKyW1Tr447jMCuFx/zHppG4jqHo/rPRuH6s8maR2rZD4zwf+xn2/1rr0qRULJX0n9mXYj8i3bpE0FNn3o7xeSGhzptPT89Kv5ZU96f4o/4TzDbJP83HC+uQwzw5Dv5oRync3+nfKr6IZCfnFc+kNK8OfF1ozV7lG8TfLJxGc+uniV9PN4CP3lT3nihSODr+3KUq7l9Dt6f0DyANhvTfM0HxuMeCaqtrv5sv0TA8h4Zl+pP51XdvyH+L+TzTn5rDaSUxy903X1gvT9G0h8CMR4PfD8Mwe1Y84cb8dsiPwG+7/8xE/5CvPx8kPBPQ/7zR/DeAyaB4Z/gWVuH1/Fv5L03P/3yH8Z3+oMiXXgVpTsHQRgg/xGEvw7aOryIfyHvnWf4/905vgtf6w+KdEEQBEEQCtDWQRD+BSjSBUEQBEEoQFuHFKYf0rzlMXFhxLt4/jY5XwBFuiAIgiAIBWjrIAj/AhTpgiAIgiAUoK2DIPwLUKQLgiAIglCAtg6C8C9AkS4IgiAIQgHH1kF/9Ed//oU/v51yBEEQBEH4G/DrWxb90R/9+ZN/wpywhq8PdHC8cflzMK/mM8Ms7IV03bv8MvLP/pmfp18bF71zE76Hc/jlfP9O8ImEWP+1De1c/3jG3ftvPaPPbw115kW+7naB3EUSM/w7YPa9Zsd5XFefgh1xvLwPzA/ZuN680GfMnxn/vl1mPbE+xbiO/Hmy10W7z0KpVa/wj3Ah3p35Dl8r5YFem3GA6rzYd67Mt5JPruWqcXg3bw/51tM/kTd+fcuiP/qjP3/yTy4vvfCWh4++TM684nJtQ2pHOXdt51q+ZDRb29F/bYk0Xh932fY5rS/bbtck8y7ubord67vbelvbOCNX/23/aVuvMNGfgsj3qF22fXInNi8Gpwvk00GJfwZm36odBz7j9oe0rB1pvLwJzA/ZuA7P2I7E3xj/gV2ieDk/Lca178+zvSgPrt1nsLigdi/KvxLvznwn/qt5gOar6ryK/lbNP++yL/VbJ2/jfMuLhI/mjeJ6mvTbkM/yulnlIadPhjc4cHVZrqK6P6EKkfak/2fWBab/V/lJeX2p56svgDlRaGtXJJijBnJys2/Wze5/7+T18ucTuH1bnp/XdnthkTD0741xn8EwKdMlE6nVcVEGH8SwxRKMBDZnVP+HLy4vFQlM/rbv2wrOLu2Z1mB6Ni/oV0w+45PJucA/83P7XZtcSnYcTWQSN2y/vWDHMV6gUYzB6E0duhlz0tzACeaH2CUTj4xn0D7pSXm+kk8emP15thfjgepD7ELjgszR96vZz6vxzuZrBqR+kskDjg9QO0K/LfpbNf+U7VuNL6eIRfnW07+UN85BUstGeT0l4zI5jj8fyKybVR6YPrX5dtc4HafLtrcNXKEe35i29T+tXbhKe3w9zMMXigTo/xk7pg4LyFi/7Cfp9SWjz7fBFjOrfdxo22n182MMYJ3WFAknD4Mfgn3JvWVtaQfvme3ibZDxzJyjJRJFQmi86rhr21tDi2u3C8ADJRZdpv9Zqc+bS6o/AJPfPWZmjwVuEY1TJQP9isonfFL/LPLv+Dnxq6IdTUx1IUPaX7DjUPmTeFxXE7P4JHWM35u/+RsyIOGH2tGJRz+uc5stxv+VfPL8Yu/72F6cB+IPzC487sycQ/nMz6vxzud7m8T1yOUBahcyL+7nNX+r5p+qfcvx5fgnyrdU/2LeMFO7VCQUNkNDBUvksHi5mc4ZPyzywPS5MF92k3DYa940wxa7RuQ22ZX9CeOH8sbWU8ZPtG46s/geP8mvL6E+X4dhbge54Azh3tErfR4s0h08yO+TT48HIvB5ZMorua/KXOsQJ4v0SY+72GzdO7GppBs8rgay00XCMAHMw6k/na+ziNJzbTdXJktqJp/wSeUU+U+V+MM1YtWOZ3Ds20Db3F60YycLXyMM/tx/ZLZ+rv+zzd8cp4QfZpfMosviGrQjPRn/naB0PplrQGwv7idYH2gXP+5uiH8sn/p5Nd4D/+RFwpWjtd4uYF7cb6v+Vsw/Rfteji/EAysSPD+Bcm7Yf0q4WCRM4zpy/PhNrpvDpyEPf6BIOFvR+bR3Yu3PbkJqfzJ8gZRR9nlQ6P/OJt5fNzG+zU8q60u87nwX3l0ktNZm22VHj9AdLHGJlXbbo99L1WwXjzucEuFrfUQI275MRUKkvzep6/wMWStfJIx7KlokYPmET7dIKPCfvAc8u1204yQnbk/bEcULH8j0NCNEPNDk6O4SzT+4XTLxmOYtSDXVvAHaqVXmXbDrJ718Zhcv7ip50ikS6vFOmzz+k3mA6M/aL+SB2nwv5/OknpGEQc5TOeCWkb0mOQn/iXFhPSV542IeSK6bjpySPhfm++tFQmZ/wvqzdub/nLdMvHT4Nj+50j9o/y6Mv/HonmW0V6jmZLqzane4tW/L0EZP+55Ssz5tdbOwj0Ut5tjEnqCsbTrVnM9sumPk+BDlwrj9XfUZnF6QM3dHPQP9e1FMfwoi/x6w5xDJx41AFYr9ypEP+XT9s8C/ow+070U7PuWEBzRMFLMjixccj9ZS/SFNcAuHilU4rsMPtIvjb5B/N96DVDPIr+cT1779R5lF/dSH24XFBeOfzZfGy4V4R/P1Z1rKA2G+4vPqUPW3Uv6p2bceXx4P5CYB6l/NG8eH+f8bq7Se0nETckB7et0s88D0Kc73ZrdYXdh9tkio7k+cfAjbqT6En6r+3+YnGc1r6/s3wt4T338Cc1wmkEtQ+HDWcVt1yDtY669U798YH2sJyJquZbE+nfc4z7rhZHfKyZnu6rjDCJ0BQh8eieuqf6r/bBSuf2K+NpP+mEcZjXjPYCQ2Z79y5A9khP7J+1P+iRxm35odaXdPTNqOyXgxn5hvPAjvQ3vsj/3QG9eZWGRHUOG7/HfPqiI9XXXqce2sNcc3lqktRQ+0C40Lh//MfCP+o3gE83XyFeMt1scKyc2Lxdhr863kk6KeqTwPG40YX/9q3qgVCaX1NJmv0nny4rpZzZ/MjkmGJvc5hm3r+elx8jprxPunRs3tT+g+Ic6TbD1l+Twk7tv85ML6UpmuIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAjCfxb/DwLd9njqvnRlAAAAAElFTkSuQmCC" alt="" />
可以看到这里为discard_passdown,表示dm会将DISCARD传给底层设备(loopback device),queue_if_no_space表示如果thin pool没有空闲空间后,IO请求会被排队。另外,error_if_no_space表示如果thin pool没有空闲空间后,直接报错。
[root@yy3 ~]# ls -slh /var/lib/docker/devicemapper/devicemapper/data
2.5G -rw-------. 1 root root 100G Nov 12 06:14 /var/lib/docker/devicemapper/devicemapper/data
[root@yy3 ~]# docker rmi dbyin/httpd
[root@yy3 ~]# ls -slh /var/lib/docker/devicemapper/devicemapper/data
2.0G -rw-------. 1 root root 100G Nov 12 06:15 /var/lib/docker/devicemapper/devicemapper/data
可以看到删除image前后,稀疏文件大小的变化。
dm.blkdiscard
docker还提供这个参数,默认值为true,即删除image后,会调用DISCARD,真正释放HOST上空间。
func (devices *DeviceSet) deleteDevice(info *DevInfo) error {
if devices.doBlkDiscard {
// This is a workaround for the kernel not discarding block so
// on the thin pool when we remove a thinp device, so we do it
// manually
if err := devices.activateDeviceIfNeeded(info); err == nil {
if err := BlockDeviceDiscard(info.DevName()); err != nil {
log.Debugf("Error discarding block on device: %s (ignoring)", err)
}
}
}
...
}
func BlockDeviceDiscard(path string) error {
...
if err := ioctlBlkDiscard(file.Fd(), 0, size); err != nil {
return err
}
...
}
func ioctlBlkDiscard(fd uintptr, offset, length uint64) error {
var r [2]uint64
r[0] = offset
r[1] = length
if _, _, err := syscall.Syscall(syscall.SYS_IOCTL, fd, BlkDiscard, uintptr(unsafe.Pointer(&r[0]))); err != 0 {
return err
}
return nil
}
主要参考
https://www.kernel.org/doc/Documentation/device-mapper/thin-provisioning.txt https://github.com/docker/docker/tree/master/daemon/graphdriver/devmapper
Docker内部存储结构(devicemapper)解析(续)的更多相关文章
- [转]关于docker包存储结构说明
原文:http://blog.csdn.net/w412692660/article/details/49005631 前段时间与同事交流docker的安装包层次结构,并沟通相关每个文件的作用,但是一 ...
- Redis-基本数据类型与内部存储结构
1-概览 Redis是典型的Key-Value类型数据库,Key为字符类型,Value的类型常用的为五种类型:String.Hash .List . Set . Ordered Set 2- Redi ...
- docker的存储结构,和以前有了很大不同
在网上学习这一块知识点时,有一个URL讲得很详细, docker 镜像与容器存储目录结构精讲 http://blog.csdn.net/wanglei_storage/article/details/ ...
- 【SQL SERVER重新认识】数据内部存储结构简单探索
数据库经常需要打交道,但是从来没想过数据库内部是如何存储数据. 今天探索一下数据库内部如何存储数据,从下面几个方面探索 数据库内部如何存储数据 索引数据如何存储 操作数据对存储影响 总结 数据库内部如 ...
- docker 内部组件结构 -- docker daemon, container,runC
Docker, Containerd, RunC : 从 Docker 1.11 开始, docker 容器运行已经不是简单地通过 Docker Daemon 来启动, 而是集成了Container, ...
- SQLServer2012 表IAM存储结构探究
SQLServer2012 表IAM存储结构探究 Author:zfive5(zidong) Email: zfive5@163.com 引子 国庆节期间,一直在翻阅<程序猿的自我修养-链接.装 ...
- 深入PHP变量存储结构 标签: PHP存储
1.深入PHP变量存储结构 标签: PHP存储 分类: 编程语言(10) 首先声明,我并没有去读PHP的源码,只是对于PHP的有时候诡异的表现感兴趣,找了一下开发人员laruence的博客结合PH ...
- PHP变量存储结构
首先声明,我并没有去读PHP的源码,只是对于PHP的有时候诡异的表现感兴趣,找了一下开发人员laruence的博客结合PHP提供的函数debug_zval_dump刺探得到了本博客所阐述的工作机理.如 ...
- docker存储结构解析
由于aufs并未并入内核,故而目前只有Ubuntu系统上能够使用aufs作为docker的存储引擎,而其他系统上使用lvm thin provisioning(overlayfs是一个和aufs类似的 ...
随机推荐
- FPS
游戏的FPS跟什么有关 http://zhidao.baidu.com/link?url=jZJoN_-MTC9fWCBSBaL5vezBp13MRSv8PdCcQMbwX5tiFGzxNEHe7UB ...
- autofac + owin + webform + mvc + webapi集成demo
http://git.oschina.net/shiningrise/AutofacOwinDemo using Microsoft.Owin; using Owin; using System.We ...
- 为什么要做url encode
因为 url 对字符有限制,比如把一个邮箱放入 url,就需要使用 urlencode 函数,因为 url 中不能包含 @ 字符.
- jQuery 取值、赋值的基本方法
转载:http://www.cnblogs.com/huanhuan86/archive/2012/06/13/2548071.html 获取元素的value值: /*获得TEXT.AREATEXT的 ...
- 黄永成-thinkphp讲解-个人博客讲解25集
整个网站的根目录用blog你要跟别人说起,自己好识别的文件夹名字. 下面的项目名称 就不再重复的写了, 直接用App就好了. 网站访问: ...../index.php(入口文件)/Admin(模块名 ...
- objective-c与c++的差异
oc的编译指令为 clang -fobjc-arc -framework Foundation test.m -o test oc中,1表示YES,0表示NO.并不是非0值都是YES,这是因为BOOL ...
- linux socket
linux 的 C 库路径为 /usr/include,可以直接查看源码,也可以通过 "man 头文件名" 来学习,需要查看某个函数如 bind() ,则只需要 man 2 bi ...
- InnerClass内部类
1,内部类概述 定义:把A类定义在B类内部,则A类是内部类.如下所示: class Outer1{外部类 String name1; public void show(){ System.out.pr ...
- Win7与XP共享互相访问及共享注意事项!
win7共享方法和XP类似,主要需要检查以下操作: 1,首先将Guest账户打开 2,右击文件夹-属性-共享选项-高级共享 3,将共享文件√打上,应用-确定即可! 4,查看自己IP(开始-运行-cmd ...
- 如何去各型MCU的官网上下载正确的数据手册
一.背景 感谢老司机左栋,虽然他一直很排斥这个名号 : ) ,可就技术上来说,还是当之无愧的. 弄了1年多单片机了,数据手册不是老员工或者头头给,就是从开发板资料拿.一直没有意识到,官网的东西才是最可 ...