| CARVIEW |
Select Language
HTTP/2 200
date: Wed, 08 Oct 2025 10:51:59 GMT
content-type: text/html; charset=utf-8
cache-control: max-age=0, private, must-revalidate
cf-cache-status: DYNAMIC
link: ; rel=preload; as=style; nopush,; rel=preload; as=script; nopush,; rel=preload; as=style; nopush,; rel=preload; as=script; nopush,; rel=preload; as=script; nopush
nel: {"report_to":"heroku-nel","response_headers":["Via"],"max_age":3600,"success_fraction":0.01,"failure_fraction":0.1}
referrer-policy: strict-origin-when-cross-origin
report-to: {"group":"heroku-nel","endpoints":[{"url":"https://nel.heroku.com/reports?s=WA2TUzfktgy%2Bu7YLSejUaRWdT71n%2BbyrlO0s9xKBWFM%3D\u0026sid=e11707d5-02a7-43ef-b45e-2cf4d2036f7d\u0026ts=1759920717"}],"max_age":3600}
reporting-endpoints: heroku-nel="https://nel.heroku.com/reports?s=WA2TUzfktgy%2Bu7YLSejUaRWdT71n%2BbyrlO0s9xKBWFM%3D&sid=e11707d5-02a7-43ef-b45e-2cf4d2036f7d&ts=1759920717"
server: cloudflare
strict-transport-security: max-age=0; includeSubDomains
vary: Accept,Accept-Encoding
via: 2.0 heroku-router
x-content-type-options: nosniff
x-permitted-cross-domain-policies: none
x-request-id: 8bb0bde6-0bd5-137a-d849-b4b4ccfceb9b
x-runtime: 0.303140
x-xss-protection: 0
content-encoding: gzip
set-cookie: _secure_speakerd_session=Py24PGei0iiyhCsIWl8FVWNN2LmHuhGC%2FTUzXwElNtGWchW3QSviXLs3UNbtGy85BoTl71zT6Pfo82%2BifFseRs2kInH9ca7oRCEQb5j5%2FFt4RSmv8jc9jvxO6xLyjMXSudTxYhCGURQLJ7oqSEbK%2BxzuW5Fu3yYFaCa0XOieoJtnSbt%2ByS7%2FY0RjeGkD6Eo0z7pOrOvrpL1J%2BrW%2FP%2B6O1KBk2AmhHCZrFeNvej8ij0adVIoMVBIVo0rdWg03C4yGRI7tAnfK2nhuCu%2B22rz6IbTe7QBidNWk0Xci5sP0q6%2BTAF8WyZDJPtwYC%2FL7H02PTL8y78ClthyrCo9bdfacd2S4nJtpg9QtDuFcZFPlYPdbUUBK4yTgV2AZXG1OU8vIuPDtjTEyue9Ld6ZNShcNAf4o--EdwZG0uddOw8XZKr--usL9ksWvs2kwwRzDJrj0eg%3D%3D; HttpOnly; SameSite=Lax; Secure; Path=/; Expires=Wed, 22 Oct 2025 10:51:58 GMT
cf-ray: 98b515ffee105917-BLR
Docker使ってたらサーバがゴミ捨て場みたいになってた話 - Speaker Deck
Docker使ってたらサーバがゴミ捨て場みたいになってた話
Immutable Infrastructure Conference #1
Yuuki Tsubouchi (yuuk1)
March 25, 2014
More Decks by Yuuki Tsubouchi (yuuk1)
yuukit
Other Decks in Technology
_awache
databricksjapan
.jpg){kind=avatar}
mdalmijn
task4233
tanoyusuke
henteko
takaking22
askua
spiderplus_cb
pfn
nrinetcom
mhlyc
Featured
shpigford
chriscoyier
maltzj
marcduiker
panda_program
tammyeverts
tanoku
cassininazir
moore
addyosmani
carmenintech
csswizardry
Transcript
-
• curl https://<jenkins>:8000/job/project • timeout • ssh <jenkins> • timeout
• Dom0 ͔Β ήετOSʹϩάΠϯͯ͠ top -
UPQVQEBZT VTFST MPBEBWFSBHF 5BTLTUPUBM SVOOJOH TMFFQJOH TUPQQFE
[PNCJF $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU $QVVT TZ OJ JE XB IJ TJ TU ,J#.FNUPUBM VTFE GSFF CV⒎FST ,J#4XBQUPUBM VTFE GSFF DBDIFE -
w QSPDFTTFT w -9$ͷϓϩηε͕͏Α͏Αͯ͠ɺϝϞϦ৯͍ͭ Ϳͯ͠TXBQ |-docker -d -p /var/run/docker.pid -H
tcp://0.0.0.0:5555 -H unix:///var/run/docker.sock | |-lxc-start -n 718e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652abfs -f… | |-lxc-start -n 542e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652a2b3 -f… | |-lxc-start -n 983e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652ad00 -f… | |-lxc-start -n 271e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652a111 -f… | |-lxc-start -n 323e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652a1f2 -f… | |-lxc-start -n 789e7f86d172fc7af7599e0f6472205c986fcc8bf51336ed1a5030900652a1f2 -f... w EPDLFSTUPQLJMM DPOUBJOFS@JE ͚ͩͩͱϓϩ ηεࢮΜͰ͘Εͳ͍ -
# lsof -p <docker pid> | wc -l 1021 !
# cat /proc/<docker pid>/limits | grep files Max open files 1024 4096 files -
root 31084 0.0 0.0 0 0 ? S< 2013 0:00
[ext4-dio-unwrit] root 31105 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31107 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31109 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1793-8] root 31110 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] root 31126 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31128 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31131 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1794-8] root 31132 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] root 31153 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31155 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31157 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1795-8] root 31159 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] root 31176 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31178 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31180 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1796-8] root 31181 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] root 31203 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31204 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31206 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1797-8] root 31207 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] root 31229 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31231 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31234 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1798-8] root 31235 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] root 31247 0.0 0.0 0 0 ? S< 2013 0:00 [kdmflush] root 31249 0.0 0.0 0 0 ? S< 2013 0:00 [bioset] root 31251 0.0 0.0 0 0 ? S 2013 0:00 [jbd2/dm-1799-8] root 31252 0.0 0.0 0 0 ? S< 2013 0:00 [ext4-dio-unwrit] -
• lsof -p <docker pid> • /dev/mapper/docker-xxx … Έ͍ͨͳΛͨΒ ͱั·͑ͯΔ
• mount /dev/mapper/docker-202:1-3019790-<sha1> on /var/lib/docker/devicemapper/mnt/ <sha1> type ext4 (rw,relatime,discard,stripe=16,data=ordered) /dev/mapper/docker-202:1-3019790-<sha1> on /var/lib/docker/devicemapper/mnt/ <sha1> type ext4 (rw,relatime,discard,stripe=16,data=ordered) /dev/mapper/docker-202:1-3019790-<sha1> on /var/lib/docker/devicemapper/mnt/ <sha1> type ext4 (rw,relatime,discard,stripe=16,data=ordered) …….. -
docker lxc-start … lxc-start … lxc-start … lxc-start … /dev/mapper/docker-xxx:…
ɾ ɾ ɾ /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… /dev/mapper/docker-xxx:… fork open open open open -
Device mapper • Docker ͷΠϝʔδࠩཧΛࢧ͑Δٕज़ͷ1ͭ • aufs, btrfs ͳͲͰସՄೳ •
ԾతͳϒϩοΫσόΠεΛͭ͘ΕΔ • ԾతͳϒϩοΫσόΠεΛελοΫͰ͖Δ • ࠩཧ • ࠩ͋ͨΓɺ1 mount -> 1 open ?? -
0 * * * * root docker rm -v $(docker
ps -a -q) 1 * * * * root df | cut -d" " -f1 | grep docker | xargs umount /FX -
• κϯϏʹͳͬͨLXCϓϩηε͕௫ΜͰΔ /dev/ mapper-xxx ͕ Resource Busy ʹͳͬͯͯ umount Ͱ͖ͳ͍
• ଟɺdocker rm ͰLXCϓϩηεΛ͏·͘ࡴͤͯ ͳͯ͘κϯϏʹͳͬͯΔ -
0 * * * * root docker rm -v $(docker
ps -a -q) 1 * * * * root df | cut -d" " -f1 | grep docker | xargs umount 0 10 * * * root service docker restart /FX -
0 * * * * root docker rm -v $(docker
ps -a -q) 1 * * * * root df | cut -d" " -f1 | grep docker | xargs umount 0 10 * * * root service docker restart 59 * * * * root docker ps | grep Ghost | cut -d" " -f1 | xargs docker kill 2 * * * * root docker rmi $(docker images | awk '/^<none>/ { print $3 }') /FX /FX -
• Refactor the devicemapper code to avoid many mount/unmount race
conditions and failures! • Remove directory when removing devicemapper device. This cleans up leftover mount directories! • Avoid extra mount/unmount during container registration. This removes an unneeded mount/unmount operation which was causing problems with devicemapper! • Clean up archive closing. This fixes and improves archive handling! • Avoid temporarily unmounting the container when restarting it. This fixes a race for devicemapper during restart -
• Avoid extra mount/unmount during build. This fixes mount/unmount related
errors during build.! • Make --rm the default for docker build.! • Devicemapper: cleanups and fix for unmount. Fixes two problems which were causing unmount to fail intermittently.! • Devicemapper: remove directory when removing device. Directories don't get left behind when removing the device.! • Add -f flag to docker rm to force removal of running containers.! • Kill ghost containers and restart all ghost containers when the docker daemon restarts.
