Docker Performance on Web Application

Transcript

1. Docker Performance on Web Application JEZ@VVLJ %PDLFS
   .FFUVQ
   5PLZP
   

2. id:y_uuki ӡ༻ @y_uuk1

3. Docker Engine্Ͱɹ ΞϓϦέʔγϣϯΛɹ ಈ͔ͯ͠ੑೳྼԽ ͠ͳ͍ͷʁ

4. None

5. 1 DockerͷύϑΥʔϚϯεʹ ͓͍ͯॏཁͳ͜ͱ͸ͳʹ͔

6. Linux Containers

7. /FUXPSL
   /BNFTQBDF *1$
   /BNFTQBDF 654
   /BNFTQBDF .PVOU
   /BNFTQBDF 6TFS
   /BNFTQBDF

   1*%
   /BNFTQBDF

8. /FUXPSL
   /BNFTQBDF *1$
   /BNFTQBDF 654
   /BNFTQBDF .PVOU
   /BNFTQBDF 6TFS
   /BNFTQBDF

   1*%
   /BNFTQBDF -JOVYΧʔωϧͷ໊લۭؒػೳͷ
   ू·Γ

9. Linux Containers Χʔωϧͷ֤ϦιʔεΛ໊લۭؒͰִ཭͠ ۭͨؒͰϓϩηεΛ࣮ߦ͢Δͱ͍͏Ϟσϧ ϑΝΠϧγεςϜɼϢʔβɼϓϩηεςʔ ϒϧɼωοτϫʔΫ… etc LXC ͸ Linux

   Containers ͷϑϩϯτΤϯυ

10. Linux Containers Overhead ୯ମͷLinux ΧʔωϧͰ׬݁͢ΔͷͰ ΦϒδΣΫτͷڞ༗ޮ཰͕Α͍ Hypervisor ͷΑ͏ʹ֤ϦιʔεΛɹɹ ೋॏॲཧ͠ͳͯ͘Α͍

11. example/packet receive /*$ ,FSOFM 6TFSMBOE $PQZ $PQZ 4JOHMF
    
    ,FSOFM

12. example/packet receive /*$ ,FSOFM 6TFSMBOE $PQZ $PQZ /*$ ,FSOFM ,FSOFM

    $PQZ $PQZ 4JOHMF
    
    ,FSOFM )ZQFSWJTPS
    9FO ,7. 6TFSMBOE $PQZ ˞ 43*07ͳͲ)ZQFSWJTPSͰ΋ύέοτίϐʔΦʔόϔουΛগͳ͘ ͢Δ࣮૷͸͋Δ

13. Linux Containers ͩͱ Single Kernel Ͱ଎͍

14. Docker Filesystem -JOVY
    $POUBJOFST

15. UNION Filesystem -JOVY
    $POUBJOFST %PDLFS
    'JMFTZTUFN

16. IUUQTEPDTEPDLFSDPNUFSNTJNBHFTEPDLFSpMFTZTUFNTNVMUJMBZFSQOH UNION Filesystem Image

17. 8SJUF
    *0 3FBE
    *0
    ADBU
    FUDEFCJBO@WFSTJPOA

18. 8SJUF
    *0 3FBE
    *0
    8SJUF
    *0
    ͸࠷্૚΁ॻ͖ࠐΈ 3FBE
    *0
    ͸֘౰ϑΝΠϧΛ
    ֨ೲ͢Δ૚Λ୳ࡧ

19. UNION Filesystem • AUFS, Btrfs, OverlayFS … ͳͲΛ࢖࣮ͬͯݱ • Copy

    On Write: ݪຊΛෳ੡ͨ͠;ΓΛͯ͠ ͓͍ͯ৽نॻ͖ࠐΈ͕͋ͬͨΒෳ੡Λ࡞Δ • LinuxΧʔωϧඪ४ͷ Device MapperΛ࢖ͬ ͯ΋࣮ݱͰ͖Δ

20. Storage drivers -JOVY
    $POUBJOFST %PDLFS
    'JMFTZTUFN 6/*0/
    'JMFTZTUFN

21. EFWJDFNBQQFS BVGT CUSGT PWFSMBZGT

22. EFWJDFNBQQFS BVGT CUSGT PWFSMBZGT 6OJPO
    '4ͷ࣮૷Λબ୒Ͱ͖Δ

23. Storage drivers devicemapper, aufs, btrfs, overrayfs (vfs) Χʔωϧඪ४ͳͨΊRHELܥͰͷɹɹ σϑΥϧτ͸ devicemapper

24. Device Mapper ϒϩοΫσόΠε΁ͷI/Oʹ༷ʑͳม׵ΛՃ͑Δ͜ͱ͕ Ͱ͖Δ (҉߸ԽɺετϥΠϓɺϛϥʔ ͳͲ) ϒϩοΫσόΠευϥΠόϨϕϧͷ࣮૷ͳͷͰɹɹɹɹ ಛఆͷϑΝΠϧγεςϜʹґଘ͠ͳ͍ LVMͳͲͰ࢖ΘΕ͍ͯΔ docker

    commit ͕૸ΔͱsnapshotσόΠεΛ࡞੒ͯ͠Π ϝʔδ૚ͱ͢Δ

25. Volume -JOVY
    $POUBJOFST %PDLFS
    'JMFTZTUFN 6/*0/
    'JMFTZTUFN 4UPSBHF
    %SJWFST

26. Volume ίϯςφؒͰσΟϨΫτϦΛڞ༗͢ΔͨΊͷ΋ͷ ίϯςφ͝ͱͰ͸ͳ͘DockerάϩʔόϧͳྖҬʹ֨ ೲ͞ΕΔ ࢦఆͨ͠σΟϨΫτϦҎԼͷΠϝʔδ૚ΛόΠύε͠ ͯΞΫηε I/Oཁٻ͕Union FS෦෼Λ௨Βͳ͍ͷͰΦʔόϔου ͕গͳ͍

27. howto/Volume EPDLFS
    SVO
    W
    WBSMJCNZTRM
    NZTRM 70-6.&
    WBSMJCNZTRM EPDLFS
    SVO
    NZTRM %PDLFSpMF 4IFMM 4IFMM PS

28. Docker Network -JOVY
    $POUBJOFST %PDLFS
    'JMFTZTUFN 6/*0/
    'JMFTZTUFN 4UPSBHF
    %SJWFST 7PMVNF

29. Portmapper -JOVY
    $POUBJOFST %PDLFS
    'JMFTZTUFN 6/*0/
    'JMFTZTUFN 4UPSBHF
    %SJWFST 7PMVNF %PDLFS
    /FUXPSL

30. Portmapper ίϯςφؒ௨৴΍ίϯςφɾϗετؒ௨৴͸ϗετଆ ͷ iptables ͰNAT ! iptables͕ͳ͍؀ڥͩͱಠࣗͷϢʔβϥϯυͷϓϩη ε(docker-proxy)Ͱίϯςφؒ௨৴ "
    %0$,&3
    
    J
    EPDLFS
    Q
    UDQ
    N
    UDQ
    ŠEQPSU
    
    K
    %/"5
    
    ŠUPEFTUJOBUJPO
    

    EPDLFSQSPYZ
    QSPUP
    UDQ
    IPTUJQ
    
    IPTUQPSU
    
     DPOUBJOFSJQ
    
    DPOUBJOFSQPSU
    

31. Host Networking -JOVY
    $POUBJOFST %PDLFS
    'JMFTZTUFN 6/*0/
    'JMFTZTUFN 4UPSBHF
    %SJWFST 7PMVNF %PDLFS
    /FUXPSL 1PSUNBQQFS

32. Host Networking ίϯςφ༻ͷNetwork NamespaceΛ࡞Β ͣʹϗετͷωοτϫʔΫΛͦͷ··࢖͏ ίϯςφ͸ϗετଆͷportΛlisten͢Δ iptables΍docker-proxyΛܦ༝͠ͳͯ͘Α ͘ͳΓɺΦʔόϔου͕খ͘͞ͳΔ

33. howto/Host Networking ! ŠFYFDESJWFSOBUJWF
    
    PS
    
    -9$
    
    Ҏ্Ͱ
    ŠFYFDESJWFSMYD
    ͕ඞཁ EPDLFS
    SVO
    ŠOFUIPTU
    NZTRM

34. -JOVY
    $POUBJOFST %PDLFS
    'JMFTZTUFN 6/*0/
    'JMFTZTUFN 4UPSBHF
    %SJWFST 7PMVNF %PDLFS
    /FUXPSL 1PSUNBQQFS )PTU
    /FUXPSLJOH DockerͷύϑΥʔϚϯεʹ ͓͍ͯॏཁͳ͜ͱ͸ͳʹ͔

35. 2 DockerԽͨ͠ ISUCONΞϓϦέʔγϣϯ ͷϕϯνϚʔΫ

36. ISUCON *JLBOKJOJ
    4QFFE
    6Q
    $0/UFTU

37. /HJOY CFODI
    NBSLFS .Z42- "QQ .FNDBDIFE ISUCON4 ༧બ

38. Machine & Software Spec instance type: m3.xlarge CPU: Xeon E5-2670

    v2 @ 2.50GHz 4 vCPU Memory: 16GB RAM Storage: Magnetic volume OS: Ubuntu 14.04 LTS Kernel 3.18.0 Docker: 1.4.1 (latest) MySQL: 5.5.40, memcached: 1.4.14, Nginx:1.4.6

39. νϡʔχϯά಺༰ • είΞ 38446 (໿ 3000 req/s) • ࢼߦ͝ͱʹ +-1000

    είΞఔ౓ͷޡࠩ • ༧બಥഁϨϕϧ • σʔλ͸શ෦ϝϞϦʹ৐Δ • ηογϣϯ৘ใͳͲ͸ memcached • Nginx Ͱ੩తϑΝΠϧΛฦ͢ • ωοτϫʔΫελοΫɺNginx, MySQL ͸ී௨ͷ νϡʔχϯά

40. Evaluation

41. • Nginx ͱ MySQL ΛͦΕͧΕDockerԽ • ҎԼͷ֤ํࣜΛnative(default)ͱൺֱ • Nginx͚ͩDockerԽ •

    —net=host ͱ —net=bridge • MySQL͚ͩDockerԽ • storage-driver=devicemapper ͱ storage-driver=overlayfs • Volume ͷ ON/OFF

42. Result

43. EFGBVMU /HJOY
    OFUCSJEHF /HJOY
    OFUIPTU   

44. /HJOY
    ʹύέοτ͕ू໿͢Δ
    /"15
    ͢ΔΦʔόϔου͕ߴ͍

45. EFGBVMU .Z42-
    EFWJDFNBQQFS
    WPMVNF
    P⒎ .Z42-
    EFWJDFNBQQFS
    WPMVNF
    PO .Z42-
    PWFSMBZGT
    WPMVNF
    P⒎

    .Z42-
    PWFSMBZGT
    WPMVNF
    PO     

46. 7PMVNF
    ͷ
    0/0''ͱ
    TUPSBHF
    ESJWFSʹΑΔ
    ੑೳྼԽͳ͠

47. 7PMVNF
    
    0''
    ͰมΘΒͳ͍ 3FBE
    *0͸ϝϞϦʹશ෦ͷΔ
    8SJUF
    *0
    ͸࠷্૚͚ͩ

48. NAPTͷߴ଎Խ

49. -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j

    DOCKER

50. -A OUTPUT 
    E
     -m addrtype --dst-type LOCAL -j DOCKER

51. 127.0.0.1Ѽͩͱiptablesͷ DockerνΣʔϯʹೖΒͳ͍

52. docker-proxy EPDLFSQSPYZ
    QSPUP
    UDQ
    IPTUJQ
    
    
    IPTUQPSU
    
    
    DPOUBJOFSJQ
    
    DPOUBJOFSQPSU
    A

53. docker-proxy EPDLFSQSPYZ
    QSPUP
    UDQ
    IPTUJQ
    
    IPTUQPSU
    
    DPOUBJOFSJQ
    
    DPOUBJOFSQPSU
    A CPU 50%

54. benchmarker bench --host 172.31.8.235 --init ~/init.sh --workload 8

55. 32172 37931

56. ͳͥ userland ͷ proxy ͕͍Δͷ͔

57. IUUQTHJUIVCDPNEPDLFSEPDLFSQVMM

58. Hairpin NAT

59. )PTU $POUBJOFS EPDLFS  -*45&/
     FUI FUI MP
    
    
    
    
     FUI
     FUI
    

60. ETU
     MP
    
    
    
    
     FUI
     FUI
     )PTU $POUBJOFS EPDLFS  -*45&/
     FUI

    FUI

61. ETU
     "
    %0$,&3
    
    J
    EPDLFS
    Q
    UDQ
    N
    UDQ
    EQPSU
    
    
    K
    %/"5
    UPEFTUJOBUJPO
     )PTU $POUBJOFS MP
    
    
    
    
     FUI
     FUI
     EPDLFS 

    -*45&/
     FUI FUI

62. ETU
     "
    %0$,&3
    
    J
    EPDLFS
    Q
    UDQ
    N
    UDQ
    EQPSU
    
    
    K
    %/"5
    UPEFTUJOBUJPO
     )PTU $POUBJOFS MP
    
    
    
    
     FUI
    FUI
    EPDLFS 

    -*45&/
     -*45&/
     Ծ૝bridge(docker0)Λ Hairpin NAT modeʹ͠ͳ͍ͱ NAT͞Εͳ͍

63. FDIP
    
    
    TZTDMBTTOFUCSCSJGFUIIBJSQJO@NPEF

64. IUUQTHJUIVCDPNEPDLFSEPDLFSQVMM

65. None

66. 3)&-
    
    BOE
    $FOU04
     SFBEPOMZ
    TZT

67. 1. DockerͷύϑΥʔϚϯεʹ͍ͭͯ ॏཁͳࣄ͸ͳʹ͔ 2. DockerԽͨ͠ISUCONΞϓϦέʔ γϣϯͷϕϯνϚʔΫ

68. Linux Containers Docker Filesystem UNION Filesystem Storage drivers Volume Docker

    Network Portmapper Host Networking

69. Linux Containers Docker Filesystem UNION Filesystem Storage drivers Volume Docker

    Network Portmapper (Performance issue) Host Networking
70. None

71. IUUQIBUFOBDPSQKQSFDSVJUDBSFFSTBMFTFOHJOFFS ηʔϧεΤϯδχΞ৬

72. ! 8FCΦϖϨʔγϣϯΤϯδχΞ
    Λืू͓ͯ͠Γ·͢