超個体型データセンターを目指したネットワークサービス間依存関係の自動追跡の構想 / A concept of superorganism tracing

Transcript

1. ͘͞ΒΠϯλʔωοτ גࣜձࣾ (C) Copyright 1996-2019 SAKURA internet Inc ͘͞ΒΠϯλʔωοτ ݚڀॴ

   ௒ݸମܕσʔληϯλʔΛ໨ࢦͨ͠ ωοτϫʔΫαʔϏεؒґଘؔ܎ͷࣗಈ௥੻ͷߏ૝ 2019/07/04 ݚڀһ ௶಺ ༎थ ௶಺༎थ*1, ݹ઒խେ*2, দຊ྄հ*1 DICOMO2019γϯϙδ΢Ϝ *1) ͘͞ΒΠϯλʔωοτ, *2) ͸ͯͳ

2. 2 1. എܠͱ໨త 2. ωοτϫʔΫαʔϏεͷґଘؔ܎ͷൃݟͷ՝୊ 3. ఏҊख๏ 4. ·ͱΊ ໨࣍

3. 1. എܠͱ໨త

4. 4 ɾاۀ͕؅ཧ͢Δσʔληϯλʔ಺ͷωοτϫʔΫ͕େن໛ԽɾෳࡶԽ ɾ௕ظؒͷΠϯλʔωοταʔϏεͷఏڙʹΑΔػೳ௥Ճ ɾར༻ऀ͔ΒͷΞΫηε૿ʹΑΔϗετ਺ͱछྨͷ૿Ճ ɾωοτϫʔΫαʔϏε(OSϓϩηε)ͷґଘͷؔ܎͕ෳࡶԽ ɾෳࡶͳґଘؔ܎ʹΑΓɼγεςϜͷมߋ࣌ʹ໰୊͕͋ͬͨ৔߹ɼ༧ ૝ΑΓେ͖ͳো֐ʹͭͳ͕Δ͜ͱ΋͋Δ ɾো֐ൃੜΛڪΕͯɼௐࠪʹ࣌ؒΛཁ͠ɼ݁Ռతʹมߋ଎౓͕௿Լ ɾγεςϜ؅ཧऀ͕ωοτϫʔΫαʔϏεؒͷґଘؔ܎Λ೺Ѳ͢Δ͜ͱ ͕ॏཁ

   ωοτϫʔΫαʔϏεґଘؔ܎ͷෳࡶԽ

5. 5 ௒ݸମܕσʔληϯλʔ ɾதԝूݖͷΫϥ΢υίϯϐϡʔςΟϯάͷΈͰ͸ղܾͰ͖ͳ͍໰୊ ɾར༻ऀͱσʔληϯλʔؒͷωοτϫʔΫ஗Ԇ ɾখɾதن໛ͷσʔληϯλʔ΍খܕͷϥοΫ܈Λ஍ཧతʹ෼ࢄͤ͞Δ ɾ෼ࢄ֤ͨ͠σʔληϯλʔ͸ಠཱͯ͠ػೳ͠ͳ͕Β΋ɼ૯ମͱͯ͠͸ ౷཰͞Ε͍ͯΔΑ͏ʹݟ͑Δ௒ݸମతͳσʔληϯλʔ[12]Λ໨ࢦ͢ ɾωοτϫʔΫαʔϏεґଘؔ܎͕ΑΓෳࡶԽ͠ɼγεςϜ؅ཧऀ͕ґ ଘؔ܎ͷશମΛৄࡉʹ೺ѲͰ͖ͳ͍͜ͱ͕લఏͱͳΔ ɾγεςϜͷґଘؔ܎Λࣗಈతʹ௥੻͢Δඞཁ͕͋Δ

   [12] দຊ྄հ, ௶಺༎थ, ٶԼ߶ี: ෼ࢄܕσʔληϯλʔOSΛ໨ࢦͨ͠ϦΞΫςΟϒੑΛ࣋ͭίϯςφ࣮ߦج൫ٕज़ɼ৘ใॲཧֶձݚڀใࠂΠϯλʔωο τͱӡ༻ٕज़(IOT) ɼVol. 2019-IOT-44, No. 27, pp. 1–8 2019.

6. 6 1. ௒ݸମܕσʔληϯλʔͰ͸ɼγεςϜ؅ཧऀ͕ωοτϫʔΫґଘؔ ܎Λ೺ѲͰ͖ͳ͍ ↪ ະ஌ͷωοτϫʔΫαʔϏεͱͷ઀ଓΛ௥੻Մೳ 2. ௒ݸମܕσʔληϯλʔͰ͸ɼσʔληϯλʔΛ෼ࢄͤ͞ΔͨΊɼத ԝूݖతߏ଄ΛͱΓͮΒ͍ ↪

   ௥੻ͨ݁͠ՌΛ෼ࢄͨ͠σʔληϯλʔ্ʹ෼ࢄ഑ஔՄೳ ௒ݸମܕτϨʔγϯάͷཁ݅ ࠓճ͸1ͷཁ݅Λຬͨ͢ख๏ΛఏҊ͢Δ

7. 7 ຊݚڀͷ໨త ະ஌ͷωοτϫʔΫαʔϏεͱͷґଘΛൃݟ͢ΔͨΊʹɼLinuxͷΧʔ ωϧͰ໢ཏతʹ઀ଓ৘ใΛൃݟͰ͖Δτϥϯεϙʔτ઀ଓ(TCP/UDP) ΛࣗಈͰ௥੻͢Δ Host Kernel Process Process Transport

   (TCP/UDP) … . . . User ௥੻ϙΠϯτ

8. 2. ωοτϫʔΫαʔϏεͷґଘؔ܎ͷ ൃݟͷ՝୊

9. 9 ɾ؍ଌࢤ޲Ξϓϩʔν: ֎෦͔ΒγεςϜͷมಈͷ૬ؔΛ؍ଌ͢Δ [7,8,9,10,20] ɾ໢ཏతͳґଘؔ܎ͷ௥੻͕Մೳ ɾհೖࢤ޲Ξϓϩʔν: ΞϓϦέʔγϣϯॲཧ΍ωο τϫʔΫ௨৴ͷؒ ʹܭଌ఺Λઃఆ͠ɼॲཧͷҰ࿈ͷྲྀΕͷதͰܭଌ఺Λ௨աͤ͞Δ [21,22,23,24,11,25,26]

   ɾґଘΛ௚઀ݕग़͍ͯ͠Δ͜ͱ͔Βِཅੑ͕খ͍͞ ઌߦݚڀͷ෼ྨ [7] P Bahl, et.al.: Towards Highly Reliable Enterprise Network Services via Inference of Multi-Level Dependencies, ACM SIGCOMM Review, Vol. 37, No. 4, pp.13–24 2007. [8] X Chen, et.al.: Automating Network Application Dependency Discovery: Experiences, Limitations, and New Solutions, USENIX Symposium on OSDI, pp.117–130 2008. [9] P Lucian, etl.al.: Macroscope: End-Point Approach to Networked Application Dependency Discovery, CoNEXT, pp.229–240 2009. [10] A Natarajan, et.al.: NSDMiner: Automated Discovery of Network Service Dependencies, IEEE INFOCOM, pp. 2507–2515 2012. [11] A Zand, et.al.: Rippler: Delay Injection for Service Dependency Detection, IEEE INFOCOM, pp. 2157–2165 2014. [21] M Y Chen, et.al.: Pinpoint: Problem Determination in Large, Dynamic Internet Services, IEEE/IFIP International Conference on DSN, pp. 595–604 2002. [22] P Barham, et.al.: Magpie: Online Modelling and Performance-aware Systems, 17th Workshop on HotOS, pp. 85–90 2003. [23] R Fonseca, et.al.: X-Trace: A Pervasive Network Tracing Framework, USENIX Conference on NSDI, pp. 20–20 2007. [24] B H Sigelman, et.al.: Dapper, a Large-Scale Distributed Systems Tracing Infrastructure, Technical report, Google 2010. [25] J K Clawson: Service Dependency Analysis via TCP/UDP Port Tracing, Master’s thesis, Brigham Young University-Provo 2015. [26] W Li, Y Lemieux: Service Mesh: Challenges, State of the Art, and Future Research Opportunities, IEEE International Conference on SOSE, pp. 122–1225 2019.

10. 10 ɾ؍ଌࢤ޲Ξϓϩʔν ɾਫ਼౓: ِཅੑͱِӄੑ ɾਫ਼౓͕ѱ͍΄ͲɼมߋͷӨڹௐ͕ࠪඞཁͱͳΔ ɾґଘํ޲: ୭͕୭ʹґଘ͢Δͷ͔ͱ͍͏ґଘͷํ޲ΛࣝผͰ͖ͳ͍ ɾґଘํ޲͕Θ͔Βͳ͍ͱɼมߋͷӨڹൣғΛਖ਼͘͠༧૝Ͱ͖ͳ͍ ɾհೖࢤ޲Ξϓϩʔν ɾܭଌΦʔόʔϔου:

    ΞϓϦέʔγϣϯͷॲཧʹհೖ͢ΔͨΊɼ Φʔόʔϔου͕ൃੜ ɾະ஌ͷωοτϫʔΫαʔϏεͷґଘΛ௥੻͢ΔͨΊʹ͸໢ཏੑ͕ ඞཁͱͳΓɼΦʔόʔϔουͷӨڹ͕େ͖͘ͳΓ΍͍͢ ઌߦݚڀͷ՝୊

11. 3. ఏҊख๏

12. 12 ఏҊख๏ʹΑΔ՝୊ͷղܾ 1. ܭଌΦʔόʔϔου ௿ΦʔόʔϔουͰ͋Δ؍ଌࢤ޲Ξϓϩʔνͱ͢Δ 2. ਫ਼౓: ِཅੑ TCP/UDP઀ଓΛऴ୺͢Δϗετ্Ͱൃੜ͢Δ઀ଓΛ௚઀ݕग़͢Δ 3.

    ґଘํ޲ ઀ଓͷ྆୺Λɼ઀ଓΛཁٻ͢Δଆͱ઀ଓΛ଴ͪड͚Δଆʹ෼཭ ະ஌ͷωοτϫʔΫαʔϏεͱͷґଘΛൃݟ͢ΔͨΊʹɼLinuxΧʔω ϧϥϯυͰ໢ཏతʹ઀ଓ৘ใΛൃݟͰ͖ΔTCP/UDP઀ଓΛର৅

13. 13 ఏҊख๏ͷߏ੒ Host 1 Host 2 Host N CMDB Tracer

    Tracer Tracer Systems Administrator ɾϗετ্ʹTracerϓϩηεΛ഑ஔ ɾ֤Tracerϓϩηε͸औಘͨ͠઀ଓ৘ใΛ CMDB(઀ଓ৘ใ؅ཧσʔλϕʔε)ʹอଘ ɾγεςϜ؅ཧऀ͸CMDBʹΞΫηε͠ɼ ෳ਺ͷϗετʹ·͕ͨΓґଘؔ܎Λऔಘ

14. 14 TCP/UDPͷ઀ଓ৘ใͷऩू Host Kernel Process Process Transport … Tracer Polling

    ɾTracerϓϩηε͕LinuxΧʔωϧʹ໰͍߹Θ ͤɼTCP/UDPιέοτ৘ใΛϙʔϦϯάऔಘ ɾ઀ଓΛऴ୺͢ΔOSϓϩηε৘ใ΋͋Θͤͯ औಘ ɾιέοτ৘ใ: /proc/net/tcp΍Netlink sock_diag ɾϓϩηε৘ใ: /proc/<pid>/{stat,fd} . . . ॲཧʹհೖ͠ͳ͍ͨΊ ௿Φʔόʔϔου

15. 15 TCP/UDP઀ଓͷґଘͷํ޲ͷܾఆ Host Y Port N Process B CONNECT Host

    X Port M Process A LISTEN ɾ઀ଓΛཁٻ͢ΔϗετY͸ɼ઀ଓΛड͚෇͚ΔϗετXʹґଘ͢Δ ɾϗετY͔ΒΈͯѼઌϙʔτ͕LISTENϙʔτMͰ͋Ε͹ɼHost Y ͔Β઀ଓΛཁٻ͍ͯ͠Δ͜ͱ͕Θ͔Δ ɾLISTENϙʔτ͸ɼϗετXͷOSʹ໰͍߹Θͤͯऔಘ͢Δ

16. 16 ɾ͢΂ͯͷ઀ଓ৘ใΛऩू͢ΔͱɼCMDBʹ֨ೲ͢Δσʔλྔ͕େ͖͘ ͳΔͨΊɼ৑௕ͳ৘ใΛ࡟ݮ͢Δ ɾΤϑΣϝϥϧϙʔτ: Χʔωϧ͔ΒׂΓ౰ͯΒΕΔϥϯμϜͳૹ৴ݩ ϙʔτ ɾಛఆͷLISTENϙʔτ΁ෳ਺ͷΤϑΣϝϥϧϙʔτ͔Β઀ଓ͞ΕΔ ɾ͜ΕΒͷ઀ଓΛू໿͠ɼ1ݸͷ઀ଓͱΈͳ͢ ΤϑΣϝϥϧϙʔτͷू໿ Host

    Port Service Port Port Host Port Service 1ݸͷ઀ଓ
 ͱͯ͠ू໿ LISTEN ΤϑΣϝϥϧ

17. 17 ɾಛఆͷωοτϫʔΫαʔϏε΍ϗετΛى఺ͱͯ͠ɼґଘؔ܎Λऔ ಘ͢Δ ɾશମਤ͸ෳࡶͳ΋ͷͱͳΔͨΊɼࢹ֮తʹݟͮΒ͍ ɾγεςϜ؅ཧऀ͸ɼγεςϜมߋ࣌ʹCLIπʔϧͰมߋର৅Λࢦఆ ͯ͠ӨڹൣғΛ೺Ѳ ఏҊख๏ͷར༻ มߋର৅ ɾιϑτ΢ΣΞ͕ࣗಈͰมߋ͢Δ ৔߹͸ɼAPIʹΑΓӨڹൣғΛ

    औಘ͠ɼҰఆҎ্ͷӨڹൣғͰ ͋ͬͨ৔߹ʹɼ؅ཧऀʹ௨஌

18. 4. ·ͱΊ

19. 19 ɾ໰୊ҙࣝ: ωοτϫʔΫαʔϏεͷґଘؔ܎͕ෳࡶԽ͢Δͱมߋ଎ ౓͕௿Լ ɾ໨త: ະ஌ͷωοτϫʔΫαʔϏεͱͷ઀ଓΛൃݟ͢ΔͨΊʹɼ LinuxΧʔωϧ಺ͷTCP/UDP઀ଓΛ௥੻ ɾ՝୊: ܭଌΦʔόʔϔουɼਫ਼౓ɼґଘํ޲ͷࣝผ ɾղܾ:

    ɾܭଌΦʔόʔϔου: ؍ଌࢤ޲Ξϓϩʔνͷ࠾༻ ɾਫ਼౓: ِཅੑʹ͍ͭͯɼTCP/UDP઀ଓͷऴ୺ϗετ্Ͱ௚઀ݕ஌ ɾґଘํ޲: TCP/UDP઀ଓͷ྆୺Λཁٻଆͱ଴ͪड͚ଆʹ෼཭ ·ͱΊ

20. 20 ɾఏҊख๏ͷ࣮૷ͱධՁ ɾTranstracer※1 ͱ࣮ͯ͠૷த ɾِӄੑͷ௿ݮ ɾ઀ଓͷ։࢝࣌ͷΈհೖࢤ޲ΞϓϩʔνΛద༻ ɾίϯςφܕԾ૝Խ؀ڥʹର͢Δख๏ͷద༻ ɾຊདྷଘࡏ͠ͳ͍͸ͣͷґଘΛൃݟͨ͠ͱ͖ͷҟৗݕ஌ ɾ௥੻ͨ݁͠ՌΛ෼ࢄͨ͠σʔληϯλʔ্ʹ෼ࢄ഑ஔ ࠓޙͷల๬

    ※1 https://github.com/yuuki/transtracer