ネットワークサービスの依存発見に向いた
TCP/UDP通信の低負荷なトレース手法 / Low Overhead TCP-UDP Tracing in Kernel

Transcript

1. ωοτϫʔΫαʔϏεͷґଘൃݟʹ޲͍ͨ 
 TCP/UDP௨৴ͷ௿ෛՙͳτϨʔεख๏ ୈ8ճWebSystemArchitectureݚڀձ 2021೥6݄04೔ @yuuk1t ͘͞ΒΠϯλʔωοτ id:masayoshi ͸ͯͳ 


   @matsumotory ͘͞ΒΠϯλʔωοτ

2. 2 Ϋϥ΢υ্ͷΞϓϦέʔγϣϯͷґଘؔ܎ͷෳࡶԽ ɾ ɾ ɾ ΞΫηε૿Ճ εέʔϧΞ΢τʹΑΔ ϗετ਺ͷ૿Ճ ػೳͷ૿Ճ ϚΠΫϩαʔϏεͷ࠾༻

   ϛυϧ΢ΣΞͷ૿Ճ ɾ ɾ ɾ RDBαʔό KVαʔό ݕࡧαʔό Webαʔό ωοτϫʔΫ 
 αʔϏε TCP/UDP

3. 3 ґଘؔ܎ͷෳࡶԽ͕΋ͨΒ͢໰୊ ωοτϫʔΫαʔϏεؒͷ௨৴ͷґଘؔ܎Λ ࣗಈͰτϨʔε͢Δඞཁ͕͋Δ ɾґଘؔ܎Λ஌ΒͣʹγεςϜΛमਖ਼͢Δͱɺ૝ఆ֎ͷൣғͷӨڹ͕ൃੜ͢Δ ɾґଘؔ܎͕ෳࡶ͔ͭಈతʹมߋ͞ΕΔͨΊɺهԱ͢Δ͜ͱ΍υΩϡϝϯτԽ ͸೉͍͠ ґଘάϥϑͷ׬શੑ ௿Φʔόϔου ύέοτϕʔεΑΓ΋ίωΫγϣϯϕʔε

   ΞϓϦέʔγϣϯมߋෆཁ

4. ιέοτϕʔεͷτϨʔεख๏ͷൺֱ . . . Kernel User Service Socket Tracing 


   Process … Event Event Event ετϦʔϛϯά๏(Weave Scope) ϑϩʔू໿๏ ([Datadog], [SAC 20]) ϑϩʔूଋ๏ʢఏҊʣ . . . Kernel User Service Socket Tracing 
 Process . . . Event Flow Event Event Event … … . . . . . . User Service Socket Tracing 
 Process Event Event . . . Event Event Event Event … … . . . Event Event … … . . . ✗ ΧʔωϧˠϢʔβۭؒؒ ͷΠϕϯτͷίϐʔίετ ✗ TCP୹໋઀ଓϨʔτ͕૿ Ճ͢Δͱɺίϐʔίετ͕ ૿Ճ ෳ਺ͷϑϩʔΛूଋ Flow = ྆୺ͷΞυϨεͱϙʔτͷ ૊ʢλϓϧʣ͕ಉҰͷ௨৴୯Ґ

5. 5 TCP୹໋઀ଓ τϨʔεॲཧͷCPUෛՙ ɾX࣠ TCP୹໋઀ଓ਺ 5k ~ 35k ɾY࣠ ίΞ͋ͨΓͷCPUར༻཰

   (0-25%) 0 5 10 15 20 25 5 10 15 20 25 30 35 CPU usage / core (%) TCP round trips / sec (x103) Snap-Poll(client) Snap-Poll(server) Streaming(client) Streaming(server) In-Kernel-Aggr(client) In-Kernel-Aggr(server) In-Kernel-Bundling(client) In-Kernel-Bundling(server) ɾఏҊख๏ͷCPUར༻཰͸2.2%ҎԼ ɾx࣠૿Ճʹରͯ͠ɺ௿CPUར༻཰Λҡ࣋ ɾετϦʔϛϯά๏͸ɺX࣠ʹൺྫͯ͠Y͕࣠ ૿Ճɻ࠷େ20%ఔ౓ͷCPUར༻཰ɻ ɾΧʔωϧ಺ू໿๏͸ɺετϦʔϛϯά๏Α ΓY࣠஋͕௿͍ɻ

6. 6 ݚڀͷߩݙ 1. TCP୹໋઀ଓ΍UDPͷϨʔτ͕େ͖ͳ؀ڥʹ͓͍ͯ΋ɺτϨʔεͷͨΊͷ CPUෛՙΛ௿ݮ͢ΔͨΊͷΧʔωϧ಺ϑϩʔूଋ๏ΛఏҊ͢Δɻ 2. Linux eBPFʹΑΔద༻ੑΛॏࢹ࣮ͨ͠૷Λࣔ͢ɻ 3. TCP/UDPϕϯνϚʔΧʔΛ༻͍࣮ͨݧʹΑΓɺϑϩʔ਺͕૿େͨ͠ͱ͠

   ͯ΋ɺίΞ͋ͨΓͷCPUར༻཰͕2.2%ҎԼͱͳΔ͜ͱΛݕূ͢Δɻ

7. 7 Χʔωϧ಺Ͱͷෳ਺ϑϩʔͷूଋ๏ ɾωοτϫʔΫαʔϏεؒͷґଘΛ஌Δ͚ͩͳΒɺ୹໋ϙʔτ͸ෆཁ ɾsrc, destΞυϨεɺϦοεϯϙʔτΛҰҙΩʔͱͯ͠ूଋ TCP/UDP Connections Hash map .

   . . . . . Key Kernel User Service Service Value Socket daddr saddr lport protocol 4-tuple Tracing 
 Process ɾΧʔωϧۭؒ಺ͷϋογϡ දʹ஝ੵ ɾTracingϓϩηε͕Ұఆपظ (1sఔ౓)Ͱू໿ΠϕϯτΛ όονऔಘ connect, 
 accept, 
 read, write,… . . . . . .

8. 8 Χʔωϧ಺ϑϩʔूଋ๏ͷΞϧΰϦζϜ ೖྗ: ιέοτߏ଄ମ S, ଴ͪड͚ϙʔτϦετ P ू໿ͷͨΊͷϋογϡද H ͷ࡞੒

   function PROBE__TCP_CONNECT (S) INSERT_FLOW_EVENT(S, TCP) end function function PROBE__TCP_ACCEPT (S) INSERT_FLOW_EVENT(S, TCP) end function function PROBE__TCP_SENDMSG (S) INSERT_FLOW_EVENTS(S, TCP) end function function PROBE__TCP_RECVMSG (S) INSERT_FLOW_EVENTS(S, TCP) end function ग़ྗ: ूଋࡁΈͷϑϩʔϦετ function GET_LISTENING_PORTS_AND_DIRECTIONS (S) if P.lookup(S.sport) then return S.sport, INCOMING else return S.dport, OUTCOMING end if end function function INSERT_FLOW_EVENTS (S, proto) lport, dir = GET_LISTENING_PORTS_AND_DIRECTIONS(S) key ← {S.saddr, S.daddr, lport, dir, proto} H.update(key, stats) end function UDPলུ

9. 9 Χʔωϧ಺ूଋ๏ͷ࣮૷ 1. Χʔωϧؔ਺Λ๣ड͢Δ → LinuxͷkprobeΛ࢖༻ ɾΧʔωϧ಺ͷΦϒδΣΫτʹ೚ҙͷϋϯυϥΛઃఆՄೳ 2. Χʔωϧ಺Ͱ҆શʹϋϯυϥΛهड़͢Δ →

   Linux eBPFΛ࢖༻ ɾִ཭͞ΕͨαϯυϘοΫε؀ڥͰϓϩάϥϜΛ࣮ߦՄೳ 3. ϋογϡදʹूଋ͞ΕͨϑϩʔΛ֨ೲ → eBPF MapΛ࢖༻ ɾMapʹ͸Ϣʔβۭ͔ؒΒΞΫηεՄೳ ࣮૷͸ https://github.com/yuuki/go-conntracer-bpf ʹͯެ։

10. 10 ࣮૷ͷུ֓ਤ TCP/UDP Connections Flow table . . . .

    . . 4-tuple Kernel User Service Service Stat Socket Tracing 
 Process connect, 
 accept, 
 sendto, 
 recv . . . . . . tcp_v4_connect() inet_csk_accept() ىಈ࣌ʹ଴ͪड ͚ϙʔτͷ 
 ϦετΛऔಘ tcp_sendmsg() tcp_cleanup_rbuf() UDPলུ kprobeͰΞλον͢Δ 
 Χʔωϧؔ਺ insert Batch APIʹΑ Γෳ਺Ϩίʔυ ಉ࣌औಘ MapΛ࢖༻

11. 11 ࣮ݧͷͨΊͷܭࢉػ؀ڥ Client (ཁٻଆ) Server (଴डଆ) Tracing Process Tracing Process

    benchmark process benchmark process ؀ڥ1: 1ର1 ωοτϫʔΫαʔϏε ؀ڥ2: 1ରN ωοτϫʔΫαʔϏε Client (ཁٻଆ) Server (଴डଆ) Tracing Process benchmark process ɾɾɾ ɾɾɾ ɾɾɾ • CPU: Intel Xeon Gold 6212U @2.40GHz x 6ίΞ • ϝϞϦ: 16GB • OS: Ubuntu 20.10 Kernel 5.8.0-33 Connperf: https://github.com/yuuki/connperf ෛՙੜ੒ • ࣗ࡞ͷΤίʔαʔόɾΫϥΠΞϯτ • ඵؒͷ৽ن઀ଓ਺ΛݻఆՄೳ

12. 12 τϨʔεॲཧͷCPUෛՙʢ1ର1؀ڥʣ ɾఏҊख๏͸ɺCPUར༻཰Λ2.2%ҎԼ·Ͱ௿ݮ TCP୹໋઀ଓ UDP ఏҊख๏ 0 5 10 15

    20 25 5 10 15 20 25 30 35 CPU usage / core (%) Number of TCP connections (x103) Snap-Poll(client) Snap-Poll(server) Streaming(client) Streaming(server) In-Kernel-Aggr(client) In-Kernel-Aggr(server) In-Kernel-Bundling(client) In-Kernel-Bundling(server) 0 5 10 15 20 25 5 10 15 20 25 30 35 CPU usage / core (%) TCP round trips / sec (x103) Snap-Poll(client) Snap-Poll(server) Streaming(client) Streaming(server) In-Kernel-Aggr(client) In-Kernel-Aggr(server) In-Kernel-Bundling(client) In-Kernel-Bundling(server) 0 5 10 15 20 25 5 10 15 20 25 30 35 CPU usage / core (%) UDP round trips / sec (x103) Streaming(client) Streaming(server) In-Kernel-Aggr(client) In-Kernel-Aggr(server) In-Kernel-Bundling(client) In-Kernel-Bundling(server) TCPӬଓ઀ଓ

13. 13 ௨৴ઌͷαʔϏε਺ʹର͢ΔCPUෛՙʢ1ରN؀ڥʣ ɾඵؒ઀ଓ਺Λ10kʹݻఆ (T=10k) ɾαʔϏε਺Λ200ʙ1000·Ͱ૿Ճ 
 (R=0.98 .. 0.9) ɾαʔϏε਺ͷ૿େʹରͯ͠ɺCPUར༻཰͸2%Ҏ

    Լ ɾҟͳΔ଴ͪड͚ϙʔτΛ΋ͭ௨৴ઌ͕૿͑Δͱɺूଋ཰͕௿Լ ↪ CPUෛՙ͕૿େ͢Δ͸ͣ 0 1 2 3 4 5 200 400 600 800 1000 CPU usage per core (%) Number of network services In-Kernel-Bundling(client,TCP short-lived) In-Kernel-Bundling(server,TCP short-lived) In-Kernel-Bundling(client,TCP persistent) In-Kernel-Bundling(server,TCP persistent) In-Kernel-Bundling(server,UDP) In-Kernel-Bundling(client,UDP) ूଋ཰ R = 1 - B / T B: ूଋ͞Εͨϑϩʔ਺ T: ϢχʔΫϑϩʔ਺

14. 14 ΞϓϦέʔγϣϯͷ஗ԆΦʔόϔου TCP୹໋઀ଓ UDP Ping RTT 300μs ʹରͯ͠ɺఏҊख๏ͷΦʔόϔου͸ 5.8 μsɻ

    
 2%ͷΦʔόϔου૿Ճɻ 0 1 2 3 4 5 6 5 10 15 20 RTT overhead (µs) UDP round trips / sec Streaming 3.6 3.2 3.1 3.1 In−Kernel−Aggr 4.6 4.1 4.0 4.2 In−Kernel−Bundling 4.1 3.4 3.3 3.4 0 1 2 3 4 5 6 5 10 15 20 RTT overhead (µs) Number of TCP connections Streaming 0 0 0 0 In−Kernel−Aggr 2.4 2.1 2.3 2.2 In−Kernel−Bundling 2.4 2.2 2.1 2.1 0 1 2 3 4 5 6 5 10 15 20 RTT overhead (µs) TCP round trips / sec Streaming 2.2 2.1 2.1 2.5 In−Kernel−Aggr 4.7 4.7 5.0 5.7 In−Kernel−Bundling 4.8 4.9 5.0 5.8 TCP࣋ଓత઀ଓ

15. 15 ·ͱΊ ɾTCP/UDP௨৴Λ௿ΦʔόϔουͰτϨʔε͢ΔͨΊʹɺΧʔωϧ಺ ϑϩʔूଋ๏ΛఏҊͨ͠ ɾ࣮ݧͷ݁ՌɺTCPͷ୹໋ͳ઀ଓ਺ͷ૿Ճʹରͯ͠ɺఏҊख๏͸2.2% ҎԼͷCPUར༻཰Λҡ࣋ͨ͠ ɾࠓޙͷ༧ఆ ɾπʔϧͱͯ͠ͷ׬੒౓ΛߴΊͯɺ࣮ࡍʹར༻ͯ͠΋Β͏