はてなのログ運用 これまでとこれから / Hatena Engineer Seminar #6

Transcript

1. ͸ͯͳͷϩάӡ༻ ͜Ε·Ͱͱ͜Ε͔Β id:wtatsuru 2016/08/31 Hatena Engineer Seminar #6

2. ࣗݾ঺հ • Tatsuru Watabane (id:wtatsuru) • ͸ͯͳγεςϜϓϥοτϑΥʔϜ෦௕ • 2011೥ೖࣾ •

   2014೥·ͰͷαʔϏε͸͍͍ͩͨશ෦ӡ༻

3. ຊ೔ͷ಺༰ • લఏͱͳΔγεςϜߏ੒ • ͸ͯͳͷϩάͷߏ੒ͱӡ༻Λ঺հ • ࠓޙ

4. લఏͱͳΔγεςϜߏ੒

5. 3૚ߏ଄ reverse proxy App server DB

6. ѻ͏ϩά reverse proxy App server DB access log error log

   various logs

7. ѻ͏ϩά • αʔϏεͷΞΫηεϩά • ˏϦόʔεϓϩΩγ • ΞϓϦέʔγϣϯ͕ు͘ϩά • Τϥʔϩάɺߦಈϩά etc.

   reverse proxy App server DB access log error log various logs

8. લఏɿଟ͘ͷαʔϏε • ৽αʔϏε͔Β10೥Ҏ্ଓ͘αʔϏε·Ͱଟ༷ • ֤αʔϏεΛ·͍ͨͩڞ௨ج൫͕ඞཁ • ։ൃνʔϜͱҰॹʹվળͰ͖Δମ੍Λ࡞Γ͍ͨ

9. ͸ͯͳͷϩάӡ༻

10. reverse proxy App server DB access log error log structured

    log syslog server storage server SSD server rsyslog aggregator

11. ͸ͯͳͷϩάӡ༻ • ΞΫηεϩάͷӡ༻ • ΞϓϦέʔγϣϯϩάͷӡ༻ • ߏஙͷྺ࢙ͱͱ΋ʹΏΔ͘঺հ͠·͢

12. ΞΫηεϩάग़ྗ • ΞΫηεϩάͷϚελͱͳΔϩάͷग़ྗ • ֤छௐࠪɺૢ࡞ϩάɺetc. • ϩʔΧϧʹϩάΛు͘@ϦόʔεϓϩΩγ • LTSV (Labeled

    Tab-separated Values) https://ltsv.org/ host:127.0.0.1 ident:- user:frank time:[10/Oct/2000:13:55:36 -0700] req:GET /apache_pb.gif HTTP/1.0 status:200 size:2326 referer:https:// www.example.com/start.html ua:Mozilla/4.08 [en] (Win98; I ;Nav)

13. reverse proxy App server DB access log ΞΫηεϩάग़ྗ

14. ϩάΛอଘ͍ͨ͠ • ผͷαʔόʹอଘ͓͖ͯͨ͘͠ͳͬͨ • rsyslog Ͱඈ͹ͯ͠ετϨʔδαʔόʹอଘ • ϩʔΧϧ͔ΒS3ʹ΋อଘͯ͠ೋॏԽ • ͠͹Β͘͢ΔͱGlacierʹ௜Ή

15. reverse proxy App server DB access log syslog server rsyslog

    storage server

16. ؆୯ʹ෼ੳ͍ͨ͠ • SSDͷαʔόʹ΋ඈ͹͢ • SSD͕҆͘ͳͬͨͷͰಋೖ • ࡶଟʹ grep ΍෼ੳ •

    ΞΫηεϩά͔ΒϨΠςϯγ෼ੳ • 2೥ॻ͍ͨΒSSD͕յΕͨͷͰަ׵

17. reverse proxy App server DB access log syslog server storage

    server SSD server rsyslog

18. ΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠ • S3 ͔Β Amazon EMR (ElasticMapReduce) ΁ • LTSVΛಡΈࠐΉσγϦΞϥΠβ

    
 github.com/hatena/KeyValuePairsDeserializer • SQLͰ؆୯ʹ෼ੳ select count(distinct regexp_extract(referer,"https://([^/]+)/",1)) from logs where req rlike 'GET /hoge‘

19. reverse proxy App server DB access log syslog server storage

    server SSD server rsyslog

20. ΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠ • ֤छ՝୊ • ॳظԽ͕஗͍ • ؀ڥߏங͕೉͍͠ • ఆظతʹճ͢ͱίετ͕ •

    ࢖ΘΕͳ͘ͳ͍ͬͯ͘…

21. ΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠ • EMRஔ͖׵͑ɿGoogle BigQuery Λݕূ • LTSV ͸ embulk Ͱલॲཧͯ͠ಡΈࠐΉ

    • ଎͍ɾ·͋·͍͋҆ • ։ൃऀҎ֎Ͱ΋࢖͍΍͍͢

22. reverse proxy App server DB access log syslog server storage

    server SSD server rsyslog

23. ͸ͯͳͷϩάӡ༻ • ΞΫηεϩάͷӡ༻ɹɹɹɹɹɹˡ͜͜·Ͱ • ΞϓϦέʔγϣϯϩάͷӡ༻ɹɹˡ͔͜͜Β • ߏஙͷྺ࢙ͱͱ΋ʹΏΔ͘঺հ͠·͢

24. ΞϓϦέʔγϣϯͷϩά • ૉ๿ͳੲ • /var/log/httpd/access.log • /var/log/httpd/error.log • ֤छͪ͝Ό·ͥ •

    Τϥʔௐ͕ࠪͭΒ͍ɺूܭ͕ͭΒ͍ • DBʹೖΕΔ

25. reverse proxy App server DB

26. ߏ଄ԽϩάΛग़͢ • શαʔϏεͰಛఆͷ path ʹ JSON Ͱు͘ • ඪ४ग़ྗɾΤϥʔग़ྗͱ͸෼͚Δ •

    ΞϓϦέʔγϣϯ͔Β໌ࣔతʹు͘ • ߦಈϩάɺDBΞΫηεɺͦͷଞͳΜͰ΋

27. reverse proxy App server DB structured log

28. fluentdͰ഑ૹ • ֤ϗετͰ in_tail +out_forward Ͱඈ͹͚ͩ͢ • ͜͜͸΄ͱΜͲ͍͡Βͳ͍ • தԝͷ

    aggregator ʹ౉ͯ͠Α͠ͳʹॲཧ • ඼࣭͸ϕετΤϑΥʔτ • ඞཁͳ৔ॴʹ͸ผ్อূͨ͠࢓૊ΈΛཱͯΔ • ͍ͭͰʹϦόʔεϓϩΩγ͔Β΋ૹΓ࢝ΊΔ

29. reverse proxy App server DB access log error log structured

    log aggregator

30. ֤ॴʹ഑ૹ • S3 ʹੜσʔλΛอଘ • Կ͔ίέͯ΋࠶ݱ • شൃੑͷอଘઌ͕ଟ͍ͨΊ͜͜Ͱ୲อ͢Δ

31. reverse proxy App server DB access log error log structured

    log aggregator

32. ϩάͷ഑ૹ • Mackerelʹ౤ߘ • ֤छϝτϦοΫՄࢹԽ • ϨΠςϯγ • ΤϥʔϨʔτ •

    KPI

33. reverse proxy App server DB access log error log structured

    log aggregator

34. ϩάͷ഑ૹ • Slackʹ௨஌ • ΤϥʔϩάΛूΊΔ • ݹ͍αʔϏεͰϩά͕ଟ͍ˠ grep Ͱ៉ྷʹ……

35. reverse proxy App server DB access log error log structured

    log aggregator

36. ϩάͷ഑ૹ • Elasticsearch→Kibana ͰՄࢹԽ • ֤νʔϜͰμογϡϘʔυΛ࡞ͬͯݟΔ

37. reverse proxy App server DB access log error log structured

    log aggregator

38. ϩάͷ഑ૹ • MongoDB capped collection ʹ౤ೖ • ͦͷ··ݟͨΓɺࡶʹूܭͨ͠Γ • ʢ࠷ۙ͋·Γ࢖ΘΕ͍ͯͳ͍ʣ

39. reverse proxy App server DB access log error log structured

    log aggregator

40. શܠ

41. reverse proxy App server DB access log error log structured

    log syslog server storage server SSD server rsyslog aggregator

42. ࠓޙ • syslog ͔Βͷ୤٫ • ։ൃऀҎ֎΋׆༻Ͱ͖Δ࢓૊Έ࡞Γ • ߦಈϩάج൫

43. ຊ೔ͷ಺༰ • ͸ͯͳͰͷϩάͷӡ༻ͷݱঢ়Λ঺հ͠·ͨ͠ • ࠓޙ΋஍ಓͳվળΛଓ͚͍͖ͯ·͢

44. Ҏ্Ͱ͢ • Ұॹʹվળͯ͘͠ΕΔํΛืूதͰ͢ʂ
 https://hatenacorp.jp/recruit/career/operation-engineer • ౦ژɾژ౎ͲͪΒͰ΋

45. ࣭ٙɾTwitterΑΓ • fluentd aggregator ٧·Γ·ͤΜ͔ • aggregator ͷઌʹɺͦΕͧΕͷσʔλετΞ͝ ͱʹ fluentd

    Λཱͯͯอଘ͍ͯ͠·͢ • ͜͏͢Δ͜ͱͰӨڹΛσʔλετΞ୯ҐʹہॴԽ ͍ͯ͠·͢

46. ࣭ٙɾTwitterΑΓ • ߦಈϩάͰ͸ܽଛ΍੔߹ੑͳͲ͍Ζ͍Ζߟ͑Δͱ͜ Ζ͋Δͱࢥ͍·͕͢Կ͔ߟ͍͑ͯ·͔͢ • কདྷతʹͦ͏͍͏ͷ͕Ͱ͖ͨΒ͍͍ͳɺͱ͍͏ஈ ֊Ͱ͢… • ࣮ࡍඞཁͱͳͬͨΒɺผʹ౸ୡอূΛͨ͠ܥΛ࡞ͬ ͯܽଛ͠ͳ͍࢓૊ΈΛ࡞Γ·͢ɻKafka

    ͷΑ͏ͳ ΩϡʔΛڬΉͳͲ

47. ࣭ٙɾTwitterΑΓ • յΕͨSSD͸Ͳ͜ͷͰ͔͢ • ॳ୅͸ Crucial M4 Λ࢖͍ͬͯ·ͨ͠