サーバモニタリング向け時系列データベースの探究 / The study of time-series database for server monitoring

Transcript

1. αʔόϞχλϦϯά޲͚ ࣌ܥྻσʔλϕʔεͷ୳ڀ גࣜձࣾ͸ͯͳ ΢ΣϒΦϖϨʔγϣϯΤϯδχΞ TSUBOUCHI, Yuuki @y_uuk1 2016/12/01 ୈ9ճΠϯλʔωοτͱӡ༻ٕज़γϯϙδ΢ϜʢIOTS2016ʣট଴ߨԋ

2. ʮӡ༻ͰΧόʔ͢Δʯ ʮӡ༻୲౰ऀͷਓபͷ্ʹ੒ཱ͢Δ લ࣌୅తͰ໺൮ͳߏ଄෺ʯ ʮෆշͳ൰͍͠࢓ࣄΛ΍Δඞཁ͕ͳ͘ͳΔͷ ͸ɺਓؒʹͱͬͯͻ͡ΐ͏ͳ෱ࢱ͔΋͠Εͳ ͍͕ɺ͋Δ͍͸ͦ͏Ͱͳ͍͔΋͠Εͳ͍ɻʯ ୈ 9 ճΠϯλʔωοτͱӡ༻ٕज़γϯϙδ΢Ϝ ։࠵Ҋ಺

   https://www.iot.ipsj.or.jp/iots/2016/announcement
3. None

4. We define toil as mundane, repetitive operational work providing no

   enduring value, which scales linearly with service growth. Betsy Beyer, Chris Jones, Jennifer Petoff, etc. Site Reliability Engineering. Oreilly & Associates Inc. April, 2016.

5. Software Engineering System Engineering

6. ࣌ܥྻDBͷ։ൃΛྫʹͱΓ WebαʔϏεӡ༻ٕज़ͷ ΤοηϯεΛ఻͑Δ

7. WebαʔϏε

8. ͸ͯͳͷ୅දతWebαʔϏε

9. 10೥੒௕͢Δ WebαʔϏε

10. 2001೥ தݹPC1୆ https://www.hatena.ne.jp/company/hatena10th/history

11. 2006೥ αʔό350୆ https://www.slideshare.net/stanaka/how-to-use-virtualization-technology-in-hatena

12. 2008೥ XenʹΑΔԾ૝ԽͰ୆਺૿Ճ https://www.slideshare.net/stanaka/how-to-use-virtualization-technology-in-hatena

13. 2010೥ 1000୆ಥഁ https://www.slideshare.net/stanaka/how-to-use-virtualization-technology-in-hatena

14. ݱࡏͰ͸਺ઍ୆ن໛

15. αʔϏε਺50+ (಺෦޲ؚ͚Ή)

16. αʔϏεಉ͕࢜ෳࡶʹ࿈ܞ͢Δ

17. γεςϜ͕ෳࡶԽͨ݁͠Ռ ͳʹ͕ى͖͍ͯΔ͔Θ͔Βͳ͘ͳΔ

18. None

19. ຊ೔͸ֶੜ͞Μ΋ͨ͘͞Μ͍Βͬ͠ΌΔͷͰɺWebαʔϏεاۀͷऔΓ૊ΈΛ஌ͬͯ΋Βͬͯɺؔ৺Λ ΋͍͚ͬͯͨͩΕ͹ͱࢥ͍·͢ɻ ෳࡶԽ͢ΔγεςϜΛલʹ ਓʑͷٙ໰͸ਚ͖ͳ͍

20. ؂ࢹΛࣗಈԽ͢Δ େਓ਺ͷΦϖϨʔλΛޏ༻͠ ਓྗͰղܾ

21. ҟৗͷݕ஌ͱ௨஌

22. ҟৗΛൃݟͨ࣌͠ʹ γεςϜͷաڈΛ஌Γ͍ͨ

23. CPUར༻཰ͷάϥϑ - 6࣌ؒલʙݱࡏ

24. CPUར༻཰ͷάϥϑ - Ұिؒલʙݱࡏ

25. ࣌ܥྻσʔλΛ௕ظؒอଘ͢Δඞཁ ͕͋Δ

26. ࣌ܥྻσʔλͷอଘ TSDB αʔό (ܥྻ໊, λΠϜελϯϓ, ஋) (ܥྻ໊, λΠϜελϯϓ, ஋) (ܥྻ໊,

    λΠϜελϯϓ, ஋) … ؅ཧαʔό αʔό αʔό ΤʔδΣϯτ ΤʔδΣϯτ ΤʔδΣϯτ

27. ࣌ܥྻσʔλͷن໛ײ TSDB αʔό (ܥྻ໊, λΠϜελϯϓ, ஋) (ܥྻ໊, λΠϜελϯϓ, ஋) (ܥྻ໊,

    λΠϜελϯϓ, ஋) … ؅ཧαʔό αʔό αʔό ΤʔδΣϯτ ΤʔδΣϯτ ΤʔδΣϯτ 1000୆ 100 datapoints/min/host 100,000 writes/min

28. ແࢹͰ͖ͳ͍ ؂ࢹγεςϜͷӡ༻ίετ

29. Mackerel https://mackerel.io

30. https://mackerel.io

31. αʔόͷϝτϦοΫՄࢹԽ

32. MackerelͷΞʔΩςΫνϟ

33. Mackerelͷ࣌ܥྻσʔλͷن໛ײ • ΤʔδΣϯτ͕Ϣʔβ͞Μͷϗετ͔Βຖ෼ϝτϦοΫ ౤ߘ • 2016/01࣌఺ͰΞΫςΟϒΤʔδΣϯτ਺ 10,000+ • 1ΤʔδΣϯτ͋ͨΓͷϝτϦοΫ਺͸࠷େ200 •

    ԾʹฏۉϝτϦοΫ਺Λ100 datapoints/hostͱ͢Δͱ ߹ܭૹ৴ϝτϦοΫ਺ 1,000,000 datapoints/min+ • ϝτϦοΫͷେྔॻ͖ࠐΈʹ଱͑ΒΕΔσʔλϕʔε͕ ඞཁ

34. MackerelΛࢧ͑ΔTSDB Λ୳ڀ͢Δ

35. TSDBͱͷࢥ͍ग़

36. 2012೥ ֶੜ࣌୅ʹ͸ͯͳͰΞϧόΠτ • Mackerelͷલ਎ͱͳΔΞϓϦέʔγϣϯ։ൃʹܞΘΔ • RRDtoolͱ͍͏2000೥୅ॳ಄ͷTSDBͰάϥϑඳը • ͦ΋ͦ΋ωοτϫʔΫσʔϞϯͰͳ͍ͨΊɺίϚϯυ Λ͍ͨͨͯ஋ͷߋ৽΍ࢀরΛ͍ͯͨ͠ •

    CPANϞδϡʔϧͷ࡞੒ https://metacpan.org/pod/RRDTool::Rawish • ϝτϦοΫͷΫϩʔϦϯάͱRRDtoolϕʔεͰάϥϑද ࣔͰ͖Δπʔϧͷ࡞੒ https://github.com/yuuki/ Monitorel

37. https://shoichimasuhara.hatenablog.com/entry/2014/04/10/003617

38. 2014೥ ͸ͯͳೖࣾ • Mackerelͷ։ൃͱ͋ΘͤͯGraphiteͷݕূΛ։࢝ • RRDtool͸εέʔϥϏϦςΟ΍৴པੑͳͲ͞·͟·ͳ ఺Ͱ໰୊͕͋ͬͨ • Graphite͸γϯϓϧͰ࣮૷Λཧղ͠΍͘͢ɺӡ༻ʹख ؒ͸͔͔Δ͕ͦΕͳΓʹΑ͘εέʔϧ͍ͯ͠Δ

    • ϐʔΫ࣌Ͱඵؒ8ສ write IOPS (͍҆SSD ਺ेຕ෼) • ϑΝΠϧγεςϜ΍ετϨʔδͷಛੑʹ͋Θͤͯνϡʔ χϯά͠ଓ͚͍ͯΔ

39. https://blog.yuuk.io/entry/high-performance-graphite

40. https://speakerdeck.com/yuukit/performance-improvement-of-tsdb-in-mackerel

41. ۙ೥Ͱ͸ ߴ౓ͳཁٻΛٻΊΒΕΔΑ͏ʹ

42. https://speakerdeck.com/sugiyama88/mackerel-meetup-number-8

43. ߴ଎ͳ਺ઍݸͷܥྻͷදࣔ

44. 1෼ղ૾౓ͷσʔλΛ௕ظؒอ࣋ https://mackerel.io/ja/docs/entry/overview

45. 1෼ҎԼͷղ૾౓ͷରԠ https://mackerel.io/ja/docs/entry/overview

46. ΞʔΩςΫνϟͷ ࡮৽͕ඞཁ

47. ݱߦͷΞʔΩςΫνϟ(Graphite)ͷ՝୊ • εέʔϧΞ΢τʹେ͖ͳਓख͕͔͔Δ • ݱࡏ͸ioDriveͷΑ͏ͳߴ଎ͳϑϥογϡετϨʔδΛ༻͍ ͯεέʔϧΞοϓ • ߴ଎ͳετϨʔδ͕ඞཁͳͨΊɺGB୯Ձ͕ඇৗʹߴ͍ • ϥ΢ϯυϩϏϯσʔλϕʔεͰ͋ΔͨΊɺݹ͍σʔλΛ্ॻ

    ͖͢Δ • ͕ͨͬͯ͠ɺݹ͍σʔλΛผͷετϨʔδ΁ୀආͤ͞ΔΑ͏ ͳ࣮૷ΛՃ͑Δ͜ͱ͕೉͍͠ • ΦϯϝϞϦͳϦϨʔϓϩΩγΛॻ͖ࠐΈཁٻΛதܧ͢ΔͨΊɺ ϦϨʔϓϩΩγ͕མͪΔͱσʔλ͚͕ܽى͖ΔՄೳੑ͕͋Δ

48. https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCEWdPtAoccJ4a-IuZv4fXDHxM/pubhtml 20ݸͷOSSͷTSDB࣮૷

49. TSDBʹؔ͢Δֶज़ݚڀ࿦จʢGorillaʣ Tuomas Pelkonen etc. Gorilla: a fast, scalable, in-memory time

    series database. In Proceedings of the 41st International Conference on Very Large Data Bases, Kohala Coast, Hawaii. Volume 8 Issue 12, August 2015. Pages 1816-1827 • Facebook͕։ൃ͍ͯ͠ΔTSDB • ௚ۙ26࣌ؒͷΈΠϯϝϞϦͰ֨ೲ • ࣌ܥྻσʔλͷղ૾౓͸15ඵ • ϝϞϦʹͷͤΔͨΊʹಠࣗͷѹॖํࣜΛఏҊ • ුಈখ਺఺਺͸ಉ͡ܥྻʹ͓͍ͯಉ͡஋͕࿈ଓ͢Δ έʔε͕ଟ͍͜ͱʹண໨ͨࠩ͠෼ූ߸Խ • ݹ͍σʔλ͸HBaseʹಀ͕͢

50. TSDBʹؔ͢Δֶज़ݚڀ࿦จʢBTrDBʣ Michael P Andersen and David E. Culler, University of

    California, Berkeley. BTrDB: Optimizing Storage System Design for Timeseries Processing. In Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST 16). Feb 2016. Pages 39-52. • IoT޲͚ͷTSDB • nanoඵʹରԠ • σʔλߏ଄͸copy-on-write treeϕʔεͷ΋ͷͰI/Oޮ཰ ͕ඇৗʹߴ͍ • 16.7 million writes/s and 19.8 million reads/s (EC2 c3.8xlarge) • GoݴޠʹΑΔ࣮૷͕ެ։͞Ε͍ͯΔ • https://github.com/SoftwareDefinedBuildings/btrdb

51. https://speakerdeck.com/rrreeeyyy/time-series-database-world

52. ͦͷଞͷTSDBͷ՝୊ • InfluxDB: RaftϓϩτίϧΛ༻͍ͨ৑௕ԽػߏΛ঎༻൛ͷΈͰα ϙʔτ • ӡ༻͕೉͍͠෼ࢄDB্ʹߏங͞Ε͍ͯΔ • OpenTSDB͸HBase, KairosDB͸Cassandra্ʹߏஙͳͲ

    • 1೥ͷϨϙʔτද͕ࣔ஗͍ • (ະ֬ೝ͕ͩ) OpenTSDBɺInfluxDBͳͲ • ޙड़͢Δ௿ղ૾౓ςʔϒϧ࡞੒ʹ૬౰͢Δ࣮૷͕ඞཁ • ϓϩμΫγϣϯʹར༻Մೳͳ඼࣭ͷιϑτ΢ΣΞͰ͸ͳ͍ • BTrDB͸development version • ίϛϡχςΟͰͷ࣮੷͕ͳ͍

53. ࣮͸·ܾͩఆ൛͕ͳ͍

54. TSDBʹٻΊΒΕΔ΋ͷ͸ ͳʹ͔

55. TSDBʹٻΊΒΕΔੑ࣭ • TSDBͱͯ͠ͷجຊతͳػೳ • ৴པੑ • εέʔϥϏϦςΟ • ߴ଎ͳϨεϙϯελΠϜ •

    ௿ίετ

56. ࠓޙͷTSDBʹٻΊΒΕΔػೳ • ߴղ૾౓σʔλͷ௕ظอଘ • ௒ߴղ૾౓σʔλͷରԠ • େྔͷܥྻͷߴ଎දࣔ • (ܥྻʹର͢Δϝλσʔλඥ෇͚) •

    (ػցֶशΛ༻͍ͨҟৗݕ஌΍ྨࣅϝτϦοΫ ݕࡧ)

57. ͜ΕΒΛ͢΂ͯຬͨ͢ʹ͸ Ͳ͏͢Δ͔

58. ύϒϦοΫΫϥ΢υͷ ϚωʔδυαʔϏεΛར༻

59. ݹ͍σʔλΛS3ͷΑ͏ͳ ͍҆σʔλετϨʔδʹୀආ

60. TSDBʹٻΊΒΕΔੑ࣭ • TSDBͱͯ͠ͷجຊతͳػೳ • ৴པੑ • εέʔϥϏϦςΟ • ߴ଎ͳϨεϙϯελΠϜ •

    ௿ίετ ← Graphiteϕʔεͷ ΠϯλϑΣʔε ← ϚωʔδυαʔϏεͰ୲อ ← ϚωʔδυαʔϏεͰ୲อ ← σʔλߏ଄΍ ɹσʔλ഑ஔͷ޻෉ ← ݹ͍σʔλΛ͍҆ετϨʔδ΁ୀආ ← ͦͷଞɺෳ਺ͷ࠷దԽΛ࣮ࢪ

61. ৽TSDBΞʔΩςΫνϟ

62. write datapoints ϝοηʔδΩϡʔ ίϯγϡʔϚ ΠϯϝϞϦ DB Webαʔό ΦϯσΟεΫ DB େ༰ྔ

    ετϨʔδ read datapoints subscribe flush & rollup put purge table read read read

63. write datapoints ϝοηʔδΩϡʔ ίϯγϡʔϚ ΠϯϝϞϦ DB Webαʔό ΦϯσΟεΫ DB େ༰ྔ

    ετϨʔδ read datapoints subscribe flush & rollup put purge table read read read

64. write datapoints ϝοηʔδΩϡʔ ίϯγϡʔϚ ΠϯϝϞϦ DB Webαʔό ΦϯσΟεΫ DB େ༰ྔ

    ετϨʔδ read datapoints subscribe flush & rollup put purge table read read read 3छͷDB ௚ۙ1࣌ؒ ίʔϧυσʔλ ϗοτσʔλ

65. write datapoints ϝοηʔδΩϡʔ ίϯγϡʔϚ ΠϯϝϞϦ DB Webαʔό ΦϯσΟεΫ DB େ༰ྔ

    ετϨʔδ read datapoints subscribe flush & rollup put purge table read read read ΩϡʔͷલޙͰܥΛ෼ׂ

66. write datapoints ϝοηʔδΩϡʔ ίϯγϡʔϚ ΠϯϝϞϦ DB Webαʔό ΦϯσΟεΫ DB େ༰ྔ

    ετϨʔδ read datapoints subscribe flush & rollup put purge table read read read 3छDB͔ΒಡΈ෼͚Δ WebΞϓϦέʔγϣϯ Graphiteޓ׵ͷΠϯλϑΣʔε

67. write datapoints ϝοηʔδΩϡʔ ίϯγϡʔϚ ΠϯϝϞϦ DB Webαʔό ΦϯσΟεΫ DB େ༰ྔ

    ετϨʔδ read datapoints subscribe put & rollup put purge table read read read ,JOFTJT -BNCEB 3FEJT
    $MVTUFS %ZOBNP%# 4 ϚωʔδυαʔϏεͳͷͰ جຊతʹεέʔϧΞ΢τ΍
 ৑௕ੑ͸อূ͞Ε͍ͯΔ

68. ৽TSDBͷσʔλߏ଄

69. ΦϯσΟεΫDB (DynamoDB)ͷεΩʔϚઃܭ ߦΩʔ ྻσʔλ ϝτϦοΫ໊ [ timestamp(int32):value(float64), … ] TFSWFSMPBEBWH

    <
    
     ʜ
    > TFSWFSMPBEBWH <
    
     ʜ
    > ಉ͡ܥྻͷϝτϦοΫͷϦετ

70. Time-based range partitioning (DynamoDB) ߦΩʔ ྻσʔλ ϝτ ϦοΫ ໊ [

    timestamp(int 32):value(float 64), … ] TFSWFS MPBEB WH <
    ʜ
    > 2016/11/30 2016/11/29 2016/11/22 ߦΩʔ ྻσʔλ ϝτ ϦοΫ ໊ [ timestamp(int 32):value(float 64), … ] TFSWFS MPBEB WH <
    ʜ
    > ߦΩʔ ྻσʔλ ϝτ ϦοΫ ໊ [ timestamp(int 32):value(float 64), … ] TFSWFS MPBEB WH <
    ʜ
    > …… େྔϨίʔυ࡟আॲཧ͸ෛՙ͕ߴ͍ͨΊ ݹ͍೔෇ςʔϒϧΛυϩοϓ͢ΔΑ͏ʹ େ༰ྔ ετϨʔδ ྻσʔλΛҰఆαΠζʹऩΊΔ෭࣍తޮՌ

71. Write-through cache (Redis +DynamoDB) Redis ߦΩʔ ྻσʔλ ϝτϦο Ϋ໊ [

    timestamp(int32):va lue(float64), … ] TFSWFSMP BEBWH <
    ʜ
    > DynamoDB ίϯγϡʔϚ ҰఆҎ্ಉ͡ܥྻͷσʔλϙΠϯτ͕ ͨ·ͬͨΒϑϥογϡ ౸ணͨ͠ σʔλϙΠϯτΛ ΠϯϝϞϦDBʹ֨ೲ

72. Graphiteޓ׵WebαʔόͷI/OଟॏԽ Redis DynamoDB S3    … … BatchGetItem

    API (100items) εϨου εϨουʹΑΓ I/OଟॏԽ͠ ϨεϙϯελΠϜ Λߴ଎Խ

73. ৽TSDBʹ͓͚Δͦͷଞͷ࠷దԽΞΠσΞ • ௿ղ૾౓ςʔϒϧͷߏஙʹΑΔɺ௕ظϨϙʔ τදࣔͷϨεϙϯεߴ଎Խ • Amazon AthenaΛ༻͍ͨS3ͷσʔλ഑ஔ࣮ ૷ͷ؆қԽ • SIMDԋࢉʹΑΔ࣌ܥྻσʔλू໿ԋࢉͷߴ

    ଎Խ

74. ৽TSDBͷܽ఺

75. ৽TSDBΞʔΩςΫνϟͷܽ఺ • ΦϯϓϨϛε؀ڥͰ͸࣮ݱͰ͖ͳ͍ • ͨͩ͠ɺӡ༻ίετΛແࢹ͢Ε͹ࣅͨΑ͏ͳOSS࣮ ૷ʹஔ͖׵͑͸Մೳ • Kinesis=>KafkaɺDynamoDB=>CassandraͳͲ • ίϯϙʔωϯτ͕ଟ͍ͨΊɺҰݟෳࡶͰ͋Δ

    • ൚༻DBʹ͍ͭͯੑೳ΍ίετΛ࠷దԽ͠Α͏ͱ͢Δ ͱɺͲ͏ͯ͠΋ෳ਺ͷಛੑΛ΋ͭDBΛ૊Έ߹ΘͤΔ ඞཁ͕͋Δ

76. ৽TSDBΞʔΩςΫνϟͷ֓ཁ·ͱΊ • ৴པੑ޲্ͱӡ༻ίετ࡟ݮͷͨΊϚωʔδ υαʔϏεΛར༻ • ੑೳͱۚમίετͷ࠷దԽͷͨΊʹɺΞΫη εύλʔϯʹԠͯ͡ෳ਺ͷҟͳΔDBΛซ༻ • ϝοηʔδΩϡʔΛڬΉ͜ͱͰγεςϜΛૄ ݁߹Խ͢Δ

77. WebαʔϏεӡ༻ ͷΤοηϯε

78. ܭࢉػΞʔΩςΫνϟ/OSͷجຊͷԠ༻ • ϝϞϦͱετϨʔδ • write-through cache • ௿ղ૾౓ςʔϒϧ͸ࣄલܭࢉ΍Ωϟογϡͷ ֓೦ʹ૬౰

79. ҟͳΔෳ਺ͷDBͷ૊Έ߹Θͤ • WebΞϓϦέʔγϣϯͷੈքͰ͸Α͘࠾༻͞Ε Δख๏ • ϝΠϯσʔλϕʔε͸RDBMSͱͯ͠MySQL΍ PostgreSQLΛར༻ • ੑೳ͕ٻΊΒΒΕΔͱ͜Ζ͸ɺKVSͱͯ͠Redis ΍memcachedΛར༻

    • ݕࡧΤϯδϯͱͯ͠ɺElasticsearch΍SolrΛར༻

80. ٕज़બ୒ • OSSͰ͋Ε͹ίϛϡχςΟϕʔε͔Ͳ͏͔ • ੈͷதʹӡ༻ࣄྫ͕͋Δ͔ • ୤ग़Մೳ͔Ͳ͏͔ • ଞͷࣄۀʹطʹ࣮੷͕͋Δ͔΋͘͠͸ԣల։ Ͱ͖Δ͔

    • ϏδωεΠϯύΫτΛߟྀ͢Δ

81. https://developer.hatenastaff.com/entry/2015/12/25/140233

82. https://blog.yuuk.io/entry/infra-for-newlang

83. ϞμϯͳΞʔΩςΫνϟ • Serverless Architecture • Microservices • Immutable Infrastructure

84. WebαʔϏεӡ༻ͷಛੑͱ՝୊ • WebαʔϏε͸σʔλʹՁ஋͕͋Δ • σʔλΛ΋ͭ(stateful)αʔόͷӡ༻ʹ͸෼ࢄ γεςϜͷ؍఺Ͱ՝୊͕ଟ͍ • ϚωʔδυαʔϏεΛ࢖͏͚ͩͰ͸ɺ݁ہӡ ༻͕Ϋϥ΢υͷԼଆʹҠߦ͢Δ͚ͩ •

    ੈͷத͔Βӡ༻͕ফ͑ΔΘ͚Ͱ͸ͳ͍

85. WebαʔϏεӡ༻ͷະདྷ

86. ਓ͕ؒཧղͰ͖ΔγεςϜ ਓ͕ؒཧղͰ͖ͳ͍γεςϜ

87. ਓ͕ؒཧղͰ͖ΔγεςϜ • γεςϜΛཧղՄೳͳখ͞ͳ୯Ґʹ෼ׂ͢Δ • খ͞ͳίϯϙʔωϯτΛཁٻʹԠͯ͡૊Έ߹ ΘͤΔ • ਓखʹΑΔӡ༻ίετΛ࠷খԽ͢Δ

88. ਓ͕ؒཧղͰ͖ͳ͍γεςϜ • શࣗಈͳγεςϜ • ੜ໋׆ಈͷΑ͏ʹγεςϜ͕ࣗ཯తʹम෮Ͱ ͖Δ • ͨͩ͠ɺશମ૾͸ਓ͕ؒཧղͰ͖ͳ͍ͷͰɺ ͨͱ͑όά͕͋ͬͯ΋ຊ౰ʹಈ͖ଓ͚ΔͳΜ Β͔ͷอূ͕ඞཁ

89. ·ͱΊ • αʔόϞχλϦϯά͸γεςϜӡ༻ͷॏཁͳج൫ • αʔόϞχλϦϯάͷίϯϙʔωϯτͷ͏ͪɺಛʹ TSDBʹ՝୊͕͋Δ • TSDBͷཁٻϨϕϧ͕ߴ·͍ͬͯΔҰํͰɺܾఆ൛͕ͳ͍ • ϚωʔδυαʔϏε͔ͭෳ਺ͷ൚༻ετϨʔδΛ૊Έ߹

    ΘͤͨTSDBΛ࣮૷͢Δख๏Λ঺հ • TSDBͷ։ൃΛྫʹWebαʔϏεͷӡ༻ࣄ৘Λؒ઀తʹ ঺հ

90. https://hatenacorp.jp/recruit/fresh/operation-engineer

91. ຊεϥΠυͷKeynoteςϯϓϨʔτͱͯ͠ shoya140͞ΜͷZebra(https://shoya.io/blog/zebra/) Λ࢖Θ͍͖ͤͯͨͩ·ͨ͠ αʔόϞχλϦϯά޲͚ ࣌ܥྻσʔλϕʔεͷ୳ڀ גࣜձࣾ͸ͯͳ ΢ΣϒΦϖϨʔγϣϯΤϯδχΞ TSUBOUCHI, Yuuki @y_uuk1