ヘビーなGraphite運用

Transcript

1. ϔϏʔͳ Graphite ӡ༻ id:y_uuki Monitoring Casual Talks #6

2. @y_uuk1 id:y_uuki ӡ༻

3. Graphite

4. Graphite Data Flow Graphite Webapp Carbon Whisper Collector TCP HTTP

   metrics Graph JSON Store Get Graphite

5. carbon-cache • 1ϓϩηε/2εϨου • ίωΫγϣϯϋϯυϦϯάεϨου • ॻ͖ࠐΈεϨου • carbon-relay: carbon-cache

   ͷલஈ • (carbon-aggregator)

6. whisper • RRDtool ͱ΄΅ಉ͡σʔλߏ଄ • 1ϑΝΠϧ1ϝτϦοΫ (xxxhost ͷ loadavg) •

   ಠࣗܗࣜͷϑΝΠϧΛ࡞Δ python ϥΠϒϥϦ

7. graphite-web • Web ΞϓϦέʔγϣϯαʔό • ศརάϥϑURLه๏ (ը૾ or json) •

   Python Django • ApacheඞਢͰ͸ͳ͍ (uwsgi, gunicorn) • ͍ͩ͞μογϡϘʔυ

8. app ॳظߏ੒ XSJUF
   SFRVFTU
   QMBJO
   UFYU
   QSPUPDPM SFBE
   SFRVFTU
   )551 carbon-cache graphite-web whisper

9. app LVS carbon-relay LVS carbon-relay carbon-cache graphite-web carbon-cache graphite-web SFQMJDBUJPO

   XSJUF
   SFRVFTU
   QMBJO
   UFYU
   QSPUPDPM SFBE
   SFRVFTU
   )551 whisper whisper XSJUF
   SFRVFTU
   QJDLF
   QSPUPDPM ৑௕Խ
   DBSCPOSFMBZ

10. όοΫΞοϓ app LVS carbon-relay LVS carbon-relay carbon- cache graphite- web

    master whisper carbon- cache graphite- web master whisper carbon- cache graphite- web backup whisper

11. carbon-cache ͕CPUͰ٧·Δ

12. multiple carbon-cache app LVS LVS carbon-relay carbon-relay carbon- cache graphite-web

    tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-backup carbon- cache carbon- cache carbon- cache carbon- relay DPOTJTUFOU
    IBTIJOH

13. carbon-relay ͕CPUͰ٧·Δ

14. ! ! tsdb-relay-lb app LVS LVS carbon-relay carbon-relay carbon- relay

    LVS LVS carbon- relay carbon- relay carbon- relay carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web backup carbon- cache carbon- cache carbon- cache carbon- relay

15. 10k scale Write IOPS ioDrive CPUϘτϧωοΫ

16. ύϥϝʔλνϡʔχϯά

17. carbon-cache MAX_CACHE_SIZE = inf MAX_UPDATES_PER_SECOND = inf MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = inf

    MAX_CREATES_PER_MINUTE = inf CACHE_WRITE_STRATEGY = naive

18. carbon-cache MAX_CACHE_SIZE = MAX_UPDATES_PER_SECOND = MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = MAX_CREATES_PER_MINUTE = CACHE_WRITE_STRATEGY

    = $16ࢗ͞ΔͷͰ
    ͳʹ΋΍Βͤͳ͍

19. σʔλͷҰ؏ੑ

20. DRBD

21. app %3#% XSJUF
    SFRVFTU
    QMBJO
    UFYU
    QSPUPDPM SFBE
    SFRVFTU
    )551 carbon-cache graphite-web primary whisper

    carbon-cache graphite-web secondary whisper

22. ఆظతͳόοΫΞοϓ • Ұ؏ੑͪΌΜͱ͢ΔͳΒߋ৽ΛͱΊΔ࢓૊Έ͕ ผ్ඞཁ (stop slave IO_THREAD;ͱ͔ͳ͍) • ߋ৽ͱΊͯΔؒʹ rsync

    / nc • ϝοηʔδΩϡʔతͳ΋ͷʹஷΊ͜Ή • ΦϯϥΠϯͰόοΫΞοϓͨ͠ͱͯ͠΋Ұఆظ ؒޙwhisper͕aggregate͢Δ

23. ͦͷଞ • ҙ֎ͱηοτΞοϓ໘౗ • த਎͸ಁ͚ͯΔʢPythonʣ • whisper ͷσʔλߏ଄I/Oޮ཰ѱ͍ • MySQL

    ͱ͔࢖͑Δ͚ͲϢʔβ৘ใͱ͔͍ΕΔ ͚ͩ

24. 
    ȦȀȀ✣Ǐ

25. https://mackerel.io “৽͍͠ΞϓϦέʔγϣϯύϑΥʔϚϯε Ϛωδϝϯτ”

26. None

27. ϞχλϦϯά͕޷͖ͳΤϯδχΞΛ
    ืू͓ͯ͠Γ·͢