Setup Weave Net monitoring using kube-prometheus

Weave Net is a resilient and simple to use CNI provider for Kubernetes. A well monitored and observed CNI provider helps in troubleshooting Kubernetes networking problems. Weave Net emits prometheus metrics for monitoring Weave Net. There are many ways to install Weave Net in your cluster. One of them is using kops.

Following this document, you can setup Weave Net monitoring for your cluster using kube-prometheus.

Service for Weave Net The service which the service monitor scrapes.
ServiceMonitor for Weave Net Service monitor to scrape the Weave Net metrics and bring it to Prometheus.
Prometheus Alerts for Weave Net This will setup all the important Weave Net metrics you should be alerted on.
Grafana Dashboard for Weave Net This will setup the per Weave Net pod level monitoring for Weave Net.
Grafana Dashboard for Weave Net(Cluster) This will setup the cluster level monitoring for Weave Net.

Instructions

You can monitor Weave Net using an example like below. Please note that some alert configurations are environment specific and may require modifications of alert thresholds. For example: The FastDP flows have never gone below 15000 for us. But if this value is say 20000 for you then you can use an example like below to update the alert. The alerts which may require threshold modifications are WeaveNetFastDPFlowsLow and WeaveNetIPAMUnreachable.

local kp = (import 'kube-prometheus/main.libsonnet') +
           (import 'kube-prometheus/addons/weave-net/weave-net.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
  },
  kubernetesControlPlane+: {
    prometheusRuleWeaveNet+: {
      spec+: {
        groups: std.map(
          function(group)
            if group.name == 'weave-net' then
              group {
                rules: std.map(
                  function(rule)
                    if rule.alert == 'WeaveNetFastDPFlowsLow' then
                      rule {
                        expr: 'sum(weave_flows) < 20000',
                      }
                    else if rule.alert == 'WeaveNetIPAMUnreachable' then
                      rule {
                        expr: 'weave_ipam_unreachable_percentage > 25',
                      }
                    else
                      rule
                  ,
                  group.rules
                ),
              }
            else
              group,
          super.groups
        ),
      },
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

After you have the required yamls file please run

kubectl create -f prometheus-serviceWeaveNet.yaml
kubectl create -f prometheus-serviceMonitorWeaveNet.yaml
kubectl apply -f  prometheus-rules.yaml
kubectl apply -f grafana-dashboardDefinitions.yaml
kubectl apply -f grafana-deployment.yaml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!