Prometheus - Grafana ACI monitor
prometheus is an open-source metrics collection and processing tool.
ACI sandbox
Use sandpit for an example to monitor ACI in Grafana
https://sandboxapicdc.cisco.com/
username:
admin
password:
!v3G@!4@Ycurl 'http://localhost:9643/probe?target=cisco_sandbox'
{"fabric":"cisco_sandbox","level":"error","msg":"Class request fvSyntheticIp failed - ACI api returned 400.","requestid":"2mDp6WTllL4ul2cNnrA8otjh7qt","time":"2024-09-18T10:53:50+10:00"}
{"level":"info","msg":"could not convert value to float, will return 0.0 ","time":"2024-09-18T10:53:50+10:00","value":""}
{"class":"eqptLC","exec_time":785518,"fabric":"cisco_sandbox","length":107,"level":"info","method":"GET","msg":"api call fabric","requestid":"2mDp6WTllL4ul2cNnrA8otjh7qt","status":200,"time":"2024-09-18T10:53:51+10:00","uri":"https://sandboxapicdc.cisco.com/api/class/eqptLC.json?rsp-subtree-include=count"}
{"exec_time":5956897,"fabric":"cisco_sandbox","level":"info","msg":"total scrape time ","requestid":"2mDp6WTllL4ul2cNnrA8otjh7qt","time":"2024-09-18T10:53:51+10:00"}
{"exec_time":5956925,"fabric":"cisco_sandbox","level":"info","msg":"total query collection time","requestid":"2mDp6WTllL4ul2cNnrA8otjh7qt","time":"2024-09-18T10:53:51+10:00"}
{"exec_time":604,"fabric":"cisco_sandbox","level":"info","msg":"metrics to prometheus format","requestid":"2mDp6WTllL4ul2cNnrA8otjh7qt","time":"2024-09-18T10:53:51+10:00"}
{"exec_time":5976152,"fabric":"cisco_sandbox","length":19167,"level":"info","method":"GET","msg":"api call","requestid":"2mDp6WTllL4ul2cNnrA8otjh7qt","status":200,"time":"2024-09-18T10:53:51+10:00","uri":"/probe?target=cisco_sandbox"}
aci_object_instances{aci="ACI Fabric1",class="fvAEPg",fabric="cisco_sandbox"} 10
aci_object_instances{aci="ACI Fabric1",class="fvBD",fabric="cisco_sandbox"} 75
aci_object_instances{aci="ACI Fabric1",class="fvTenant",fabric="cisco_sandbox"} 11
aci_object_instances{aci="ACI Fabric1",class="vnsCDev",fabric="cisco_sandbox"} 0
aci_object_instances{aci="ACI Fabric1",class="vnsGraphInst",fabric="cisco_sandbox"} 0
aci_object_instances{aci="ACI Fabric1",class="fvIP",fabric="cisco_sandbox"} 0
aci_object_instances{aci="ACI Fabric1",class="fvSyntheticIp",fabric="cisco_sandbox"} 0
aci_object_instances{aci="ACI Fabric1",class="eqptLC",fabric="cisco_sandbox"} 0
# HELP aci_scrape_duration_seconds The duration, in seconds, of the last scrape of the fabric
# TYPE aci_scrape_duration_seconds gauge
aci_scrape_duration_seconds{aci="ACI Fabric1",fabric="cisco_sandbox"} 5.956897104
# HELP aci_up The connection state 1=UP, 0=DOWN
# TYPE aci_up gauge
aci_up{aci="ACI Fabric1",fabric="cisco_sandbox"} 1
Prometheus node exportor
The Prometheus Node Exporter exposes a wide variety of hardware- and kernel-related metrics.
curl http://localhost:9100/metrics
promhttp_metric_handler_requests_total
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection # HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.13
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 524288
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 8
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.9550208e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.72660336979e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.650778112e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
Prometheus.yml Configuration
The Prometheus configuration is done through YAML files, the main one being located at /etc/prometheus/prometheus.yml.
[Unit]
Description=Prometheus exporter for machine metrics
Requires=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Restart=on-failure
EnvironmentFile=-/etc/conf.d/prometheus-node-exporter
ExecStart=/usr/bin/prometheus-node-exporter $NODE_EXPORTER_ARGS
ExecReload=/bin/kill -HUP $MAINPID
NoNewPrivileges=true
ProtectHome=read-only
ProtectSystem=strict
[Install]
WantedBy=multi-user.target Loaded: loaded (/usr/lib/systemd/system/prometheus-node-exporter.service; enabled; preset: disabled)
Active: active (running) since Wed 2024-09-18 06:02:50 AEST; 47s ago
Invocation: 37eeab53543a4b788d8d4fcd8d3ee530
Main PID: 9810 (prometheus-node)
Tasks: 6 (limit: 17706)
Memory: 6.7M (peak: 7.7M)
CPU: 14ms
CGroup: /system.slice/prometheus-node-exporter.service
└─9810 /usr/bin/prometheus-node-exporter