1.环境 environment

centos6.5 jdk 1.8.0_201
docker部署的prometheus+alertmanager+grafana

2.部署jmx-exporter

到jar包路径下下载jmx-exporter和示例配置文件

wget -O tomcat.yml https://github.com/prometheus/jmx_exporter/blob/main/example_configs/tomcat.yml
wget -O jmx_prometheus_javaagent-0.20.0.jar https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.20.0/jmx_prometheus_javaagent-0.20.0.jar
#这步Catalina替换Tomcat很重要 不然prometheus的mertic接口报异常
#参考文章https://www.jianshu.com/p/8a5e681b18ce
sed -i 's#Catalina#Tomcat#g' ./tomcat.yml

javaagent启动jar包

java -javaagent:./jmx_prometheus_javaagent-0.20.0.jar=12345:tomcat.yml -jar admin.jar

错误启动踩坑

jmx_prometheus_javaagent-0.20.0.jar tomcat.yml不在当前目录的启动报错

[root@t01 xxl-job]# java  -Xms512m -Xmx512m -Xmn256m -Dserver.port=8080  CATALINA_OPTS="-javaagent:/opt/prometheus/jmx_prometheus_javaagent-0.20.0.jar=12345:/opt/prometheus/tomcat.yaml" -jar admin.jar 
Error: Could not find or load main class CATALINA_OPTS=-javaagent:.opt.prometheus.jmx_prometheus_javaagent-0.20.0.jar=12345:.opt.prometheus.tomcat.yaml

[root@t01 xxl-job]# java -javaagent:/opt/prometheus/jmx_prometheus_javaagent-0.20.0.jar=30001:/opt/prometheus/tomcat.yaml -Xms512m -Xmx512m -Xmn256m -Dserver.port=8080 -jar admin.jar 
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
        at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.io.FileNotFoundException: /opt/prometheus/tomcat.yaml (No such file or directory)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at java.io.FileReader.<init>(FileReader.java:72)
        at io.prometheus.jmx.JmxCollector.<init>(JmxCollector.java:116)
        at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:55)
        ... 6 more
FATAL ERROR in native method: processing of -javaagent failed

3.prometheus配置增加

  - job_name: 'java'
    static_configs:
      - targets: ['172.30.171.61:12345']  #exporter地址
        labels:
          instance: 'test.com[admin]' #添加的标签

4.prometheus重载

前提你的prometheus开启了web.enable-lifecycle 热重载

curl -X POST -u user1:password http://localhost:9090/-/reload

如果你的prometheus没有密码去掉-u user1:password

curl -X POST  http://localhost:9090/-/reload

5.grafana添加dashboard仪表盘

打开 URL/dashboard/import 输入14845 导入,数据源选择prometheus

这边grafana使用的https://grafana.com/grafana/dashboards/14845-jmx-dashboard-basic/

14845 仪表盘优点

上面的label 值 instance: ‘test.com[admin]’ 配置好 仪表盘可以选择

14845 标准配置

这个模板的标准prometheus配置

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
# Задание сбора данных самой службы
- job_name: 'prometheus'
  static_configs:
  - targets: ['localhost:9090']

# Задание для сбора метрик Java приложений
- job_name: 'jmx'
  metrics_path: /
  scheme: http
  static_configs:
  # Targets for multi-nodes apps
  - targets: ['srv-apps-001.domain.loc:19000']
    labels:
      instance: 'multi-node.domain.com[node-01.domain.loc]'
  - targets: ['srv-apps-002.domain.loc:19000']
    labels:
      instance: 'multi-node.domain.com[node-02.domain.loc]'
  # Target for apps witch one node:
  - targets: ['srv-apps-003.domain.loc:19000', 'srv-apps-004.domain.loc:19000']
  - targets: ['srv-apps-005.domain.loc:19000']
    labels:
      instance: 'one-node.domain.com'

如果你使用的网上教程大多使用的8563模板,感觉不如14845美观
最新的grafana还可能可能会提示

This dashboard depends on Angular, which is deprecated and will stop working in future releases of Grafana.
Read our deprecation notice and migration advice.
该仪表板依赖于 Angular，后者已被弃用，并将在 Grafana 的未来版本中停止工作。
请阅读我们的弃用通知和迁移建议。