ICode9
精准搜索请尝试： 精确搜索
首页
编程语言>
数据库
系统相关
互联网
其他分享
Python
Java
JavaScript
android
PHP
首页 > 其他分享> 文章详细

使用 filebeat原生处理日志时间，就是使用日志文件中message字段值开头的时间覆盖默认的@timestamp时间，使用filebeat processor配置，pipeline不符合要求

2022-05-28 17:33:16  阅读：530  来源： 互联网

标签：filebeat java handlers org io 使用 undertow 日志 servlet


情况说明

Filebeat 收集的日志发送到 ElasticSearch 后，会默认添加一个 @timestamp 字段做为时间戳用于检索，而日志中的信息会所有添加到 message 字段中，可是这个时间是 Filebeat 采集日志的时间，不是日志生成的实际时间，因此为了便于检索日志，须要将 @timestamp 替换为 message 字段中的时间。

首先日志格式以下：

# info文件的日志内容

2022-05-24 00:12:19.196 INFO  com.jdd.parking.modules.chargefee.standard.rest.GetCarChargeFeeController Line:131 - 有牌车出口码计费,plateNo:鄂A98J51
2022-05-24 00:12:19.196 INFO  com.jdd.parking.modules.chargefee.standard.service.impl.ChargeFeeV2ServiceImpl Line:64  - 计费服务类,出场码计费,传入参数:OutParkChargeFeeDTO(parkCode=58818, serialNo=c79e5114-7103aa77, plateNo=鄂A98J51, imgPath=https://jdd-parking.oss-cn-qingdao.aliyuncs.com/picture/1/c79e5114-7103aa77/20220524/00/20220524_001218_813.jpg)
2022-05-24 00:12:19.205 INFO  com.jdd.parking.modules.chargefee.standard.utils.ChargeFeeUtils Line:444 - 计费服务类,计费开始,车牌号：鄂A98J51,openId:null
2022-05-24 00:12:19.208 INFO  com.jdd.parking.modules.chargefee.standard.utils.ChargeFeeUtils Line:877 - 计费服务类,单次入场纪录计费,传入参数：计费区域Id：59a3ef3754d87c85b02c51e4ed0e03ae,计费开始时间：2022-05-23 22:44:04,计费结束时间：2022-05-24 00:12:19
2022-05-24 00:12:19.209 INFO  com.jdd.parking.modules.chargefee.standard.utils.ChargeFeeUtils Line:897 - 计费服务类,单次入场纪录计费,按24小时计算一天,停车整天数：0天,不足整天的停车时间：88分钟

# error文件的日志内容

2022-05-24 00:12:19.210 ERROR com.jdd.parking.modules.chargefee.standard.service.impl.ChargeFeeV2ServiceImpl Line:129 - 计费服务类,异常信息
com.jdd.parking.common.exception.JddException: 计费规则模块不存在
	at com.jdd.parking.modules.chargefee.standard.utils.ChargeFeeUtils.lessThanOneDayFee(ChargeFeeUtils.java:85)
	at com.jdd.parking.modules.chargefee.standard.utils.ChargeFeeUtils.chargeEnterLogFee(ChargeFeeUtils.java:920)
	at com.jdd.parking.modules.chargefee.standard.utils.ChargeFeeUtils.chargeTotalFeeMonthlyV3(ChargeFeeUtils.java:525)
	at com.jdd.parking.modules.chargefee.standard.service.impl.ChargeFeeV2ServiceImpl.outParkChargeFeeV2(ChargeFeeV2ServiceImpl.java:97)
	at com.jdd.parking.modules.chargefee.standard.rest.GetCarChargeFeeController.getOutParkChargeFeeV2(GetCarChargeFeeController.java:133)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:517)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:584)
	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
	at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
	at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130)
	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78)
	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449)
	at java.lang.Thread.run(Thread.java:748)
2022-05-24 00:12:19.211 ERROR com.jdd.parking.modules.chargefee.standard.rest.GetCarChargeFeeController Line:137 - 有牌车出口码计费失败,错误信息：{}
com.jdd.parking.common.exception.JddException: 计费服务类,出场码计费系统错误,计费规则模块不存在
	at com.jdd.parking.modules.chargefee.standard.service.impl.ChargeFeeV2ServiceImpl.outParkChargeFeeV2(ChargeFeeV2ServiceImpl.java:130)
	at com.jdd.parking.modules.chargefee.standard.rest.GetCarChargeFeeController.getOutParkChargeFeeV2(GetCarChargeFeeController.java:133)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:517)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:584)
	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
	at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
	at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130)
	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78)
	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449)
	at java.lang.Thread.run(Thread.java:748)

当前的日志传输路径是filebeat->es，没有使用logstash

解决办法

使用elasticsearch pipeline功能

参考文章：http://www.javashuo.com/article/p-ghefxmal-nu.html

索引文件名上的时间是当前时间

pipeline中没有加"timezone": "Asia/Shanghai", 可以看到，有的是当前系统时间，有的是UTC时间，但是真正想要的时间是日志message开头的时间

pipeline中加上"timezone": "Asia/Shanghai", 可以看到，有的是当前系统时间，有的是UTC时间，但是真正想要的时间是日志message开头的时间

结合以上实践，使用elasticsearch pipeline功能不符合要求，建议使用使用filebeat processor配置

使用filebeat processor配置

参考网址：https://blog.csdn.net/qq_34083066/article/details/115354511

索引文件名上的时间是日志message开头截取的时间

# cat /etc/filebeat/filebeat.yml
filebeat.inputs:

- type: log
  enabled: true
  paths:
    - /opt/log_error.log
  fields:
    log_source: fee
    log_type: error
  fields_under_root: true
  tags: ["application"]
  multiline.pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d* ' # 匹配日志开头的时间，划分为一行
  multiline.negate: true
  multiline.match: after

- type: log
  enabled: true
  paths:
    - /opt/log_info.log
  fields:
    log_source: fee
    log_type: info
  fields_under_root: true
  tags: ["application"]
  multiline.pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d* '
  multiline.negate: true
  multiline.match: after

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
  reload.period: 10s

setup.ilm.enabled: false
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 0
  index.codec: best_compression

setup.dashboards.enabled: true

setup.kibana:
  host: "192.168.20.250:5601"

output.elasticsearch:
  hosts: ["localhost:9200"]
  indices:
    - index: "work-%{+yyyy.MM.dd}"
      when.contains:
        tags: "application"

processors:
  - decode_json_fields:
      fields: ["message"]
      target: "" 
      overwrite_keys: false
      process_array: false
      max_depth: 1
  # 这个脚本的内容是根据message字段值开始的俩空格截取值然后拼接在一起，也就是时间：2022-05-24 00:12:19.211，赋给新变量：start_time
  - script:
      lang: javascript
      id: my_filter
      tag: enable
      source: >
        function process(event) {
            var str= event.Get("message");
            var time =str.split(" ").slice(0,2).join(" ");
            event.Put("start_time",time);
        }
  # 使用新变量的值start_time替换默认的@timestamp的值
  - timestamp:
      field: start_time
      #timezone: Asia/Shanghai
      target_field: "@timestamp"
      layouts:
        - '2006-01-02 15:04:05'
        - '2006-01-02 15:04:05.999'
      test:
        - '2006-01-02 15:04:05'
        - '2006-01-02 15:04:05.999'
  # 移除无用的字段
  - drop_fields:
      fields: ["input","agent","ecs","start_time"]

效果显示



本文中timezone说明

@timestamp 这个字段是系统自带的，默认时间就是采集log的UTC时间 。例如 现在时间是2020-12-30 18:59:58.234 ，那么他的值为2020-12-30T10:59:58.234Z

现在我们手动替换@timestamp为日志时间（东八区），需要将时间字符串解析成UTC时间，赋值给@timestamp，那么我们必须显示的指定timezone 信息。

1.指定timezone后的表现
原先log时间字符串：2020-08-05 16:21:51.824。重新赋值的@timestamp值： 2020-08-05T08:21:51.824Z （正确）

2.未指定timezone后的表现

原先log时间字符串：2020-08-05 16:21:51.824。重新赋值的@timestamp值： 2020-08-05T21:51.824Z （默认使用了UTC零时区解析）

按上面配置是没有指定timezone的，这个一定要注意。若是指定了则会跟log时间相差8个小时

标签：filebeat,java,handlers,org,io,使用,undertow,日志,servlet
来源： https://www.cnblogs.com/sanduzxcvbnm/p/16321325.html

本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享；
2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关；
3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关；
4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除；
5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。

关于我们 | 联系我们 | 留言反馈

专注分享技术，共同学习，共同进步。侵权联系[81616952@qq.com]

Copyright (C)ICode9.com, All Rights Reserved.

ICode9版权所有