Htykuut

I want make a log management system through EFK. I'm trying for days now to get my multiline mycat log parser to work with fluent-bit.

My fluentbit configuration:

parsers.conf:

    [PARSER]

      Name        mycat_error_log_parser_head

      Format      regex

      Regex       ^(?<time>(d)+(-d+)+(S)+W(S)+)(s+)(?<action>S+)(s+)(?<on>S+)

      Time_Key    time

      Time_Format %Y-%m-%d %H:%M:%S.%L

      Time_Keep   On



    [PARSER]

      Name        mycat_error_log_parser

      Format      regex

      Regex       ^(?<time>(d)+(-d+)+(S)+W(S)+)(s+)(?<action>S+)(s+)(?<on>S+)(s+)(?<content>(.|s)*)$

      Time_Key    time

      Time_Format %Y-%m-%d %H:%M:%S.%L

      Time_Keep   On



fluent-bit.conf

    [INPUT]

        Name tail

        tag mycat

        path /var/log/mycat.log

        Multiline On

        Parser_Firstline mycat_error_log_parser_head

        Parser mycat_error_log_parser

        Path_Key file

and the mycat.log looks like this:

mycat.log

2018-08-02 09:59:41.205  INFO [$_NIOConnector] (io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostS1 of schema mysql

2018-08-02 09:59:53.939  INFO [Timer0] (io.mycat.backend.datasource.PhysicalDatasource.getConnection(PhysicalDatasource.java:529)) - no ilde connection in pool,create new connection for hostS1 of schema mysql

2018-08-02 10:00:01.173 ERROR [$_NIOConnector] (io.mycat.net.NIOConnector.finishConnect(NIOConnector.java:155)) - error:

java.net.ConnectException: Connection timed out

    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.7.0_111]

    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744) ~[?:1.7.0_111]

    at io.mycat.net.NIOConnector.finishConnect(NIOConnector.java:165) ~[Mycat-server-1.6.5-release.jar:?]

    at io.mycat.net.NIOConnector.finishConnect(NIOConnector.java:143) ~[Mycat-server-1.6.5-release.jar:?]

    at io.mycat.net.NIOConnector.run(NIOConnector.java:98) ~[Mycat-server-1.6.5-release.jar:?]

2018-08-02 10:00:01.173  INFO [$_NIOConnector] (io.mycat.net.AbstractConnection.close(AbstractConnection.java:514)) - close connection,reason:java.net.ConnectException: Connection timed out ,MySQLConnection [id=0, lastTime=1533175073937, user=root, schema=mysql, old shema=mysql, borrowed=false, fromSlaveDB=true, threadId=0, charset=utf8, txIsolation=3, autocommit=true, attachment=null, respHandler=null, host=parse1, port=3306, statusSync=null, writeQueue=0, modifiedSQLExecuted=false]

2018-08-02 10:00:01.173  INFO [$_NIOConnector] (io.mycat.sqlengine.SQLJob.connectionError(SQLJob.java:117)) - can't get connection for sql :select user()

I think about i'm close now, but no luck so far.

In my kibana. I get follow result:
enter image description here

My multi-line error log is missing. but i testing in rubular it normal parsing
enter image description here

Any help would be appreciated.