4.3.4 XML based log formats

Ascolog Insight is able to work with log formats that use XML tags to markup the information of a log record.

Please consider that XML tags increase the size of the log files and that it's hard to read a log file with XML tags without an appropriate tool. The advantage is that XML tags facilitate the processing of such log files with other software tools. However, it's also possible to use a log format without XML and to use Ascolog Insight to convert such a format to a format that contains XML tags (see chapter Exporting Logs, section XML).

<log4j:event logger="XML" timestamp="1355320268501" level="ERROR" thread="9">
  <log4j:message>File not found</log4j:message>
  <log4j:locationInfo class="Logging.Logger" method="LogError"
  file="C:\data\Logger.java" line="50"/>

<log4j:event logger="XML" timestamp="1355320269284" level="INFO" thread="9">
  <log4j:message>Entering Method</log4j:message>
  <log4j:locationInfo class="Logging.Logger" method="CreateFile" 
  file="C:\data\Logging\CreateFile.java" line="38" />

Example 1: A simple log4j log record based on XML

The layout log4j which is located in the folder “log4jXML” delivered with Ascolog Insight is a sample layout which demonstrates how to create layouts that can work with log formats based on XML tags. The XML log files are created with the log4j™ logging framework. The sample layout is able to process log records as listed in Example 1. The used timestamps are based on the UNIX time.

The purpose of the log4j sample layout is to be used as a template for other layouts that have to deal with XML based logs. The approach is to use a grammar definition which just identifies the first tag of a log record, in the log4j sample it's the tag <log4j:event. The complete grammar is shown in Figure 4.3. Please note that the terminal in the grammar does not specify the closing angle bracket of the log4j:event tag because the tag has attributes (logger, timestamp, level).

Sample grammar definition for log record based on XML
Sample grammar definition for log record based on XML

The grammar is used to define what is a log record. So far nothing is different to layouts for log formats without XML. However, in order to extract all the other pieces of information marked by XML tags a custom columns definition is used. More information can be found in the chapter Custom Columns Definitions.You can find the commands which extract the information in the file “log4jXML_basecolumns.cdf” which is also located in the folder of the log4j layout. Example 2 shows the “log4jXML_basecolumns.cdf” file.

#pragma once

#include <basemacros.cdf>

#define FMT(text) rpl(text, "<br/>", "%0d%0a", "&lt;", "<", "&gt;", ">", "&amp;", "&")

LOGGER = xtr(TEXT, "logger=\"", "[^\"]*", "");

TIMESTAMP = cdt(dat($tmpUTC, xtc(TEXT, "timestamp=\"", "[^\"]*", "")), TIME_UTC2TS_MS(dat($tmpUTC)));

LEVEL = map(xtc(TEXT, "level=\"", "[^\"]*", ""), null, "DEBUG", "0", "INFO", "1", "WARN", "2", "ERROR", "3", "FATAL", "4");

THREAD = xtc(TEXT, "thread=\"", "[^\"]*", "");


#include <extendedcolumns.cdf>

MESSAGE = chk(xtr(TEXT, "<log4j:message><!\\[CDATA\\[", "", "\\]\\]>"),
FMT(xtr(TEXT, "<log4j:message>", "[^<]*", "")));

CALLSTACK = chk(xtr(TEXT, "<log4j:throwable><!\\[CDATA\\[", "", "\\]\\]>"),
FMT(xtr(TEXT, "<log4j:throwable>", "[^<]*", "")));

CLASS = xtr(TEXT, "locationInfo class=\"", "[^\"]*", "");

METHOD = FMT(xtc(TEXT, "method=\"", "[^\"]*", ""));

SOURCEFILE = xtc(TEXT, "file=\"", "[^\"]*", "");

LINE = xtc(TEXT, "line=\"", "[^\"]*", "");

Example 2: A custom columns definition is used to extract the logged information marked by XML tags