4.1 An Introduction to Logs and Layouts

This is a very crucial section because it explains the basic concepts which are needed to work with Ascolog Insight effectively. Events which occur in a software system are often documented by log records. The format (structure) of log records from various sources (e.g. software applications) can be very different and depends on the following two aspects:

  • the information that is provided

  • the way how the information is provided

The following examples show two simple logs that have a different format. Example 1 and Example 2 show log records of two different types.

2011-10-10, 15:01, ERROR, File not found.
2011-10-10, 15:10, WARNING, Screen saver is turned on.

Example 1: First example for a log record type

The log records in Example 2 provide more information than the records in Example 1 because they also contain a severity information (“High” or “Low”). Moreover, the logs use a different way to display the timestamp and to separate the different pieces of information (comma vs. space).

10.10.2011 15:01:12 High Warning <File error>
10.10.2011 15:10:13 Low Information <User A not found>

Example 2: Second example for a log record type

In order to analyze logs Ascolog Insight must identify the pieces of information which are provided by a log record.

A log record usually consists of two parts (see Example 3):

  • a header which provides recurring meta information about the logged event (e.g. severity, time of occurrence, etc.)

  • a description of the event

Headers will be highlighted in red, descriptions will be highlighted in green in the following examples.

10.10.2011 15:01:12 High Warning <Access denied!>
10.10.2011 15:01:13 High Info <Created User A, User B, User C, User D, User E, User F, User G, User H, User I, User J>
10.10.2011 15:01:14 High Error <File not found!>

Example 3: Second example for a log record type

In order to identify the different pieces of information that a log record provides Ascolog Insight uses a formal grammar (often simply called grammar). The application handles all types of log formats that can be described by a context-free grammar of type LL1. Further log formats can be handled by using custom columns definitions which are explained in chapter 10 Custom Columns Definitions (CCD).

In most cases the grammar (as part of a layout) which describes your log records already exists. If the needed grammar does not exist you can use Ascolog Insight's layout creation wizard to automatically create a grammar for your purposes. This means that most users can effectively analyze their logs without having a knowledge of formal grammars.

If you need support when creating a grammar don't hesitate to contact us. We will be glad to help you.

Since there are many good introductions to formal grammars in the world wide web this documentation won't provide an introduction to grammars but focuses on how grammars are used in Ascolog Insight (see chapter 13 Grammars -Advanced Topics). A good introduction to formal grammars can be found in the article Formal Grammars in the English Wikipedia (http://en.wikipedia.org/wiki/Formal_grammar).

As stated already above a grammar is part of a layout and describes the format of a log (e.g. the log format used by your software application or the log format used by a certain web server).

However, to be more precisely, a grammar only describes the header of a log record.

A header must be located at the beginning of a line. This is useful because so it's possible to log data that has the same structure as the header like in Example 4 where the header is just a timestamp (red text) and the description of the event (green text) contains a date in the same format as the header's timestamp:

2014-04-05 13:14:12,123 The time of the last registration was: 2014-04-05 13:12:13,123

Example 4: Header and description contain a timestamp of the same format

However, if the header is known the description is also defined. The description is all data between two headers (or the end of the log for the last log record in a log file). Log records that span several lines are supported but the header part cannot span several lines. The pieces of information that are identified by a grammar are organized in columns. For the description the column TEXT is used.

A layout stores all information which relates to working with a certain log type. It consists of the following main components:

  • Grammar definitions

  • Custom columns definition

  • Configuration settings

Besides columns created by the nonterminal symbols of a grammar definition Ascolog Insight also knows custom columns which are generated by the commands of a custom columns definition.

In order to get an idea of a custom columns definition have a look at the extracts shown in Example 5 and Example 6. It is not necessary to understand the command's syntax in detail.

In Example 5 the command map will allow proper sorting by mapping different type strings to values, e.g. if the column _TYPE contains the string "T" or "Trace", then the column TYPE will contain the string “0”.

TYPE = map(_TYPE, _TYPE, "T(race)?", "0", "I(nfo)?", "1", "W(arning)?", "2", "E(error)?", "3");

Example 5: How the map command can be used in a custom columns definition

In Example 6 the command xtr will extract the memory usage expressed as a percentage from the FULLROW column and create a new column called MEMORY for the extracted values.

MEMORY = xtr(FULLROW, "Memory:[ %09]+", "-?[0-9]+", "\\%");

Example 6: How the xtr command can be used in a custom columns definition

The configuration settings mentioned in the enumeration above store information like paths to files, macro defines or pattern definitions.

Usually the layout you need already exists and just has to be selected by using the Load... command from the Layout menu. How to create a layout yourself is described in Creating a Layout with the Wizard respectively Creating a Layout without the Wizard. There is also the possibility to purchase customized layouts from Ascolog.

In Ascolog Insight a table is used to display logs records. A row represents a log record. A column contains information of a certain type, e.g. a timestamp, the type of a log record (Error, Warning, Information), etc.