Introduction to tags
Devo uses tags to identify the type of data and store it in the proper place. When events arrive to the platform, each of them is assigned a timestamp or event date, used to order events chronologically as they are received. Events are then classified using the tag they come with, and saved to a file used to generate a virtual data table when querying. Events collected by Devo are included in different tables depending on their tags.
Tables can be consulted in the Finder of the Data Search area of the application, where you will see the different tag elements to access the necessary table. Each tag identifies a parser, that is to say, events corresponding to a tag are parsed and broken down into different data fields when accessing a data table, using that specific tag parser. After parsing, each column can contain a different data type: string, IP, date, etc. Go to Running a search to learn more about how to access data tables and start querying.
Devo has pre-defined tags for a wide range of common data sources. For more details, see List of supported technologies.
Events can be associated with tags in two different ways:
- In the data source (for example, a rsyslog configuration file).
- By a rule defined in the Devo In-house Relay.
It is not important if events are tagged at the source or by the relay. What is important is that they are tagged before they are delivered to the Devo repository.
In Devo, tags are composed of elements separated by periods following this format:
The two first elements, technology and brand, are mandatory and help to identify the data source.
- Technology identifies the general application type. For example, the technology element of the tag for a web application that sends data to Devo will be web. And events from firewall applications will have tags that start with firewall.
- Brand identifies the specific source of the events. For example, logs generated by an Apache server will have tags starting with web.apache. However, to ensure differentiation, Apache Tomcat logs will have tags starting with web.tomcat.
For some types of logs, it may be enough to apply a tag using only these two mandatory elements. This is the case for the system logs of Unix machines, which are identified with the tag box.unix. However, in most cases, the data will require additional levels of identification, namely type and subtype. While the type element is a single element, subtype can be extended using periods to separate the parts of the subtype, but with a maximum of three parts. For example, web.apache.error.pro.logtrust.server1 is a tag composed of the following elements:
- Technology is web
- Brand is apache
- Type is error to describe the type of events that will be received
- Subtype is pro.logtrust.server1 to identify the specific machine or clone that generated the event
There are some additional rules for the structure of tags:
- Each tag can have up to six levels of detail, the first three being the technology, brand, and type. The subtype can then contain up to three parts.
- The tag can only be made of alphanumeric characters and periods. They cannot contain special characters or symbols.
- The maximum length of a tag is 50 characters.
Dos and don'ts for tags
These are examples of syntactically correct labels:
These are incorrectly designed labels:
- box.unix. is incorrect because it ends with a period.
- web.apache.error.pro.log/trust.server1 is incorrect because one of the elements is a special character ( "/" ).
- a[ ].b.c is also incorrect because it contains special characters ("").
Not all syntactically correct tags are permitted, however. Devo only recognizes a limited set of tags that correspond to the system or user-defined formats.
- box.unix is a permitted tag that identifies the system log of a Unix machine.
- web.apache.error.pro.logtrust.server1 is a permitted tag that Devo recognizes as identifying an error log on an Apache Web server in production (pro) and that has sent the clone server1.
- web.apache is not an acceptable tag, however, because it does not have enough elements to identify the corresponding log. It is too general.
When Devo does not recognize an event (due to poorly designed tags or the absence of tags), it will save the entire event as raw data in a single column, with one event per line in the unknown.unknown table. Here is an example of an unrecognized event where Devo records the receipt timestamp and the event as a single message string:
Proprietary logs or data from a product not yet supported by Devo can be marked with the tag my.app. This tag has been defined specifically for all types of common data coming from unknown sources. When Devo receives an event with this tag for the first time, it will generate a notification to inform you, also providing some help tips on how to separate values in columns assigning the correct values.
Some elements in the structure of a tag need to match a set of legal values predefined by Devo. This is true for technology and brand, but is also sometimes the case for type and subtype. Tag elements that must use a pre-defined value are called prefixed elements.
For example, the third element in Apache log tags is type and identifies the type of log event. Its values are limited to the log types produced by Apache and recognized by Devo:
- If the type is error (as in web.apache.error...), Devo recognizes the events as Apache log error events and will parse it according to the Apache error format.
- If the type is access-clf (as in web.apache.access-clf...), Devo recognizes the events as generated by an Apache access log and will parse it according to the corresponding Apache log format.
In most cases, the values of the tag elements can be user-defined to suit the client installation environment. These are called free elements:
- Free elements can help the Administrator to identify where the event was generated (For example, the machine and/or the environment).
- Free elements might be dependant upon the technology and/or brand elements of the tag.
The Apache logs tags always start with three prefixed elements (web.apache.type). The subtype element can be made of up to three parts using the following guide:
- Working environment (for example, development or production)
- Application name
- Name of the machine or clone that generated the event
Here are some examples of tags using prefixed and free elements:
In this case, the production logs will be searched by adding the restriction environment = pro, while the developer dev1 can see the errors coming out of its job with a combined filter of environment = des and clone = dev1
- If a company has only one web application, the tag for the application is unnecessary, but it has to be specified by assigning it a certain value (for example web.apache.error.pro.X.clon1).
- If a company has multiple Apache servers distributed in multiple data centers, it would require a tag divided into 2 parts (data center and machine). These could be the possible labels:
These tags are correct and legal, and allow you to run searches about the data center and the server, properly decomposing the clone camp in the web.apache table.