Special Devo tags and data tables
This article explains the difference between tags and tables and describes some of the special tags used for sending data to Devo and the tables you might see in your finder.
About tags and tables
In Devo, tags are used primarily for ingesting and storing data sent from a source system to Devo, while tables are used to retrieve, parse, and display data saved in Devo. In effect, a table offers a view of the data that you've saved. For example, it's possible to create two tables that retrieve data from the same file but display that data differently. One could display fewer columns or display the columns in a different order. Similarly, a table can be created that draws data from multiple files that contain similar data in a similar structure. An example would be the
firewall.fortinet.traffic table. It is a parent table and you can read more about them later in this article.
Most of the time, a tag and its resulting table have the same name, as is the case with
cms.wordpress.stdout. However, there are a number of tags that contain hyphens, a character not allowed in table names. In these cases, the table name differs from the tag and uses camel case instead. For example, the
web.apache.access-lt tag corresponds to the
web.apache.accessLt table in Devo. If you are ever in doubt, you can see the table's name in the operations' path of the search window like this:
Why is this important?
Tags and tables beginning with
my.app can describe two types of data.
Events for which there is no Devo tag
When you want to send events to Devo from a source for which there is no Devo tag, you can create your own tag using
my.app as the prefix. This can happen with proprietary data sources or publicly-available sources for which Devo has not yet created a tag (and therefore, there's no associated parser).
A note about creating Devo tags
The Devo professional services team can create new tags for any kind of data source. Just contact customer support for details.
The full tag should have at least four levels and may have up to six. The first two must be
my.app. The third and fourth levels are free and should describe the application type and event type respectively. The fifth and sixth levels are optional and should be used to identify the actual source of the events. For example, if there are several servers running the application and reporting events to Devo, these levels can help identify the event's specific event source.
free and required
free, not required but highly recommended
free, not required
|free, not required|
When Devo receives an event with a tag that begins with
my.app, it saves the event to a file and location determined by the tag levels, adds the first four levels of the tag to the finder (this will be the table), and saves the fifth and sixth levels (if any) to the event in the data table. However, since Devo is not equipped with a parser for this event type, when you open the data table, each event row will have only a few fields:
- eventdate - This is the date/time the event was received by Devo.
- cluster - This is the fifth level of the tag (if it was used).
- instance - This is the sixth level of the tag (if it was used).
- message - This contains the unparsed content as it was received.
You can manually parse the content of the message field using the Autoparser tool in the search window, or using the column operations available (for example, creating new columns using the Split (split) operation). Then, when you've parsed all the fields into columns, create a custom table. From this point, you can use the custom table to consult the data parsed into columns.
Tables created by injecting existing data
When you inject data into a new table, the new table will always be assigned the prefix
Tags and tables beginning with
my.upload are created when uploading static files to Devo, either from your local machine or from a Dropbox account.
Uploading static files can be useful to log historical data as opposed to real-time data. These files will be stored in tables starting with the
my.upload tag levels. The first two levels of the tag are predefined as
my.upload. The third and fourth levels are up to you.
For more information about sending event files to a
my.upload table, see Uploading log files.
Lookup tables created by uploading a .csv file or using query data are saved using the tag
Lookup tables enrich the information in raw data tables, added to the virtual data table at query time as new columns. The original data tables are never modified.
For more information, see Data Enrichment.
A table in the finder that begins with
my.synthesis, my.parser or my.tech identifies a custom table; one that a user has created from another table's contents or by forming a union of two tables. When creating a custom table, the user can choose which prefix they prefer to use.
For more information about custom tables, see Custom and union tables.
This tag is only used to test event sending after you have set up a new event source, Devo relay, or to test a relay rule. Events that are received with this tag are saved in the
test.keep.free table which you can consult to confirm that your events have been sent correctly and with the correct data.
While we recommend sending data to Devo in syslog format whenever possible, we support the ingestion of events received in common event format (CEF) via syslog for some technologies. A prime example of the use of this particular format is when ArcSight is used as a log management solution and events are going to be forwarded from ArcSight directly to Devo.
When events are sent in CEF syslog format, they don't require a Devo tag to enable ingestion. However, it is necessary for the platform to be equipped with a parser in order for you to access the data tables and see the events parsed correctly at query time.
Learn more about CEF syslog and check out the list of technologies that Devo currently supports when sent in this special format.
This is a table that Devo automatically generates when it receives any inbound event that has no tag or has a tag that it doesn't recognize. Instead of dropping the event, it is saved to this table. When this table appears in your finder, open it to investigate the nature of the problem, and address it.
Create an alert that notifies the domain administrators when new events are saved to this table.
There are several technologies for which, regardless of brand, the log events contain very similar, or identical fields. When this is the case, as with web servers, firewalls, proxies, and several other technologies, Devo automatically generates a union table that contains the events from several different data sources. Union tables are indicated in the finder by the union icon. Hover over the icon to see a full list of the tables that the union table will collect if available in the deployment.
If your deployment is collecting these events, you will see these tables in your finder (they have the same names as the tags used for ingestion). Open any of these tables to see the collection of events for the corresponding event type.
However, you will also find a parent table called
network.f5.bigip. If you open this table, you will see all the events collected using the
network.f5.bigip.* tags in a single view.
Parent tables exist for several technologies but only when the different event formats and event collection policy allow for it. Here are some other examples of parent tables: