Special Devo tags and data tables
This article explains the difference between tags and tables and describes some the special tags used for sending data to Devo and the tables you might see in your Finder.
About tags and tables
In Devo, tags are used primarily for ingesting and storing data send from a source system to Devo, while tables are used to retrieve, parse, and display data saved in Devo. In effect, a table offers a view of the data that you've saved. For example, it's possible to create two tables that retrieve data from the same file but display that data differently. One could display fewer columns or display the columns in a different order. Similarly, a table can be created that draws data from multiple files that contain similar data in a similar structure. An example would be the firewall.fortinet.traffic table. It is a parent table and you can read more about them later in this article.
Most of the time, a tag and its resulting table have the same name, as is the case with box.win or cms.wordpress.stdout. However, there are a number of tags that contain hyphens, a character not allowed in table names. In these cases, the table name differs from the tag and uses camel case instead. For example, the web.apache.access-lt tag corresponds to the web.apache.accessLt table in Devo. If you are ever in doubt, you can see the table's name in the operations path of the search window like this:
Why is this important? Because in both Data Search and in Dashboards many users prefer to write their LINQ statements directly. In this case, it is essential to use the correct table name and not to confuse it with the tag.
my.sythesis and my.blend
A table in the Finder that begins with my.synthesis or my.blend identifies a custom table; one that a user has created from another table's contents or by forming a union of two tables. When creating a custom table, the user can choose which prefix they prefer to use.
To help differentiate between the types of tables created in your domain, use my.synthesis for custom tables and my.blend for union tables.
For more information about custom tables, see Custom and union tables.
Tags and tables beginning with my.app can describe two types of data.
Events for which there is no Devo tag
When you want to send events to Devo from a source for which there is no Devo tag, you can create your own tag using my.app as the prefix. This can happen with proprietary data sources or publicly-available sources for which Devo has not yet created a tag (and therefore, there's no associated parser).
A note about creating Devo tags
The Devo professional services team can create new tags for any kind of data source. Just contact customer support for details.
The full tag should have at least four levels and may have up to six. The first two must be my.app. The third and fourth levels are free and should describe the application type and event type respectively. The fifth and sixth levels are optional and should be used to identify the actual source of the events. For example, if there are several servers running the application and reporting events to Devo, these levels can help identify the event's specific event source.
|my||app||free and required|
free, not required but highly recommended
|free, not required||free, not required|
When Devo receives an event with a tag that begins with my.app, it saves the event to a file and location determined by the tag levels, adds the first four levels of the tag to the finder (this will be the table), and saves the fifth and sixth levels (if any) to the event in the data table. However, since Devo is not equipped with a parser for this event type, when you open the data table, each event row will have only a few fields:
- eventdate This is the date/time the event was received by Devo.
- cluster This is the fifth level of the tag (if it was used).
- instance This is the sixth level of the tag (if it was used).
- message This contains the unparsed content as it was received.
You can manually parse the content of the message field using the column operations available in the query window. Then, when you've parsed all the fields into columns, create a custom table. From this point, you can use the custom table to consult the data parsed into columns.
Tables created by injecting existing data
When you inject data into a new table, the new table will always be assigned the prefix my.app.
This tag is only used to test event sending after you have set up a new event source, Devo relay, or to test a relay rule. Events that are received with this tag are saved in the test.keep.free table which you can consult to confirm that your events have been sent correctly and with the correct data.
While we recommend sending data to Devo in syslog format whenever possible, we support the ingestion of events received in common event format (CEF) via syslog for some technologies. A prime example of the use of this particular format is when ArcSight is used as a log management solution and events are going to be forwarded from ArcSight directly to Devo.
When events are sent in CEF syslog format, they don't require a Devo tag to enable ingestion. However, it is necessary for the platform to be equipped with a parser in order for you to access the data tables and see the events parsed correctly at query time.
Learn more about CEF syslog and check out the list of technologies that Devo currently supports when sent in this special format.
This is a table that Devo automatically generates when it receives any inbound event that has no tag or has a tag that it doesn't recognize. Instead of dropping the event, it is saved to this table. When this table appears in your finder, open it to investigate the nature of the problem and address it.
Create an alert that notifies the domain administrators when new events are saved to this table.
There are several technologies for which, regardless of brand, the log events contain very similar, or identical fields. When this is the case, as with web servers, firewalls, proxies, and several other technologies, Devo automatically generates a union table that contains the events from several different data sources. Union tables are indicated in the Finder by the union icon. Hover over the icon to see a full list of the tables that the union table will collect if available in the deployment.
If your deployment is collecting these events, you will see these tables in your Finder (they have the same names as the tags used for ingestion). Open any of these tables to see the collection of events for the corresponding event type.
However, you will also find a parent table called network.f5.bigip. If you open this table, you will see all the events collected using the network.f5.bigip.* tags in a single view.
Parent tables exist for several technologies but only when the different event formats and event collection policy allow for it. Here are some other examples of parent tables: