Autoparser
Overview
When data enters Devo through a non-standardized method, it cannot be automatically parsed at that moment of reception due to the lack of Devo tags to interpret its structure. However, it can still be parsed automatically thanks to the Autoparser. This functionality analyzes the data patterns in search of possible ways to parse your data, allowing you to choose the one you consider most adequate to resemble the originally intended structure.
The following video is an introduction to the autoparser concept:
How does it work?
The autoparser uses an internal logic to analyze the content of up to 200 sample events in an unparsed table in order to identify:
- Non-alphanumeric ASCII characters that might be field delimiters.
- Units of data that follow a common and fixed syntax; i.e. IPv4 and IPv6 addresses, strings enclosed in quotation marks, floating-point decimal values, and more.
Based on this analysis, it recommends a selection of characters that are likely (and less likely) to be delimiters. Using the autoparser controls, you can select the delimiters that you need and deselect the rest. By testing the same delimiter pattern on different sample events, you can confirm that it will parse your data table as needed.
The Autoparser will only offer as delimiters those characters that appear in all the 200 logs analyzed. For example, in this series of logs...
- a, b, c, d
- x, y, z
- sampleLog
...the comma will not be offered as a delimiter since the third log does not include any.
What type of tables can be autoparsed?
The Autoparser will be available for the following types of tables:
my.app→ tables created by sending data from a new, proprietary data source.my.upload→ tables created by manually uploading a file containing data.
Be aware that the autoparser will not be available for these tables unless they have at least four tag levels.
It will not be available either if they were created by injecting data from another table, since these are already properly parsed.
What data do I need for a successful autoparse?
In certain cases, the autoparser can be used to parse these data tables quickly and easily. However, for the autoparser to work optimally, the log events must:
- Contain the same number of fields in the same order.
- Use delimiters in the same pattern in every event.
Valid timestamp formats
When the table to be parsed contains timestamp data, it must present a valid format, otherwise it will be parsed as a string. Check the valid formats in the table below:
Valid timestamp formats | Example |
|---|---|
ddd MMM DD HH:mm:ss YYYY | Thu Mar 29 00:21:05 2012 |
DD-MMM-YYYY HH:mm:ss.SSS | 27-Aug-2012 09:44:09.378 |
DD/MM/YYYY HH:mm:ss | 23/07/2019 07:55:00 |
DD/MM/YYYY H:mm:ss | 14/09/2012 9:42:05 |
YYYY-MM-DD HH:mm:ss.SSS | 2000-12-17 01:01:01.123 |
YYYY/MM/DD HH:mm:ss | 2012/08/25 06:48:18 |
YYYY/MM/DD | 2012/08/25 |
yyyy-MM-dd:hh:mm:ss+gmt | 2012-08-16:10:29:17+0200 |
yyyy-MM-dd hh:mm:ss.micros | 2000-12-17 01:01:01.123456 |
Epoch.millis This format is recognized by default as a float so it needs to be manually changed. | 1234567890.123 |
Millis This format is recognized by default as an integer so it needs to be manually changed. | 1584100816544 |
Disparate data
If the log events you need to parse do not conform to these requirements, the autoparser may not be your best way forward. This data results too disparate for the Autoparser to extract a pattern so it will not open and an error message will pop up to further specify the reason.
In those cases, you can manually parse the content of the message field using the column operations available (for example, creating new columns using the Split (split) operation). Then, you can create a custom table and use it to consult the data parsed into columns.
You can also contact customer support to request a custom parser for your my.app data.
Using the Autoparser
- Go to Data Search, and select the new table using the finder.
Click the Additional Tools icon in the toolbar and select Column Operations → AutoParser.
The option will not appear for you to select if the requirements explained in the type of tables section are not met.
- The autoparser window opens so you can select the desired settings to transform the raw data contained in the message column into a fully classified table.
If you want a different set of samples to better analyze the adequacy of the delimiters you chose, click the Reload samples button
next to the No. of Samples dropdown. This will lead to two possible scenarios depending on the temporal aspects of your query:Fixed period query: if you have the query running for a fixed period so new events are not being received, you will get a warning message telling you that "no new events were found" so changing the samples is not possible
Real-time query: if you have the query running in real-time so new events are being received, this will load the last 200. If the structure changes, new delimiters will be considered. If the structure is too different, you will get the "Disparate Data" error message explained before.
Select or deselect the symbols identified as possible field Delimiters. You can either select them by clicking in the Delimiters area or one by one in the Sample area. The symbols are displayed in colors to show if they are going to be used when parsing:
Color Meaning 
All the symbols in the sample are going to be used. 
None of the symbols in the sample are going to be used. 
Some of the symbols in the sample are going to be used. A number is displayed below to specify how many of them. - Once you have selected the desired pattern of delimiters, you can assign names and select the required data type for each column using the dropdown menu whenever possible.
- Select the Exclude checkbox for any columns you do not want to include in the parsed table. When you do so, the corresponding name field will be automatically disabled.
- Click Confirm Settings.
The process of parsing will not create a new table but transform the original table instead so it can be fully used in Devo.
Autoparsing an already parsed table
If you are not happy with the result for whatever reason, you can use the Autoparser again. However, it is not possible to apply changes selectively; the table will be reset to its original unparsed state for you to start over.
You just need to open the Autoparser as instructed before. Don't worry if you have second thoughts after clicking because you will receive a warning message. You can either cancel and keep the table as it is or restore it to parse it again with different settings.
If you choose to restore it, you will be forced out while the table goes back to its original unparsed state. To parse it anew, you need to access it again through the Data Search and open the Autoparser once more.
Autoparsing special objects
It is possible to autoparse special objects such as JSON. Although the concept is the same, the procedure varies. Check the article Autoparse a JSON object to know more.
.