Autoparse a my.app table
When you create a new my.app data table by sending data from a new, proprietary data source for which there is no Devo tag, the event fields are not parsed when you first open the table. This is because Devo hasn't been equipped with an internal parser for the data received.
In certain cases, the autoparser can be used to parse these data tables quickly and easily. For the autoparser to work optimally, the log events must:
- contain the same number of fields in the same order.
- use delimiters in the same pattern in every event.
If the log events you need to parse do not conform to these requirements, the autoparser may not be your best way forward. In those cases, contact customer support to request a custom parser for your my.app data.
Note that the autoparser is not available in my.app tables created by injecting data from another table, since these are already properly parsed.
The following video is an introduction to the autoparser concept:
.
How it works
The autoparser uses an internal logic to analyze the content of up to 200 sample events in an unparsed my.app table in order to identify:
- non-alphanumeric ASCII characters that might be field delimiters.
- units of data that follow a common and fixed syntax; i.e. IPv4 and IPv6 addresses, strings enclosed in quotation marks, floating-point decimal values, and more.
Based on this analysis, it recommends a selection of characters that are likely (and less likely) to be delimiters. Using the autoparser controls, you can select the delimiters that you need and deselect the rest. By testing the same delimiter pattern on different sample events, you can confirm that it will parse your data table as needed.
The Autoparser will only offer as delimiters those characters that appear in all the 200 logs analyzed. For example, in this series of logs...
- a, b, c, d
- x, y, z
- sampleLog
...the comma will not be offered as a delimiter since the third log does not include any.
Autoparsing a my.app data table
- Go to Data Search, and select the new
my.apptable using the finder. Click the Additional Tools icon in the toolbar and select Column Operations → AutoParser.
In order to use the autoparser function, you need to use a table with at least four levels, otherwise the option will be disabled.
The autoparser window opens so you can select the desired settings to transform the raw data contained in the message column into a fully classified table.
Disparate data
If the data is too different for the autoparser to extract a pattern to split and classify, you will get an error message. You can manually parse the content of the message field using the column operations available (for example, creating new columns using the Split (split) operation). Then, you can create a custom table and use it to consult the data parsed into columns.
Select or deselect the symbols identified as possible field Delimiters. You can either select them by type in the Delimiters area or one by one in the Sample area. The symbols are displayed in colors to show if they are going to be used when parsing:
Color Meaning 
All the symbols in the sample are going to be used. 
None of the symbols in the sample are going to be used. 
Some of the symbols in the sample are going to be used. A number is displayed below to specify how many of them. - Select the number of sample events you want to analyze in the No. of Samples dropdown menu. Include additional samples to test the same delimiter pattern on different events contained in the table and confirm that it will be parsed as needed.
- If you want a different set of samples to better analyze the adequacy of the delimiters you chose, click the Reload samples button
next to the No. of Samples dropdown. This will lead to two possible scenarios:- If you have the query running so new events are being received, this will load the last 200. If the structure changes, new delimiters will be considered. If the structure is too different, you will get the "Disparate Data" error message explained before.
- If you have the query running for a fixed period so new events are not being received, you will get a warning message telling you that "no new events were found" so changing the samples is not possible.
Once you have selected the desired pattern of delimiters, assign names and select the required data type for each column using the dropdown menu when possible.
Select the Exclude checkbox for any columns you do not want to include in the parsed table. When you do so, the corresponding name field will be automatically disabled.
Click Confirm Settings.
The action of parsing will not create a new table but transform the original table instead so it can be fully used in Devo.
Using the Autoparser with an already parsed table
If you try to use the autoparser with an already parsed table, you will receive a warning message. You will lose the changes made during the previous autoparsing process if you proceed. You can either cancel and keep the current table or restore it to its original state to parse it again with different settings.
If you choose to restore it and parse it again, you need to go again to the Data Search, select the my.app table using the finder and click Additional Tools → Column Operations → AutoParser.