When data enters Devo through a non-standardized method, it cannot be automatically parsed at that moment of reception due to the lack of Devo tags to interpret its structure. However, it can still be parsed automatically thanks to the Autoparser. This functionality analyzes the data patterns in search of possible ways to parse your data, allowing you to choose the one you consider most adequate to resemble the originally intended structure.
How does it work?
The autoparser uses internal logic to analyze the content of up to 200 sample events in an unparsed table in order to identify:
- Non-alphanumeric ASCII characters that might be field delimiters.
- Units of data that follow a common and fixed syntax; i.e. IPv4 and IPv6 addresses, strings enclosed in quotation marks, floating-point decimal values, and more.
Based on this analysis, it recommends a selection of characters that are likely (and less likely) to be delimiters. Using the autoparser controls, you can select the delimiters that you need and deselect the rest. By testing the same delimiter pattern on different sample events, you can confirm that it will parse your data table as needed.
The Autoparser will only offer as delimiters those characters that appear in all the 200 logs analyzed. For example, in this series of logs...
- a, b, c, d
- x, y, z
...the comma will not be offered as a delimiter since the third log does not include any.
What type of tables can be autoparsed?
The Autoparser will be available for the following types of tables:
my.app→ tables created by sending data from a new, proprietary data source.
my.upload→ tables created by manually uploading a file (either locally or from a drive) containing data.
Not available for tables...
Be aware that the autoparser will not be available for these tables unless they have at least four tag levels.
It will not be available either if they were created by injecting data from another table, since these are already properly parsed.
What data do I need for a successful autoparse?
In certain cases, the autoparser can be used to parse these data tables quickly and easily. However, for the autoparser to work optimally, the log events must:
- Contain the same number of fields in the same order.
- Use delimiters in the same pattern in every event.
Valid timestamp formats
When the table to be parsed contains timestamp data, it must present a valid format, otherwise, it will be parsed as a string. Check the valid formats in the table below:
ddd MMM DD HH:mm:ss YYYY
Thu Mar 29 00:21:05 2012
By default recognized as a float so it needs to be manually changed.
By default recognized as an integer so it needs to be manually changed.
If the log events you need to parse do not conform to these requirements, the autoparser may not be your best way forward. These data may result too disparate for the Autoparser to extract a pattern so it will not open and an error message will pop up to further specify the reason.
In those cases, you can manually parse the content of the message field using the column operations available (for example, creating new columns using the Split (split) operation). Then, you can create a custom table and use it to consult the data parsed into columns.
my.app data, you can also contact customer support to request a custom parser.
Using the Autoparser
Inside the table that you want to parse, click the gear icon in the toolbar and select Source table → Autoparse.
The autoparser window opens so you can select the desired settings to transform the raw data contained in the message column into a fully classified table.
Autoparse option not showing
The option will not appear for you to select if the requirements explained in the type of tables section are not met.
If you want a different set of samples, click the Reload samples buttonnext to the No. of Samples dropdown. This will lead to two possible scenarios depending on the temporal aspects of your query:
Fixed period query: new events are not being received so you will get a message informing you that "no new events were found" and thus changing the samples is not possible.
Real-time query: new events are being received so this will load the last 200. If the structure changes, new delimiters will be considered. If the structure is too different, you will get the "Disparate Data" error message explained before.
Select or deselect the symbols identified as possible field delimiters by clicking in the Delimiters area or one by one in the Sample area. The symbols are displayed in colors to show if they are going to be used when parsing.
Assign names and select the required data type for each column using the dropdown menu.
Mark the Exclude checkbox for any columns you do not want to include in the parsed table. When you do so, the corresponding name field will be automatically disabled.
|All the symbols in the sample are going to be used.|
|None of the symbols in the sample are going to be used.|
|Some of the symbols in the sample are going to be used. A number is displayed below to specify how many of them.|
The process of parsing will not create a new table but transform the original table instead so it can be fully used in Devo.
Autoparsing an already parsed table
If you are not happy with the result for whatever reason, you can use the Autoparser again. However, it is not possible to apply changes selectively; the table will be reset to its original unparsed state for you to start over.
You just need to open the Autoparser as instructed before. Don't worry if you have second thoughts after clicking because you will receive a warning message. You can either cancel and keep the table as it is or restore it to parse it again with different settings.
If you choose to restore it, you will be forced out while the table goes back to its original unparsed state. To parse it anew, you need to access it again through the Data Search and open the Autoparser once more.
Autoparsing special objects
It is possible to autoparse special objects such as JSON. Although the concept is the same, the procedure varies. Check the article Autoparse a JSON object to know more.
Watch the following video to guide you through the process of parsing your proprietary data using the Autoparser, including data in JSON format.