Uploading log files
While Devo is designed to receive real-time data feeds, it's sometimes useful to upload static files if you want historical log data to be accessible in Devo for querying and analysis.
.
The process is simple and this 3-minute video takes you through it step by step.
Upload a data file
You have the option of uploading a CSV file from your local machine or from a Dropbox account. Note that there is a 2GB limit on files uploaded from the local machine so if you have an extremely large file, you should first upload it to Dropbox.
- Go to Data upload in the navigation panel.
Select either Dropbox or Local files.
If you select Dropbox, you will be prompted to grant Devo access to your Dropbox files, then log into Dropbox. Navigate to the folder and select the file you want to upload.
If you select Local files, you can either drag and drop the file or navigate to the necessary folder and select it for upload. You can try the process using this file.
When you have selected the file, click the Upload File button. The settings page appears.
This page contains a couple of fields that will determine how the data will be saved in Devo.
Tag The first two levels of the tag are predefined as my.upload. The third and fourth levels are up to you. Try to name them in a way that will help you identify the data contained in the data table.Date parsing type This determines the date that will be used when you query the data by time range. If the log information that you are uploading contains a date field, you can choose from Current date or Date from a log field.
- If you choose Current date, each event will be assigned a date field with the upload date and time.
- If you choose Date from a log field, you must then indicate the parsing type in the Parsing log type field, so that Devo can scans the events and highlight the field values that it recognizes as possible date/time information. Select the field whose values you want to use for querying by time range.
Click the Confirm Settings button.
To work with the new data, go to Data search and use the finder to locate the my.upload table that you just created.
The above table will always be created with the following three columns:
| eventdate | This is the date in which the data arrives into Devo. |
|---|---|
| uploadid | This is the coding that identifies the data according to the moment it is uploaded. Each new upload will have a unique uploadid. |
| message | Use the Split (split) function to manually parse the message of the column. |
Recognized date formats
Devo is capable of recognizing a wide range of date formats in the files you upload.
| Format | Sample | Regular Expression |
|---|---|---|
| DD/MM/YYYY | 30/05/2017 | \d{2}\/\d{2}\/\d{4} |
| DD/MM/YYYY HH:mm:ss | 30/05/2017 18:00:00 | \d{2}\/\d{2}\/\d{4}\s\d{2}\:\d{2}\:\d{2} |
| DD:MM:YYYY | 30:05:2017 | \d{2}\:\d{2}\:\d{4} |
| DD:MM:YYYY-HH:mm:ss | 30:05:2017-18:00:00 | \d{2}\:\d{2}\:\d{4}\-\d{2}\:\d{2}\:\d{2} |
| ddd MMM DD HH:mm:ss YYYY | Tue May 30 18:00:00 2017 | \w{3}\s\w{3}\s\d{2}\s\d{2}\:\d{2}\:\d{2}\s\d{4} |
| DD-MMM-YYYY HH:mm:ss.SSS | 30-May-2017 18:00:00.123 | \d{2}\-\w{3}\-\d{4}\s\d{2}\:\d{2}\:\d{2}\.\d{3} |
| DD-MM-YYYY | 30-05-2017 | \d{2}\-\d{2}\-\d{4} |
| MMM dd HH:mm:ss | May 30 18:00:00 | \w{3}\s\d{2}\s\d{2}\:\d{2}\:\d{2} |
| x (epoch milliseconds with dot separator) | 1496167200 | \d{10}\.\d{3} |
| YYYY/MM/DD | 2017/05/30 | \d{4}\/\d{2}\/\d{2} |
| YYYY/MM/DD HH:mm:ss | 2017/05/30 18:00:00 | \d{4}\/\d{2}\/\d{2}\s\d{2}\:\d{2}\:\d{2} |
| YYYY/MM/DD hh:mm:ss | 2017/09/14 15:10:23 | \d{4}\/\d{2}\/\d{2}\s\d{2}\:\d{2}\:\d{2} |
| YYYY:MM:DD | 2017:05:30 | \d{4}\:\d{2}\:\d{2} |
| YYYY:MM:DD-HH:mm:ss | 2017:05:30-18:00:00 | \d{4}\:\d{2}\:\d{2}\-\d{2}\:\d{2}\:\d{2} |
| YYYY-MM-DD | 2017-05-30 | \d{4}\-\d{2}\-\d{2} |
| YYYY-MM-DD hh:mm:ss | 2017-09-14 15:10:23 | \d{4}\-\d{2}\-\d{2}\ s\d{2}\:\d{2}\:\d{2} |
| YYYY-MM-DD HH:mm:ss.SSS | 2017-05-30 18:00:00.123 | \d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{3} |
| YYYY-MM-DD HH:mm:ss.SSSSSS | 2013-01-17 00:54:01.744909 | \d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{6} |
| YYYY-MM-DDThh:mm:ss | 2017-09-13T19:01:15 | \d{4}-\d{2}-\d{2}T\d{2}\:\d{2}\:\d{2} |
| YYYY-MM-DDThh:mm:ss.SSSZ | 2017-09-13T19:01:15.000Z | \d{4}-\d{2}-\d{2}T\d{2}\:\d{2}\:\d{2}\.\d{3}Z |
| YYYY-MM-DDThh:mm:ssZ | 2017-09-13T19:01:15Z | \d{4}-\d{2}-\d{2}T\d{2}\:\d{2}\:\d{2}Z |