Best practices for data search
By incorporating these ideas when searching data in Devo, you can avoid common problems and improve the effectiveness of the application when working with data tables. In case you need additional help, please contact the Devo support team.
Browser recommendations
When dealing with massive amounts of data, you must consider browser restrictions. To get an optimal performance:
Restart browser to refresh / restore maximum memory availability.
Limit the number of open tabs to maximize memory availability.
Data search tips
Select Additional tools → Query info → Toggle Execution Info in data tables to get stats like row counts and memory utilization.
Loaded rows in this search The number of rows loaded in the current query. Unloaded holes in this search Periods of time when data is not loaded. These are represented as 'holes' in the chart over the table. Total loaded rows The number of rows loaded in all the queries open in your session. Total rows limit Limit of total rows displayed. Estimated memory usage Current memory consumption. The View Selected Events tool lets you check the details of individual events in your data table. First, select an event by clicking on the event row. To select multiple events, use ALT+ click when selecting rows. Then, select the View Selected Events icon in the toolbar. All the retrieved information can be viewed in different formats (CSV, text, JSON...) and it can also be copied to clipboard or downloaded. If you select an event that includes a JSON field, you can activate the Rich views toggle to see the field in JSON format. If not, it will be displayed as raw data.
Limit the number of running queries to allow maximum memory availability for individual queries.
If you need to have multiple queries open, use a second incognito mode to have multiple sessions.
By default, data tables show the data from the last 24 hours. Before starting to search or applying any filters, narrow your search selecting a smaller time range. For example, if you want to see the events coming in real-time, select Last hour.
The following are some examples and recommended actions when working on a data table.
Even if you want to search in a broader range of time, apply the filters and operations before selecting a previous date in the From field.
✓ Define the query and then select Last 30 days.
X Avoid modifying the time range selecting Last 30 days before defining the query.
Apply all the filters before grouping. This reduces the memory and computation required for the search activity.
✓ Run a filter prior to grouping data, as in this example where the country US is applied before grouping.
X Avoid grouping data before applying a filter, as shown here. This causes many more rows to be part of the grouping, which is memory-intensive and can severely impact browser performance.
When grouping data, use fields that have a low variability/cardinality, i.e. not many unique values. You can see the distinct values clicking the column header and selecting the arrow at the left.
✓ In this example, the grouping is done using the country column (13 distinct values):
X Avoid grouping by numeric fields with a large number of distinct values, like the responseLength column in this example that has 2001 distinct values. Instead, execute an aggregation to sum, average, or min/max, etc. in order to understand the values.
When the data table that you are working with has a large number of columns, hide the columns you don't need.
✓ You can do this when selecting the table in the Finder. Find out how.
✓ You can also do this in the query window using the Search column layout tool. Learn more here.
Summary
Time scope
Use time scope to limit overall volume of data.
Filtering
Use filters to limit overall density of data
Use filters to limit potential variations in results. This helps to cut down on the possible number of unique combinations that the grouping will have to produce, and makes any resulting visualization more practical.
Minimize number of columns included to cut down on the possible number of unique combinations that the grouping will have to produce.
Grouping
Never group by columns with a great number of distinct values, like the timestamp column (maximum number of distinct values).
Minimize number of columns included to cut down on the possible number of unique combinations that the grouping will have to produce.
Use the smallest time segmentation that make sense for the query when grouping data. Larger settings use more memory to complete the grouping and any subsequent aggregation.
Applications and dashboards
Use applications and dashboards to push grouping and aggregation calculate to be 100% server side.
.