Best practices for data search

By incorporating these ideas when searching data in Devo, you can avoid common problems and improve the effectiveness of the application when working with data tables. In case you need additional help, please contact the Devo support team.

Browser recommendations

When dealing with massive amounts of data, you need to consider the browser memory limitations. For optimal performance:

  • Restart the browser to refresh/restore maximum memory availability.

  • Limit the number of open tabs to maximize available memory.

Data search tips

  • Select Additional tools → Query info → Toggle Execution Info in data tables to get stats like row counts and memory utilization.


    Loaded rows in this search The number of rows loaded in the current query.
    Unloaded holes in this search Periods of time when data is not loaded. These are represented as 'holes' in the chart over the table.
    Total loaded rows The number of rows loaded in all the queries open in your session.
    Total rows limit  Limit of total rows displayed.
    Estimated memory usage Current memory consumption.
  • Limit the number of running queries to optimize performance by minimizing memory use.

  • If you need to have multiple queries open, use a second incognito mode to have multiple sessions.

  • By default, data tables show the data from the last 24 hours. Before starting to search or applying any filters, narrow your search selecting a smaller time range. For example, if you want to see the events coming in real-time, select Last hour.


The following are some examples and recommended actions when working on a data table.

  • Even if you want to search in a broader range of time, apply the filters and operations before selecting a previous date in the From field.

 Define the query and then select Last 30 days.

    

X Avoid modifying the time range selecting Last 30 days before defining the query.

    

  • Apply all the filters before grouping. This reduces the memory and computation required for the search activity.

 Run a filter prior to grouping data, as in this example where the country US is applied before grouping.

    

X Avoid grouping data before applying a filter, as shown here. This causes many more rows to be part of the grouping, which is memory-intensive and can severely impact browser performance.

    

  • When grouping data, use fields that have a low variability/cardinality, i.e. not many unique values. You can see the distinct values clicking the column header and selecting the arrow at the left.

 In this example, the grouping is done using the country column (13 distinct values): 

    

X Avoid grouping by numeric fields with a large number of distinct values, like the responseLength column in this example that has 2001 distinct values. Instead, execute an aggregation to sum, average, or min/max, etc. in order to understand the values.


    
  • When the data table that you are working with has a large number of columns, hide the columns you don't need.

✓ You can do this when selecting the table in the Finder. Find out how.

✓ You can also do this in the query window using the Search column layout tool. Learn more.  

Summary

Time scope

  • Use time scope to limit overall volume of data.

Filtering

  • Use filters to limit overall density of data

  • Use filters to limit potential variations in results. This helps to cut down on the possible number of unique combinations that the grouping will have to produce, and makes any resulting visualization more practical.

  • Minimize number of columns included to cut down on the possible number of unique combinations that the grouping will have to produce.

Grouping

  • Never group by columns with a great number of distinct values, like the timestamp column (maximum number of distinct values).

  • Minimize number of columns included to cut down on the possible number of unique combinations that the grouping will have to produce.

  • Use the smallest time segmentation that make sense for the query when grouping data. Larger settings use more memory to complete the grouping and any subsequent aggregation.

Applications and dashboards

  • Use applications and dashboards to push grouping and aggregation calculate to be 100% server side.

.

Have we answered your question?

If not, please contact our technical support team via email by clicking the button below.

CONTACT US