Searching data / Working in the search window

Working in the search window

Introduction

Once you open a data table, you are redirected to the search window, where all the events corresponding to the tag you have selected appear arranged in rows and columns, forming the data table. This is where you can start to query your data, apply operations using the set of tools in the toolbar and customize the aspect of the data table, rearranging columns, filtering the data, etc.

The search window contains the following elements, as exemplified in the image below:

Search window elements

You can find detailed information about each element shown in the picture above by clicking on the corresponding number in the tabs below.

1. Event timeline

The timeline appears embedded at the top left of the search window by default. You can pop it out of the window so you can place it freely on your screen, or you can close it. After closing it, you can open it again by clicking the gear icon in the toolbar and then selecting Tools → Event timeline.

This useful graph shows a count of the queried events over the period of time set in the Time range selector (more info on the Time range selector in tab 4). The data counts represented in the timeline are plotted before the actual events are loaded in the browser. To avoid overloading the browser's memory, not all the events in the data table are downloaded to the browser. Instead, Devo downloads events in interval blocks within the time range selected. This is important to understand, especially with respect to using the Get Server Counts and Event Loading Indicator described below.

The timeline is a dynamic graph and gives you the ability to:

Hover over the timeline to show the count of events at a specific time.
Click and drag the mouse across a segment of the timeline to display only the event count and data for that period, narrowing the range of analysis. Use the Back button to go back to previously selected periods.
Click on the timeline to rerender the data table with the events available at that date and time. In this way, you can use the timeline to navigate the events in the table. If the events from the selected date/time have not yet been downloaded to the browser, this will download them. When this occurs, a blue band appears in the timeline indicating those events that are being downloaded to the browser. Alternatively, you can use the table scroll bar to download events to the data table.

The following table describes the settings above the timeline:

Events per	This determines the intervals at which event counts are totaled and plotted on the timeline. When you hover over the timeline, tooltips appear reporting event totals. The points along the timeline are determined by the Events per set. Auto sets the interval based on the query's current time range. Use this setting if you want to plot event counts at a specific interval.
Logarithmic scale	This applies a logarithmic scale to the y-axis of the timeline chart instead of the default scale, which uses uniform intervals of units. This can be especially helpful when outlying data is causing significant spikes or dives, distorting your ability to visualize the detail of the timeline.
Full counts	This toggle appears after applying a filter to your data. When your data is filtered, the green line automatically adjusts to represent the number of events with the filter applied. Activate this toggle to display a comparison between the count of filtered events (green line) and the full count of events (yellow line).
Get server counts	This button appears after applying a filter to your data. Select it to plot the real count of events in the timeline after applying a filter. When you apply a filter, segments of the timeline may appear as dotted lines, indicating that the counts are actually extrapolated values for those subintervals that have not been downloaded to the browser. Click this button to obtain the actual counts for the dotted segments. The line will change from dotted to continuous. Note that this doesn’t mean the actual events are downloaded to the browser, just that the real event count is reflected in the timeline.

2. Query code editor

The query code editor appears embedded at the top right of the search window by default. You can click the window icon at the top right to detach it from the search window so that you can place it wherever you prefer. You can click the X icon at the top right corner to close it. To open it again, click the Query code editor icon on the toolbar.

The Query code editor allows users to easily modify their queries and see the results immediately on the table below. Besides, the query editor will update your query any time you perform a modification to your query using the search window interface. In Devo, queries are written in LINQ language. Learn more about LINQ in Build a query using LINQ.

To modify your query, just click the editor. The window will expand and you can start adding the required modifications. After clicking Apply, the query results will be shown on the table.

3. Event loading indicator

Devo automatically controls how events are loaded in order to maintain optimal browser performance while at the same time fulfilling user requests for viewing and working with their data.

This reports what percentage of the query's time range has been loaded in the browser so far. For example, if the time range is set to 24 hours and 12 hours of data has been loaded, the progress indicator will report 50%.

Click the indicator to open the Event Loading Preferences. This shows you a more detailed summary of the event loading status and gives you access to some preferences that give you greater control over how events are loaded.

Keep in mind that both the Event count and Browser memory are limited by the thresholds established in the domain preferences, with the inferior value being the limit.

Exercise caution when modifying these preferences. By forcing Devo to load and maintain large amounts of data in the browser, you are likely to experience performance degradation and even browser failure.

Event Loading Preferences
Smart event loading	This is the default behavior. When on, it loads and manages a subset of the query's events to maintain browser performance and satisfy the user's requests for data. Turn this off to stop loading the query's remaining events into the browser.
Load all events	Turn ON to load all of the query's events in the selected date range. Exercise caution with this setting because when turned on, there is a risk of overloading the browser and causing it to crash.
Load all only when sorting	To sort a column, the data needs to be downloaded to the browser in order to take into account all of the column's values. Turn this setting on to load all events only when you sort the contents of a column.
Load all only when chart-building	To build a chart that plots data from individual events, all of the query's events need to be downloaded to the browser. Turn this setting on to load all events only when you build one of these types of charts. Examples of charts that are built using individual event data (not grouped events with aggregate functions) are scatter charts and some world maps.
Retain all events	By default, Devo employs a memory management process that can remove events from the browser's memory in order to make room for events that are more relevant. Turn this setting on to prevent loaded events from being removed from the browser's memory. This also prevents the progress indicator to fluctuate. Fluctuation may occur due to the fact that memory is liberated and used again. When all the events are retained, no event is removed and therefore the progress won't fluctuate.
Set thresholds. Event count, Browser memory (MB)	This becomes available when you turn Retain all events on. Turn on Set thresholds to enforce an upper limit to the amount of data to load. This amount can be expressed in number of events, Event count, or in MB used by the query in the Browser memory. If you define upper limits in both fields, event loading will stop when either one is met.
Event Loading Status
Progress	This reports what percentage of the query's time range has been loaded in the browser so far. For example, if the time range is set to 24 hours and 12 hours of data has been loaded, the progress indicator will report 50%.
Events loaded	This reports the number of events loaded and the corresponding use of memory.
Gaps remaining	Devo loads a query's events to the browser in blocks. This leaves gaps in the event timeline that contain the missing events. This tells you how many gaps exist in the current query.

4. Time range selector

You can use this menu to select a time period for the data shown on the table. You can select a short time range to narrow down your search or you can use an extended period to analyze long-term patterns like an advanced persistent threat. You can perform the following actions:

Set a new time interval using the interface

You can set a time interval following the steps described in the picture below. When setting time ranges, it is important to consider different aspects related to the type of time range specified and the method chosen to do it. You can use the interface to set absolute, relative, or snap-to dates:

Absolute: a specific interval with fixed start and end dates to see data from a specific time period.
Relative: a period of time relative to the current date (last 5 minutes, last day, etc.) to see data progression up to the present.
Snap to: a period of time that goes back to the starting point of the selected time frame to see data without unrepresentative data samples resulting from analyzing incomplete periods. For example, if it is 10:53:17 on a Tuesday:

Snap to the day: you will see data beginning at 00:00 on that same Tuesday.
Snap to the hour: you will see data beginning at 10:00.
Snap to the minute: you will see data beginning at 10:53:00.

Set a new time interval using date language expressions

You can also introduce time ranges manually using date language expressions, which gives you more flexibility and precision when searching your data. Simply click on the date field and write the desired time expression or edit the existing one. The field turns red and an explanatory message appears until a valid date is entered. Click Apply when you finish and the expressions will be translated into the corresponding dates.

Invalid expressions

Your from date cannot be after your to date and your to date cannot be in the future.

You can use a mix of both absolute and date language expressions in any given time range (for example, the to date can be relative and the from date absolute, and vice versa). For date language expressions, the current moment "now()" is used as the reference point.

Click here to see the operators...

You can establish absolute dates in the required format:

Operator	Action	Example
yyyy-MM-dd hh:mm:ss	Establishes the specified absolute date	2021-06-30 15:35:23

With date language expressions, use a series of mathematical operations to move away from the current time which is used as the reference point. You can use multiple operators at once and the execution order is from left to right:

Operator	Action	Example
Snap to (@) or \|<	Rounds the date to the beginning of a time unit. Note that this operator only works with 1m, 1d, 1h, 1w, 1W, 1M and 1y.	now() @ 1m or now() \|< 1m
Arithmetics (+/-)	Applies an offset to the date (date + offset or date - offset)	now() - 3h
Replace (^)	Replaces part of the date by a time unit (date ^ time_unit)	now() ^ 6d
Backward & forward (>>/<<)	Shifts the date to the next/past time unit (date >> time_unit or date << time_unit)	now() << 11M

Click here to see examples on time expressions...

Let's suppose the current time (which we refer to as "now()") is Sunday, 05 February 2017, 13:37:05. The table below shows the resulting time when different expressions are applied. Note that this isn't an exhaustive list:

Time expression	Description	Resulting time
now() - 60m	60 minutes ago	Sunday, 05 February 2017, 12:37:05
now() @ 1h	Now (rounded to the beginning of the hour)	Sunday, 05 February 2017, 13:00:00
now() - 24h	24 hours ago	Saturday, 04 February 2017, 13:37:05
(now() - 1d) @ 1d	Yesterday (rounded to the beginning of the day)	Saturday, 04 February 2017, 00:00:00
(now() - 2d) @ 1d	2 days ago (rounded to the beginning of the day)	Friday, 03 February 2017, 00:00:00
(now() - 2d) @ 1m	2 days ago (rounded to the beginning of the minute)	Friday, 03 February 2017, 13:37:00
((now() - 2d) @ 1d) - 2h	2 days ago (rounded to the beginning of the day minus 2 hours)	Thursday, 02 February 2017, 22:00:00
now() @ 1w	Locale week	Sunday, 05 February 2017, 00:00:00
now() @ 1W	ISO week	Monday, 30 January 2017, 00:00:00
now() ^ 6d	Replace the day with 6	Monday, 06 February 2017, 13:37:05
now() ^ 2018y3M6d15h30m20s	Replaces the year with 2018 Replaces the month with 3 Replaces the day with 6 Replaces the hour with 15 Replaces the minutes with 30 Replaces the seconds with 20	Tuesday, 06 March 2018, 15:30:20
now() >> 2M	Forward to next second month	Monday, 05 February 2018, 13:37:05
now() << 2M	Backward to previous second month	Friday, 05 February 2016, 13:37:05
now() >> 2M6d15h20m10s	Forward to next second month, sixth day, fifteenth hour, twentieth minute and 10 seconds	Tuesday, 06 February 2018, 15:20:10
now() << 1h/1d	Goes back to the first hour of the current day. Minutes and seconds don't change.	Sunday, 05 February 2017, 01:37:05

Activate or deactivate real-time data flow

Click the RT icon to suspend or reestablish the flow of real-time data. In some cases of extremely large volumes of data, real-time data flow will stop automatically and a warning message will be shown above the table. This is done to prevent the browser from crashing.

Users with the necessary permissions can determine if real-time data flow is active or inactive by default when users run searches. Go to Preferences → Domain Preferences → Global to access this setting. For more information, see Domain preferences.

Apply previously used time intervals

Use the Back button to apply previously selected time intervals in your query.

Additionally, the Time interval history tool allows you to easily apply previously selected time periods in the current or other data tables, to facilitate the analysis of data over time. The results can be used in reports or to create dashboard data sources from different time intervals.

Select the required interval in the Available Time Intervals area. When there are multiple active queries, checkboxes will be available to let you apply the interval to more than one query. The current query is selected by default.

5. Time control settings

This feature has limited availability. If you have any doubts or questions do not hesitate to reach out to us.

Data in Devo is indexed by ingestion time (eventdate) by default, which means that filtering and grouping by the event date are optimized. With this new feature, we have added a new time index over the creation time of the events (creationdate field), so queries over this field are also optimized.

The event date is when Devo receives and ingests data. On the other hand, the creation date is considered as when data is generated at its source. For this reason, the event date is always after the creation date, although in many cases there is a negligible difference between both. Sometimes there is a large difference between both dates and a user is interested in the creation date being the time reference for their data. For example, imagine a machine that generates events, which is then switched off during a whole day and the events arrive into Devo the next day: in this case the event date and creation date would differ.

The time control button is located to the left of the date picker in the search window and serves as a visual indicator of whether time control is activated, as well as which time reference you are using for your table. We recommend you refrain from switching between both time reference options when running queries as the results returned might not be correct.

Once activated, use this button to select the time reference for your current search:

Creation date mode (CrD): This sets the creation date as the time reference for the current search.
Ingestion date mode (InD): This sets the event date as the time reference for the current search. Note that this is the default in Devo.

Get more information about the time control feature in this article.

6. Search window toolbar

This toolbar offers a rich set of tools to work with the table data including grouping, aggregation, data download, and more. Hover over each icon to see its tooltip. These are the default tools displayed in the toolbar:

(1) Time interval history	Apply time intervals previously set in any active queries. Learn more about this in tab 4.
(2) Column manager	Hide or show columns in the data table to work only with the necessary ones. Check out Hide and show columns to learn how to do it.
(3) Selected events	It is the clipboard icon. It is useful to see and be able to download information for a few selected events instead of the whole table. First, select the rows you want to see or download, by clicking on their rows. Then click the clipboard icon. This allows you to check information about specific events in the data table. Select one or several events in the data table to make this option available.
(4) Column	Access a set of operations to edit and arrange the table columns. Learn more about these operations in the articles in Modifying the column layout.
(5) Query code editor	Open a query editor where you can build or modify the current query using LINQ.
(6) Node tree	Display a treemap representing all the operations applied to the original data table. See the below section to learn more.
(7) CyberChef	Use this tool to analyze and decode your data before building your query. Learn more in Manipulate your data using CyberChef.
(8) Alert definition	Define alerts to monitor active queries and receive notifications when certain conditions occur. Check the instructions in Creating new alerts.
(9) Aggregate	Perform aggregation operations on table data that has been already grouped by time interval.
(10) Group	Group data to get all the different row value combinations of the grouped columns.
(11) Filter OR	You can use an OR filter to get records that have any of the values for a given property.
(12) Filter	Filter data to retrieve certain values or exclude them from the table.
(13) Create column	Create columns in your data tables transforming the already existing data.
(14) Download	Download query data in different formats. Go to Download query data for further information.
(15) Server mode	Check this box to activate server mode in your searches. The default search mode is recommended for small queries, while server mode is recommended for queries that process a large amount of data. Learn more in Best practices for data search. You can set server mode as default in your User preferences.
(16) Additional tools	Access a set of additional operations that do not appear in the default toolbar.
(17) Close search	Close the current search.

You can customize the default toolbar configuration to provide quick access to the tools that you use most frequently. You can perform the following actions to customize your toolbar as needed.

Add new tools to the toolbar and manage them

Select the gear icon on the toolbar (Additional tools) and navigate to the required tool in the list. Hover over its icon and when the cursor changes shape, drag and drop the tool into the table toolbar. The icon will appear as the first tool in the toolbar.

To change the order of the tools, select the tool you want to move and drag it to the new position in the toolbar. To remove an icon from the table toolbar, select, hold and move it to the dynamic trash bin that appears, as seen above.

Save and restore the toolbar configuration

After adding the tools you use the most to the toolbar and move them as required, click the gear icon on the toolbar and select Workspace → Save current Workspace. The custom toolbar will appear the next time you access the search window, no matter the data table you open. Select Workspace→ Reset Workspace to default to restore the original toolbar configuration.

7. Applied operations bar

Any operations you apply on the table when building your query will appear listed above the data table (see Build a query in the search window to learn more about how to transform and work with query data). This way, you can easily consult the operations affecting the data, modify them, or undo operations. You can go back to any of the operations applied and start a new path of actions from there.

Each tab in the bar/node in the tree appears in a color that denotes the type of action taken. The color code is described here:

Select the Node tree tool to display a visual record of all the modifications applied to the original data table. The actions and their sequences are displayed in a tree, known as the search tree. Select any point in the tree to display the query results at that point in the sequence of modifications. If you select an operation in a branch different from the current one, that path will be shown in the applied operations bar.

8. Column header menu

Hover over any column header and click the arrow icon that appears to show the column header menu. You will see the name of the column and the data type of its values. The icons at the top of the menu allow you to perform the following actions:

Highlight the column. You can also do this by clicking the column header.
Expand or shrink the column to the default size. You can also adjust the size of the columns dragging the top right side of the header. Learn more here.
Hide the column. To display it again, select Column → Show hidden columns on the toolbar. Learn more here.

The menu also shows the top 10 distinct values with the highest number of instances in the column.

Missing values

This list shows only the data that has been loaded in the browser so far and corresponds to the percentage indicated in the Event loading indicator (more info on the Event loading indicator in tab 3). Due to this, note that you may not see some of the values in your query when you use the search box to filter the list. Filter the data in the table if you want to verify that the values you are looking for exist or not.

Filter data by a specific value

Click one of the values in the list to get only events with that specific value in that column. The Operations over columns window will be open in the Filter tab, and the Equal (eq, =) operation selected.

Apply an Or filter using one or several values

Select the checkbox of any value from this list. Now select at least one more value from the list to add it to an Or filter. You can also select column values from other columns.

9. Data table

In the data table, each row represents an event and each column represents a data value correctly recognized by Devo. If the data is not separated by several columns or is shown in the unknown tag structure of the search view, it is normally due to missing or incorrect tags. Learn more about tags here. The data displayed in the table will change according to the operations you apply to build your query (filters, new columns...). Learn more about building queries here.

Data table shortcuts

You can perform the following actions in the data table:

Shortcut	Action
Select a row + SPACE BAR	Click one or several rows to select them and hit the space bar to open the Selected events window, where you can see the event(s) content in detail. You can copy the content to your clipboard or download it in several formats (csv, txt or json) using the controls in the top right corner. You can also select the required event(s) and click the Selected events icon in the search window toolbar to open the Selected events window.
Hover over a cell + ENTER	You can hover over a value in your table and hit the ENTER key to apply a filter and get only events with that specific value in that column. The Operations over columns window will be open in the Filter tab, and the Equal (eq, =) operation selected.
Hover over a cell + P	Hover over any cell in your data table and click P on your keyboard to open a window displaying the cell's contents. For JSON content, this is especially useful. If the content of the cell is of json data type, the window displays it in a reader-friendly way: name/value pairs are shown on separate lines and values are color-coded by data type. Learn more about this in Working with JSON objects in data tables.
Hover over a cell + C	Use this shortcut to add cell values as input data in the CyberChef tool. Select the CyberChef icon in the toolbar to see the cells added. Learn more in Manipulate your data using CyberChef.

Download as PDF