- The Devo data analytics platform
- Getting started
- Domain administration
-
Sending data to Devo
-
The Devo In-House Relay
- Installing the Devo Relay
- Configuring the In-House Relay
- Relay migration
- Sending SSL/TLS encrypted events to the Devo relay
- Relay troubleshooting tips (v1.4.2)
- Event sources
- Other data collection methods
- Uploading log files
- Devo software
-
The Devo In-House Relay
-
Searching data
- Accessing data tables
-
Building a query
- Data types in Devo
- Build a query in the search window
- Build a query using LINQ
- Working with JSON objects in data tables
- Subqueries
-
Operations reference
-
Aggregation operations
- Average (avg)
- Count (count)
- First (first)
- First not null (nnfirst)
- HyperLogLog++ (hllpp)
- HyperLogLog++ Count Estimation (hllppcount)
- Last (last)
- Last not null (nnlast)
- Maximum (max)
- Median / 2nd quartile / Percentile 50 (median)
- Minimum (min)
- Non-null average (nnavg)
- Non-null standard deviation (biased) (nnstddev)
- Non-null standard deviation (unbiased) (nnustddev)
- Non-null variance (biased) (nnvar)
- Non-null variance (unbiased) (nnuvar)
- Percentile 10 (percentile10)
- Percentile 25 / 1st quartile (percentile25)
- Percentile 5 (percentile5)
- Percentile 75 / 3rd quartile (percentile75)
- Percentile 90 (percentile90)
- Percentile 95 (percentile95)
- Standard deviation (biased) (stddev)
- Standard deviation (unbiased) (ustddev)
- Sum (sum)
- Sum Square (sum2)
- Variance (biased) (var)
- Variance (unbiased) (uvar)
-
Arithmetic group
- Absolute value (abs)
- Addition, sum, plus / Concatenation (add, +)
- Ceiling (ceil)
- Cube root (cbrt)
- Division (div, \)
- Division remainder (rem, %)
- Floor (floor)
- Modulo (mod, %%)
- Multiplication, product (mul, *)
- Power (pow)
- Real division (rdiv, /)
- Rounding (round)
- Sign (signum)
- Square root (sqrt)
- Subtraction, minus / Additive inverse (sub, -)
-
Conversion group
- Duration (duration)
- Format date (formatdate)
- From base16, b16, hex (from16)
- From base64, b64 (from64)
- From UTF8 (fromutf8)
- From Z85, base85 (fromz85)
- Human size (humanSize)
- Make byte array (mkboxar)
- Parse date (parsedate)
- Regular expression, regexp (re)
- Template (template)
- Timestamp (timestamp)
- To base16, b16, hex (to16)
- To base64, b64, hex (to64)
- To BigInt (bigint)
- To boolean (bool)
- To Float (float)
- To image (image)
- To Int (int)
- To IPv4 (ip4)
- To IPv4 net (net4)
- To IPv6 (ip6)
- To IPv6 compatible (compatible)
- To IPv6 mapped (mapped)
- To IPv6 net (net6)
- To IPv6 translated (translated)
- To MAC address (mac)
- To string (str)
- To string (stringify)
- To UTF8 (toutf8)
- To Z85, base85 (toz85)
- Cryptography group
- Date group
- Flow group
- General group
-
Geolocation group
- Coordinates distance (distance)
- Geocoord (geocoord)
- Geographic coordinate system (coordsystem)
- Geohash (geohash)
- Geohash string (geohashstr)
- Geolocated Accuracy Radius with MaxMind GeoIP2 (mm2accuracyradius)
- Geolocated ASN (mmasn)
- Geolocated ASN with MaxMind GeoIP2 (mm2asn)
- Geolocated AS Organization Name with MaxMind GeoIP2 (mm2asorg)
- Geolocated AS owner (mmasowner)
- Geolocated City (mmcity)
- Geolocated City with MaxMind GeoIP2 (mm2city)
- Geolocated Connection Speed (mmspeed)
- Geolocated connection type with MaxMind GeoIP2 (mm2con)
- Geolocated Coordinates (mmcoordinates)
- Geolocated coordinates with MaxMind GeoIP2 (mm2coordinates)
- Geolocated Country (mmcountry)
- Geolocated Country with MaxMind GeoIP2 (mm2country)
- Geolocated ISP (mmisp)
- Geolocated ISP name with MaxMind GeoIP2 (mm2isp)
- Geolocated Latitude (mmlatitude)
- Geolocated Latitude with MaxMind GeoIP2 (mm2latitude)
- Geolocated Level 1 Subdivision with MaxMind GeoIP2 (mm2subdivision1)
- Geolocated Level 2 Subdivision with MaxMind GeoIP2 (mm2subdivision2)
- Geolocated Longitude (mmlongitude)
- Geolocated Longitude with MaxMind GeoIP2 (mm2longitude)
- Geolocated Organization (mmorg)
- Geolocated organization name with MaxMind GeoIP2 (mm2org)
- Geolocated Postal Code (mmpostalcode)
- Geolocated Postal Code with MaxMind GeoIP2 (mm2postalcode)
- Geolocated Region (mmregion)
- Geolocated Region Name (mmregionname)
- ISO-3166-1 Continent Alpha-2 Code (continentalpha2)
- ISO-3166-1 Continent Name (continentname)
- ISO-3166-1 Country Alpha-2 Code (countryalpha2)
- ISO-3166-1 Country Alpha-2 Continent (countrycontinent)
- ISO-3166-1 Country Alpha-3 Code (countryalpha3)
- ISO-3166-1 Country Latitude (countrylatitude)
- ISO-3166-1 Country Longitude (countrylongitude)
- ISO-3166-1 Country Name (countryname)
- Latitude (latitude)
- Latitude and longitude coordinates (latlon)
- Longitude (longitude)
- Parse geocoord format (parsegeo)
- Represent geocoord format (reprgeo)
- Round coordinates (gridlatlon)
- JSON group
- Logic group
-
Mathematical group
- Arc cosine (acos)
- Arc sine (asin)
- Arc tangent (atan)
- Bitwise AND (band, &)
- Bitwise left shift (lshift, <<)
- Bitwise NOT (bnot, ~)
- Bitwise OR (bor, |)
- Bitwise right shift (rshift, >>)
- Bitwise unsigned right shift (urshift, >>>)
- Bitwise XOR (bxor, ^)
- Cosine (cos)
- e (mathematical constant) (e)
- Exponential: base e (exp)
- Hyperbolic cosine (cosh)
- Hyperbolic sine (sinh)
- Hyperbolic tangent (tanh)
- Logarithm: base 2 (log2)
- Logarithm: base 10 (log10)
- Logarithm: natural / arbitrary base (log)
- Pi (mathematical constant) (pi)
- Sine (sin)
- Tangent (tan)
- Meta Analysis group
- Name group
-
Network group
- HTTP Status Description (httpstatusdescription)
- HTTP Status Type (httpstatustype)
- IP Protocol (ipprotocol)
- IP Reputation Score (reputationscore)
- IP Reputation Tags (reputation)
- IPv4 legal use (purpose)
- IPv6 host number (host)
- IPv6 routing number (routing)
- Is IPv4 (ipip4)
- Is Private IPv4 (isprivate)
- Is Public IPv4 (ispublic)
- Squid Black Lists Flags (sbl)
- Order group
-
Packet group
- Ethernet destination MAC address (etherdst)
- Ethernet payload (etherpayload)
- Ethernet source MAC address (ethersrc)
- Ethernet status (etherstatus)
- Ethernet tag (ethertag)
- EtherType (ethertype)
- Has Ethernet frame (hasether)
- Has IPv4 datagram (hasip4)
- Has TCP segment (hastcp)
- Has UDP datagram (hasudp)
- IPv4 destination address (ip4dst)
- IPv4 differentiated services (ip4ds)
- IPv4 explicit congestion notification (ip4ecn)
- IPv4 flags (ip4flags)
- IPv4 fragment offset (ip4fragment)
- IPv4 header checksum (ip4cs)
- IPv4 header length (ip4hl)
- IPv4 identification (ip4ident)
- IPv4 payload (ip4payload)
- IPv4 protocol (ip4proto)
- IPv4 source address (ip4src)
- IPv4 status (ip4status)
- IPv4 time to live (ip4ttl)
- IPv4 total length (ip4len)
- IPv4 type of service (ip4tos)
- TCP ACK (tcpack)
- TCP checksum (tcpcs)
- TCP destination port (tcpdst)
- TCP flags (tcpflags)
- TCP header length (tcphl)
- TCP payload (tcppayload)
- TCP sequence number (tcpseq)
- TCP source port (tcpsrc)
- TCP status (tcpstatus)
- TCP urgent pointer (tcpurg)
- TCP window size (tcpwin)
- UDP checksum (udpcs)
- UDP destination port (udpdst)
- UDP length (udplen)
- UDP payload (udppayload)
- UDP source port (udpsrc)
- UDP status (udpstatus)
- Statistical group
-
String group
- Contains (has, ->)
- Contains - case insensitive (weakhas)
- Contains tokens (toktains)
- Contains tokens - case insensitive (weaktoktains)
- Edit distance: Damerau (damerau)
- Edit distance: Hamming (hamming)
- Edit distance: Levenshtein (levenshtein)
- Edit distance: OSA (osa)
- Ends with (endswith)
- Format number (formatnumber)
- Hostname public suffix (publicsuffix)
- Hostname root domain (rootdomain)
- Hostname root prefix (rootprefix)
- Hostname root suffix (rootsuffix)
- Hostname subdomains (subdomain)
- Hostname top level domain (topleveldomain)
- Is empty (isempty)
- Is in (`in`, <-)
- Is in - case insensitive (weakin)
- Length (length)
- Locate (locate)
- Lower case (lower)
- Matches (matches, ~)
- Peek (peek)
- Replace all (replaceall)
- Replace first (replace)
- Shannon entropy (shannonentropy)
- Split (split)
- Split regexp (splitre)
- Starts with (startswith)
- Substitute (subs)
- Substitute all (subsall)
- Substring (substring)
- Trim both sides (trim)
- Trim the left side (ltrim)
- Trim the right side (rtrim)
- Upper case (upper)
-
Web group
- Absolute URI (absoluteuri)
- Opaque URI (opaqueuri)
- URI authority (uriauthority)
- URI fragment (urifragment)
- URI host (urihost)
- URI path (uripath)
- URI port (uriport)
- URI query (uriquery)
- URI scheme (urischeme)
- URI ssp (urissp)
- URI user (uriuser)
- URL decode (urldecode)
- User Agent Company (uacompany)
- User Agent Company URL (uacompanyurl)
- User Agent Device Icon (uadeviceicon)
- User Agent Device Information URL (uadeviceinfourl)
- User Agent Device Type (uadevicetype)
- User Agent Family (uafamily)
- User Agent Icon (uaicon)
- User Agent Information URL (uainfourl)
- User Agent is Robot (uaisrobot)
- User Agent Name (uaname)
- User Agent OS Company (uaoscompany)
- User Agent OS Company URL (uaoscompanyurl)
- User Agent OS Family (uaosfamily)
- User Agent OS Icon (uaosicon)
- User Agent OS Name (uaosname)
- User Agent OS URL (uaosurl)
- User Agent Type (uatype)
- User Agent URL (uaurl)
- User Agent Version (uaversion)
-
Aggregation operations
-
Working in the search window
-
Generate charts
- Affinity chord diagram
- Availability timeline
- Bipartite chord diagram
- Bubble chart
- Chart aggregation
- Custom date chart aggregation
- Flame graph
- Flat world map by coordinates
- Flat world map by country
- Google animated heat map
- Google area map
- Google heat map
- Graph diagram
- Histogram
- Pew Pew map
- Pie chart
- Pie layered chart
- Punch card
- Robust Random Cut Forest chart
- Sankey diagram
- Scatter plot
- Time heatmap
- Triple exponential chart
- Voronoi treemap
- Data enrichment
- Setting up a data table
- Advanced data operations
- Use case: eCommerce behavior analysis
-
Generate charts
- Managing your queries
- Best practices for data search
- Monitoring tables
-
Parsers and collectors
- About Devo tags
- Special Devo tags and data tables
-
List of Devo parsers
- Business & Consumer
- Cloud technologies
- Databases
- Host and Operating Systems
-
Network and application security
- auth.secureauth
- auth.securenvoy
- av.mcafee
- av.sophos
- box.iptables
- edr.cylance
- edr.fireeye.alerts
- edr.minervalabs.events
- endpoint.symantec
- firewall.checkpoint
- firewall.cisco firepower and vpn.cisco
- firewall.fortinet
- firewall.huawei
- firewall.juniper
- firewall.paloalto
- firewall.pfsense
- firewall.sonicwall
- firewall.sophos
- firewall.sophos.xgfirewall
- firewall.stonegate
- firewall.windows
- mail.proofpoint
- nac.aruba
- network.meraki
- network.versa
- proxy.bluecoat
- proxy.forcepoint
- proxy.squid
- uba.varonis
- vuln.beyondtrust
- vpn.pulsesecure.sa
- Network connectivity
- Web servers
- Technologies supported in CEF syslog format
- Collectors
- Activeboards
-
Dashboards
- Create a new dashboard
-
Working with dashboard widgets
- Availability timeline widget
- Chord diagram widget
- Circle world map widget
- Color key value widget
- Color world map widget
- Column chart widget
- Comparative chart widget
- Funnel widget
- Gauge meter widget
- Google heatmap widget
- Heat calendar widget
- Line chart widget
- Monitoring widget
- Pie chart widget
- Punch card widget
- Sectored pie chart widget
- Table widget
- Time heatmap widget
- Tree diagram widget
- Voronoi tree widget
- Configuring and sharing dashboards
- Alerts and notifications
- Panels
- Applications
- Tools
- Social Intelligence
- API reference
- Release notes
web.jboss
The tags beginning with web.jboss identify log events generated by the JBoss web application server and the access logs of the applications the server hosts. The JBoss v7 server generates a boot and server log. You can configure access logs for each of the applications that runs on the JBoss server.
This article describes the tag naming structure, a little about logging in JBoss v7, and how to send log events from the JBoss server to Devo using rsyslog.
Tag structure
The full tag must have at least six levels. The first two are fixed as web.jboss. The third level identifies the log type/format and currently must be one of boot, access-clf, access-combined, access-lt, or server.
The fourth, fifth and sixth levels are required and should identify the environment type, web application, and instance respectively.
- environment - Describes the environment in when the event occurred. For example, development, testing, or production.
- web application - The name of the web application.
- clon - This is the instance that generated the event. Depending on your network, this can be a machine name, or the virtual name of a JBoss process.
The values of these levels should be guided by the structure we propose because they will be saved in the events when saved in Devo. When you open the resulting data table, these will appear in the environment, site and clon columns.
technology | brand | log type/format | environment | web application | clon |
---|---|---|---|---|---|
web | jboss |
| free but required | free but required | free but required |
Therefore, the valid tags include:
- web.jboss.boot.env.app.clon
- web.jboss.access-clf.env.app.clon
- web.jboss.access-combined.env.app.clon
- web.jboss.access-lt.env.app.clon
- web.jboss.server.env.app.clon
For more information, read more about Devo tags.
JBoss logs
Server.log and boot.log
For general information about these log files and their default directories, see the JBoss Documentation.
You should review the configuration for the generation of boot.log and modify it as needed. You want to ensure it is generated using the default PATTERN formatter and that it is written in append mode. The configuration file is usually found in standalone/configuration/logging.properties or domain/configuration/logging.properties depending on if the server is started in managed domain or standalone server mode.
These are the lines you should review and edit as needed:
boot.log JBoss 7.x configuration
# File handler configuration
handler.FILE=org.jboss.logmanager.handlers.FileHandler
handler.FILE.level=DEBUG
handler.FILE.properties=autoFlush,append,fileName
handler.FILE.autoFlush=true
handler.FILE.append=true
handler.FILE.fileName=${org.jboss.boot.log.file:boot.log}
handler.FILE.formatter=PATTERN
Access logs (access_log.yyyy-MM-dd)
JBoss v7 access log files are identical to those of the Apache Tomcat application server but you can control the actual event content in terms of format and fields included.
The access log is not enabled by default so you need to edit the urn:jboss:domain:web:1.1 subsystem in the standalone/configuration/standalone.xml file to add the access-log definition as in this example:
JBoss 7.x access-log configuration
<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">
<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
<virtual-server name="default-host" enable-welcome-root="true">
<alias name="localhost"/>
<access-log rotate="false" prefix="access_log.txt" pattern="combined" >
<directory path="." relative-to="jboss.server.log.dir" />
</access-log>
</virtual-server>
</subsystem>
Devo supports three formats for access logs; two of them are based upon standard formats and the other is defined by Devo to offer a format that contains more details. Each of these formats corresponds with the third level of an access log tag.
The log format that corresponds to the web.jboss.access-clf tag is based on the Common Log Format (CLF). The specification of this format is:
pattern="%h %l %u %t "%r" %s %b" pattern="common"
The log format that corresponds to the web.jboss.access-combined tag is based on the NCSA Combined log format. The specification of this format is:
pattern="%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-Agent}i"" pattern="combined"
The log format that corresponds to the web.jboss.access-lt tag is a custom format defined by Devo to offer more detailed logging. The specification of this format that includes cookie names is:
pattern="%t %a %l %u %S %v:%p %m "%U%q" %H "%{Referer}i" "%{User-Agent}i" "%{cookieName1}c:%{cookieName2}c" %s %D %B %I"
However, if you prefer not to include cookie names in your events, use this specification instead.
pattern="%t %a %l %u %S %v:%p %m "%U%q" %H "%{Referer}i" "%{User-Agent}i" "" %s %D %B %I"
Sending to Devo using file monitoring
Using rsyslog in Unix-like environments
You can read more about using rsyslog to monitor and send files to a Devo endpoint in the Sending data to Devo section of our documentation. Here we offer a sample rsyslog configuration file that is set up to monitor the server and boot logs and one access log, and forward them to a Devo Relay.
/etc/rsyslog.d/45-jboss.conf file
$template jboss,"<%PRI%>%timegenerated% %HOSTNAME% %syslogtag% %msg%"
# File server.log
$InputFileName JBOSS_PATH/standalone/log/server.log
$InputFileTag web.jboss.server.ENV.APP.CLON:
$InputFileStateFile stat-jboss-server1
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
# File boot.log
$InputFileName JBOSS_PATH/standalone/log/boot.log
$InputFileTag web.jboss.boot.ENV.APP.CLON:
$InputFileStateFile stat-jboss-boot1
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
# File access log
$InputFileName JBOSS_PATH/standalone/log/access_log.txt
$InputFileTag web.jboss.access-combined.ENV.APP.CLON:
$InputFileStateFile stat-jboss-accessCombined1
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
# SSL config for DEVO cloud
#$DefaultNetstreamDriver gtls # use gtls netstream driver
#$DefaultNetstreamDriverCAFile /etc/rsyslog.d/ca.crt
#$DefaultNetstreamDriverCertFile /etc/rsyslog.d/user.crt
#$DefaultNetstreamDriverKeyFile /etc/rsyslog.d/user.key
#$ActionSendStreamDriverMode 1 # require TLS for the connection
#$ActionSendStreamDriverAuthMode x509/name
#$ActionSendStreamDriverPermittedPeer collector
if $syslogtag contains 'web.jboss.' and $syslogfacility-text == 'local7' then @@DEVO-RELAY:PORT;jboss
:syslogtag, contains, "web.jboss." ~
Note the following placeholder values in the sample file above:
- JBOSS_PATH should be the absolute path where your JBoss log files reside.
- Replace ENV.APP.CLON with the values that represent the source environment, application, and instance.
- In this example, the access log uses the accessCombined tag. You should specify the tag that corresponds to the format you have elected to use.
- Replace DEVO-RELAY and PORT with the IP address and port of your Devo Relay.
You can uncomment the SSL section of the file to send the events directly to the Devo Cloud. In this case, you should replace DEVO-RELAY and PORT with the hostname of your Devo domain and port 443.
Make sure that the file you are sending and the directory where resides can be read by the user running rsyslog. If not, you can assign the necessary permissions by changing the file group to syslog:
chown :syslog /opt/jboss/standalone/log /opt/jboss/standalone/log/* chmod 750 /opt/jboss/standalone/log chmod 640 /opt/jboss/standalone/log/*
To configure file rotation, you can use the logrotate utility. The following in an example of a jboss.conf might be set up.
/etc/logrotate.d/jboss.conf
/opt/jboss/standalone/log/server.log /opt/jboss/standalone/log/boot.log /opt/jboss/standalone/log/access_log.txt
{
rotate 14
daily
copytruncate
missingok
notifempty
delaycompress
compress
sharedscripts
lastaction
service rsyslog stop
rm -f /var/spool/rsyslog/stat-jboss-*
service rsyslog start
endscript
}
When you have made all the configuration changes necessary, don't forget to restart the rsyslog process:
/etc/init.d/rsyslog restart
Windows environments
When running JBoss on Windows, Devo recommends using the MagicLog component of the Devo Agent for Windows to forward the log events to Devo. You can also use the third-party tool, Snare Epilog.
In both cases:
- Make sure the logs are written in text files.
- Have the complete paths to the log files on hand when setting up the sending.