JSON Template Layout
JsonTemplateLayout
is a customizable, efficient, and garbage-free JSON generating layout.
It encodes LogEvent
s according to the structure described by the JSON template provided.
In a nutshell, it shines with its
-
Customizable JSON structure (see
eventTemplate[Uri]
andstackTraceElementTemplate[Uri]
layout configuration parameters) -
Customizable timestamp formatting (see
timestamp
event template resolver) -
Feature rich exception formatting (see
exception
andexceptionRootCause
event template resolvers) -
Customizable object recycling strategy
JSON Template Layout is intended for production deployments where the generated logs are expected to be delivered to an external log ingestion system such as Elasticsearch or Google Cloud Logging. While running tests or developing locally, you can use Pattern Layout for human-readable log output. |
Usage
Adding log4j-layout-template-json
artifact to your list of dependencies is enough to enable access to JSON Template Layout in your Log4j configuration:
-
Maven
-
Gradle
We assume you use log4j-bom
for dependency management.
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-layout-template-json</artifactId>
<scope>runtime</scope>
</dependency>
We assume you use log4j-bom
for dependency management.
runtimeOnly 'org.apache.logging.log4j:log4j-layout-template-json'
JSON Template Layout is primarily configured through an event template describing the structure log events should be JSON-encoded in.
Event templates themselves are also JSON documents, where objects containing $resolver
members, such as,
{
"$resolver": "message", (1)
"stringified": true (2)
}
1 | Indicating that this object should be replaced with the output from the message event template resolver |
2 | Passing a configuration to the message event template resolver |
are interpreted by the JSON Template Layout compiler, and replaced with the referenced event or stack trace template resolver rendering that particular item.
For instance, given the following event template stored in MyLayout.json
in your classpath:
{
"instant": { (1)
"$resolver": "timestamp",
"pattern": {
"format": "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",
"timeZone": "UTC"
}
},
"someConstant": 1, (2)
"message": { (3)
"$resolver": "message",
"stringified": true
}
}
1 | Using the timestamp event template resolver to populate the instant field |
2 | Passing a constant that will be rendered as is |
3 | Using the message event template resolver to populate the message field |
in combination with the below layout configuration:
-
XML
-
JSON
-
YAML
-
Properties
log4j2.xml
<JsonTemplateLayout eventTemplateUri="classpath:MyLayout.json"/>
log4j2.json
"JsonTemplateLayout": {
"eventTemplateUri": "classpath:MyLayout.json"
}
log4j2.yaml
JsonTemplateLayout:
eventTemplateUri: "classpath:MyLayout.json"
log4j2.properties
appender.0.layout.type = JsonTemplateLayout
appender.0.layout.eventTemplateUri = classpath:MyLayout.json
JSON Template Layout generates JSON as follows:
{"instant":"2017-05-25T19:56:23.370Z","someConstant":1,"message":"Hello, error!"} (1)
1 | JSON pretty-printing is not supported for performance reasons. |
Good news is JSON Template Layout is perfectly production-ready without any configuration!
The event template defaults to EcsLayout.json
, bundled in the classpath, modelling
the Elastic Common Schema (ECS) specification.
JSON Template Layout bundles several more predefined event templates modeling popular JSON-based log formats.
Configuration
This section explains how to configure JSON Template Layout plugin element in a Log4j configuration file.
Are you trying to implement your own event (or stack trace) template and looking for help on available resolvers? Please refer to Template configuration instead. |
Plugin attributes
JSON Template Layout plugin configuration accepts the following attributes:
charset
Type |
|
---|---|
Default value |
|
Configuration property |
|
Charset
used for encoding the produced JSON into bytes
While RFC 4627, the JSON specification, supports multiple Unicode encodings, we strongly advise you to stick to the layout default, and use UTF-8 for several practical reasons. |
locationInfoEnabled
Type |
|
---|---|
Default value |
|
Configuration property |
|
Toggles access to the LogEvent
source; file name, line number, etc.
See also location information.
stackTraceEnabled
Type |
|
---|---|
Default value |
|
Configuration property |
|
Toggles access to the stack traces
eventTemplate
Type |
|
---|---|
Default value |
|
Configuration property |
|
Inline event template JSON for rendering LogEvent
s.
If present, this configuration overrides eventTemplateUri
.
eventTemplateUri
Type |
|
---|---|
Default value |
|
Configuration property |
|
URI pointing to the event template JSON for rendering LogEvent
s.
This configuration is overriden by eventTemplate
, if present.
eventTemplateRootObjectKey
Type |
|
---|---|
Default value |
|
Configuration property |
|
If present, the event template is put into a JSON object composed of a single member with the provided key.
stackTraceElementTemplate
Type |
|
---|---|
Default value |
|
Configuration property |
|
Inline stack trace element template JSON for rendering StackTraceElement
s
If present, this configuration overrides stackTraceElementTemplateUri
.
stackTraceElementTemplateUri
Type |
|
---|---|
Default value |
|
Configuration property |
|
URI pointing to the stack trace element template JSON for rendering StackTraceElement
s.
This configuration is overriden by stackTraceElementTemplate
, if present.
eventDelimiter
Type |
|
---|---|
Default value |
|
Configuration property |
|
Delimiter used for separating rendered LogEvent
s.
if nullEventDelimiterEnabled
is true
, this value will be suffixed with \0
(null) character.
nullEventDelimiterEnabled
Type |
|
---|---|
Default value |
|
Configuration property |
|
If true
, eventDelimiter
will be suffixed with \0
(null) character.
maxStringLength
Type |
|
---|---|
Default value |
|
Configuration property |
|
Causes truncation of string values longer than the specified limit.
When a string value is truncated, its length will be shortened to the maxStringLength
provided and truncatedStringSuffix
will be appended to indicate the truncation.
Note that this doesn’t cap the maximum length of the JSON document produced!
Consider a JSON document rendered using a An example JSON document exceeding 16,384 characters in length, yet subject to no truncation
|
truncatedStringSuffix
Type |
|
---|---|
Default value |
|
Configuration property |
|
Suffix to append to strings truncated due to exceeding maxStringLength
recyclerFactory
Type |
|
---|---|
Default value |
Refer to Recycling strategy |
Configuration property |
|
Name of the Recycling strategy employed
Plugin elements
JSON Template Layout plugin configuration accepts the following elements:
EventTemplateAdditionalField
Additional event template fields are convenient shortcuts to add custom fields to a template or override existing ones.
You can specify an additional event template field using an EventTemplateAdditionalField
element composed of following attributes:
key
-
Entry key
value
-
Entry value
format
-
Format of the value. Accepted values are:
STRING
(default)-
Indicates that the entry value will be string-encoded
JSON
-
Indicates the the entry value is already formatted in JSON and will be appended to the event template verbatim
Below we share an example configuration overriding the GelfLayout.json
event template with certain custom fields:
-
XML
-
JSON
-
YAML
-
Properties
log4j2.xml
<JsonTemplateLayout eventTemplateUri="classpath:GelfLayout.json">
<EventTemplateAdditionalField
key="aString"
value="foo"/>(1)
<EventTemplateAdditionalField
key="marker"
format="JSON"
value='{"$resolver": "marker", "field": "name"}'/>
<EventTemplateAdditionalField
key="aNumber"
format="JSON"
value="1"/>
<EventTemplateAdditionalField
key="aList"
format="JSON"
value='[1, 2, "three"]'/>
</JsonTemplateLayout>
log4j2.json
"JsonTemplateLayout": {
"eventTemplateUri": "classpath:GelfLayout.json",
"eventTemplateAdditionalField": [
{
"key": "aString",
"value": "foo" (1)
},
{
"key": "marker",
"value": "{\"$resolver\": \"marker\", \"field\": \"name\"}",
"format": "JSON"
},
{
"key": "aNumber",
"value": "1",
"format": "JSON"
},
{
"key": "aList",
"value": "[1, 2, \"three\"]",
"format": "JSON"
}
]
}
log4j2.yaml
JsonTemplateLayout:
eventTemplateUri: "classpath:GelfLayout.json"
eventTemplateAdditionalField:
- key: "aString"
value: "foo" (1)
- key: "marker"
value: '{"$resolver": "marker", "field": "name"}'
format: "JSON"
- key: "aNumber"
value: "1"
format: "JSON"
- key: "aList"
value: '[1, 2, "three"]'
format: "JSON"
log4j2.properties
appender.0.layout.type = JsonTemplateLayout
appender.0.layout.eventTemplateUri = classpath:GelfLayout.json
appender.0.layout.eventTemplateAdditionalField[0].type = EventTemplateAdditionalField
appender.0.layout.eventTemplateAdditionalField[0].key = aString
appender.0.layout.eventTemplateAdditionalField[0].value = foo (1)
appender.0.layout.eventTemplateAdditionalField[1].type = EventTemplateAdditionalField
appender.0.layout.eventTemplateAdditionalField[1].key = marker
appender.0.layout.eventTemplateAdditionalField[1].value = {"$resolver": "marker", "field": "name"}
appender.0.layout.eventTemplateAdditionalField[1].format = JSON
appender.0.layout.eventTemplateAdditionalField[2].type = EventTemplateAdditionalField
appender.0.layout.eventTemplateAdditionalField[2].key = aNumber
appender.0.layout.eventTemplateAdditionalField[2].value = 1
appender.0.layout.eventTemplateAdditionalField[2].format = JSON
appender.0.layout.eventTemplateAdditionalField[3].type = EventTemplateAdditionalField
appender.0.layout.eventTemplateAdditionalField[3].key = aList
appender.0.layout.eventTemplateAdditionalField[3].value = [1, 2, "three"]
appender.0.layout.eventTemplateAdditionalField[3].format = JSON
1 | Since the format attribute is not explicitly set, the default (i.e., STRING ) will be used |
Template configuration
Templates are JSON documents, where objects containing $resolver
members, such as,
{
"$resolver": "message", (1)
"stringified": true (2)
}
1 | Indicating that this object should be replaced with the output from the message event template resolver |
2 | Passing a configuration to the message event template resolver |
are interpreted by the JSON Template Layout compiler, and replaced with the referenced event or stack trace template resolver rendering that particular item.
Templates are configured by means of the following JSON Template Layout plugin attributes:
-
eventTemplate
andeventTemplateUri
(for encodingLogEvent
s) -
stackTraceElementTemplate
andstackTraceElementTemplateUri
(for encodingStackStraceElement
s) -
EventTemplateAdditionalField
(for extending the event template)
Event templates
eventTemplate
and eventTemplateUri
describe the JSON structure JSON Template Layout uses to encode LogEvent
s.
JSON Template Layout contains the following predefined event templates:
EcsLayout.json
-
The default event template modelling the Elastic Common Schema (ECS) specification
LogstashJsonEventLayoutV1.json
GelfLayout.json
-
Models the Graylog Extended Log Format (GELF) payload specification with additional
_thread
and_logger
fields.If used, it is advised to override the obligatory
host
field with a user provided constant via additional event template fields to avoidhostName
property lookup at runtime, which incurs an extra cost. GcpLayout.json
-
Models the structure described by Google Cloud Platform structured logging with additional
_thread
,_logger
and_exception
fields. The exception trace, if any, is written to the_exception
field as well as themessage
field – the former is useful for explicitly searching/analyzing structured exception information, while the latter is Google’s expected place for the exception, and integrates with Google Error Reporting. JsonLayout.json
-
Models the exact structure generated by the deprecated
JsonLayout
with the exception ofthrown
field.JsonLayout
serializes theThrowable
as is via JacksonObjectMapper
, whereasJsonLayout.json
event template employs theStackTraceElementLayout.json
stack trace template for stack traces to generate a document-store-friendly flat structure.This event template is only meant for existing users of the deprecated
JsonLayout
to migrate to JSON Template Layout without much trouble, and other than that purpose, is not recommended to be used!
Event template resolvers
Event template resolvers consume a LogEvent
and render a certain property of it at the point of the JSON where they are declared.
For instance, marker
resolver renders the marker of the event, level
resolver renders the level, and so on.
An event template resolver is denoted with a special object containing a $resolver
key:
level
resolver{
"version": "1.0",
"level": {
"$resolver": "level",
"field": "name"
}
}
Here version
field will be rendered as is, while level
field will be populated by the level
resolver.
That is, this template will generate JSON similar to the following:
{
"version": "1.0",
"level": "INFO"
}
The complete list of available event template resolvers are provided below in detail.
counter
Resolves a number from an internal counter
counter
event template resolver grammarconfig = [ start ] , [ overflowing ] , [ stringified ]
start = "start" -> number
overflowing = "overflowing" -> boolean
stringified = "stringified" -> boolean
Unless provided, start
and overflowing
are respectively set to 0
(zero) and true
by default.
When overflowing
is set to true
, the internal counter is created using a long
, which is subject to overflow while incrementing, though faster and garbage-free.
Otherwise, a BigInteger
is used, which does not overflow, but incurs allocation costs.
When stringified
is enabled, which is set to false
by default, the resolved number will be converted to a string.
See examples
Resolves a sequence of numbers starting from 0. Once Long.MAX_VALUE
is reached, counter overflows to Long.MIN_VALUE
.
{
"$resolver": "counter"
}
Resolves a sequence of numbers starting from 1000. Once Long.MAX_VALUE
is reached, counter overflows to Long.MIN_VALUE
.
{
"$resolver": "counter",
"start": 1000
}
Resolves a sequence of numbers starting from 0 and keeps on doing as long as JVM heap allows.
{
"$resolver": "counter",
"overflowing": false
}
caseConverter
Converts the case of string values
caseConverter
event template resolver grammarconfig = case , input , [ locale ] , [ errorHandlingStrategy ]
input = JSON
case = "case" -> ( "upper" | "lower" )
locale = "locale" -> (
language |
( language , "_" , country ) |
( language , "_" , country , "_" , variant )
)
errorHandlingStrategy = "errorHandlingStrategy" -> (
"fail" |
"pass" |
"replace"
)
replacement = "replacement" -> JSON
input
can be any available template value; e.g., a JSON literal, a lookup string, an object pointing to another resolver.
Unless provided, locale
points to the one returned by JsonTemplateLayoutDefaults.getLocale()
, which is configured by the log4j.layout.jsonTemplate.locale
system property and by default set to the default system locale.
errorHandlingStrategy
determines the behavior when either the input doesn’t resolve to a string value or case conversion throws an exception:
fail
-
Propagates the failure
pass
-
Causes the resolved value to be passed as is
replace
-
Suppresses the failure and replaces it with the
replacement
, which is set tonull
by default
errorHandlingStrategy
is set to replace
by default.
Most of the time JSON logs are persisted to a storage solution (e.g., Elasticsearch) that keeps a statically-typed index on fields.
Hence, if a field is always expected to be of type string, using non-string replacement
s or pass
in errorHandlingStrategy
might result in type incompatibility issues at the storage level.
Unless the input value is |
See examples
Convert the resolved log level strings to upper-case:
{
"$resolver": "caseConverter",
"case": "upper",
"input": {
"$resolver": "level",
"field": "name"
}
}
Convert the resolved USER
environment variable to lower-case using nl_NL
locale:
{
"$resolver": "caseConverter",
"case": "lower",
"locale": "nl_NL",
"input": "${env:USER}"
}
Convert the resolved sessionId
thread context data (MDC) to lower-case:
{
"$resolver": "caseConverter",
"case": "lower",
"input": {
"$resolver": "mdc",
"key": "sessionId"
}
}
Above, if sessionId
MDC resolves to a, say, number, case conversion will fail.
Since errorHandlingStrategy
is set to replace
and replacement is set to null
by default, the resolved value will be null
.
One can suppress this behavior and let the resolved sessionId
number be left as is:
{
"$resolver": "caseConverter",
"case": "lower",
"input": {
"$resolver": "mdc",
"key": "sessionId"
},
"errorHandlingStrategy": "pass"
}
or replace it with a custom string:
{
"$resolver": "caseConverter",
"case": "lower",
"input": {
"$resolver": "mdc",
"key": "sessionId"
},
"errorHandlingStrategy": "replace",
"replacement": "unknown"
}
exception
Resolves fields of the Throwable
returned by LogEvent#getThrown()
exception
event template resolver grammarconfig = field , [ stringified ] , [ stackTrace ]
field = "field" -> ( "className" | "message" | "stackTrace" )
stackTrace = "stackTrace" -> (
[ stringified ]
, [ elementTemplate ]
)
stringified = "stringified" -> ( boolean | truncation )
truncation = "truncation" -> (
[ suffix ]
, [ pointMatcherStrings ]
, [ pointMatcherRegexes ]
)
suffix = "suffix" -> string
pointMatcherStrings = "pointMatcherStrings" -> string[]
pointMatcherRegexes = "pointMatcherRegexes" -> string[]
elementTemplate = "elementTemplate" -> object
stringified
is set to false
by default.
stringified
at the root level is deprecated in favor of stackTrace.stringified
, which has precedence if both are provided.
pointMatcherStrings
and pointMatcherRegexes
enable the truncation of stringified stack traces after the given matching point. If both parameters are provided, pointMatcherStrings
will be checked first.
If a stringified stack trace truncation takes place, it will be indicated with a`suffix`, which by default is set to the configure truncatedStringSuffix
in the layout, unless explicitly provided.
Every truncation suffix is prefixed with a newline.
Stringified stack trace truncation operates in Caused by:
and Suppressed:
label blocks.
That is, matchers are executed against each label in isolation.
elementTemplate
is an object describing the template to be used while resolving the StackTraceElement
array.
If stringified
is set to true
, elementTemplate
will be discarded.
By default, elementTemplate
is set to null
and rather populated from the layout configuration.
That is, the stack trace element template can also be provided using stackTraceElementTemplate
and stackTraceElementTemplateUri
layout configuration attributes.
The template to be employed is determined in the following order:
-
elementTemplate
provided in the resolver configuration -
stackTraceElementTemplate
layout configuration attribute (The default is populated from thelog4j.layout.jsonTemplate.stackTraceElementTemplate
system property.) -
stackTraceElementTemplateUri
layout configuration attribute (The default is populated from thelog4j.layout.jsonTemplate.stackTraceElementTemplateUri
system property.)
See Stack trace element templates for the list of available resolvers in a stack trace element template.
Note that this resolver is toggled by the stackTraceEnabled
layout configuration attribute.
Since Each |
See examples
Resolve LogEvent#getThrown().getClass().getCanonicalName()
:
{
"$resolver": "exception",
"field": "className"
}
Resolve the stack trace into a list of StackTraceElement
objects:
{
"$resolver": "exception",
"field": "stackTrace"
}
Resolve the stack trace into a string field:
{
"$resolver": "exception",
"field": "stackTrace",
"stackTrace": {
"stringified": true
}
}
Resolve the stack trace into a string field such that the content will be truncated after the given point matcher:
{
"$resolver": "exception",
"field": "stackTrace",
"stackTrace": {
"stringified": {
"truncation": {
"suffix": "... [truncated]",
"pointMatcherStrings": ["at javax.servlet.http.HttpServlet.service"]
}
}
}
}
Resolve the stack trace into an object described by the provided stack trace element template:
{
"$resolver": "exception",
"field": "stackTrace",
"stackTrace": {
"elementTemplate": {
"class": {
"$resolver": "stackTraceElement",
"field": "className"
},
"method": {
"$resolver": "stackTraceElement",
"field": "methodName"
},
"file": {
"$resolver": "stackTraceElement",
"field": "fileName"
},
"line": {
"$resolver": "stackTraceElement",
"field": "lineNumber"
}
}
}
}
See Stack trace element templates for further details on resolvers available for StackTraceElement
templates.
exceptionRootCause
Resolves the fields of the innermost Throwable
returned by LogEvent#getThrown()
.
Its syntax and garbage-footprint are identical to the exception
resolver.
level
Resolves the fields of the LogEvent#getLevel()
level
event template resolver grammarconfig = field , [ severity ]
field = "field" -> ( "name" | "severity" )
severity = severity-field
severity-field = "field" -> ( "keyword" | "code" )
See examples
Resolve the level name:
{
"$resolver": "level",
"field": "name"
}
Resolve the Syslog severity keyword:
{
"$resolver": "level",
"field": "severity",
"severity": {
"field": "keyword"
}
}
Resolve the Syslog severity code:
{
"$resolver": "level",
"field": "severity",
"severity": {
"field": "code"
}
}
logger
Resolves LogEvent#getLoggerFqcn()
and LogEvent#getLoggerName()
.
logger
event template grammarconfig = "field" -> ( "name" | "fqcn" )
See examples
Resolve the logger name:
{
"$resolver": "logger",
"field": "name"
}
Resolve the logger’s fully qualified class name:
{
"$resolver": "logger",
"field": "fqcn"
}
main
Performs Main Argument Lookup for the given index
or key
main
event template resolver grammarconfig = ( index | key )
index = "index" -> number
key = "key" -> string
See examples
Resolve the 1st main()
method argument:
{
"$resolver": "main",
"index": 0
}
Resolve the argument coming right after --userId
:
{
"$resolver": "main",
"key": "--userId"
}
map
Resolves MapMessage
s.
See Map resolver for details.
marker
Resolves the marker of the event
marker
event template resolver grammarconfig = "field" -> ( "name" | "parents" )
See examples
Resolve the marker name:
{
"$resolver": "marker",
"field": "name"
}
Resolve the names of the marker’s parents:
{
"$resolver": "marker",
"field": "parents"
}
mdc
Resolves the thread context map, aka. Mapped Diagnostic Context (MDC). See Map resolver for details.
|
message
Resolves the message of the event
message
event template resolver grammarconfig = [ stringified ] , [ fallbackKey ]
stringified = "stringified" -> boolean
fallbackKey = "fallbackKey" -> string
For simple string messages, the resolution is performed without allocations.
For |
See examples
Resolve the message into a string:
{
"$resolver": "message",
"stringified": true
}
Resolve the message such that if it is an ObjectMessage
or a MultiformatMessage
with JSON support, its type (string, list, object, etc.) will be retained:
{
"$resolver": "message"
}
Given the above configuration, a SimpleMessage
will generate a "sample log message"
, whereas a MapMessage
will generate a {"action": "login", "sessionId": "87asd97a"}
.
Certain indexed log storage systems (e.g.,
Elasticsearch)
will not allow both values to coexist due to type mismatch: one is a string
while the other is an object
.
Here one can use a fallbackKey
to work around this problem:
{
"$resolver": "message",
"fallbackKey": "formattedMessage"
}
Using this configuration, a SimpleMessage
will generate a {"formattedMessage": "sample log message"}
, and a MapMessage
will generate a {"action": "login", "sessionId": "87asd97a"}
.
Note that both emitted JSONs are of type object
and have no type-conflicting fields.
messageParameter
Resolves LogEvent#getMessage().getParameters()
messageParameter
event template resolver grammarconfig = [ stringified ] , [ index ]
stringified = "stringified" -> boolean
index = "index" -> number
Regarding garbage footprint, |
See examples
Resolve the message parameters into an array:
{
"$resolver": "messageParameter"
}
Resolve the string representation of all message parameters into an array:
{
"$resolver": "messageParameter",
"stringified": true
}
Resolve the first message parameter:
{
"$resolver": "messageParameter",
"index": 0
}
Resolve the string representation of the first message parameter:
{
"$resolver": "messageParameter",
"index": 0,
"stringified": true
}
ndc
Resolves the thread context stack – aka. Nested Diagnostic Context (NDC), aka – String[]
returned by LogEvent#getContextStack()
ndc
event template resolver grammarconfig = [ pattern ]
pattern = "pattern" -> string
See examples
Resolve all NDC values into a list:
{
"$resolver": "ndc"
}
Resolve all NDC values matching with the pattern
regex:
{
"$resolver": "ndc",
"pattern": "user(Role|Rank):\\w+"
}
pattern
Resolver delegating to Pattern Layout
pattern
event template resolver grammarconfig = pattern , [ stackTraceEnabled ]
pattern = "pattern" -> string
stackTraceEnabled = "stackTraceEnabled" -> boolean
Unlike providing the pattern
attribute to Pattern Layout in a configuration file, property substitutions found in the pattern
will not be resolved.
The default value of stackTraceEnabled
is inherited from the parent JSON Template Layout.
This resolver is mostly intended as an emergency lever when all other JSON Template Layout resolvers fall short of addressing your need. If you find yourself using this, it is highly likely you are either doing something wrong, or JSON Template Layout needs some improvement. In either case, you are advised to share your use case with maintainers in one of our support channels. |
See examples
Resolve the string produced by %p %c{1.} [%t] %X{userId} %X %m%ex
pattern:
{
"$resolver": "pattern",
"pattern": "%p %c{1.} [%t] %X{userId} %X %m%ex"
}
source
Resolves the fields of the StackTraceElement
returned by LogEvent#getSource()
source
event template resolver grammarconfig = "field" -> (
"className" |
"fileName" |
"methodName" |
"lineNumber" )
Note that this resolver is toggled by the locationInfoEnabled
layout configuration attribute.
Capturing the source location information is an expensive operation, and is not garbage-free.
The |
See examples
Resolve the line number:
{
"$resolver": "source",
"field": "lineNumber"
}
thread
Resolves LogEvent#getThreadId()
, LogEvent#getThreadName()
, LogEvent#getThreadPriority()
thread
event template resolver grammarconfig = "field" -> ( "name" | "id" | "priority" )
See examples
Resolve the thread name:
{
"$resolver": "thread",
"field": "name"
}
timestamp
Resolves LogEvent#getInstant()
timestamp
event template resolver grammarconfig = [ patternConfig | epochConfig ]
patternConfig = "pattern" -> ( [ format ] , [ timeZone ] , [ locale ] )
format = "format" -> string
timeZone = "timeZone" -> string
locale = "locale" -> (
language |
( language , "_" , country ) |
( language , "_" , country , "_" , variant )
)
epochConfig = "epoch" -> ( unit , [ rounded ] )
unit = "unit" -> (
"nanos" |
"millis" |
"secs" |
"millis.nanos" |
"secs.nanos" |
)
rounded = "rounded" -> boolean
See examples
Configuration |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Map resolver
ReadOnlyStringMap
is Log4j’s Map<String, Object>
equivalent with garbage-free accessors, and is heavily employed throughout the code base.
It is the data structure backing both thread context map (aka., Mapped Diagnostic Context (MDC)) and MapMessage
implementations.
Hence template resolvers for both of these are provided by a single backend: ReadOnlyStringMapResolver
.
Put another way, both mdc
and map
resolvers support identical configuration, behaviour, and garbage footprint, which are detailed below.
config = singleAccess | multiAccess
singleAccess = key , [ stringified ]
key = "key" -> string
stringified = "stringified" -> boolean
multiAccess = [ pattern ] , [ replacement ] , [ flatten ] , [ stringified ]
pattern = "pattern" -> string
replacement = "replacement" -> string
flatten = "flatten" -> ( boolean | flattenConfig )
flattenConfig = [ flattenPrefix ]
flattenPrefix = "prefix" -> string
singleAccess
resolves a single field, whilst multiAccess
resolves a multitude of fields.
If flatten
is provided, multiAccess
merges the fields with the parent, otherwise creates a new JSON object containing the values.
Enabling stringified
flag converts each value to its string representation.
Regex provided in the pattern
is used to match against the keys.
If provided, replacement
will be used to replace the matched keys.
These two are analogous to Pattern.compile(pattern).matcher(key).matches()
and Pattern.compile(pattern).matcher(key).replaceAll(replacement)
calls.
Regarding garbage footprint,
Writing certain non-primitive values (e.g., |
See examples
"$resolver"
is left out in the following examples, since it is to be defined by the actual resolver, e.g., map
, mdc
.
Resolve the value of the field keyed with user:role
:
{
"$resolver": "…",
"key": "user:role"
}
Resolve the string representation of the user:rank
field value:
{
"$resolver": "…",
"key": "user:rank",
"stringified": true
}
Resolve all fields into an object:
{
"$resolver": "…"
}
Resolve all fields into an object such that values are converted to string:
{
"$resolver": "…",
"stringified": true
}
Resolve all fields whose keys match with the user:(role|rank)
regex into an object:
{
"$resolver": "…",
"pattern": "user:(role|rank)"
}
Resolve all fields whose keys match with the user:(role|rank)
regex into an object after removing the user:
prefix in the key:
{
"$resolver": "…",
"pattern": "user:(role|rank)",
"replacement": "$1"
}
Merge all fields whose keys are matching with the user:(role|rank)
regex into the parent:
{
"$resolver": "…",
"flatten": true,
"pattern": "user:(role|rank)"
}
After converting the corresponding field values to string, merge all fields to parent such that keys are prefixed with _
:
{
"$resolver": "…",
"stringified": true,
"flatten": {
"prefix": "_"
}
}
Stack trace element templates
exception
and exceptionRootCause
event template resolvers can encode an exception stack trace (i.e., StackTraceElement
returned by Throwable#getStackTrace()
) into a JSON array.
While doing so, they employ the underlying JSON templating infrastructure again.
The stack trace template used by these event template resolvers to encode StackTraceElement
s can be configured in following ways:
-
elementTemplate
provided in the resolver configuration -
stackTraceElementTemplate
layout configuration attribute (The default is populated from thelog4j.layout.jsonTemplate.stackTraceElementTemplate
system property.) -
stackTraceElementTemplateUri
layout configuration attribute (The default is populated from thelog4j.layout.jsonTemplate.stackTraceElementTemplateUri
system property.)
By default, stackTraceElementTemplateUri
is set to classpath:StackTraceElementLayout.json
, which references to a classpath resource bundled with JSON Template Layout:
StackTraceElementLayout.json
bundled as a classpath resource{
"class": {
"$resolver": "stackTraceElement",
"field": "className"
},
"method": {
"$resolver": "stackTraceElement",
"field": "methodName"
},
"file": {
"$resolver": "stackTraceElement",
"field": "fileName"
},
"line": {
"$resolver": "stackTraceElement",
"field": "lineNumber"
}
}
Stack trace element template resolvers
Similar to Event template resolvers consuming a LogEvent
and rendering a certain property of it at the point of the JSON where they are declared, stack trace element template resolvers consume a StackTraceElement
.
The complete list of available event template resolvers are provided below in detail.
Property substitution
Property substitutions (e.g., ${myProperty}
), including lookups (e.g., ${java:version}
, ${env:USER}
, ${date:MM-dd-yyyy}
), are supported, but extra care needs to be taken.
We strongly advise you to carefully read the configuration manual before using them.
Lookups are intended as a very generic, convenience utility to perform string interpolation for, in particular, configuration files and components (e.g., layouts) lacking this mechanism. JSON Template Layout has a rich template resolver collection, and you should always prefer it whenever possible over lookups. Which resolvers can I use to replace lookups?
|
Property substitution in event templates
JSON Template Layout performs property substitution in string literals in templates, except if they are located in configuration object of resolvers.
Consider the following event template file provided using the eventTemplateUri
attribute:
{
"java-version": "${java:version}", (1)
"pid": {
"$resolver": "pattern",
"pattern": "${env:NO_SUCH_KEY:-%pid}" (2)
}
}
1 | This works. ${java:version} will be replaced with the corresponding value. |
2 | This won’t work! That is, ${env:NO_SUCH_KEY:-%pid} literal will not get substituted, since it is located in a configuration object of a resolver. |
Property substitution in configuration files
If the very same event template shared above is inlined in a configuration file using the eventTemplate
attribute or additional event template fields, then all substitutions will be replaced, once, at configuration-time.
This has nothing to do with the JSON Template Layout, but the substitution performed by the configuration mechanism when the configuration is read.
Consider the following example:
-
XML
-
JSON
-
YAML
-
Properties
log4j2.xml
<JsonTemplateLayout eventTemplate='{"instant": {"$resolver": "pattern", "pattern": "${env:LOG4J_DATE_PATTERN:-%d}"}}'> (1)
<EventTemplateAdditionalField
key="message"
format="JSON"
value='{"$resolver": "pattern", "pattern": "${env:LOG4J_MESSAGE_PATTERN:-%m}"}'/> (2)
</JsonTemplateLayout>
log4j2.json
"JsonTemplateLayout": {
"eventTemplate": "{\"instant\": {\"$resolver\": \"pattern\", \"pattern\": \"${env:LOG4J_DATE_PATTERN:-%d}\"}}", (1)
"eventTemplateAdditionalField": [
{
"key": "message",
"format": "JSON",
"value": "{\"$resolver\": \"pattern\", \"pattern\": \"${env:LOG4J_MESSAGE_PATTERN:-%m}\"}" (2)
}
]
}
log4j2.yaml
JsonTemplateLayout:
eventTemplate: '{"instant": {"$resolver": "pattern", "pattern": "${env:LOG4J_DATE_PATTERN:-%d}"}}' (1)
eventTemplateAdditionalField:
- key: "message"
format: "JSON"
value: '{"$resolver": "pattern", "pattern": "${env:LOG4J_MESSAGE_PATTERN:-%m}"}' (2)
log4j2.properties
appender.0.layout.type = JsonTemplateLayout
appender.0.layout.eventTemplate = {"instant": {"$resolver": "pattern", "pattern": "${env:LOG4J_DATE_PATTERN:-%d}"}} (1)
appender.0.layout.eventTemplateAdditionalField[0].type = EventTemplateAdditionalField
appender.0.layout.eventTemplateAdditionalField[0].key = message
appender.0.layout.eventTemplateAdditionalField[0].format = JSON
appender.0.layout.eventTemplateAdditionalField[0].value = {"$resolver": "pattern", "pattern": "${env:LOG4J_MESSAGE_PATTERN:-%m}"} (2)
1 | eventTemplate will be passed to the layout with ${env:…} substituted |
2 | value will be passed to the layout with ${env:…} substituted |
External values injected this way can corrupt your JSON schema. It is your responsibility to ensure the sanitization and safety of the substitution source. |
Recycling strategy
Encoding a LogEvent
without a load on the garbage-collector can yield significant performance benefits.
JSON Template Layout contains the Recycler
interface to implement Garbage-free logging.
A Recycler
is created using a RecyclerFactory
, which implements a particular recycling strategy.
JSON Template Layout contains the following predefined recycler factories:
dummy
-
It performs no recycling, hence each recycling attempt will result in a new instance. This will obviously create a load on the garbage-collector. It is a good choice for applications with low and medium log rate.
threadLocal
-
It performs the best, since every instance is stored in
ThreadLocal
s and accessed without any synchronization cost. Though this might not be a desirable option for applications running with a considerable number of threads, e.g., a web servlet. queue
-
It is the best of both worlds. It allows recycling of objects up to a certain number (
capacity
). When this limit is exceeded due to excessive concurrent load (e.g.,capacity
is 50 but there are 51 threads concurrently trying to encode), it starts allocating.queue
is a good strategy wherethreadLocal
is not desirable.queue
accepts following optional parameters:supplier
-
of type
java.util.Queue
, defaults toorg.jctools.queues.MpmcArrayQueue.new
if JCTools is in the classpath; otherwisejava.util.concurrent.ArrayBlockingQueue.new
capacity
-
of type
int
, defaults tomax(8,2*cpuCount+1)
Some example configurations of
queue
recycling strategy are as follows:-
queue:supplier=org.jctools.queues.MpmcArrayQueue.new
(useMpmcArrayQueue
from JCTools) -
queue:capacity=10
(set the queue capacity to 10) -
queue:supplier=java.util.concurrent.ArrayBlockingQueue.new,capacity=50
(useArrayBlockingQueue
with a capacity of 50)
The default RecyclerFactory
is threadLocal
, if log4j2.enable.threadlocals=true
; otherwise, queue
.
The effective recycler factory can be configured using the recyclerFactory
plugin attribute.
Next to predefined ones, you can introduce custom RecyclerFactory
implementations too.
See Extending recycler factories for details.
Extending
JSON Template Layout relies on the Log4j plugin system to compose the features it provides. This makes it possible for users to extend the plugin-based feature set as they see fit. As of this moment, following features are implemented by means of plugins:
-
Event template resolvers (e.g.,
exception
,message
,level
event template resolvers) -
Event template interceptors (e.g., injection of
eventTemplateAdditionalField
) -
Recycler factories
Following sections cover how to extend these in detail.
While existing features should address most common use cases, you might find yourself needing to implement a custom one. If this is the case, we really appreciate it if you can share your use case in a user support channel. |
Plugin preliminaries
Log4j plugin system is the de facto extension mechanism embraced by various Log4j components. Plugins provide extension points to components, that can be used to implement new features, without modifying the original component. It is analogous to a dependency injection framework, but curated for Log4j-specific needs.
In a nutshell, you annotate your classes with @Plugin
and their (static
) factory methods with @PluginFactory
.
Last, you inform the Log4j plugin system to discover these custom classes.
This is done using running the PluginProcessor
annotation processor while building your project.
Refer to Plugins for details.
Extending event template resolvers
All available event template resolvers are simple plugins employed by JSON Template Layout.
To add new ones, one just needs to create their own EventResolver
and instruct its injection via a @Plugin
-annotated EventResolverFactory
class.
For demonstration purposes, below we will create a randomNumber
event template resolver.
Let’s start with the actual resolver:
package com.acme.logging.log4j.layout.template.json;
import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.layout.template.json.resolver.EventResolver;
import org.apache.logging.log4j.layout.template.json.util.JsonWriter;
/**
* Resolves a random floating point number.
*
* <h3>Configuration</h3>
*
* <pre>
* config = ( [ range ] )
* range = number[]
* </pre>
*
* {@code range} is a number array with two elements, where the first number
* denotes the start (inclusive) and the second denotes the end (exclusive).
* {@code range} is optional and by default set to {@code [0, 1]}.
*
* <h3>Examples</h3>
*
* Resolve a random number between 0 and 1:
*
* <pre>
* {
* "$resolver": "randomNumber"
* }
* </pre>
*
* Resolve a random number between -0.123 and 0.123:
*
* <pre>
* {
* "$resolver": "randomNumber",
* "range": [-0.123, 0.123]
* }
* </pre>
*/
public final class RandomNumberResolver implements EventResolver {
private final double loIncLimit;
private final double hiExcLimit;
RandomNumberResolver(TemplateResolverConfig config) {
List<Number> rangeArray = config.getList("range", Number.class);
if (rangeArray == null) {
this.loIncLimit = 0D;
this.hiExcLimit = 1D;
} else if (rangeArray.size() != 2) {
throw new IllegalArgumentException(
"range array must be of size two: " + config);
} else {
this.loIncLimit = rangeArray.get(0).doubleValue();
this.hiExcLimit = rangeArray.get(1).doubleValue();
if (loIncLimit > hiExcLimit) {
throw new IllegalArgumentException("invalid range: " + config);
}
}
}
static String getName() {
return "randomNumber";
}
@Override
public void resolve(LogEvent value, JsonWriter jsonWriter) {
double randomNumber = loIncLimit + (hiExcLimit - loIncLimit) * Math.random();
jsonWriter.writeNumber(randomNumber);
}
}
Next, create an EventResolverFactory
class to admit RandomNumberResolver
into the event resolver factory plugin registry.
RandomNumberResolver
into the event resolver factory plugin registrypackage com.acme.logging.log4j.layout.template.json;
import org.apache.logging.log4j.core.config.plugins.Plugin;
import org.apache.logging.log4j.core.config.plugins.PluginFactory;
import org.apache.logging.log4j.layout.template.json.resolver.EventResolverContext;
import org.apache.logging.log4j.layout.template.json.resolver.EventResolverFactory;
import org.apache.logging.log4j.layout.template.json.resolver.TemplateResolver;
import org.apache.logging.log4j.layout.template.json.resolver.TemplateResolverConfig;
import org.apache.logging.log4j.layout.template.json.resolver.TemplateResolverFactory;
/**
* {@link RandomNumberResolver} factory.
*/
@Plugin(name = "RandomNumberResolverFactory", category = TemplateResolverFactory.CATEGORY)
public final class RandomNumberResolverFactory implements EventResolverFactory {
private static final RandomNumberResolverFactory INSTANCE = new RandomNumberResolverFactory();
private RandomNumberResolverFactory() {}
@PluginFactory
public static RandomNumberResolverFactory getInstance() {
return INSTANCE;
}
@Override
public String getName() {
return RandomNumberResolver.getName();
}
@Override
public RandomNumberResolver create(EventResolverContext context, TemplateResolverConfig config) {
return new RandomNumberResolver(config);
}
}
Done! Let’s use our custom event resolver:
randomNumber
event resolver<?xml version="1.0" encoding="UTF-8"?>
<Configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="https://logging.apache.org/xml/ns"
xsi:schemaLocation="
https://logging.apache.org/xml/ns
https://logging.apache.org/xml/ns/log4j-config-2.xsd">
<!-- ... -->
<JsonTemplateLayout>
<EventTemplateAdditionalField
key="id"
format="JSON"
value='{"$resolver": "randomNumber", "range": [0, 1000000]}'/>
</JsonTemplateLayout>
<!-- ... -->
</Configuration>
All available event template resolvers are located in org.apache.logging.log4j.layout.template.json.resolver
package.
It is a fairly rich resource for inspiration while implementing new resolvers.
Intercepting the template resolver compiler
JSON Template Layout allows interception of the template resolver compilation, which is the process converting a template into a Java function performing the JSON encoding.
This interception mechanism is internally used to implement eventTemplateRootObjectKey
and EventTemplateAdditionalField
features.
In a nutshell, one needs to create a @Plugin
-annotated class extending from the EventResolverInterceptor
interface.
To see the interception in action, check out the EventRootObjectKeyInterceptor
class which is responsible for implementing the eventTemplateRootObjectKey
feature:
eventTemplateRootObjectKey
, if presentimport org.apache.logging.log4j.layout.template.json.resolver.EventResolverContext;
import org.apache.logging.log4j.layout.template.json.resolver.EventResolverInterceptor;
import org.apache.logging.log4j.layout.template.json.resolver.TemplateResolverInterceptor;
/**
* Interceptor to add a root object key to the event template.
*/
@Plugin(name = "EventRootObjectKeyInterceptor", category = TemplateResolverInterceptor.CATEGORY)
public class EventRootObjectKeyInterceptor implements EventResolverInterceptor {
private static final EventRootObjectKeyInterceptor INSTANCE =
new EventRootObjectKeyInterceptor();
private EventRootObjectKeyInterceptor() {}
@PluginFactory
public static EventRootObjectKeyInterceptor getInstance() {
return INSTANCE;
}
@Override
public Object processTemplateBeforeResolverInjection(EventResolverContext context, Object node) {
String eventTemplateRootObjectKey = context.getEventTemplateRootObjectKey();
return eventTemplateRootObjectKey != null
? Collections.singletonMap(eventTemplateRootObjectKey, node)
: node;
}
}
Here, processTemplateBeforeResolverInjection()
method checks if the user has provided an eventTemplateRootObjectKey
.
If so, it wraps the root node
with a new object; otherwise, returns the node
as is.
Note that node
refers to the root Java object of the event template read by JsonReader
.
Extending recycler factories
recyclerFactory
layout configuration attribute is converted from String
to a RecyclerFactory
using the default RecyclerFactoryConverter
extending from TypeConverter<RecyclerFactory>
.
If one wants to change this behavior, they simply need to add their own TypeConverter<RecyclerFactory>
implementing
Comparable<TypeConverter<?>>
to prioritize their custom converter.
TypeConverter
for RecyclerFactory
package com.acme.logging.log4j.layout.template.json;
import org.apache.logging.log4j.core.config.plugins.Plugin;
import org.apache.logging.log4j.core.config.plugins.convert.TypeConverter;
import org.apache.logging.log4j.core.config.plugins.convert.TypeConverters;
@Plugin(name = "ExampleRecyclerFactoryConverter", category = TypeConverters.CATEGORY)
public final class ExampleRecyclerFactoryConverter implements TypeConverter<RecyclerFactory>, Comparable<TypeConverter<?>> {
@Override
public RecyclerFactory convert(String recyclerFactorySpec) {
return ExampleRecyclerFactory.ofSpec(recyclerFactorySpec);
}
@Override
public int compareTo(final TypeConverter<?> ignored) {
return -1;
}
}
Here note that compareTo()
always returns -1 to rank it higher compared to other matching converters.
Features
Below is a feature comparison matrix between JSON Template Layout and alternatives.
Feature |
|
|||
Java version |
8 |
8 |
8 |
6 |
Dependencies |
None |
Jackson |
None |
None |
Schema customization? |
✓ |
✕ |
✕ |
✕ |
Timestamp customization? |
✓ |
✕ |
✕ |
✕ |
(Almost) garbage-free? |
✓ |
✕ |
✓ |
✓ |
Custom typed |
✓ |
✕ |
✕ |
?[1] |
Custom typed |
✓ |
✕ |
✕ |
✕ |
Rendering stack traces as array? |
✓ |
✓ |
✕ |
✓ |
Stack trace truncation? |
✓ |
✕ |
✕ |
✕ |
JSON pretty print? |
✕ |
✓ |
✕ |
✕ |
Additional string fields? |
✓ |
✓ |
✓ |
✓ |
Additional JSON fields? |
✓ |
✕ |
✕ |
✕ |
Custom resolvers? |
✓ |
✕ |
✕ |
✕ |
Performance
JSON Template Layout is a heavily optimized piece of software to encode a log event as fast as possible. To get the most out of it, mind the following checklist:
-
Enable garbage-free logging
-
Choose a recycling strategy that suits best to your deployment environment
-
Don’t give too much slack to
maxStringLength
and try to keep it relatively tight
F.A.Q.
Are recursive collections supported?
No. Consider a Message
containing a recursive value as follows:
Object[] recursiveCollection = new Object[1];
recursiveCollection[0] = recursiveCollection;
While the exact exception might vary, you will most like get a StackOverflowError
for trying to render recursiveCollection
into a String
.
Note that this is also the default behaviour for other Java standard library methods, e.g., Arrays.toString()
.
Hence mind self references while logging.
Is JSON Template Layout garbage-free?
Yes, if garbage-free logging is enabled. Take into account the following caveats:
-
The configured Recycling strategy might not be garbage-free.
-
Since
Throwable#getStackTrace()
clones the originalStackTraceElement[]
, access to (and hence rendering of) stack traces are not garbage-free. -
Serialization of
MapMessage
s andObjectMessage
s are mostly garbage-free except for certain types (e.g.,BigDecimal
,BigInteger
,Collection
s, exceptList
). -
Property substitutions (that is,
${…}
variables) might not be garbage-free.
Event template resolvers contain notes on their garbage footprint. Make sure to check those notes of resolvers you employ in templates.
ObjectMessage
s and if Jackson is in the classpath.