Skip to content

Latest commit

 

History

History
822 lines (786 loc) · 23.2 KB

data-model-appendix.md

File metadata and controls

822 lines (786 loc) · 23.2 KB

Data Model Appendix

Note: this document is NOT a spec, it is provided to support the Logs Data Model specification. These examples provided purely for demonstrative purposes and are not exhaustive or canonical; please refer to the respective exporter documentation if exact details are required.

Appendix A. Example Mappings

This section contains examples of mapping of other events and logs formats to this data model.

RFC5424 Syslog

Property Type Description Maps to Unified Model Field
TIMESTAMP Timestamp Time when an event occurred measured by the origin clock. Timestamp
SEVERITY enum Defines the importance of the event. Example: Debug Severity
FACILITY enum Describes where the event originated. A predefined list of Unix processes. Part of event source identity. Example: mail system Attributes["syslog.facility"]
VERSION number Meta: protocol version, orthogonal to the event. Attributes["syslog.version"]
HOSTNAME string Describes the location where the event originated. Possible values are FQDN, IP address, etc. Resource["host.name"]
APP-NAME string User-defined app name. Part of event source identity. Resource["service.name"]
PROCID string Not well defined. May be used as a meta field for protocol operation purposes or may be part of event source identity. Attributes["syslog.procid"]
MSGID string Defines the type of the event. Part of event source identity. Example: "TCPIN" Attributes["syslog.msgid"]
STRUCTURED-DATA array of maps of string to string A variety of use cases depending on the SDID:
Can describe event source identity.
Can include data that describes particular occurrence of the event.
Can be meta-information, e.g. quality of timestamp value.
SDID origin.swVersion map to Resource["service.version"]. SDID origin.ip map to Attributes["client.address"]. Rest of SDIDs -> Attributes["syslog.*"]
MSG string Free-form text message about the event. Typically human readable. Body

Windows Event Log

Property Type Description Maps to Unified Model Field
TimeCreated Timestamp The time stamp that identifies when the event was logged. Timestamp
Level enum Contains the severity level of the event. Severity
Computer string The name of the computer on which the event occurred. Resource["host.name"]
EventID uint The identifier that the provider used to identify the event. Attributes["winlog.event_id"]
Message string The message string. Body
Rest of the fields. any All other fields in the event. Attributes["winlog.*"]

SignalFx Events

Field Type Description Maps to Unified Model Field
Timestamp Timestamp Time when the event occurred measured by the origin clock. Timestamp
EventType string Short machine understandable string describing the event type. SignalFx specific concept. Non-namespaced. Example: k8s Event Reason field. Attributes["com.splunk.signalfx.event_type"]
Category enum Describes where the event originated and why. SignalFx specific concept. Example: AGENT. Attributes["com.splunk.signalfx.event_category"]
Dimensions map<string, string> Helps to define the identity of the event source together with EventType and Category. Multiple occurrences of events coming from the same event source can happen across time and they all have the value of Dimensions. Resource
Properties map<string, any> Additional information about the specific event occurrence. Unlike Dimensions which are fixed for a particular event source, Properties can have different values for each occurrence of the event coming from the same event source. Attributes

Splunk HEC

We apply this mapping from HEC to the unified model:

Field Type Description Maps to Unified Model Field
time numeric, string The event time in epoch time format, in seconds. Timestamp
host string The host value to assign to the event data. This is typically the host name of the client that you are sending data from. Resource["host.name"]
source string The source value to assign to the event data. For example, if you are sending data from an app you are developing, you could set this key to the name of the app. Resource["com.splunk.source"]
sourcetype string The sourcetype value to assign to the event data. Resource["com.splunk.sourcetype"]
event any The JSON representation of the raw body of the event. It can be a string, number, string array, number array, JSON object, or a JSON array. Body
fields map<string, any> Specifies a JSON object that contains explicit custom fields. Attributes
index string The name of the index by which the event data is to be indexed. The index you specify here must be within the list of allowed indexes if the token has the indexes parameter set. Attributes["com.splunk.index"]

When mapping from the unified model to HEC, we apply this additional mapping:

Unified model element Type Description Maps to HEC
SeverityText string The severity of the event as a human-readable string. fields['otel.log.severity.text']
SeverityNumber string The severity of the event as a number. fields['otel.log.severity.number']
Name string Short event identifier that does not contain varying parts. fields['otel.log.name']
TraceId string Request trace id. fields['trace_id']
SpanId string Request span id. fields['span_id']
TraceFlags string W3C trace flags. fields['trace_flags']

Log4j

Field Type Description Maps to Unified Model Field
Instant Timestamp Time when an event occurred measured by the origin clock. Timestamp
Level enum Log level. Severity
Message string Human readable message. Body
All other fields any Structured data. Attributes

Zap

Field Type Description Maps to Unified Model Field
ts Timestamp Time when an event occurred measured by the origin clock. Timestamp
level enum Logging level. Severity
caller string Calling function's filename and line number. Attributes, key=TBD
msg string Human readable message. Body
All other fields any Structured data. Attributes

Apache HTTP Server access log

Field Type Description Maps to Unified Model Field
%t Timestamp Time when an event occurred measured by the origin clock. Timestamp
%a string Client address Attributes["network.peer.address"]
%A string Server address Attributes["network.local.address"]
%h string Client hostname. Attributes["client.address"]
%m string The request method. Attributes["http.request.method"]
%v,%p,%U,%q string Multiple fields that can be composed into URL. Attributes["url.full"]
%>s string Response status. Attributes["http.response.status_code"]
All other fields any Structured data. Attributes, key=TBD

CloudTrail Log Event

Field Type Description Maps to Unified Model Field
eventTime string The date and time the request was made, in coordinated universal time (UTC). Timestamp
eventSource string The service that the request was made to. This name is typically a short form of the service name without spaces plus .amazonaws.com. Resource["service.name"]?
awsRegion string The AWS region that the request was made to, such as us-east-2. Resource["cloud.region"]
sourceIPAddress string The IP address that the request was made from. Attributes["client.address"]
errorCode string The AWS service error if the request returns an error. Attributes["cloudtrail.error_code"]
errorMessage string If the request returns an error, the description of the error. Body
All other fields * Attributes["cloudtrail.*"]

Google Cloud Logging

Field Type Description Maps to Unified Model Field
timestamp string The time the event described by the log entry occurred. Timestamp
resource MonitoredResource The monitored resource that produced this log entry. Resource
log_name string The URL-encoded LOG_ID suffix of the log_name field identifies which log stream this entry belongs to. Attributes["gcp.log_name"]
json_payload google.protobuf.Struct The log entry payload, represented as a structure that is expressed as a JSON object. Body
proto_payload google.protobuf.Any The log entry payload, represented as a protocol buffer. Body
text_payload string The log entry payload, represented as a Unicode string (UTF-8). Body
severity LogSeverity The severity of the log entry. Severity
trace string The trace associated with the log entry, if any. TraceId
span_id string The span ID within the trace associated with the log entry. SpanId
labels map<string,string> A set of user-defined (key, value) data that provides additional information about the log entry. Attributes
http_request HttpRequest The HTTP request associated with the log entry, if any. Attributes["gcp.http_request"]
trace_sampled boolean The sampling decision of the trace associated with the log entry. TraceFlags.SAMPLED
All other fields Attributes["gcp.*"]

Elastic Common Schema

Field Type Description Maps to Unified Model Field
@timestamp datetime Time the event was recorded Timestamp
message string Any type of message Body
labels key/value Arbitrary labels related to the event Attributes[*]
tags array of string List of values related to the event ?
trace.id string Trace ID TraceId
span.id* string Span ID SpanId
agent.ephemeral_id string Ephemeral ID created by agent **Resource
agent.id string Unique identifier of this agent **Resource
agent.name string Name given to the agent Resource["telemetry.sdk.name"]
agent.type string Type of agent Resource["telemetry.sdk.language"]
agent.version string Version of agent Resource["telemetry.sdk.version"]
source.ip, client.ip string The IP address that the request was made from. Attributes["client.address"]
cloud.account.id string ID of the account in the given cloud Resource["cloud.account.id"]
cloud.availability_zone string Availability zone in which this host is running. Resource["cloud.zone"]
cloud.instance.id string Instance ID of the host machine. **Resource
cloud.instance.name string Instance name of the host machine. **Resource
cloud.machine.type string Machine type of the host machine. **Resource
cloud.provider string Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. Resource["cloud.provider"]
cloud.region string Region in which this host is running. Resource["cloud.region"]
cloud.image.id* string Resource["host.image.name"]
container.id string Unique container id Resource["container.id"]
container.image.name string Name of the image the container was built on. Resource["container.image.name"]
container.image.tag Array of string Container image tags. **Resource
container.labels key/value Image labels. Attributes[*]
container.name string Container name. Resource["container.name"]
container.runtime string Runtime managing this container. Example: "docker" **Resource
destination.address string Destination address for the event Attributes["destination.address"]
error.code string Error code describing the error. Attributes["error.code"]
error.id string Unique identifier for the error. Attributes["error.id"]
error.message string Error message. Attributes["error.message"]
error.stack_trace string The stack trace of this error in plain text. Attributes["error.stack_trace]
host.architecture string Operating system architecture **Resource
host.domain string Name of the domain of which the host is a member.
For example, on Windows this could be the host’s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host’s LDAP provider.
**Resource
host.name string Hostname of the host.
It normally contains what the hostname command returns on the host machine.
Resource["host.name"]
host.id string Unique host id. Resource["host.id"]
host.ip Array of string Host IP Resource["host.ip"]
host.mac array of string MAC addresses of the host Resource["host.mac"]
host.name string Name of the host.
It may contain what hostname returns on Unix systems, the fully qualified, or a name specified by the user.
Resource["host.name"]
host.type string Type of host. Resource["host.type"]
host.uptime string Seconds the host has been up. ?
service.ephemeral_id string Ephemeral identifier of this service **Resource
service.id string Unique identifier of the running service. If the service is comprised of many nodes, the service.id should be the same for all nodes. **Resource
service.name string Name of the service data is collected from. Resource["service.name"]
service.node.name string Specific node serving that service Resource["service.instance.id"]
service.state string Current state of the service. Attributes["service.state"]
service.type string The type of the service data is collected from. **Resource
service.version string Version of the service the data was collected from. Resource["service.version"]

* Not yet formalized into ECS.

** A resource that doesn’t exist in the OpenTelemetry resource semantic convention.

This is a selection of the most relevant fields. See for the full reference for an exhaustive list.

Appendix B: SeverityNumber example mappings

Syslog WinEvtLog Log4j Zap java.util.logging .NET (Microsoft.Extensions.Logging) SeverityNumber
TRACE FINEST LogLevel.Trace TRACE
Debug Verbose DEBUG Debug FINER LogLevel.Debug DEBUG
FINE DEBUG2
CONFIG DEBUG3
Informational Information INFO Info INFO LogLevel.Information INFO
Notice INFO2
Warning Warning WARN Warn WARNING LogLevel.Warning WARN
Error Error ERROR Error SEVERE LogLevel.Error ERROR
Critical Critical Dpanic ERROR2
Alert Panic ERROR3
Emergency FATAL Fatal LogLevel.Critical FATAL

References