feat[opentelemetry-collector]: add support for multiline recombine from container parser #1364
Conversation
…om container parser Signed-off-by: Paulo Dias <[email protected]>
Signed-off-by: Paulo Dias <[email protected]>
paulojmdias
requested review from
dmitryax,
TylerHelmuth,
povilasv and
a team
as code owners
| # combine multiline logs (e.g. stack traces) | ||
| - id: recombine-multiline | ||
| type: recombine | ||
| combine_field: body | ||
| is_first_entry: body matches "^[^\\s]" | ||
| source_identifier: attributes["log.file.path"] | ||
| max_log_size: {{ $.Values.presets.logsCollection.maxRecombineLogSize }} |
There was a problem hiding this comment.
What happens to logs that don't need recombined? Does this add performance overhead?
There was a problem hiding this comment.
Regarding resources usage, we don't see any performance penalty. Below you can see the average CPU and memory usage before and after this add.
Before:
After:
Is there any other metric that we can take an eye?
Does it make sense to add this using a feature toggle on values.yaml instead of force use it?
There was a problem hiding this comment.
Since this is a preset config, I'm trying to understand if recombine is universally helpful and wont impact any users who don't need it.
If it negatively affects users who don't need to recombine logs, then it would be better for you to configure the filelog's operators manually instead of including it automatically in the preset's config.
There was a problem hiding this comment.
Technically isn't seems so, but I understand your point.
I still see value on this recombine or dynamically support custom operator additions to the Kubernetes filelog receiver without copying paste everything and lose future updates.
Let me also give you some possibilities to think about it:
- Add a feature toggle on values to add this block on this preset config
- Add a new hash on values with this new value as default to be used and anyone can make it empty if they want to disable
- Add a new hash on values to support custom operators for this Kubernetes filelog receiver with an empty value as default
Let me know what do you think about it
The current
containeroperator cannot handle multiline messages like Java stack traces.This PR adds the needed recombine operator to make it work. Probably in the future, this makes sense to be moved into the container operator itself.