Level property keeps getting renamed and added to the LokiLogEvent if the grafana server is down #273
Description
Which version of Serilog.Sinks.Grafana.Loki are you using?
v8.3.0
Which version of .NET are you using?
net6.0
Describe the bug
If a socketException is thrown, the field _waitingBatch in the class LokiSink is not cleared (while it is cleared if an error StatusCode is returned). Next time the timer is fired, the same logs get sent out, the level property gets renamed to "level" and a new "level" property is added. Every following tick a new level property gets added and and the old one gets renamed with an ever increasing number of pre-pended "".
First timer tick: The first level gets added
Second timer tick: The second level property gets added
Fourth tick and beyond: A new level property keeps getting added
To Reproduce
Connect to an address with no Loki Instance attached to it
Expected behavior
The number of "level" properties should not grow.
Log/SelfLog output or exception with stacktrace
No response
Application or code sample, which could be used to reproduce a bug
No response
Additional context
Our grafana instance went down for the weekend and the memory consumption went from 400MB~ to 3GB~ for each instance of our deployed application. Every single logevent in the memory dump had hundreds of level properties with an increasing number of underscores.
A screenshot of a log property extracted from our app:
We fixed the problem by adding a custom property renamer:
public class GrafanaPropertiesRenamingStrategy:IReservedPropertyRenamingStrategy
{
public string Rename(string originalName) => originalName=="level"?originalName:$"_{originalName}";
}I have read the documentation
- I have read Common Problems section in project wiki