Which Serilog sink to use for sending to Logstash?
We started using Serilog in combination with Elasticsearch, and it's a very efficient way to store structure log data (and later visualize them using tools like Kibana). However, I see the advantage of not writing log data directly to the backend but instead configure a log broker such as Logstash that can take responsibility for adding tags to log messages, selecting indexes etc. With this setup applications won't need to have knowledge of log data distribution.
With Logstash in the middle the question is what Serilog sink is best to use so Logstash can import its data without applying advanced and CPU-intensive filters. I've seen Redis mentioned as a good companion to Logstash, but Serilog doesn't have a Redis sink. Any recommendations for Serilog sink which data can be easily transferred by Logstash to an Elasticsearch index?
There is even an approach to use Elasticsearch sink first and then loopback it to Elasticsearch again after some arrangements and applying extra tags.
The accepted answer was written before the sink Serilog.Sinks.Http existed.
Instead of logging to file and having Filebeat monitoring it, one could have the HTTP sink post log events to the Logstash HTTP input plugin. This would mean fewer moving parts on the instances where the logs where created.
I received a suggestion from Nicholas Blumhardt (Serilog creator) to use RollingFileSink with JsonFormatter.
I've created a sink for Serilog that supports RabbitMQ, which ties extremely well into logstash, using logstash's rabbitmq input-plugin:
If you run an instance of RabbitMQ on your application-server, you can then log with Serilog to this RabbitMQ instance using the RabbitMQSink, ultimately by-passing the network-segregation/downage scenarioes.
Logstash will then pick-up the messages on the queue when network is back up.
Update: The Sink is now in V2.0 and supports ASP.NET Core 1.0