Send Application/Server logs from EC2 to SigNoz

Introduction

This guide provides detailed instructions on how to send application and server logs from an EC2 instance to SigNoz Cloud. By integrating with SigNoz, you can efficiently collect, monitor, and analyze your logs for better insights into your applications and servers.

Prerequisites

• A Linux-based EC2 instance
• An active SigNoz Cloud account

Sending your server/application logs to SigNoz Cloud broadly involves these two simple steps:

• Install OpenTelemetry Collector(OTel collector)
• Configure filelog receiver

Install OpenTelemetry Collector

The OpenTelemetry collector provides a vendor-neutral way to collect, process, and export your telemetry data such as logs, metrics, and traces.

You can install OpenTelemetry collector as an agent on your Virtual Machine by following this guide.

Dummy log file

As an example, we can use a sample log file called app.log with the following dummy data:

This is log line 1
This is log line 2
This is log line 3

This file represents a log file of your application/server.

Configure filelog receiver

Receivers are used to get data into the collector. A filelog receiver collects logs from files. Modify the config.yaml file that you created while installing OTel collector in the previous step to include the filelog receiver. This involves specifying the path to your app.log file (or your log file) and setting the start_at parameter. For more fields that are available for filelog receiver please check this link.

receivers:
  ...
  filelog/app:
    include: [ /tmp/app.log ] #include the full path to your log file
    start_at: end
...

Update pipeline configuration

Receivers must be enabled via pipelines within the service section of the collector config file. In the same config.yaml file mentioned above, update the pipeline settings to include the new filelog receiver. This step is crucial for ensuring that the logs are correctly processed and sent to SigNoz.

    service:
        ....
        logs:
            receivers: [otlp, filelog/app]
            processors: [batch]
            exporters: [otlp]

Now restart the OTel collector so that new changes are applied. The steps to run the OTel collector can be found here

Verifying the exported logs

The logs will be exported to SigNoz UI. If you add more entries to your app.log file they will also be visible in SigNoz UI.