It’s day 1 of SigNoz Launch Week 2.0, and we’re releasing Ingest Guard, a feature that will help platform and finops teams have granular control over data ingestion and observability costs.
Let’s dig in deeper.
The Motivation Behind Ingest Guard
Ingestion keys are usually built to enhance security, and they typically serve as authentication tokens that allow users to send data securely. They help manage and authenticate data ingestion, ensuring that only authorized sources are allowed to push data into the system. Security was a key consideration while building Ingest Guard.
But we didn’t want to stop there. We gathered feedback from platform teams using SigNoz and found out they struggle to manage data ingestion from different teams. That’s when we thought of extending the capabilities of ingestion keys in SigNoz and building Ingest Guard.
With Ingest Guards, you can:
- Create ingestion keys for enhanced security using auth
- Set data ingestion limits based on the type of signal, engineering teams, environment, etc.
How does it work?
Here’s a quick demo of how the Ingest Guard feature works in SigNoz:
Prioritizing Security from the Start
The expiration feature forces users to think about security proactively from the time an ingestion key is created. By setting up a mandatory expiration, teams can avoid long-lived credentials that often become security loopholes.
Ingestion Limits for Better Control on Observability Costs
We extended the functionality of the ingestion keys to include data ingestion limits. It enables a number of use cases around control of data ingestion and observability costs:
- Set data ingestion limits based on the type of signal. For example, a daily limit of 100GB on logs data ingestion.
- Set data ingestion limits based on engineering teams. For example, you can allot 100GBs per day to Team A and 200GBs per day to Team B based on use case.
- Create separate ingestion keys for various environments (e.g., production, staging, development) to control data sent from each environment individually.
- Prevent Data Spikes with Second-Level Limits
And many more.
Prevent Data Ingestion Spikes with Second-level limits
Observability systems commonly experience data spikes during certain events, such as application errors, sudden traffic increases, or large-scale log generation during a production push. These unexpected surges can quickly use up your daily data limit in a short period, leaving the system unable to ingest further data for the rest of the day. This can cause critical gaps in monitoring, lead to missed alerts, and disrupt the ability to troubleshoot effectively.
We have solved this by introducing Second-level limits. You can configure these to manage sudden data spikes, ensuring that unexpected surges don’t exhaust daily quotas too quickly.
Empowering FinOps with Data Ingestion Control
FinOps teams are under increasing pressure to optimize costs while maintaining service quality. Ingest Guard is tailored for FinOps teams, enabling them to monitor and control the amount of observability data each team sends. By creating separate ingestion keys for different teams, environments, or even microservices, FinOps teams can set clear usage boundaries. This ensures that no single team exceeds their allotted data budget, keeping costs in check and avoiding unnecessary resource waste.
Using tags for better categorization
Tags in Ingest Guard can be a powerful tool for better cost control by providing a way to organize, monitor, and allocate resource usage across different teams, projects, environments, or cost centers.
Future Roadmap
We will actively collect feedback from our users for our next iteration on Ingest Guard. But here are a few things that we think we can possibly do with it.
Improving usage visibility
Lots of our users ask us how they can know how much of the observability data being ingested is used by engineering teams. With Ingest Guard, we can improve usage visibility for platform teams by enabling them to see how much data ingested was queried and at what frequency. This can help them optimize data ingestion from sources that are not frequently used.
Alerts on data ingestion
We also plan to integrate alerting mechanisms that notify teams when they approach or exceed their ingestion limits, allowing for proactive management. Additionally, we’re exploring the possibility of adding more granular control options, such as minute-based limits or monthly quotas, to provide even more flexibility to our users.
Get Started with Ingest Guard
Ingest Guard is a powerful step forward in giving teams the control and visibility they need to manage their observability data. Focusing on security and cost control helps platform and FinOps teams work more efficiently while reducing waste and improving overall security. We’re excited to see how our users implement this feature, and we welcome feedback to help make it even better.
Join us for SigNoz Launch Week 2.0
From Sep 16 to Sep 20, we will announce a feature everyday at 9AM PT to level up your observability.
Join Us for SigNoz Launch Week 2.0
Day 4 - Alerts History & Scheduled Maintenance
