Customer Challenge:
A cyber security company specializing in solutions for potential security breaches faced a significant challenge. High operational overhead and difficulty scaling their legacy system complicated their operations. To address this issue the cyber security company chose to partner with Jahnel Group. We are a software consulting firm renowned for its expertise in AWS DynamoDB with a proven track record of optimizing cost and time savings for clients.
The Jahnel Group team provided a distributed system using AWS services, including DynamoDB, AWS Lambda, and SQS. The solution comprised five core components for efficient message processing and analysis. The benefits included reduced operational overhead, estimated cost savings of about 75%, improved performance with rapid message delivery, and seamless scalability to handle varying workloads.
About Our Customer:
A prominent figure in the cybersecurity industry, our customer excels in providing advanced security solutions to protect organizations against cyber threats. They are dedicated to offering cutting-edge technologies and expertise to safeguard their clients' digital assets and data in an ever-evolving threat landscape. Their focus on defending organizations of all sizes aligns with their vision of making world-class security accessible to every entity, regardless of their scale or industry.
Navigating the Challenge:
Our customer faced several challenges with their existing system for detecting potential security breaches based on messages received from agents running on machines. The legacy solution had high operational overhead, required dedicated engineering resources for support, and had difficulties scaling in response to bursty traffic. This led to outages in critical situations, compromising the effectiveness of the product. Also, keeping servers provisioned at maximum capacity was not cost-effective.
If the challenge of replacing the legacy system with a scalable and efficient one was not addressed, our customer would continue to face high operational overhead and difficulty in scaling during bursts of traffic. The system could suffer from outages during critical situations, which could lead to increased security risks and potential security breaches going undetected. The cost-effectiveness of the product would also be compromised, as they would have to maintain costly provisioned servers at maximum capacity.
The JG Solution:
To address our customer's business challenge, Jahnel Group developed a distributed system using a combination of AWS services. The cyber security company required a more efficient and scalable system to detect potential security breaches based on messages received from agents running on various machines.
The core AWS services utilized in Jahnel Group's solution were DynamoDB, AWS Lambda, and SQS. The solution comprised five essential components to process the incoming messages effectively.
The Syslog Receiver Service, built on AWS ECS, received and stored the raw Syslog messages into an SQS queue. Following that, the Syslog Parser Service, implemented as an AWS Lambda function, parsed the raw messages and converted them into JSON format. This service also provided basic parsing and data normalization capabilities, ensuring a consistent data structure.
The Syslog Hydrator, another AWS Lambda triggered by messages in an SQS queue, performed initial validation and normalized the JSON schema for further processing. This component leveraged Redis as a hot cache to apply filtering rules to quickly identify known safe patterns, effectively reducing the number of messages requiring further analysis.
Two AWS Lambdas, collectively known as the Optics Hydrator, were responsible for handling API call responses and adding them to the SQS messages. These messages were then dropped into a new queue for additional processing.
The critical component, the Guard Orchestrator, processed the collected data. It applied a set of rules stored in DynamoDB to determine whether a message matched a harmful pattern or required manual review as a potentially harmful message. The messages were subsequently filed in the ticket management system based on the analysis results.
The solution allowed users to update "fast filter" rules in Redis and create, update, or delete dynamic rulesets based on emerging threat patterns through a Lambda function. Another background Lambda flagged rules for review and potential eviction if they had not been matched within a configurable time frame, preventing outdated rules from impacting the filtering process.
DynamoDB Streams played a crucial role in maintaining fast access times to rulesets. The overloaded PK/SK table contained three core data types: compressed rulesets, a list of rulesets with multiple Global Secondary Indexes (GSIs), and a count of matched rules and last matched date used for rule eviction.
Results and Benefits:
To address the customer's business challenge, Jahnel Group developed a distributed system using a combination of AWS services. The customer required a more efficient and scalable system to detect potential security breaches based on messages received from agents running on various machines.
The implementation of Jahnel Group's solution brought positive outcomes and benefits for our customer. One of the most notable results was the substantial reduction in operational overhead and maintenance burden.
With the legacy system, the customer had to allocate dedicated engineering resources and perform weekly paging to address scaling and debugging issues. After the implementation of the new solution, these operational tasks were streamlined and automated, resulting in a considerable amount of time saved for the customer's engineering team. The ability to scale dynamically without the need for engineer paging allowed them to focus more on strategic initiatives and enhance overall productivity.
Regarding cost-effectiveness, the customer realized significant savings in their IT expenditure. Before the solution, they were faced with the challenge of choosing between costly provisioned servers that were underutilized or slow-responding scaling groups that caused outages during bursty traffic. The new solution provided cost savings of approximately 75% due to its ability to provision the appropriate capacity as needed. This ensured that IT costs were optimized, and resources were efficiently utilized, further contributing to the customer's bottom line.
Another crucial benefit was the improved performance compared to their legacy system utilizing an RDBMS. With Jahnel Group's solution, the message delivery time to the ticketing system was drastically reduced. When fully warmed, the system achieved message delivery in less than 250 milliseconds, and even from a cold start, it took less than seven seconds for every service. This significant enhancement in performance enabled the customer to respond swiftly to potential security threats, minimizing the impact of security breaches and strengthening their ability to protect their clients' environments.
The solution's scalability also played a vital role in handling bursty traffic and adapting to changing workloads. The ability to scale from zero to full capacity, as demonstrated during load testing up to 80,000 messages per second, proved essential in meeting dynamic demands and providing a seamless experience for the customer's clients. This scalability also ensured that the system could handle varying levels of traffic without compromising its performance and efficiency.
Why the Customer Chose Jahnel Group:
The customer selected Jahnel Group based on their qualifications and expertise in the cyber security industry, which suited tackling the customer's business challenge. They have a track record of successfully managing projects with large data volumes and optimizing complex technology implementations, proving their capability to handle dynamic and bursty traffic demands effectively. Their experience in working with DynamoDB Streams showcased their ability to create dynamic rule updates and ensure fast access times to rulesets, critical for the customer's solution.
Jahnel Group's proficiency in leveraging DynamoDB's features enabled them to design and implement a distributed system for efficiently processing and analyzing incoming messages from agents on various machines, meeting the customer's need for a scalable and streamlined solution.
About Jahnel Group
Jahnel Group is a consulting firm that specializes in helping companies leverage technology to improve their business operations. We provide end-to-end strategic consulting and deployment services, helping companies optimize their operations and reduce costs through the use of technology. Jahnel Group is an Advanced Tier AWS Services Partner, with expertise in AWS Lambdas, Amazon API Gateway, Amazon DynamoDB, and other AWS services.
