ELK Stack Specialist

About DeepLight:

DeepLight is a pioneering AI company committed to pushing the boundaries of innovation in artificial intelligence. Our mission is to harness the power of data and machine learning to revolutionize industries and create a brighter future. With a dynamic team of experts and a culture of relentless innovation, we are at the forefront of AI research and development.

Position Overview:

We are looking for an experienced ELK Stack Specialist to join our team. The ideal candidate will have strong expertise in managing and writing ELK pipelines, particularly with Logstash agents, Elasticsearch, and OpenSearch. You will be responsible for reading data from EMS Queues and implementing efficient data storage and retrieval mechanisms on AWS OpenSearch.

Develop and manage ELK pipelines to ensure smooth data ingestion and processing.

Expertise in configuring and managing Logstash agents for efficient data processing.

Configure and manage Elasticsearch and OpenSearch clusters on AWS.

Implement and maintain alerting mechanisms to monitor system performance and health.

Ensure proper data publishing to OpenSearch instances on AWS.

Design and implement index creation and auto-archival strategies for efficient data storage.

Troubleshoot and resolve any data ingestion or processing issues.

Collaborate with cross-functional teams to integrate data pipelines and improve overall system performance.

Strong hands-on experience with ELK stack components: Logstash, Elasticsearch, and OpenSearch.

Familiarity with data ingestion from EMS Queues.

Experience with alerting and monitoring mechanisms in ELK.

In-depth knowledge of AWS services, specifically for OpenSearch management.

Experience with creating and managing indexes, data retention, and archival strategies.

Strong problem-solving and troubleshooting skills.

Excellent written and verbal communication skills.

Preferred Experience:

Experience with AWS CloudFormation, Lambda, or other AWS automation services.

Familiarity with other monitoring and logging tools.