Senior Data Engineer OpenSearch

IntroductionYour primary responsibilities will include: Design and Develop Open. Search Solutions: Design, implement, and maintain scalable Open. Search clusters and search solutions for large-scale structured, unstructured, and vector-based data. Implement Vector Search: Develop and optimize vector search solutions using approximate nearest neighbor (ANN) techniques, with particular focus on HNSW and filtered vector search. Optimize Search Relevance: Analyze semantic and hybrid search quality and performance, including chunking strategies, embeddings, filtering, scoring, ranking, and retrieval strategies, and implement improvements based on search evaluation results. Optimize Queries and Search Performance: Profile slow or resource-intensive queries, identify bottlenecks, and rewrite queries to improve latency and throughput. Favor efficient filtering and caching strategies where appropriate over expensive scoring operations. Tune Open. Search Clusters: Right-size shards and replicas, balance workloads across nodes, tune indexing and search performance, and identify appropriate data-node instance types based on workload characteristics and search patterns. Manage Index Lifecycle: Design and maintain appropriate index lifecycle strategies, including Index State Management (ISM)/ILM concepts, rollover, retention, shard management, and data tiering, to support sustained production performance. Monitor and Troubleshoot Production Platforms: Analyze cluster health, node utilization, JVM/memory pressure, CPU, disk usage, indexing throughput, search latency, and query behavior to identify and resolve production issues. Improve Platform Efficiency: Continuously optimize Open. Search infrastructure for performance, scalability, reliability, and cost efficiency.Your role and responsibilitiesAs a Senior Open. Search Engineer specializing in large-scale search and vector search platforms, you will design, develop, optimize, and operate high-performance Open. Search solutions supporting semantic, hybrid, and traditional search use cases. The role requires strong hands-on experience with vector search, approximate nearest neighbor (ANN) algorithms, search relevance, query optimization, and Open. Search cluster performance tuning. You will work across the full search lifecycle, from index and data modeling through relevance optimization, scaling, performance troubleshooting, and production operations.Required technical and professional expertise * Strong hands-on experience with Open. Search and/or Elasticsearch in production environments. * Proven experience designing and operating large-scale Open. Search clusters, preferably with 100M+ vectors and 200GB+ of data. * Strong understanding of vector search and approximate nearest neighbor (ANN) algorithms. * Practical experience with HNSW-based vector search and associated index configuration and performance tuning. * Experience implementing and optimizing filtered vector search. * Strong understanding of semantic search, hybrid search, relevance, ranking, scoring, and retrieval strategies. * Experience evaluating and improving search relevance and search quality, including the impact of chunking strategies, embeddings, filtering, and scoring. * Strong knowledge of Open. Search query DSL and the ability to profile, analyze, and optimize slow or resource-intensive queries. * Experience with query performance optimization, caching strategies, filtering, scoring, and reducing search latency. * Strong understanding of sharding, replicas, shard sizing, node allocation, cluster topology, and workload balancing. * Experience tuning Open. Search clusters based on CPU, memory, JVM, disk I/O, indexing throughput, search throughput, and latency characteristics. * Experience with index lifecycle management, rollover, retention, and index/data lifecycle strategies. Preferred technical and professional experience * Experience with Open. Search Serverless and/or AWS-managed Open. Search. * Experience with AWS services such as Amazon Open. Search Service, Amazon S3, AWS Lambda, AWS Glue, Amazon Bedrock, or related cloud services. * Experience building RAG (Retrieval-Augmented Generation) solutions using Open. Search as a retrieval/vector search layer. * Experience working with embedding models, vector databases, semantic retrieval, and LLM-based search applications. * Experience with hybrid search architectures, combining lexical/BM25 search with vector/semantic search. * Knowledge of search relevance evaluation methodologies and experience defining search quality metrics. * Experience with Python for search evaluation, performance analysis, automation, and data processing. * Experience with Open. Search Dashboards, monitoring, observability, and operational tooling. * Experience with Terraform or other Infrastructure as Code technologies. * Experience with CI/CD, Git-based development workflows, automated deployments, and DevOps practices. * Experience designing search platforms for high availability, fault tolerance, and horizontal scalability. * Experience working with large-scale AI/ML workloads and production GenAI applications. IBM is committed to creating a diverse environment and is proud to be an equal-opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual orientation, national origin, caste, genetics, pregnancy, disability, neurodivergence, age, veteran status, or other characteristics. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.