Big Data Pipeline Engineer

Key Roles & Responsibilities:

• System Monitoring: Continuously monitor ETL jobs, workflows, and data pipelines for performance and reliability.
• Job Scheduling: Ensure jobs are scheduled appropriately and are running as per the defined schedules.
• Issue Resolution: Troubleshoot and resolve issues related to data integration, ETL jobs, and workflows promptly.
• Root Cause Analysis: Perform root cause analysis for recurring issues and implement preventive measures.
• Tuning: Optimize ETL jobs and workflows for better performance and efficiency.
• Resource Management: Ensure optimal use of system resources like CPU, memory, and storage.
• Data Validation: Ensure data accuracy, completeness, and consistency across different systems.
• Error Handling: Implement robust error handling and recovery mechanisms in ETL processes.
• Data Reconciliation: Perform data reconciliation to verify that the data loaded into th...