Data Engineer
Job Description
Role Summary
We are looking for a Data Engineer responsible for operating, monitoring, and enhancing data ingestion and processing pipelines within a Google Cloud–based data platform. The role combines day-to-day platform reliability, data validation, and development activities, supporting continuous platform evolution, new data onboarding, and automation initiatives. The ideal candidate is hands-on, detail-oriented, and comfortable working across production and pre-production environments.
Key Responsibilities
A) Data Ingestion Operations & Monitoring
- Monitor and ensure successful daily and monthly data ingestion across multiple upstream systems.
- Oversee scheduled jobs and queries, validating successful execution and investigating failures or delays.
- Maintain and enhance data ingestion monitoring checks, including:
- Updating checks when new datasets or tables are introduced.
- Adapting monitoring logic based on frequency, country, or business dimensions.
- Track and investigate unprocessed or delayed files in cloud storage, performing root cause analysis and coordinating resolution with relevant stakeholders.
- Manage and follow up on incident or service requests related to missing or delayed data until fully resolved.
B) Automation & Platform Reliability
- Support and troubleshoot automation workflows used to capture ingestion exceptions and feed monitoring logic.
- Ensure automation processes run reliably and intervene quickly in case of failures.
- Handle ad-hoc automation requests to reduce manual effort and improve operational efficiency.
- Proactively identify recurring issues and implement improvements to strengthen pipeline stability.
C) Data Validation & Reporting
- Perform data validation and reconciliation activities by comparing datasets across environments or systems when required (typically on a recurring or monthly basis).
- Develop, maintain, and execute Python-based validation scripts to identify discrepancies and data quality issues.
- Prepare recurring operational and reconciliation reports in line with agreed schedules.
D) Development & Enhancements
- Support environment setup and migration activities within Google Cloud, including configuration changes between production and pre-production environments.
- Create, test, and deploy new tables and datasets following platform standards.
- Implement schema changes and enhancements to existing tables while ensuring backward compatibility and data integrity.
- Enable onboarding of new countries, business units, or data domains into existing ingestion and monitoring frameworks.
- Continuously refine ingestion and validation logic to align with evolving business requirements.
Technologies & Tools
- Google Cloud Platform (GCP)
- SQL
- Python
- ODBC
- Postman
- Workflow and automation tools
Issue tracking tools (e.g., ticketing systems)
Qualifications
Required Skills & Competencies
2-6 years of experience
Technical Skills
- Experience with data engineering operations: ingestion monitoring, troubleshooting, and reliability ownership.
- Hands-on exposure to GCP administration (environment handling, scheduled workloads, monitoring).
- Strong SQL skills for investigation, validation, and operational queries.
- Proficiency in Python for scripting, automation, and data validation.
- Ability to work confidently across pre-production and production environments.
- Solid understanding of system architecture and end-to-end data flows.
Professional Skills
- Strong analytical and problem-solving capabilities with a structured approach to root cause analysis.
- Effective communication and stakeholder management skills, including coordination with cross-functional teams.
- Ability to balance BAU operations with development and enhancement work.
- High level of ownership, accountability, and attention to detail.
Nice to Have
- Experience with cloud data storage patterns and file-based ingestion.
- Familiarity with incident management and operational SLAs.
- Interest in improving observability, monitoring, and data quality frameworks.
