GCP Kafka BigQuery Data Engineer

## GCP Data Engineer – Roles and Responsibilities

- Design, develop, and maintain scalable data pipelines to ingest data from Kafka topics into Google Cloud Platform (GCP).
- Develop and manage Kafka-to-BigQuery integrations using Kafka BigQuery Sink Connectors.
- Configure and maintain Kafka connectors to consume messages from Kafka topics and load data into BigQuery tables.
- Work with Google Cloud services, including BigQuery and Cloud Storage, for data storage and processing.
- Develop and maintain ETL/ELT pipelines using SQL and Python.
- Deploy and manage Kafka connectors running in Kubernetes environments.
- Use GitHub Actions for connector configuration management and deployment automation.
- Handle schema evolution, schema changes, and BigQuery table structure updates to support changing data requirements.
- Monitor data ingestion pipelines and troubleshoot failures, connectivity issues, and data processing errors.
- Implement data quality checks to ensure data accuracy, consistency, and reliability.
- Optimize data pipelines for performance, scalability, and efficient data processing.
- Collaborate with cross-functional teams to understand data requirements and deliver reliable data integration solutions.

**Technical Skills:** Google Cloud Platform (GCP), BigQuery, Cloud Storage, Apache Kafka, Kafka Connect, BigQuery Sink Connector, SQL, Python, Kubernetes, GitHub Actions, ETL/ELT, Data Integration, and Data Pipeline Monitoring.