Senior Data Engineer

​

Responsibilities​

​

• Leading data engineer and complex projects to completion

• Design, plan, prioritize and be responsible for the project

• Build scalable, maintainable data pipelines (batch/streaming-ingestion/ELT/ETL/data products) and ensure their quality/reliability/integrity.

• Collaborate with analytics and business teams to improve data models that feed business intelligence tools, increasing data accessibility and fostering data-driven decision-making across the organization.

• Coaching and mentoring junior data engineer to develop their skills

 

Qualifications

​

• Bachelor’s Degree in Computer Science, Software Engineering, Information Technology, or equivalent industry experience.

• 6 years experience in Big Data technologies and their ecosystem

• Proficient in SQL, Python or Linux & Unix

• Experience in hadoop ecosystem such as HDFS, Spark, Hive, Sqoop, Airflow, Oozie, Ranger, Ambari, Flink.

• Experience in cloud computing technologies such as AWS, Azure, GCP.

• Experience working with relational database such as MySQL, PostgreSQL, SQL Server, Oracle.

• Experience working with NoSQL databases such as MongoDB, HBase, Cassandra, Bigtable, DynamoDB, and Cosmos DB.

• Experience working with seach engines tools like Elastic Search

• Experience in end to end data management solution

• Experience in data migration tools such as Fivetran, Informatica, database migration tools

• Ability to design data lake, data warehouse, data mart based on AWS, Azure, GCP and on-premise.

• Understanding of data lake management such as life cycle management, storage class design, and access control.

• Ability to optimize data warehouse and data mart such as indexing, clustering, and partitioning.

• Ability to design data modeling (schema design) such as star schema, snowflake schema, fact table, dimensional table.

• Experience in ETL/ ELT solutions for both on cloud and on-premise.

• Ability to develop ETL/ELT solution in Python, Spark, SQL.

• Understanding of real-time and batch processing.

• Experience in real-time processing (streaming) tools such as Apache Kafka, RabbitMQ, Cloud Pub/Sub, Azure Event Hubs, Amazon Kinesis

• Experience in workflow orchestration, monitoring or data pipeline tools such as Apache Airflow, Azure Data Factory, Luigi, NiFi, AWS Step Function.

• Innovative problem-solving skills with the ability to identify and resolve complex architectural issues

• Ability to communicate clearly and work closely with cross-functional teams such as Data Analyst, Data Visualization, Software Engineering, and businesses functions

• Good command of English

• Excellent organizational and leadership skills

• An outgoing personality who has proven to be a good team player

• Agile methodology mindset

​

​