Acerca de

Senior Data Engineer

Responsibilities

  • Leading data engineer and complex projects to completion

  • Design, plan, prioritize and be responsible for the project

  • Build scalable, maintainable data pipelines (batch/streaming-ingestion/ELT/ETL/data products) and ensure their quality/reliability/integrity.

  • Collaborate with analytics and business teams to improve data models that feed business intelligence tools, increasing data accessibility and fostering data-driven decision-making across the organization.

  • Coaching and mentoring junior data engineer to develop their skills

 

Qualifications

  • Bachelor’s Degree in Computer Science, Software Engineering, Information Technology, or equivalent industry experience.

  • 6 years experience in Big Data technologies and their ecosystem

  • Proficient in SQL, Python or Linux & Unix

  • Experience in hadoop ecosystem such as HDFS, Spark, Hive, Sqoop, Airflow, Oozie, Ranger, Ambari, Flink.

  • Experience in cloud computing technologies such as AWS, Azure, GCP.

  • Experience working with relational database such as MySQL, PostgreSQL, SQL Server, Oracle.

  • Experience working with NoSQL databases such as MongoDB, HBase, Cassandra, Bigtable, DynamoDB, and Cosmos DB.

  • Experience working with seach engines tools like Elastic Search

  • Experience in end to end data management solution

  • Experience in data migration tools such as Fivetran, Informatica, database migration tools

  • Ability to design data lake, data warehouse, data mart based on AWS, Azure, GCP and on-premise.

  • Understanding of data lake management such as life cycle management, storage class design, and access control.

  • Ability to optimize data warehouse and data mart such as indexing, clustering, and partitioning.

  • Ability to design data modeling (schema design) such as star schema, snowflake schema, fact table, dimensional table.

  • Experience in ETL/ ELT solutions for both on cloud and on-premise.

  • Ability to develop ETL/ELT solution in Python, Spark, SQL.

  • Understanding of real-time and batch processing.

  • Experience in real-time processing (streaming) tools such as Apache Kafka, RabbitMQ, Cloud Pub/Sub, Azure Event Hubs, Amazon Kinesis

  • Experience in workflow orchestration, monitoring or data pipeline tools such as Apache Airflow, Azure Data Factory, Luigi, NiFi, AWS Step Function.

  • Innovative problem-solving skills with the ability to identify and resolve complex architectural issues

  • Ability to communicate clearly and work closely with cross-functional teams such as Data Analyst, Data Visualization, Software Engineering, and businesses functions

  • Good command of English

  • Excellent organizational and leadership skills

  • An outgoing personality who has proven to be a good team player

  • Agile methodology mindset