Senior Data Engineer (Toronto, Remote)

Description

• The EDEV Team is chartered with building robust and resilient pipelines and data solutions that enable business users to derive key business insights
• The ideal candidate will be responsible for designing, developing, implementing, and documenting data solutions for complex ETL (Extract, Transform, Load) data pipelines, and lending technical expertise within the group. The ideal applicant will thrive in a highly collaborative workplace and actively engage in development and problem solving

Responsibilities 

• Automate data transformations against data sourced from a variety of systems including desktop, web, and mobile product usage logs and business systems data 
• Develop and deploy solutions leveraging CI/CD processes to orchestrate automated batch and NRT (Near Real Time) pipelines running PySpark and dbt data transformations using Airflow 
• Leverage programming languages used for data manipulation, including SQL, Python, Spark, PySpark, Spark SQL, Java, Jinja 
• Design, develop, execute, and document software solutions to address complex data collection, processing, transformation, and testing
• Collaborate with peer organizations, dev ops, support organizations on technical issues and provide guidance
• Address escalated production support issues related to critical data pipelines 
• Interpret and translate business requirements into technical solutions 
• Collaborate with other data engineers to help troubleshoot code level problems and performance issues 
• Develop, refine, and educate the data community on coding standards and best practices
• Participate in code reviews and documentation reviews 
• Recommend architectural standards, best practices and quality assurance processes for data related software, systems, and applications 
• Build and maintain data quality and durability tracking mechanisms to provide visibility into and address disruptions in data ingestion, processing, and storage
• Lead and/or participate in the design of data schemas to ensure common protocols and storage mechanisms are used across all data services
• Partner with data and software architects to design data models, APIs or other architectural elements as needed 
• Stay abreast of industry best practices, patterns, and reference architectures, evaluate, and analyze modern technologies, capabilities, open-source software, third party services and competitors in context of our data strategy to ensure we are adapting, extending, or replacing our own core technologies to stay ahead of the industry 

Qualifications

• Experience with data modeling, including the design of schemas optimized for data retrieval 
• In-depth knowledge and experience working with complex data structures 
• Effective communication skills and ability to translate deeply technical designs into business appropriate representations as well as analyze business needs and requirements ensuring implementation of data services directly correlates to the strategy and growth of the business
• Experience architecting and implementing data testing solutions 
• Proven experience of finding and resolving bugs and data anomalies within data pipelines
• Experience working with a big data stack environment including Hadoop, Hive, Spark and Presto
• Proficiency in executing data transformation and analysis using PySpark and dbt 
• Experience working with workflow management tools, like Airflow and Azkaban
• Advanced SQL skills and experience transforming data in relational data structures 
• Experience in cloud ETL, ELT and near-real time data collection, transport, and processing technologies
• Experience working with GitHub, Jira, Perforce, and Jenkins
• Familiarity with data governance frameworks, SDLC (Software Development Life Cycle), and Agile methodology
• Strong technical aptitude and strong interest in learning best of class technologies around data warehousing, data wrangling, data quality, data governance and data ethics
• Problem solving and analysis skills