Job description

[What the role is]

To design, build, and lead the development of reliable data ingestion and transformation pipelines for the Enterprise Data and AI platform. The Senior Data Engineer will ensure that data from multiple sources is ingested, governed, secured, and made available for analytics, AI, and business use in alignment with NP’s data architecture and governance frameworks.

This is a 2-year contract position with the Digital Services & Technology Office.

[What you will be working on]

• Design, build and lead the development of ingestion pipelines for structured and unstructured data (Excel, MSSQL, Postgres, APIs, SharePoint, PDFs, etc.).

• Architect and optimise transformation workflows across Bronze, Silver, and Gold layers using Delta Lake and Delta Live Tables.

• Define and implement data quality frameworks (schema, dedupe, lineage, metadata enrichment) and work with Governance team to enforce standards.

• Collaborate with the Security Engineer to ensure sensitive data fields are encrypted, tokenized, or masked.

• Optimize Spark/Databricks jobs for performance and cost efficiency.

• Lead sprint planning for data tasks and perform code reviews.

• Drive root cause analysis and implement long-term fixes.

• Develop dashboards using tools such as PowerBI and DataBricks

• Configure Power BI DirectQuery connections to Databricks SQL Warehouse.

• Develop backend APIs & work on backend databases.

[What we are looking for]

• Degree in Computer Science, Information Systems, Engineering or a related Technology based education.

• Good interpersonal and partner/ executive leadership skills.

• 2–4 years of experience in data engineering.

• Prior experience with cloud-based data platforms (Databricks, Snowflake, BigQuery, Synapse, Redshift) is a plus.

• Exposure to large-scale ingestion and transformation pipelines is a plus.

Skills & Certifications

• Certifications (Preferably):

  • Databricks Certified Data Engineer Associate.

  • Cloud certs such as Azure Data Engineer Associate, AWS Data Analytics Specialty, or GCP Data Engineer.

• Technical Skills:

  • PySpark, SQL, and Python programming.

  • Delta Lake, Delta Live Tables, Databricks Workflows is a plus.

  • APIs and JDBC/ODBC connectors.

  • Version control (Git) and CI/CD for data pipelines.

  • Understanding of medallion architecture, data quality frameworks, and orchestration (Airflow, Prefect, or equivalent).

Attributes

• Detail-focused and methodical.

• Problem-solver with a performance-driven mindset.

• Adaptable and collaborative within cross-functional teams.

• Strong communicator able to work across data, governance, and business stakeholders