About the Role
We are looking for a Data Engineer who enjoys solving complex problems and building scalable data solutions. This person will play a key role in designing, developing, and optimizing data platforms that support analytics, reporting, machine learning, AI initiatives, and other data-driven business needs.
Essential Functions
- Assemble large, complex data sets that meet business requirements through extraction, transformation, and loading of data from a wide variety of data sources.
- Provide operational support and troubleshooting for existing processes and systems.
- Work closely with architects, solution leads, data owners, Data Scientists and key stakeholders to facilitate and coordinate the data platform backlog grooming process, triaging new feature requests.
- Deliver automation and efficient processes to ensure high quality throughput and performance of the entire data and analytics platform.
- Ensure data extraction, transformation and loading meet data security and compliance requirements.
- Engage with data source platform leads to gain tactical and strategic understanding of data sources required by Agency Data Services AI/ML.
- Create data tools for data scientist team members that assist them in building and optimizing models.
Requirements
- BS degree in Computer Science, Data Science, Engineering, or equivalent software/services experience.
- 4+ years working with SQL, Snowflake, Databricks, Spark, and other big data technologies.
- 4+ years using Python, SQL, PySpark, or R for manipulating and extracting value from large, disconnected data sets.
- 4+ years building and optimizing data pipelines and architectures.
- 2+ years supporting large-scale data processing using Azure Data Factory, Integration Runtime, Data Lake, Databricks, Spark, Azure ML, and Cosmos DB.
- 2+ years addressing privacy, compliance, and security aspects of data storage and processing.
- 2+ years with CI/CD methodologies and automation tools including Ansible Tower, Terraform, Azure DevOps, and GitHub Actions.
- 2+ years with Salesforce FSC and Salesforce Data Cloud.
Benefits
- Health, Dental, and Vision insurance
- 401k retirement savings plan with company match
- Paid holidays and flexible time off
- Employer paid disability, life, and AD&D insurance
- Health savings account (with employer contributions) and flexible spending accounts
- Employee assistance program
- Adoption assistance and domestic partner coverage