Data AI/ML Software Engineer

About the Role

Data AI/ML (Artificial Intelligence and Machine Learning) Engineering involves the use of algorithms and statistical models to enable systems to analyze data, learn patterns, and make data-driven predictions or decisions without explicit human programming. AI/ML applications leverage vast amounts of data to identify insights, automate processes, and solve complex problems across a wide range of fields, including healthcare, finance, e-commerce, and more. AI/ML processes transform raw data into actionable intelligence, enabling automation, predictive analytics, and intelligent solutions. Data AI/ML combines advanced statistical modeling, computational power, and data engineering to build intelligent systems that can learn, adapt, and automate decisions.

Responsibilities

• Design, develop, and implement scalable data pipelines to process large datasets from diverse sources, while handling data in large-scale architectures
• Develop and implement automated data validation and testing frameworks to ensure data accuracy and consistency
• Implement best practices for security, automation, and error handling using tools like Apache Kafka and data warehousing or lake technologies
• Research and implement new tools, technologies, and methodologies and integrate these into production systems, ensuring scalability and reliability
• Apply creative problem-solving techniques to design innovative tools and optimized workflows and independently apply data-driven approaches when appropriate
• Independently manage and optimize data storage solutions such as data warehouses, data lakes, and cloud-based systems
• Understand technical tools and frameworks used by the team, including programming languages, libraries, and platforms and actively support debugging or refining code in projects
• Contribute to the design and documentation of data engineering solutions, clearly detailing methodologies, assumptions, and findings for future reference and cross-team collaboration
• Collaborate across teams to develop and implement high-quality, scalable engineering solutions that align with business goals, address user needs, and improve performance

Core Skills

• Data Integration and ETL practices: Combining data from different sources into a single, unified view, and the process of extracting data from sources, transforming it into a suitable format, and loading it into a destination system. (Proficiency Level: Proficient)
• Data Analysis: The process of inspecting, cleansing, transforming, and modeling data to discover useful information, draw conclusions, and support decision-making. (Proficiency Level: Proficient)
• Data Processing Frameworks: Tools and libraries used to process large data sets efficiently, such as Apache Hadoop and Apache Spark. (Proficiency Level: Foundational)
• Programming: Writing code to manipulate, analyze, and visualize data, often using languages like Python, R, and SQL. (Proficiency Level: Proficient)
• Data Modelling: The process of creating a data model for the data to be stored in a database, representing the data structures and their relationships. (Proficiency Level: Proficient)
• Data Quality Management: Ensuring that data is accurate, complete, reliable, and relevant for its intended use. (Proficiency Level: Proficient)

Specialized Skills

• Data Management: The development and execution of architectures, policies, practices, and procedures to manage the data lifecycle needs of an enterprise.
• Big Data Technologies: Tools and techniques used to process and analyze large and complex data sets that traditional data-processing software cannot handle.
• Database Design and Management: The process of designing, implementing, and maintaining a database system to ensure data is stored efficiently and securely.
• Data Visualization: The graphical representation of data to help people understand complex data sets and derive insights.
• Data Security: Protecting data from unauthorized access, use, disclosure, disruption, modification, or destruction.
• Data Storage solutions: Systems and technologies used to store data, such as databases, data warehouses, and cloud storage.
• Data Governance and Compliance: The management of data availability, usability, integrity, and security in an organization, based on internal data standards and policies. Ensuring that an organization adheres to external regulations and internal policies, managing risk, and maintaining ethical standards.
• Technical Documentation: Creating and maintaining documentation that explains the functionality, use, and maintenance of software or systems.
• Data Architecture: The design and structure of data systems, ensuring that data is stored, managed, and utilized efficiently.
• CI/CD & Observability: The use of continuous integration and continuous delivery (CI/CD) pipelines to automate the process of software development, including building, testing, and deploying code.