Data Engineering

Develop and implement efficient and stable processes to collect, store, extract, transform, load and integrate data at various stages in the data pipeline. This also involves processing varying amounts of data from a variety of sources and preparing data in a structure that is easily access and analysed according to business requirements



Competency Area

Development and Implementation


Utilise appropriate tools, systems and techniques to collect, store, extract, transform and load data

Apply appropriate data collection tools and techniques to collect data from various sources

Merge varying datasets from disparate sources into a common structure

Catalogue data according to set guidelines

Clean the data, checking for outliers or errors

Validate data from different data sets to verify accuracy and minimise errors

Check the structure and quality of warehouse data against standard guidelines and data purpose and usage

Utilise database management system software to perform simple data processing

Create databases to store electronic data

Maintain documentation as per the organisation's methodology for Extract, Transform and Load (ETL) processes

Implement data management processes and systems to transform and process multiple streams of data

Identify relevant data sources, processes and relationships in accordance to business requirements

Propose methods and tools to gather data, process data, and minimise confounding variables and data limitations

Apply data analysis and data profiling to improve the clarify, quality and integrity of valid data

Process multiple streams of data using data systems

Utilise data systems and platform capabilities to solve new data problems

Transform data to meet specific business requirements

Operate data warehouse systems to balance optimisation of data access with loading and resource utilisation factors

Create supporting documentation with metadata and diagrams of entity relationships, business processes and process flow

Map data between source systems, data warehouses and data marts

Standardise data, verify data reliability and validity, store, extract, transform, load and integrate data

Develop efficient processes to standardise and maintain data definitions, sources and quality

Develop data warehouse process models, including sourcing, loading, transformation and extraction

Design data validation methodology to verify reliability and validity of data

Design staging databases to store the data temporarily before moving them into the target system

Design extraction process for consolidating data from multiple data source systems

Verify extracted data with business rules specified in target system

Design the process to transform extracted data into structures that align to the business rules incorporated in the target system

Develop load process to upload transformed and integrated data to live target system

Translate complex functional and technical business requirements into detailed data structures and designs

Develop data integration procedures, managing the alignment of data availability and integration processes

Lead the creation of data management procedures and oversee the integration of data, optimise data pipeline

Maintain an updated view of the business requirements, the respective source data systems and data models in the organisation

Lead the creation, refinement and enforcing of data management procedures and conventions

Direct the design of the organisation's Extract, Transform and Load (ETL) processes to support business needs

Establish alignment among the data ETL processes throughout the pipeline to maximise efficiency for data processing

Pre-empt any gaps between the existing organisational data system features and evolving business needs

Refine the ETL processes based on data changes over time and target system business requirements

Manage the integration of data into a unified interface

Manage the optimisation of the various data processing elements in the organisation's data pipeline