Data Engineering
Develop and implement efficient and stable processes to collect, store, extract, transform, load and integrate data at various stages in the data pipeline. This also involves processing varying amounts of data from a variety of sources and preparing data in a structure that is easily access and analysed according to business requirements
Type
Domain
Competency Area
Development and Implementation
Levels
Utilise appropriate tools, systems and techniques to collect, store, extract, transform and load data
Apply appropriate data collection tools and techniques to collect data from various sources
Merge varying datasets from disparate sources into a common structure
Catalogue data according to set guidelines
Clean the data, checking for outliers or errors
Validate data from different data sets to verify accuracy and minimise errors
Check the structure and quality of warehouse data against standard guidelines and data purpose and usage
Utilise database management system software to perform simple data processing
Create databases to store electronic data
Maintain documentation as per the organisation's methodology for Extract, Transform and Load (ETL) processes
Implement data management processes and systems to transform and process multiple streams of data
Identify relevant data sources, processes and relationships in accordance to business requirements
Propose methods and tools to gather data, process data, and minimise confounding variables and data limitations
Apply data analysis and data profiling to improve the clarify, quality and integrity of valid data
Process multiple streams of data using data systems
Utilise data systems and platform capabilities to solve new data problems
Transform data to meet specific business requirements
Operate data warehouse systems to balance optimisation of data access with loading and resource utilisation factors
Create supporting documentation with metadata and diagrams of entity relationships, business processes and process flow
Map data between source systems, data warehouses and data marts
Standardise data, verify data reliability and validity, store, extract, transform, load and integrate data
Develop efficient processes to standardise and maintain data definitions, sources and quality
Develop data warehouse process models, including sourcing, loading, transformation and extraction
Design data validation methodology to verify reliability and validity of data
Design staging databases to store the data temporarily before moving them into the target system
Design extraction process for consolidating data from multiple data source systems
Verify extracted data with business rules specified in target system
Design the process to transform extracted data into structures that align to the business rules incorporated in the target system
Develop load process to upload transformed and integrated data to live target system
Translate complex functional and technical business requirements into detailed data structures and designs
Develop data integration procedures, managing the alignment of data availability and integration processes
Lead the creation of data management procedures and oversee the integration of data, optimise data pipeline
Maintain an updated view of the business requirements, the respective source data systems and data models in the organisation
Lead the creation, refinement and enforcing of data management procedures and conventions
Direct the design of the organisation's Extract, Transform and Load (ETL) processes to support business needs
Establish alignment among the data ETL processes throughout the pipeline to maximise efficiency for data processing
Pre-empt any gaps between the existing organisational data system features and evolving business needs
Refine the ETL processes based on data changes over time and target system business requirements
Manage the integration of data into a unified interface
Manage the optimisation of the various data processing elements in the organisation's data pipeline