Designing, building and supporting Data Warehouse (DW) systems is a VLDB core competence.
Since the 1980’s the aim of the Data Warehouse has remained largely the same: to reliably deliver timely, cleansed & integrated data from one or more source systems into a single platform to enable activities such as KPI reporting, Business Intelligence (BI) and analytics.
Data Warehouse Services
VLDB offers the full range of Data Warehouse related services.
This includes: platform evaluation and selection; platform configuration & deployment; Change Data Capture (CDC) strategy; Extract Transform Load (ETL) design/build/test (DBT); ETL job scheduling; ETL support; core schema & semantic/BI layer design; database administration (DBA) and training.
The Data Warehouse sits at the centre of an ecosystem that includes important supporting toolsets. VLDB also offers services covering ETL, SQL client, Business Intelligence (BI), and Data Science toolset selection, deployment & training.
Data Warehouse Platform Evaluation and Selection
There are many options when it comes to Data Warehouse platform selection.
Should you use your existing general purpose DBMS, such as SQL Server or Oracle? What about specialised analytics platforms like Teradata, Greenplum, AWS Redshift or Google BigQuery? What about open source databases such as PostgreSQL or MySQL? Should you deploy on-prem, in the cloud, or maybe go for a hybrid approach?
We firmly believe in ‘show me, don’t tell me’. This is the only way to cut through the vendor hype, reseller bias, analyst reports and tall tales from conferences.
To this end we will use our decades of vendor-neutral, real-world expertise to help you evaluate and select a Data Warehouse platform to economically meet your needs today, and for many years into the future.
Change Data Capture (CDC) & Extract/Transform/Load (ETL)
Once you’ve deployed your Data Warehouse platform, the next challenge is to keep it fed with changed & new (‘delta’) records from your operational systems of record.
Change Data Capture (CDC) describes the process of identifying and unloading delta records from operational systems. This is not always as straight forward as it sounds. Challenges include the operational importance of the system of record, or systems that operate 24x7 with no obvious ‘extract window’.
Extract/Transform/Load (ETL) applications are designed to take the delta records from the CDC (‘Extract’) process, apply the required business logic (‘Transform’) and then insert/update/delete (‘Load’) the target Data Warehouse tables as required.
The design/build/test (DBT) of CDC and ETL applications represents VLDB’s core Data Warehouse development skill-set. We’ve written ETL applications that have been running consistently, not just for years, but decades!
Business Intelligence (BI) & Data Science
Once your business data has been extracted, transformed and loaded to your Data Warehouse, it needs to be put to good use. That ROI won’t happen all on it’s own!
As a minimum, Key Performance Indicator (KPI) reports should be generated from the Data Warehouse. In addition to KPI reports, core value-generating activities to exploit the business data include executive dashboards and self-service Business Intelligence (BI).
Taking things beyond BI, the Data Warehouse should also be able to support Data Science activities in the form of Machine Learning (ML) and Artificial Intelligence (AI).
Analytic maturity is discussed on the VLDB blog:
How do you rate your organisation’s analytic maturity?
Database Administration, Application Support & Training
There are a lot of moving parts to be managed to keep Data Warehouse platforms and analytic applications alive and well. No amount of monitoring or alerting tools can be relied upon to keep everything on track. That’s where good old-fashioned folks with tech skills come into play.
VlDB provides system administration and Database Administration (DBA) services to maximise uptime and performance of the Data Warehouse platform, and application support services to diagnose and fix ETL and BI processes when they break.
In addition to system administration, Database Administration (DBA) and application support services, VLDB also run training courses to help get your own in-house team up to speed.