Services

Data Warehousing

Data Vault

Organizations with many data sources often struggle to integrate them into a single high performing, auditable, and well-understood repository. Even when they do, the complexity and cost of legacy data warehousing solutions tend to become too expensive and fail to scale over time as the business demands more and more out of the enterprise data platform.

In today's data landscape, more responsibility is placed on data warehousing teams to deliver more frequently while also having to "defend" or "prove" the information in their warehouse is accurate. Additionally, many analyitcs often require the ability to track changes over time which can be challenging even for seasoned data professionals. And, of course, all this information needs to be presented within performance thresholds across millions or even billions of rows.

So how does data vault help solve these challenges? Dan Linstedt, creator of Data Vault, defines Data Vault 2.0 as a "System of Business Intelligence containing the necessary components needed to accomplish enterprise vision in Data Warehousing and Information Delivery". It delivers on this definition through three foundational pillars: Methodology, Architecture, and Model.

Methodology essentially means adopting a consistent, repeatable, and pattern-based approach to building your data warehouse. Pulling from The Matrix movie trilogy (yes I purposely made that green), what if I told you the pattern-based approach of data vault can allow you to automatically generate 98% of all your ETL code? What if I told you that you could have full auditability of that data even when source systems do not track changes? By combining data vault with tools such as WhereScape, organizations can leverage data vault patterns to achieve the following benefits.

• Automated Models
• Automated ETL Code
• Automated Docs
• Performance Patterns
• Standard Query Pattern
• Fully Historized

Architecture refers to the multi-tiered, scalable, and technology agnostic benefits when adopting data vault. Classic data warehouse methodologies typically apply hard and soft business rules directly into the dimension and/or fact ETL pipeline. This makes it difficult to change and equally as challenging to share those rules with other information marts. Data Vault separates these two rule types.

Hard rules do not change the meaning of the data or the grain of that data from the source system. By only applying hard rules during data acquisition, your Vault can provide an auditable view of source systems at any point in time all integrated by business key.

Soft rules represent transformations or business logic that changes the meaning or grain of your source data. Instead of applying soft rules when generating dimensional models, data vault persists the output of business rules in the Business Vault. The benefit of this approach is the ability to have full auditability of business rules along with the flexibility to share those rules across multiple information marts.

So while the separation of hard rules allows you to recreate the source data at any point in time, the separation of soft business rules allows you to recreate information marts at any point in time while also leveraging the same automated ETL loading patterns. This means 50% of your information mart code can be auto-generated.

• Source Auditability
• Mart Auditability
• Enables Virtialization
• Automated ETL Code

Model refers to the flexible and scalable hub-spoke architecture of data vault. At its core, data vault models are based on three entity types: hubs which holds a distinct list of business keys, links which store distinct list of relationships, and satellites which store descriptive data about the business key or relationship over time.

Using these entities to model your data provides a cross-source system model integrated by business key within a single repository that is organized under an enterprise-wide ontology.

Another major benefit of the model is that it can be built incrementally over time. You don't have to model an entire system before building your information marts. With data vault, you can adopt an agile approach that quickly delivers value to the business while minimizing the impact of change in your warehouse to achieve the following benefits.

• Business Key Integration
• Performance Optimized
• Parallel Loading Pattern
• Historization of Data
• Master Data Synergy
• Supports Big Data

Application Architecture

.NET Development

I'm a certified consultant with over 20 years of experience building web applicaitons, web APIs, and custom software solutions using the latest cross-platform .NET frameworks. Leverage proven object-oritented design patterns to ensure the highlest levels of maintanability and scalability in any solution.

Almost every application in today's world needs data. Not only can I help you build high quality software, I can also model and implement your data access layer using using any of the latest data access platforms. Extensive experience building and tuning solutions built on the Microsoft SQL Server platform including Azure cloud editions such as Azure SQL Server, SQL Managed Instance, and Azure Synapse.

So whether you need REST APIs, ASP.NET websites using MVC/Razor, or middleware class libraries, I can help architect and build the entire solution or simply sit on your team to accelerate delivery time. Below you can find many of the skill sets that I can bring to your team to help accelerate your software project.

• C# Development
• SQL Development
• Entity Framework Core
• PowerShell
• Object Oriented Design
• Web API Architecture
• Website Development
• VB.NET Development

When architecting your solution, you should always consider how it will scale and react to various levels of failure. I can help you understand how to meet your disaster recovery and performance objectives using through services available in Azure cloud. Be sure your solutions fault tolerant, meet scalability requirements, and provide a geographically redundant hosting environment.