Building a Common Data Model across a Membership Platform under a single view, with multiple lenses whilst avoiding Vendor Lock-In
*Transcript from cloudThings recent Membership Sector Digital Conference
The Common Data Model that I’ll be discussing today is one of those things that can often get overlooked but can also massively accelerate a Digital Transformation project if considered early enough.
I've been lucky enough in my career to work on some fascinating problems dealing with data, Big Data, Data Lakes and all the associated Data Architecture that comes with it and currently I’m working with an amazing team of Data Analysts and Data Scientists to find innovative solutions for the Membership sector.
If you want to avoid rework and technical debt in your project when it comes to make use of your data, one of the first steps of a project should be to design your data architecture to consider how you will be using that data – and to make sure you have coverage of all your domain entities with a data schema. Getting this right maybe the basis of numerous workshops between different teams of stakeholders.
But what if you could skip that effort; because it has already been done for you? Read on to find out more.
So, to begin the journey lets talk about… ‘Data as the new oil’.
That’s an idiom that’s been coined in recent years; recognising the fact that data holds tremendous value for an organisation when utilised correctly.
It’s worth protecting with secure engineering practices and if set up correctly allows you to dive in for insights as to how your organisation should be making decisions.
Unfortunately, the decision as to how data is stored (rather than used) is often left quite late into a Digital Transformation project, by which time it’s often too late to make any fundamental changes to correctly optimise it for its use in AI or ML (Artificial Intelligence & Machine Learning) projects.
That’s where the Common Data Model comes in.
In a nutshell it’s there to make sure your Data Architecture is as simple, useable and transferable as possible. Trust me… you no one benefits from a complicated Data Architecture!
Talking about Data Architecture I imagine is a little like plumbers talking about plumbing to their wives… if you’re the person in the charge of it at your business, you will feel like no one else cares until it goes wrong!
That’s why I’m talking today about Data Architecture’s unsung hero… the Common Data Model (or the Common Data Service) that quietly operates behind the scenes, behind all of Microsoft’s PowerPlatform products, and why consideration of it it should be a fundamental part of your next Digital Transformation project.
Before I dive into talking about the Common Data Model specifically though, it’s worth setting the scene as to why it’s such a good thing.
Debt is a word with unpleasant connotations. In general, when it comes to creating solutions, as in life, the lower your debt the cheaper it is to deliver.
If you leave your Data Architecture as an afterthought to a project, you’ll effectively be incurring something called Solution Debt from its inception. It’s something you’ll eventually have to pay back, most likely whilst you’re trying to make improvements to your Platform in the future.
In other words, good Data Architecture will prevent conversations with your Financial Director that sound like…
“Why does this Data Analysis project cost so much?
Why can’t I have this data in real time?
Why is our GDPR compliance so complicated?
Why can’t I find the data I’m looking for?
Why are our reports so complicated to run?”
What can you do on your next project to avoid ever being asked those questions?
Plan out your Data Architecture now, preferably using the Common Data Model as the basis.
As I’ve mentioned, far too often Solution Debt isn’t considered when planning a Digital Transformation project.
Consider for a moment…
Why do ‘disruptors’ entering a marketplace for the first time do so well against large established brands?
Surely an established organisation should have the capital reserves to pivot and reinvent themselves in a competitive market?
So why is it they so often struggle to move as quickly as a newer, younger competitor?
I would argue it's because when an organisation is born, it is swift and agile. It has not yet acquired the Solution or Organisational Debt that occurs as a business grows. It may go through a merger or two, make several acquisitions or grow naturally as successful business do.
Eventually though, making a change becomes a lot more difficult because there’ll be data everywhere with duplicate data stored in different places meaning making just one, simple change involves having to access four different places.
That’s when you find can’t even move away from these systems anymore because you’re stuck in contracts that last for years with various vendors and providers… sound familiar to anyone?
It’s a difficult problem called Vendor Lock-In and it’s a problem cloudThing have a lot of experience in solving.
I will refer to a few specifically by name, as there are many different types of debt an organisation accumulates naturally as it grows, but that’s not to say with the correct architectural decisions early on you cannot aim to minimise the impact have on your organisational effectiveness.
Enter then… The Common Data Model!
This is where the Common Data Model shines. When implemented early in a programme, it can really ease the headaches in future change.
The first type of debt the Common Data Model can combat is Data Debt but to understand that we’ll need to go back to first principles and consider a concept called Data Gravity.
It’s a phrase coined originally by Dave McCrory, a wellk-known pioneer of noSQL databases back in 2010. In a nutshell… the more data you have in one place, the better your Data Gravity will be.
The better your Data Gravity is, the easier it’ll be to use with apps or to analyse it and garner insights from. If you have your data spread around, and siloed, you have low data gravity situation. And it will cost you to bring it together before you will be able to fully unlock all the value held within.
If you only take one thing away from this article, I’d hope it was this as it’s such an important concept.
For instance… Have you ever been caught in a scenario where you wanted to integrate two different membership databases but couldn't because the tables holding those members had different columns or even completely different data types, meaning it wasn't as simple as just combining all the records together?
Or maybe you’ve got the data and you want it on your application database but can’t until the data has been sanitised. For instance, there may be missing or broken records.
Those are just a couple of examples of how Data Debt can harm your organisation.
It makes the simple act of using your own data in just a slightly different form from the one its currently in difficult. You have to pay off that debt before being able to do the things you really want to with your data.
So how does the Common Data Model help with that?
First of all, it’s true the Common Data Model is a Microsoft market design but I should point out that it’s been designed to a non-vendor specific standard that they created for all their business applications.
There are elements which Microsoft consider ‘core’ such as Account and Safety, Contact, Currency, Email, Letter, Notes etc… Things that every organisation has.
Then they've got segments with things like Sales, Service, Finance, Supply Chain, Commerce, Marketing, Emails and Marketing Pages.
So contact on there would be the same as your core, which is how you can share data across both of those domains. There are tables of entities that are specifically created for that standard that include many Solutions for common sector problems. e.g. Finance, SupplyChain, Marketing, Healthcare sector.
The main takeaway I want to get across though is that any Development Team can implement this and if they follow the Common Data Model guidelines then any application that uses this standard will fit together with other apps like Lego.
However… Microsoft being Microsoft they’ve have taken it way beyond that!
One of the big benefits of the Common Data Model is that there is already a ready-to-use implementation that Microsoft has built called the Common Data Service.
That means (circling back to Data Gravity for a second) if you store your data in the CDS (Common Data Service) you literally have to do nothing to be able to use the standard. But let’s go over the other details that get uses geeks excited....
The next benefit of the Common Data Service is that you have a way of getting data from almost anywhere. Within the Power Platform ecosystem there are already over three hundred pre-built connectors and that includes 3rd parties like Gmail, Twitter, LinkedIn, Salesforce, DropBox with all the usual connectors into Microsoft Office 365.
Because you have so much Data Gravity in your CDS, these connectors let you assemble a phenomenal amount of value with relatively little effort. You can literally treat it as the centre of your ‘Octopus’ with its ‘tentacles’ pulling data from all sorts of sources.
Whilst a clear benefit to the Common Data Service is that it’s great for building apps, where you need your data in real-time; it’s also great if you want to do very, very deep Analytics. When you do offline analytics, you might be combining many different data sources to augment your source data, some of these may not be sanitised yet. You won’t want this polluting your application database. But it maybe considered ok for analysis purposes.
Step in the out-the-box feature to plug straight into Microsoft Azure’s Datalake, which is Microsoft’s store for big data and deep analytics using DataBricks. So, you get a vital component of your data architecture for free.
So, you’ve may have all your data in-one-place, but now you’ve realised it’s become a very juicy target for attack. If you rolled your own data architecture, now you to think about designing security features. If you forgot to do this from the beginning, that’s technical debt right there.
Well Microsoft have thought about that by building in a range of Enterprise features.
Security for instance… having all your data in one place makes it a massive target for cyber actors unless it’s very well secured.
Privacy features… Who’s got that data, who’s using it and for what?
All of these things are provided for by the Common Data Service as an out-of-the-Box solution so in the end your team needs to design a lot less than they would if they went down a customisation route. You still need to design the appropriate policies, but at least you know that if created correctly your data is safe.
Another type of debt to consider is Process Debt and again.
This is not so much debt caused by bad solutions architecture, as a natural debt that occurs as the processes in an organisation form; especially as they undergo mergers and end up with multiple systems doing similar jobs.
CDS is also at the heart of solving this through its ability to be the centre of your PowerPlatform universe and how you can use the connectors, PoweApps and PowerAutomate to ‘glue’ your disparate systems together in an orchestrated way. Also, AI Builder, as Microsoft’s low code way of building AI automatic decision-making modules for linking to your process automation.
Within that system we can create apps, analyse data with Power BI, create bots through Power Virtual Agent and have the ability to knit together any number of systems using Power Automate. Power Automate is a great way of stringing together multi-stage processes which can link together as many different systems as you like.
A re-occurring pattern in the way cloudThing use the Common Data Service is as a single source-truth, linking into Microsoft Dynamics 365 but also other non-Microsoft systems as well.
So if you’ve got five or six different payroll systems for example you might use Power Automate to keep those in-sync but ultimately it works much better if the Common Data Service is you're single source of truth.
Imagine an email coming in from a volunteer with an expense request for example.
Power Automate can funnel it into an approval process which triggers a PO to be raised in your finance package, a response can then automatically be sent back to the requestee saying it’s been authorised. You could even imagine PowerAutomate ordering an item from a supplier if these were re-occurring orders.
Those are the types of things that PowerAutomate is good at, whilst still bringing everything back into your Common Data Model for analysis later.
Whilst Power Automate is for gluing together other systems, there are other tools at your disposal.
Power Virtual Agent has been designed to let you run a customer facing bot that is pre-scripted for tracking conversations and their results all in one place
Finally Microsoft’s Power Apps is great for building low code apps with all that data flowing straight back into your CDS.
So you can, for instance, personalise that app for a member, tracking how they engage with it so you can understand what features, activity and information they might be showing interest in and their Power BI can report on what’s occurring in those other apps you’ve built; whether through a Power Virtual Agent, through Power Automate or through Power Apps, its ultimately all stored in one place for you to analyse.
This has been a journey that I hope you have enjoyed; we started off with so fairly tenuous concept for data nerds, but we end up concluding that the strength of Common Data Model is the strength of the Microsoft ecosystem the surrounds this universe.
Your organisation will undoubtedly care more about the fancy shapes you can build, but Common Data Model is the most important piece of Lego: the base that you start from.
Does your Data Architecture need a little bit of TLC? Could your organisation benefit from using the Common Data Model? You know what to do...