Hey Dave!

First, love the name CDepot!

Second, I’m curious if you could expand upon how you’d implement Key Point #5 (Maintaining data quality dimensions throughout the data management process.)

Third (to key into prompt #2), I think an easy best practice for data migration and integration is leveraging the unique functionality of each part of your system. For example, your organization may be using reporting and analytics systems that can handle complex dimensional models (or even thrive on them). In that case, you can use replication tools (like CData Sync) to replicate the raw data into staging tables and then use the transformations (again using Sync) to build the dimensional models from the raw data, defining facts and dimensions based on your reporting needs.

I’m looking forward to seeing where this conversation goes!

Hey Jerod!

First, I want to thank you for your thoughtful response and for furthering this conversation. It's fantastic to connect with someone who shares a similar passion for data management. I'll try my best to address some of your thoughts from my perspective, keeping in mind that my background is focused on staging, transforming, unifying, and preparing data in a SQL Server environment using Linked Servers and CData products.

Regarding data quality dimensions, let's stick to these six (if it's good enough for Collibra, it's good enough for me!): Completeness, Accuracy, Consistency, Validity, Uniqueness, and Integrity.

Now, let's add a 7th dimension - Auditability! It's especially important in migrations, where we need a rock-solid methodology to explain discrepancies between source and target data. To achieve auditability, we stage multiple versions of the data. Earlier versions resemble the source, while later versions are "target ready." Maintaining keys allows us to quickly crosswalk data between states. In my experience, if we're spending more than 10% of our time answering data questions, we're likely falling behind on deadlines and deliverables.

Your third point about leveraging the unique functionality of each part of a system is spot on! However, we should be mindful of the challenges that arise when people use presentation-tier solutions like Qlik and Tableau for data collection and transformation. Once the logic exists in the presentation tier, it's tough to produce a finished product that closely resembles what the customer sees in that tier.

In later conversations, we could also dive deeper into topics like profiling, KPIs, and no-code solutions for keeping leadership and stakeholders informed on progress, as well as allowing folks to do their own discovery of data discrepancies.

In summary, maintaining data quality dimensions throughout the data management process, leveraging the unique functionality of each part of a system, and being mindful of the challenges associated with presentation-tier solutions are key factors in successful data migration and integration projects. Those and observability. I look forward to continuing this conversation and diving deeper into these topics, as there's always more to learn and discuss in this ever-evolving field.

Best

Dave Q

This is a great insight! I’m wondering aloud if there’s an existing practice of IT/data admin teams providing line-of-business users with more raw data to perform ad-hoc querying and reporting (essentially performing the collection and transformation that you’re talking about). IT/data teams could then apply those transformations/collections on the base dataset as a derived view.

While I don’t do a ton of visualization work or reporting, I personally don’t often know what kind of report I want to build until I know what the data I’m reporting on looks like. I know a lot of our partners (and database/lake/warehouse providers in general) tout the analytical capabilities of their platforms (with good reason!). Is this practice of letting end users explore data in an ad-hoc fashion and using their discoveries to inform the model of the stored data something you see in your line of work? Are there other practices that you see played out?

Hey Jerod!

Wow! Thanks for steering the conversation in the direction of Data Prep. This is the essence of my work: acquiring, preparing, and providing data. Data Prep is a narrow yet vital discipline in large organizations, which often isn't explicitly recognized. By preparing data for consumption, we enable a wide range of users to access and analyze it in various ways.

By centralizing data and conforming it for broader use at the database tier, we can create consistent, reusable, and trustworthy data that can be consumed by presentation tools like Qlik and Tableau. This approach takes the logic out of the presentation tier, allowing data scientists, end-users, and even institutional data to benefit from accessible and reliable information. Trustworthy data can be used to enrich other datasets and unlock limitless potential.

Highly iterative and responsive, Data Prep can save corporations considerable time and money if properly supported. Data consumers shouldn't have to worry about the reusability or trustworthiness of the data they access – it should be provided for them. Without this, data silos and "data wars" arise as everyone stages and transforms data in different ways, from spreadsheets to presentation tools to the latest database solutions.

This perspective isn't meant to discredit the Qlik and Tableau communities, who have had to adapt due to the lack of easy connectivity to disparate data sources. However, CData and its connectors have now made it possible to efficiently access, compare, and conform datasets into trustworthy products. By leveraging these connectors, we can finally break down data silos, streamline data access, and promote a more collaborative data-driven culture.

None of this is easy, all of this is necessary, and much of it is happening for the first time, or at least in any fashion that could be deemed expedient and reusable thanks to CData and its connectors.
Let’s keep the conversation going.

Dave Q