Posts

Showing posts from December, 2023

Python Code for Mapping Data Flow Users: Part 1 - Multiple Inputs

Image
  Why Learn Python Microsoft has published a how-to guide for Dataflows Gen2. In it they maintain: Microsoft Fabric’s Data Factory experience provides an intuitive and user-friendly interface using Power Query Online that can help you streamline your data transformation workflows when authoring Dataflow Gen2. If you’re a developer with a background in Azure Data Factory and Mapping Data Flows, you’ll find this guide helpful in mapping your existing Mapping Data Flow transformations to the Dataflow Gen2 Power Query user interface. These dataflow do not readily avail themselves of similar functionality and as things stand in Fabric the real strength will be the use of Notebooks. From all of my investigation and exploration nothing yet has abused me of the view the Code First is the way to go for Fabric. So, I have put together this and subsequent posts to demonstrate how the functionality can be achieved from Notebooks using Python as would be used in Mapping Data flow. Loading Data...

Azure Synapse Link for Dataverse Synapse and Fabric

Image
  This blog will look the capabilities and configuration of the Power Apps Azure Synapse Link for Dataverse. It will ·         Review each of the destinations ·         Explore the resulting data structure and supporting architecture. ·         Consider the options available for specifying the data structure. ·         Look at any issues or considerations which should be addressed during implementation. This is not a ‘How To’ document and will not describe the steps required to implement the solution unless it is pertinent to the explanation of a feature, issue or consideration that is being discussed. For How-to information, I’m including these links. ·         Azure Data Lake Storage Gen2 See details here . ·         Azure Synapse Workspace. See details here...

Delta CDC: How it Compares to SQL CDC

Image
What is Meant by CDC? Change Data Capture ( CDC ) is a mechanism to record the effect of a transaction on a table within a database or lake house. It records the details of insertions, updates, and deletions to data. It is separate from the table data and has its own lifecycle. It is expected to contain data and metadata for these actions. CDC is available for SQL Server and Azure SQL. It is also available for Delta Tables within Lake Houses. There are differences between these two versions of CDC. We will look at the SQL CDC and what, if any comparison feature exists for Delta Tables. Compare CDC Feature SQL Delta Enable for the database This permits the CDC to be enabled on a database table. It does not enable this for any table. It can be set to default for all tables in a database as they are created either during a session or in a Spark Pool’s Configuration. Enable for the table Req...