Data Management Framework (DMF )
in D365 F&O

This document gives a detailed rundown of using the Data Management Framework (DMF) for getting data out of Dynamics 365 Finance & Operations (F&O).

1. What It Is

 

The Data Management Framework (DMF) is F&O's workhorse for handling high-volume data imports and exports. It's a built-in suite of tools designed for managing data in batches. Unlike OData, which handles one record at a time in real-time, DMF is all about moving large sets of data asynchronously, meaning it runs in the background without making the user wait.

You can use it to set up "data projects" to export data from F&O's Data Entities into various formats like Excel, CSV, or XML. More importantly for analytics, it's the engine behind Recurring Integrations and the "Bring Your Own Database" (BYOD) feature. BYOD allows you to set up jobs that automatically push F&O data into your own Azure SQL database on a recurring schedule.

2. Architecture & Data Flow
 

The DMF process is asynchronous and built for reliability:

​

• Job Creation

  • An administrator or user sets up an export "data project" in the F&O Data Management workspace. They select the data entity, the target format (e.g., Azure SQL for BYOD), and the fields to include.

​

• Job Execution

  • The job can be started manually, or more commonly, scheduled to run on a recurring basis (e.g., every 15 minutes). When the job kicks off, it's added to a batch processing queue.

​

• Staging

  • The F&O batch service picks up the job. It pulls the data from the source tables via the data entity and places it into a temporary "staging table" within the F&O database.

​

• Push to Destination

  • The framework then reads the data from the staging table and pushes it to the final destination. For BYOD, this means it connects to your Azure SQL database and inserts/updates the records. For a file export, it creates the file and saves it to a specified location.

​

• Status Update

  • The job status is updated within F&O to show whether it succeeded or failed. The whole process runs in the background, and F&O performance isn't directly impacted.​

​

3. Prerequisites

 

• Data Entities

  • The data you want to export must be available in a Data Entity. If not, a developer needs to create a custom one.

​

• For BYOD:

  • An Azure SQL Database. You can't use a SQL Server on a virtual machine; it must be the PaaS (Platform-as-a-Service) version.

  • A SQL user with the right permissions (db_owner is often required for the initial setup).

  • A Data Source configured within F&O that contains the connection string for your Azure SQL database.

​

• For Recurring Integrations (Files)

  • You'll need to configure the endpoints, which could be blob storage, an FTP site, or other locations.

​

​

4. Pros (The Good Bits)
 

• High-Volume Data

  • It's specifically designed to handle large datasets - think hundreds of thousands or millions of records - without timing out. This is its main advantage over OData.

​

• Asynchronous & Reliable

  • Jobs run in the background, so they don't slow down the user interface. The batch framework is reliable and can be configured to retry failed jobs.

​

• Automation

  • You can schedule export jobs to run automatically on a recurring basis, making it a "set and forget" solution for feeding data to a data warehouse.

​

• Incremental Updates

  • You can configure jobs to only push records that have changed since the last run (change tracking), which is far more efficient than exporting the full dataset every time.

​

• BYOD Simplicity

  • The BYOD feature provides a straightforward way to get your F&O data into a standard SQL database, which is easy to connect to with almost any BI tool.

​

5. Cons (The Not-So-Good Bits)
 

• Not Real-Time

  • This is a batch process. The data in your destination will always have a latency based on your schedule. The fastest you can realistically schedule it is every few minutes, so it's not suitable for live dashboards.

​

• Staging Table Bloat

  • The staging tables used by DMF can grow massive over time if not maintained. The research highlights that forgetting to run the "Staging table cleanup" job is a common mistake that can cripple the performance of your data jobs.

​

• BYOD is One-Way

  • The BYOD feature is strictly for exporting data out of F&O. You can't write data back into F&O through your Azure SQL database.

​

• Management Overhead

  • You need to monitor the batch jobs to ensure they are running successfully. If a job fails, you have to investigate the error messages in F&O, which can sometimes be a bit cryptic.

​

• BYOD is Being Superseded

  • While still supported, Microsoft's strategic direction is clearly towards Azure Synapse Link. BYOD is now considered a slightly older, though still functional, technology.

6. Key Implementation 'Gotchas' (Things to Watch Out For)

​​

• Change Tracking Nuances

  • Enabling incremental push requires that change tracking is enabled on the underlying tables and the data entity. This might require a developer to configure. Also, if you re-publish the entity to your BYOD, the change tracking information can be lost, forcing a full push.

​

• Staging Table Cleanup is a Must

  • Seriously. This is mentioned everywhere. You must schedule the cleanup job, or your DMF performance will eventually grind to a halt.

​

• API for Automation

  • While you can schedule jobs in the UI, fully automating the DMF (e.g., triggering it from an external orchestrator) requires using a specific package API, which adds another layer of complexity.

​

• Entity Design Matters

  • The performance of an export job is heavily dependent on how well the underlying data entity is designed. A poorly designed entity with complex joins can still be slow, even in DMF.

​

​

7. When Should You Use It?

 

DMF is the best choice for these scenarios:

​

• Feeding a Traditional Data Warehouse

  • When you need to populate a downstream data warehouse or BI platform with F&O data on a regular, scheduled basis (e.g., nightly or hourly).

​

• Large Data Exports

  • When you need to export a large volume of data for offline analysis, migration, or archiving.

​

• System Integration (Batch)

  • When integrating F&O with another system that works on a batch basis (e.g., sending a file of all new sales orders to a logistics partner once an hour).

​

• When Synapse Link is Overkill

  • If your analytics needs are simple and you're already comfortable with Azure SQL, BYOD can be a simpler and cheaper alternative to setting up a full Azure Synapse Link environment.

​

It should NOT be used when you need live, real-time data in an app or a dashboard, or for small, interactive updates. That's what OData is for.