Showing posts sorted by relevance for query D365. Sort by date Show all posts
Showing posts sorted by relevance for query D365. Sort by date Show all posts

10 May 2019

🧊💫Data Warehousing: Architecture (Part II: Data Warehousing and Microsoft Dynamics 365)

Data Warehousing

With Dynamics 365 (D365) Online Microsoft made an important strategical move on the ERP market, however in what concerns the BI & Data Warehousing (BI/DW) area Microsoft changed the rules of the game by allowing no direct SQL access to the production environment. This primarily means that will become challenging for organizations to use the existing DW infrastructure to access the D365 data, and for Vendors and Service Providers to provide BI/DW solutions integrated within the D365 platform.

D365 includes its own data warehouse (actually data mart) designed for financial reporting however as per now it can’t be extended to support other business areas. The solution favorited by Microsoft for DW seems to be the use of an Azure SQL Database aka BYOD (Bring Your Own Database) to which entity-based data can be exported incrementally (aka incremental push) or fully (aka full push) via the Data Management Framework (DMF) packages.

Because many of the D365 tables (e.g. Inventory Transactions, Products, Customers, Vendors) were overnormalized over the years and other tables were added as part of new functionality, to hide this complexity, Microsoft introduced a new layer of abstraction formed from data entities organized within an entity store. Data entities are view-like encapsulations of the underlying D365 table schema, the data import/export from and D365 being performed extensively over these data entities via the DMF, which extends the Data Import/Export Framework (DIXF).

One can use thus a BYOD as a direct source for other reporting tools as long they support a connection to Azure, otherwise the data can be further loaded into a database into the cloud, which seems to be the best option until now, as long the organization has other data that need to be consolidated for reporting. From here on, one deals with the traditional way of reporting and the available infrastructure can be extended to use an additional data source.

The BYOD solution comes with several restrictions: a package needs to be created for each business unit, no composite data entities can be exported, data entities that don’t have a unique key can’t be exported via an incremental push, data entities can change over times (new versions being available), while during synchronization no active locks should be on the database. In addition, organizations which followed this path report also some bugs that needed to be addressed via the Microsoft support. Even if the about 1700 available data entities facilitate to some degree data consumption, they seem to be more appropriate for data migrations and data integrations than for DW workloads.

In absence of direct SQL connectivity, in theory organizations can still use SSIS or similar integration tools to connect to D365 production databases and consume data entities via the Open Data Protocol (OData), a standard that defines a set of best practices for building and consuming RESTful APIs. Besides some architectural challenges, loading big tables with transactional data is reportedly slow and impracticable for loading a data warehouse. Therefore, the usability of such an architecture becomes limited in time.

Microsoft imposed a hard limitation upon its D365 architecture by making its production database inaccessible. Of course, there’s still time for Microsoft to do some magic and pull new solutions from the technology stack hat. Unfortunately, the constraints imposed to the production environments limit organizations’ choices of building a modern and flexible data warehouse. For the future it would be great if the DMF could be used directly with standard SQL Server databases, avoiding thus the need for the intermediary Azure database, or if a real-time operational solution could be provided out-of-the-box. We’ll see what the future brings...

30 December 2023

💫🗒️ERP Systems: Microsoft Dynamics 365's Invoice Capture (Features) [Notes]

Disclaimer: This is work in progress intended to consolidate information from the various sources and not to provide an overview of all the features. Please refer to the documentation for a complete overview!

In what concerns the releases see [2].
Last updated: 10-Feb-2025

Invoice Capture - Main Components
Invoice Capture - Main Components [3]

AI Model

  • {feature} prebuilt model (aka Invoice processing model) [by design]
    • can handle the most common invoices in various languages
    • owned by Microsoft and cannot be trained by the customers
  • {feature} custom prebuilt models [planned]
    • built on top of the prebuilt model to handle more complex invoice layouts
    • only requires to train the exceptional invoices
    • after a model is published, additional mapping is required to map the model fields to the invoice files
  • {feature} custom model [by design]
    • requires training the model from scratch 
      • costs time and effort
    • supports the Charges and Sales Tax Amount fields [1.5.0.2]
    • support lookup lists on custom fields [1.6.0.x]

Channels

  • flows that collect the invoices into one location 
    • there's always a 1:1 relationship between flows and channels
  • multiple channels can be defined using different triggers based on Power Automated connectors
  • {feature}default channel for Upload files [1.0.1.0]
  • {feature} supports multiple sources via flow templates [by design]
    • Outlook.com
    • Microsoft Outlook 365
    • Microsoft Outlook 365 shared mailbox [1.1.0.10]
      • to achieve similar behavior one had to modify the first steps of the generated flow with the "When a new email arrives in a shared mailbox (V2)" trigger
    • SharePoint
    • OneDrive
    • OneDrive for business [1.0.1.0]
  • {feature} assign legal entity on the channel
    • the LE value is automatically assigned without applying additional derivation logic [1]
  • invoices sent via predefined channels are captured and appear on the 'Received files' page

Configuration groups 

  • allow managing the list of invoice fields and the manual review settings 
  • can be assigned for each LE or Vendors. 
    • all the legal entities in the same configuration group use the same invoice fields and manual review setting
  • default configuration group 
    • created after deployment, can't be changed or deleted

Fields

  • {feature} standard fields 
    • Legal entity 
      • organizations registered with legal authorities in D365 F&O and selected for Invoice capture
      • {feature} allow to enforce role-based security model
      • {feature} synchronized from D365 F&O to CRM 
      • {missing feature} split the invoices between multiple LEs
      • {feature} Sync deleted legal entities 
        • when legal entities are deleted in D365, their status is set to 'Inactive' [1.9.0.3]
            • ⇐ are not going to be derived during Invoice capture processing
          • ⇒ invoices with inactive legal entities cannot be transferred to D365FO
          • ⇐ same rules apply to vendor accounts
    • Vendor master 
      • individuals or organizations that supply goods or services
      • used to automatically derive the Vendor account
      • {feature} synchronized from D365 F&O to CRM
        • synchronization issues solved [1.6.0.x]
      • {feature}Vendor account can be derived from tax number [1.0.1.0]
      • {feature} Synchronize vendors based on filter conditions [1.9.0.3]
        • one can set filter conditions to only synchronize vendors that are suitable for inclusion in Invoice capture
      • {feature} Sync deleted vendor accounts 
        • when vendor accounts are deleted in D365, their status is set to 'Inactive' [1.9.0.3]
    • Invoice header
      • {feature} Currency code can be derived from the currency symbol [1.0.1.0]
    • Invoice lines
    • Item master
      • {feature} Item number can be derived from External item number [1.0.1.0]
      • {feature} default the item description for procurement category item by the value from original document [1.1.0.32/10.0.39]
    • Expense types (aka Procurement categories) 
    • Charges
      • amounts added to the lines or header
      • {feature} header-level charges [1.1.0.10]
      • {missing feature}line-level charges 
    • {feature}Financial dimensions 
      • header level [1.1.0.32/10.0.39]
      • line level [1.3.0.x]
    • Purchase orders
      • {feature} PO formatting based on the number sequence settings
      • {missing feature}PO details
      • {missing feature} multiple POs, multiple receipts 
    • {missing feature} Project information integration
      • {workaround} the Project Id can be added as custom field on Invoice line for Cost invoices and with this the field should be mapped with the corresponding Data entity for transferring the value for D365 F&O
  • {feature} custom fields [1.1.0.32/10.0.39]

File filter

  • applies additional filtering to incoming files at the application level
  • with the installation a default updatable global file filter is provided
  • can be applied at different channel levels
  • {event} an invoice document is received
    • the channel is checked first for a file filter
    • if no file filter is assigned to the channel level, the file filter at the system level is used [1]
  • {configuration} maximum files size; 20 MB
  • {configuration} supported files types: PDF, PNG, JPG, JPEG, TIF, TIFF
  • {configuration} supported file names 
    • filter out files that aren't relevant to invoices [1]
    • rules can be applied to accept/exclude files whose name contains predefined strings [1]
Actions 

  • Import Invoice
    • {feature} Channel for file upload
      • a default channel is provided for directly uploading the invoice files
      • maximum 20 files can be uploaded simultaneously
  • Capture Invoice
    • {feature} Invoice capture processing
      • different derivation rules are applied to ensure that the invoices are complete and correct [1]
    • {missing feature} differentiate between relevant and non-relevant content}
    • {missing feature} merging/splitting files
      • {workaround} export the file(s), merge/split them, and import the result
  • Void files [1.0.1.0]
    • once the files voided, it is allowed to be deleted from Dataverse
      • saves storage costs
  • Classify Invoice
    • {feature} search for LEs
    • {feature} assign LE to Channel
      • AP clerks view only the invoices under the LE which are assigned to them
    • {feature} search for Vendor accounts by name or address 
  • Maintaining Headers
    • {feature} multiple sales taxes [1.1.0.26]
    • {missing feature} rounding-off [only in D365 F&O]
    • {feature} support Credit Notes [1.5.0.2]
  • Maintaining Lines
    • {feature} Add/remove lines
    • {feature} "Remove all" option for deleting all the invoice lines in Side-by-Side Viewer [1.1.0.26]
      • Previously it was possible to delete the lines one by one, which by incorrectly formatted big invoices would lead to considerable effort. Imagine the invoices from Microsoft or other cloud providers that contain 5-10 pages of lines. 
    • {feature}Aggregate multiple lines [requested]
      • Now all lines from an invoice have the same importance. Especially by big invoices, it would be useful to aggregate the amounts from multiple lines under one. 
    • {feature} Show the total amount across the lines [requested]
      • When removing/adding lines, it would be useful to compare the total amount across the lines with the one from the header. 
    • Check the UoM consistency between invoice line and linked purchase order line
      • For the Invoice to be correctly processed, the two values must match. 
    • Support for discounts [requested]
      • as workaround, discounts can be entered as separate lines
  • Transfer invoice

  • Automation
    • {parameter} Auto invoice cleanup
      •  automatically cleans up the transferred invoices and voided invoices older than 180 days every day [1]
    • Use continuous learning 
      • select this option to turn on the continuous learning feature
      • learns from the corrections made by the AP clerk on a previous instance of the same invoice [1]
        • records the mapping relationship between the invoice context and the derived entities [1]
        • the entities are automatically derived for the next time a similar invoice is captured [1]
      • {missing feature} standard way to copy the continuous learning from UAT to PROD
      • {feature} migration tool for continuous learning knowledge
        • allows us to transfer the learning knowledge from one environment to another [1.6.0.x]
      • {feature} Continuous learning for decimal format [1.9.0.3]
        • allows the system to learn from the history record and automatically apply the correct decimal format on the amount fields
          • {requirement} manually correct the first incoming invoice and do the successful transfer
      • {feature} date format handling
        • fixes the issue with date formatting, which is caused by ambiguity in date recognition [1.8.3.0]
          • when a user corrects the date on the first invoice, the corresponding date format will be automatically applied to the future invoice if it is coming from the same vendor
          •  enabled only when the "Using continuous learning" parameter is active
    • Confidence score check
      • for the prebuilt model the confidence score is always the same 
        • its value is returned by the AI Builder service
        • confidence score can only be improved only when the customer prebuilt model is used 
        •  it can be increased by uploading more samples and do the tagging accordingly
        • a low confidence score is caused by the fact that not enough samples with the same pattern have been trained
      • {parameter} control the confidence score check [1.1.0.32/10.0.39]
  • Manage file filters
User Interface
  • Side-by-side view [by design]
    • users can adjust column widths [1.8.3.0]
      • improves the review experience
  • History logs [1.0.1.0]
    • supported in Received files and Captured invoices
    • help AP clerks know the actions and results in each step during invoice processing 
  • Navigation to D365 F&O [1.0.1.0]
    • once the invoice is successfully transferred to D365 F&O, a quick link is provided for the AP clerk to open the Pending vendor invoice list in F&O
  • Reporting
    • {missing feature} consolidated overview across multiple environments (e.g. for licensing needs evaluation)
    • {missing features} metrics by Legal entity, Processing status, Vendor and/or Invoice type
      • {workaround} a paginated report can be built based on Dataverse

Data Validation

  • [Invoice Capture] derivation rules 
    • applied to ensure that the invoices are complete and correct [1]
    • {missing feature} derive vendor using the associated email address
    • {parameter} Format purchase order 
      • used to check the number sequence settings in D365 F&O to format the PO number [1]
    • {parameter} Derive currency code for cost invoice
      • used to derive the value from invoice master data in D365 F&O [1]
      • ⇐ the currency code on PO Invoices must be identical to the one on PO
    • {parameter} Validate total sales tax amount 
      • validates the consistency between the sales tax amount on the Sales tax card and the total sales tax amount, when there's a sales tax line [1]
    • {parameter} Validate total amount 
      • confirm alignment between the calculated total invoice amount and the captured total amount [1]
  • [Invoice capture] automation
    • {feature} Automatically remove invalid field value [1.9.0.3]
      • when enabled, the system automatically removes the value if it doesn't exist in the lookup list
        • eliminates the need to manually removing values during the review, streamlining the process
        • {scenario} invalid payment terms errors
  • [D365 F&O] before workflow submissions [requested]
    • it makes sense to have out-of-box rules 
    • currently this can be done by implementing extensions
  • [D365 F&O] during workflow execution [by design]
Dynamics 365 for Finance [aka D365 F&O]
  • Attachments 
    • {parameter} control document type for persisting the invoice attachment in D365 F&O [1.1.0.32/10.0.39]
  • Invoice Capture
    • {parameter} select the entities in scope
    • {parameter}differentiate by Invoice type whether Vendor invoice or Invoice journal is used to book the invoices
    • {parameter} Transfer attachment
  • Fixed Assets
    • {missing feature} create Fixed asset automatically during the time the invoice is imported
  • Vendor Invoice Journal 
    • {missing feature} configure what journal the invoice is sent to
      • the system seems to pick the first journal available
      • {parameter} control journal name for creating 'Invoice journal'  [1.1.0.32/10.0.39]
    • {missing feature}grouping multiple invoices together in a journal (e.g., vendor group, payment terms, payment of method)
  •  Approval Workflow
    • {missing feature} involve Responsible person for the Vendor [requested]
    • {missing feature} involve Buyer for Cost invoices [requested]
    • {missing feature} differentiator between the various types of invoices [requested]
    • {missing feature} amount-based approval [requested]
  • Billing schedule
    • {feature} integration with Billing schedules
    • {feature} modify or cancel Billing schedules
  • Reporting
    • {missing feature} Vendor invoices hanging in the approval workflow (incl. responsible person for current action, respectively error message)
    • {missing feature} Report for GL reconciliation between Vendor invoice and GL via Billing schedules
    • {missing feature} Overview of the financial dimensions used (to identify whether further setup is needed)
Licensing
  • licensing name: Dynamics 365 E-Invoicing Documents - Electronic Invoicing Add-on for Dynamics 365
  • customers are entitled to 100 invoice capture transactions per tenant per month
    • if customers need more transactions, they must purchase extra Electronic Invoicing SKUs for 1,000 transactions per tenant per month [5] 
      • the transaction capacity is available on a monthly, use-it-or-lose-it basis [5]
      • customers must purchase for peak capacity [5]
  • only the captured invoices are going to be considered as valid transactions [4]
    • files filtered by the file filter setting won't be counted [4]
Architecture
  •  invoice capture is supported only when the integrated Dataverse environment exists [4]
Security
  • [role] Invoice capture operator
    • must be included in the role settings to 
      • successfully run the derivation and validation logic in Invoice capture [5]
      • transfer the invoice to D365F&O [5]
      • the role must be added to the corresponding flow user on the app side [5]
    •  is only applied on the D365 side [4]
      • ⇒ it doesn't block user to login to Invoice capture [4]
      • ⇐ the corresponding access to the virtual entities will be missed [4]
        • then the user will receive an error [4]
  • [role] Environment maker 
    • must be assigned to the Accounts payable administrator if they create channels in Invoice capture [5]

Previous post <<||>> Next post

Resources:
[1] Microsoft Learn (2023) Invoice capture overview (link)
[2] Yammer (2023) Invoice Capture for Dynamics 365 Finance (link)
[3] Microsoft (2023) Invoice Capture for Dynamics 365 Finance - Implementation Guide
[4] Yammer (2025) Quota/licensing follow up [link]
[5] Microsoft Learn (2024) Dynamics 365 Finance: Invoice capture solution [link]

Release history (requires authentication)

Acronyms:
AI - Artificial Intelligence 
AP - Accounts Payable
F&O - Finance & Operations
LE - Legal Entity
PO - Purchase Order
SKU - Stock Keeping Units 
UoM- Unit of Measure

25 December 2023

💫⚙️ERP Systems: Microsoft Dynamics 365's Invoice Capture (Setup Areas)

ERP Systems

When implementing the Invoice Capture process, there are several setup areas that need to be considered into the Power App (Channels & Mapping Rules), some that need to be done in Dynamics 365 for Finance and Operations (D365 F&O) and impact the integration with the Power App (Expense types/Procurement categories, Financial dimensions, Approval workflow), respectively setup that concerns only D365 F&O but still important. In addition, there are also overlapping features like Recurring Vendor Invoice Templates and E-Invoicing.

This post focuses on the first two types of setup areas, the rest following  in a second post

Channels

In the digitalization era, it's expected that most of the Vendor invoices are received by email or an integrated way (see e-Invoicing). However there can be also several exceptions (e.g. small companies operating mostly offline).

In Invoice Capture can be defined one or more channels for Outlook, SharePoint or OneDrive on which Invoices can be imported in the App. In a common scenario, depending on the number of mailboxes intended for use, one or more channels can be set up for Outlook. Power App listens on the respective channels for incoming emails in the Inbox, in a first step processes the emails by importing all the attachments (including signature images), while in a second step processes all the documents that look like an Invoice (including Receipts) and extracts the metadata to create the Invoice record. The Invoices received in hard-paper format can be scanned and sent to one of the internal mailboxes, respectively imported manually in Invoice Capture. 

More likely, there will be also a backlog that can be imported via SharePoint or OneDrive, which is more convenient than resending the backlog by email. In the end it should make no difference for the process which channels is used as long as the Invoices are processed in a timely manner. It should make also no difference if for example in UAT were used other channels than in Production. For testing purposes it might be advantageous to have more control over which Invoices are processed, while the UAT could follow the same setup as Prod, which is generally recommended.

Mapping Rules

The mapping rules allow on one side to set default values based on a matching string checked against several attributes (e.g. Company name, Address or Tax registration number for Legal entity, Item number for Item, Item description for Expense type). Secondly, they allow to also define a configuration for a rule, which defines what fields are mandatory, respectively which Invoice types are supported, etc.

The mapping rules will not cover all scenarios, though it's enough if they cover a good percentage from the most common cases. Therefore, over time they are also good candidates for further optimization. Moreover, because Invoice Capture remembers the values used before for an instance of the same Invoice, the mapping rules will be considered only for the first occurrence of the respective Invoice or whatever is new in its processing. 

If a channel was defined for each Legal entity, this seems to make obsolete the definition of mapping rules for it. Conversely, if the number of manual uploads is not neglectable, it still makes sense to define a mapping rule. 

Mapping rules for the Expense type seem to work well when Items' descriptions are general enough to include certain words (e.g. licenses, utilities). 

One can define mapping rules also for the Vendor accounts and Items, though it's questionable whether the effort makes sense as long as the internal Vendor names and Product numbers don't deviate from the ones used by the Vendor itself.

Expense Types

Invoice Capture requires that either the Item or the Expense type are provided on the line. For PO-based invoices, an Item should be available. Cost invoices can have Items as well and they can be used on the line, though from the point of view of the setup it might be easier to use Expense types. It's the question whether the information loss has any major impact on the business. There are also cases in which the lines don't bring any benefit and can be thus in Invoice capture deleted. 

At least for Cost invoices, the Expense types (aka Procurement categories in Dynamics 365) defined can considerably facilitate the automatic processing. D365 F&O can use the Procurement category to automatically populate the Main Account in the Invoice distributions. The value is used as default and can be overwritten, if needed. 

Having for example a 1:1 mapping between Procurement categories and Main accounts, respectively the same names can make easier the work of AP Clerks and facilitate the troubleshooting.

Conversely, one can define an additional level of detail (aka an additional segment) for reporting purposes. This implies that multiple categories will point to the same Main account, which can increase the overhead, though the complexity of the structure can be simplified by using maybe a good naming convention and a consolidated Excel list with the values. The overhead resumes mainly when dealing with the first instance of an Vendor invoice.

On Procurement categories can be defined also the default Item sales tax groups (a 1:1 mapping) which can be overwritten as well. For the categories with multiple Items sales tax groups, one should decide whether the benefit of providing a default value outweighs the effort for adding the value for each Invoice line. 

Defining upfront, before the Go Live, a good hierarchical structure for the Procurement categories and the mappings to the Main accounts, respectively to the Item sales tax groups can reduce the effort of maintaining the structure later and reduces the reporting overhead. 

Financial Dimensions & the Vendor Invoice Approval Workflow

Besides their general use, the Financial dimensions can be used to implement an approval process on D365 F&O side by configuring an expenditure reviewer (see [2]) and using it in the Vendor Invoice Approval workflow, respectively of setting up the financial owners for each dimension in scope. Different owners for the Financial dimensions can be defined for each Legal entity via Legal entity overrides. From what it seems, notifications are sent then to the override as well to the default owner. 

Starting with the 1.1.0.32 (07-Nov-2023) version of Invoice capture, respectively the 10.0.39 version of D365 F&O (planned for Apr-2024), 3 financial dimensions (Cost center, Department and Legal Entity) are supported directly in the App. This would allow us to cover the example covered in [2]). This reduces the need for maintaining the values in D365 F&O. 

Unfortunately, if the approval process needs to use further dimensions (e.g. Vendor, Location) or attributes (e.g. Invoice responsible person) in the approval, one needs either to compromise or find workarounds. If there's no Purchase order as in the case of Cost invoices, involving the actual Buyer is almost impossible. For such extreme cases one needs more flexibility in the approval process and hopefully Microsoft will extend the functionality behind it.

The approval processes needed by customers require occasionally a complexity that's not achievable with the functionality available in D365 for F&O. The customers are forced then either to compromise or use external tools (e.g. Power Automate) for building the respective functionality. 

One should consider defining default Financial dimensions on the Vendor, the respective values being used when generating the Invoice in D365 F&O. Defining Financial dimensions templates can help as well when the costs need to be split across different Financial dimensions based on percentages. 

In case the financial dimensions differ across a same Vendor's invoices, one can request from the Vendors to provide the respective information on the Invoice. 

Organization Administration

Cost invoices can work without providing a Unit of Measure (UoM) in Invoice capture, however one should consider using an UoM at least for aesthetic purposes in reporting. On the other side, this can complicate the setup if the same UoM is used for other purposes. 

Vendor Invoice Automation

To be able to process automatically the Vendor invoices once they arrived in D365 F&O, it would useful to maintain the attributes which are modified manually in D365 F&O directly in Invoice Capture. The usual fields are the following:

  • Invoice description (header)
  • Financial dimensions 
  • Sales item tax group

Master Data Management

There are master data attributes which even if they are not directly involved in the process, they could make process actor's life a bit easier. It can be the case of the Responsible person and/or the Buyer group, which would allow the AP Clerks to identify the Cost Center and Department related to the Invoice. Maintaining the external Item descriptions for Vendors can help as well in certain scenarios. 

Previous post <<||>> Next post

Resources:
[1] Microsoft Learn (2023) Invoice capture overview (link)
[2] Microsoft Learn (2023) Configure expenditure reviewers (link)

30 October 2022

💎💫SQL Reloaded: Querying Azure Synapse Metadata for the D365 CRM & FO Tables Exported via Azure Synapse Link for Dataverse

Enabling Azure Synapse Link for Dataverse for Dynamics 365 CRM (D365 CRM) or for Dynamics 365 Finance & Operations (D365 FO) allows to have all the needed tables for reporting in a CDM structure, however the csv files with data don’t have any headers, which makes it challenging to use directly the files without the corresponding metadata. For example, attempting to define external tables would be useless without proper headers and data types. Fortunately, attribute’s name and data types are available in JSON files and can be queried. 

D365 CRM 

For CRM the files start with tables’ names and have the EntityMetadata.json postfix, the files being stored in the Microsoft.Athena.TrickleFeedService folder which lies with the other folders containing the data. Given that all files are in the same folder, the following query can be used to export the metadata for all or some of the tables. Just replace "(container name)" and "(data source)" with the corresponding values for your environment.
 
-- export definition for D365 CRM' CDM (all attributes, all tables)
SELECT DAT.EntityName
, DAT.AttributeName
, DAT.Timestamp
, DAT.AttributeType
, DAT.MetadataId
, DAT.Precision
, DAT.MaxLength
FROM openrowset(
        bulk '/(container name)/Microsoft.Athena.TrickleFeedService/*-EntityMetadata.json',
        data_source = '(data source)',
        format = 'csv',
        fieldterminator ='0x0b',
        fieldquote = '0x0b',
        rowterminator = '0x0b' --> You need to override rowterminator to read classic JSON
    ) with (doc nvarchar(max)) as rows
       CROSS APPLY OPENJSON(doc, '$.AttributeMetadata')--definitions[0].hasAttributes
       with (
         EntityName nvarchar(255) '$.EntityName'
        , AttributeName nvarchar(255) '$.AttributeName'
        , Timestamp nvarchar(50) '$.Timestamp'
        , AttributeType nvarchar(50) '$.AttributeType'
        , MetadataId nvarchar(100) '$.MetadataId'
        , Precision int '$.Precision'
        , MaxLength int '$.MaxLength'
     ) as DAT
WHERE DAT.EntityName IN ('lead', 'opportunity', 'product')

D365 FO
 

For Finance & Operations folders’ structure is more complex, a whole hierarchy of folders being built based on a set of predefined categories. One would need to traverse the hierarchy structure to find the files. Thus, it might be easier to generate the metadata for each table. Files’ names start with tables’ names and have the .cdm.json postfix. The below query generates the metadata for the InventTable table. Just replace the "(container name)", "(instance name)" and "(data source)" with the values for your environment.
 
-- export definition for D365 FO's CDM (all attributes, specific table)
SELECT DAT.name
, DAT.dataFormat
FROM openrowset(
        bulk '/(container name)/(instance name)/Tables/SupplyChain/ProductInformationManagement/Main/InventTable.cdm.json',
        data_source = '(data source)',
        format = 'csv',
        fieldterminator ='0x0b',
        fieldquote = '0x0b',
        rowterminator = '0x0b' --> You need to override rowterminator to read classic JSON
    ) with (doc nvarchar(max)) as rows
       CROSS APPLY OPENJSON(doc, '$.definitions[0].hasAttributes')
       with (
          name nvarchar(255) '$.name'
        , dataFormat nvarchar(50) '$.dataFormat'
     ) as DAT

Further Steps

One can obtain thus the needed metadata, however after a first inspection, the data types are too general compared with the ones considered for the data source attributes. One can work probably with these data types as well, though there's a Microsoft recommendation to minimize the row length by using the smallest possible column size, which leads to better query performance. Exporting the metadata from the source system and matching the two datasets based on tables and attributes’ names would allow addressing this recommendation, even if this implies checking from time to time whether their definition changed. The trick is to keep the same sorting order like in the Synapse files. Unfortunately, not all SQL Server data types are supported (e.g. text, ntext, sql_variant, etc.) or are ideal to work with (e.g. money), however there are alternative data types that can be used. 

Happy coding!

Previous Post <<||>> Next Post

31 January 2020

💫🧮☯ERP: Microsoft Dynamics 365 (The Good, the Bad and the Ugly)

ERP Implementation

The Good: The shift made by Microsoft by porting their Dynamics AX ERP solution to a web-based application (aka D365) hosted in the Microsoft cloud, offered them a boost on the ERP market. The integration with the Office and BI stack, as well Microsoft’s One Version strategy of pushing new features almost on a monthly basis, and of having customers at a maximum 2 releases from the current version, makes from D365 a solution to consider for small to big organizations that span over business sectors and geographies.

The Bad: Currently the monthly release cycle seems to be a challenge for the customers and service providers altogether. Even if the changes in existing functionality are minor, while the functionality is thoroughly tested before releases, the customers still need to test the releases in several systems, especially to assure that the customizations and integrations still work. This can prove to be quite a challenge in which automatic or semiautomatic tools can help when adequately used. Even then, a considerable effort needs to be addressed by the parties involved.
The burden is bigger for the service providers that build their own solutions for D365 as they need to assure in advance that after each release the applications still work. From customers’ perspective, the more such applications they use, the higher the risks of delays in adopting a release or, in extremis, to look for similar solutions. In theory, with good planning and by following best practices the risks are small, though that’s just the theory speaking.
If in the past 2-3 instances were enough to support the ERP during and post implementation, currently the requirements for the cloud-based solution more than doubled, an organization arriving to rent 5-7 D365 instances for the same. Moreover, even if the split between the main blocks (Finance, Supply Chain, Retail and Talent), plus the various Customer Engagement packages, provides some flexibility when thy are combined, this leads to a considerable price increase. Further costs are related to the gaps existing in the available functionality. More likely Microsoft will attempt closing some of the gaps, however until then the customers are forced to opt for existing solutions or have the functionality built. Microsoft pretends that their cloud-based ERP solution provides lower ownership costs, however, looking at the prices, it’s questionable on whether D365 is affordable for small and average organizations. To put it bluntly – think how many socks (aka products) one needs to sell just to cover the implementation, the licensing and infrastructure costs!
One important decision taken by Microsoft was to not allow the direct access to the D365 production database, decision that limits an organization’s choices and flexibility in addressing reporting requirements. Of course, the existing BI infrastructure can still be leveraged with a few workarounds, though the flexibility is lost, while further challenges are involved.
The Ugly: ERP implementations based on D365 make no exceptions from the general trend – given their complexity, they are predisposed to fail achieving the set objectives, and this despite Microsoft’s attempts of creating methodologies, tools and strong communities to support the service providers and customers in such projects. The reasons for failure reside with the customers and service providers altogether, the chains of implications forming a complex network of causalities with multiple levels of reinforcement. When the negative reinforcements break the balance, it can result a tipping point where the things start to go wrong – escalations, finger-pointing, teams’ restructuring, litigations, etc. In extremis, even if the project reaches the finish, the costs can easily reach an overrun of 50-150% from the initial estimation, and that’s a lot to bear.

25 April 2025

💫🗒️ERP Systems: Microsoft Dynamics 365's Business Process Catalog (BPC) [Notes]

Disclaimer: This is work in progress intended to consolidate information from the various sources and not to provide a complete overview of all the features. Please refer to the documentation for a complete overview!

Last updated: 25-Apr-2025

Business Process Catalog - End-to-End Scenarios

[Dynamics 365] Business Process Catalog (BPC)

  • {def} lists of end-to-end processes that are commonly used to manage or support work within an organization [1]
    • agnostic catalog of business processes contained within the entire D365 solution space [3]
      • {benefit} efficiency and time savings [3]
      • {benefit} best practices [3]
      • {benefit} reduced risk [3]
      • {benefit} technology alignment [3]
      • {benefit} scalability [3]
      • {benefit} cross-industry applicability [3]
    • stored in an Excel workbook
      • used to organize and prioritize the work on the business process documentation [1]
      • {recommendation} check the latest versions (see [R1])
    • assigns unique IDs to 
      • {concept} end-to-end scenario
        • describe in business terms 
          • not in terms of software technology
        • includes the high-level products and features that map to the process [3]
        • covers two or more business process areas
        • {purpose} map products and features to benefits that can be understood in business contexts [3]
      • {concept} business process areas
        • combination of business language and basic D365 terminology [3]
        • groups business processes for easier searching and navigation [1]
        • separated by major job functions or departments in an organization [1]
        • {purpose} map concepts to benefits that can be understood in business context [3]
        • more than 90 business process areas defined [1]
      • {concept} business processes
        • a series of structured activities and tasks that organizations use to achieve specific goals and objectives [3]
          • efficiency and productivity
          • consistency and quality
          • cost reduction
          • risk management
          • scalability
          • data-driven decision-making
        • a set of tasks in a sequence that is completed to achieve a specific objective [5]
          • define when each step is done in the implementation [5] [3]
          • define how many are needed [5] [3]
        • covers a wide range of structured, often sequenced, activities or tasks to achieve a predetermined organizational goal
        • can refer to the cumulative effects of all steps progressing toward a business goal
        • describes a function or process that D365 supports
          • more than 700 business processes identified
          • {goal} provide a single entry point with links to relevant product-specific content [1]
        • {concept} business process guide
          • provides documentation on the structure and patterns of the process along with guidance on how to use them in a process-oriented implementation [3]
          • based on a catalog of business process supported by D365 [3]
        • {concept} process steps 
          • represented sequentially, top to bottom
            • can include hyperlinks to the product documentation [5] 
            • {recommendation} avoid back and forth in the steps as much as possible [5]
          • can be
            • forms used in D365 [5]
            • steps completed in LCS, PPAC, Azure or other Microsoft products [5]
            • steps that are done outside the system (incl. third-party system) [5]
            • steps that are done manually [5]
          • are not 
            • product documentation [5]
            • a list of each click to perform a task [5]
        • {concept} process states
          • include
            • project phase 
              • e.g. strategize, initialize, develop, prepare, operate
            • configuration 
              • e.g. base, foundation, optional
            • process type
              • e.g. configuration, operational
      • {concept} patterns
        • repeatable configurations that support a specific business process [1]
          • specific way of setting up D365 to achieve an objective [1]
          • address specific challenges in implementations and are based on a specific scenario or best practice [6]
          • the solution is embedded into the application [6]
          • includes high-level process steps [6]
        • include the most common use cases, scenarios, and industries [1]
        • {goal} provide a baseline for implementations
          • more than 2000 patterns, and we expect that number to grow significantly over time [1]
        • {activity} naming a new pattern
          • starts with a verb
          • describes a process
          • includes product names
          • indicate the industry
          • indicate AppSource products
      • {concept} reference architecture 
        • acts as a core architecture with a common solution that applies to many scenarios [6]
        • typically used for integrations to external solutions [6]
        • must include an architecture diagram [6]
    • {concept} process governance
      • {benefit} improved quality
      • {benefit} enhanced decision making
      • {benefit} agility adaptability
      • {benefit{ Sbd alignment
      • {goal} enhance efficiency 
      • {goal} ensure compliance 
      • {goal} facilitate accountability 
      • {concept} policy
      • {concept} procedure
      • {concept} control
    • {concept} scope definition
      • {recommendation} avoid replicating current processes without considering future needs [4]
        • {risk} replicating processes in the new system without re-evaluating and optimizing [4] 
        • {impact} missed opportunities for process improvement [4]
      • {recommendation} align processes with overarching business goals rather than the limitations of the current system [4]
    • {concept} guidance hub
      • a central landing spot for D365 guidance and tools
      • contains cross-application documentations
  • {purpose} provide considerations and best practices for implementation [6]
  • {purpose} provide technical information for implementation [6]
  • {purpose} provide link to product documentation to achieve the tasks in scope [6]
Previous Post <<||>> Next Post 

References:
[1] Microsoft Learn (2024) Dynamics 365: Overview of end-to-end scenarios and business processes in Dynamics 365 [link]
[2] Microsoft Dynamics 365 Community (2023) Business Process Guides - Business Process Guides [link]
[3] Microsoft Dynamics 365 Community (2024) Business Process Catalog and Guidance - Part 2 Introduction to Business Processes [link]
[4] Microsoft Dynamics 365 Community (2024) Business Process Catalog and Guidance - Part 3: Using the Business Process Catalog to Manage Project Scope and Estimation [link]
[5] Microsoft Dynamics 365 Community (2024) Business Process Catalog and Guidance - Part 4: Authoring Business Processes [link]
[6] Microsoft Dynamics 365 Community (2024) Business Process Catalog and Guidance - Part 5:  Authoring Business Processes Patterns and Use Cases [link]
[7] Microsoft Dynamics 365 Community (2024) Business Process Catalog and Guidance  - Part 6: Conducting Process-Centric Discovery [link]
[8] Microsoft Dynamics 365 Community (2024) Business Process Catalog and Guidance  - Part 7: Introduction to Process Governance [link]

Resources:
[R1] GitHub (2024) Business Process Catalog [link]
[R2] Microsoft Learn (2024) Dynamics 365 guidance documentation and other resources [link]
[R3] Dynamics 365 Blog (2025) Process, meet product: The business process catalog for Dynamics 365 [link]

Acronyms:
3T - Tools, Techniques, Tips
ADO - 
BPC - Business Process Catalog
D365 - Dynamics 365
LCS - Lifecycle Services
PPAC - Power Platform admin center
RFI - Request for Information
RFP - Request for Proposal

13 January 2021

💎💫SQL Reloaded: Useful Queries for Dynamics 365 F&O and not only (Part I: Dependencies)

Microsoft Dynamics 365 for Finance and Operations (D365 FO or simply D365) has currently more than 12000 tables, therefore attempting to understand the data model behind might seem at first sight to be a foolhardy effort. Fortunately, only a small percentage from all the tables contain meaningful master or transactional data, fact that makes easier the task. 

Moreover, the tables have meaningful prefixes (e.g. 'Invent' for Inventory, 'Purch' for Purchae Orders, 'Sales' for Sales Orders, etc.) and postfixes (e.g. 'Table for header data, 'Line' for position data, 'Trans' for transaction data) while their names are in many cases self-explanatory if one knows the corresponding business terms.

For those coming from earlier versions of D365 (e.g. Microsoft Dynamics AX 2009), with a few exceptions in the inventory transactions, product master data, addresses and finance dimensions, the data structures remained almost the same, if one considers the main functional area of the ERP system. However as new modules and/or functionality were added, the number of new tables increased considerably between the versions. 

On the other side in D365 the data are exposed through data entities, abstractions from the physical implementation of the database tables [1], implemented as views typically having the 'Entity' postfix. In many scenarios the data entities more likely will prove to be the best way of accessing the data, as they include an important part of the logic one will need anyway to reengineer, however there are maybe also exceptions in which is needed to work with the base tables. In the end it will be needed to know the main data entities as well the important tables related to them, being needed to check which views reference a given table, respectively which tables are references by a view. This can be easily achieved via the sys.sql_expression_dependencies system catalog view. 

The following queries can be run in non-production D365 environments:

-- retrieving the views where an object was used 
SELECT SCHEMA_NAME(obj.SCHEMA_ID) + '.'+ obj.name AS referencing_object
, obj.type_desc referencing_object_type
, sed.referenced_schema_name + '.'+ sed.referenced_entity_name AS referenced_object
, obr.type_desc referenced_object_type
FROM sys.all_objects obj
     JOIN sys.sql_expression_dependencies sed 
       ON obj.OBJECT_ID = sed.REFERENCING_ID 
	      JOIN sys.all_objects obr
		    ON sed.referenced_id = obr.object_id
WHERE sed.referenced_entity_name = 'PurchTable'
  AND obj.Type = 'V'
  AND obj.name LIKE 'Purch%'
ORDER BY referencing_object

Output (12 records):
ReferencingObject type_desc ReferencedObject ReferencedObjectType
dbo.PURCHASEORDERCUBE VIEW dbo.PURCHTABLE USER_TABLE
dbo.PURCHASEORDERRESPONSEHEADERSUMMARY VIEW dbo.PURCHTABLE USER_TABLE
dbo.PURCHLINEBACKORDER VIEW dbo.PURCHTABLE USER_TABLE
dbo.PURCHLINEBACKORDERWORKSPACE VIEW dbo.PURCHTABLE USER_TABLE
dbo.PURCHLINEEXPANDED VIEW dbo.PURCHTABLE USER_TABLE
dbo.PURCHORDERAPPROVEDVIEW VIEW dbo.PURCHTABLE USER_TABLE
dbo.PURCHPREPAYOPENBASE VIEW dbo.PURCHTABLE USER_TABLE
dbo.PURCHPURCHASEORDERHEADERCHARGEENTITY VIEW dbo.PURCHTABLE USER_TABLE
dbo.PURCHPURCHASEORDERHEADERENTITY VIEW dbo.PURCHTABLE USER_TABLE
dbo.PURCHPURCHASEORDERHEADERV2ENTITY VIEW dbo.PURCHTABLE USER_TABLE
dbo.PURCHPURCHASEORDERLINECDSENTITY VIEW dbo.PURCHTABLE USER_TABLE
dbo.PURCHTABLENOTARCHIVEDVERSIONS VIEW dbo.PURCHTABLE USER_TABLE

The inverse search: 

-- retrieving the objects referenced in a view
SELECT SCHEMA_NAME(obj.SCHEMA_ID) + '.'+ obj.name AS referencing_object
, obj.type_desc referncing_object_type
, sed.referenced_schema_name + '.'+ sed.referenced_entity_name AS referenced_object
, obr.type_desc referenced_object_type
FROM sys.all_objects obj
     JOIN sys.sql_expression_dependencies sed 
       ON obj.OBJECT_ID = sed.referencing_id 
	      JOIN sys.all_objects obr
		    ON sed.referenced_id = obr.object_id
WHERE obj.name = 'PURCHPURCHASEORDERHEADERV2ENTITY'
ORDER BY referenced_object

Output:
ReferencingObject ReferncingObjectType ReferencedObject ReferencedObjectType
dbo.PURCHPURCHASEORDERHEADERV2ENTITY VIEW dbo.ACCOUNTINGDISTRIBUTIONTEMPLATE USER_TABLE
dbo.PURCHPURCHASEORDERHEADERV2ENTITY VIEW dbo.CONFIRMINGPO USER_TABLE
dbo.PURCHPURCHASEORDERHEADERV2ENTITY VIEW dbo.DIMENSIONSETENTITY VIEW
dbo.PURCHPURCHASEORDERHEADERV2ENTITY VIEW dbo.GetValidFromInContextInfo SQL_SCALAR_FUNCTION
dbo.PURCHPURCHASEORDERHEADERV2ENTITY VIEW dbo.GetValidToInContextInfo SQL_SCALAR_FUNCTION
dbo.PURCHPURCHASEORDERHEADERV2ENTITY VIEW dbo.HCMWORKER USER_TABLE
dbo.PURCHPURCHASEORDERHEADERV2ENTITY VIEW dbo.LOGISTICSPOSTALADDRESS USER_TABLE
dbo.PURCHPURCHASEORDERHEADERV2ENTITY VIEW dbo.LOGISTICSPOSTALADDRESSBASEENTITY VIEW
dbo.PURCHPURCHASEORDERHEADERV2ENTITY VIEW dbo.PURCHTABLE USER_TABLE
dbo.PURCHPURCHASEORDERHEADERV2ENTITY VIEW dbo.PURCHTABLE_W USER_TABLE
dbo.PURCHPURCHASEORDERHEADERV2ENTITY VIEW dbo.REASONTABLEREF USER_TABLE
dbo.PURCHPURCHASEORDERHEADERV2ENTITY VIEW dbo.TMSPURCHTABLE USER_TABLE
dbo.PURCHPURCHASEORDERHEADERV2ENTITY VIEW dbo.TRANSPORTATIONDOCUMENT USER_TABLE
dbo.PURCHPURCHASEORDERHEADERV2ENTITY VIEW dbo.VENDINVOICEDECLARATION_IS USER_TABLE

Notes:
1. There will be cases in which is needed to check views' definition. To understand the logic it will be needed to format the queries in a more readable form. This can take time, given that the joins and the further constraints are not always straightforward. Therefore, it might be a good idea to export a formatted copy of the view for later use.
2.  Unfortunately, the data model for D365 is not publicly available. One can use however the data model for AX 2012 (see [2] [3]) as basis and fill the gaps when needed. 
3. The important changes between the various data models versions are partially documented in several documents available via the Microsoft Dynamics CustomerSource. 
4. For non-production environments the data from views, tables or other database objects can be exported directly from the database. Unfortunately, Microsoft blocked the direct access to production environments. The data from tables can be accessed and exported via the table browser, while the data from entities can be exported to Excel typically from the display forms or via the Data Management Framework.
5. The queries can be used to explore any SQL Server database. The logic can be easily encapsulated in a view to simplify the calls.  
6. To minimize the use of redundant code, it's a good idea to encapsulate the logic within a view, when the database allows it.

-- cleaning after
--DROP VIEW admin.vobject_dependencies

-- view definition
CREATE OR ALTER VIEW admin.vobject_dependencies
-- retrieving the dependencies between objects
AS
SELECT SCHEMA_NAME(obj.SCHEMA_ID) + '.'+ obj.name AS referencing_object
, SCHEMA_NAME(obj.SCHEMA_ID) referencing_objecT_schema
, obj.name AS referencing_object_name 
, obj.type_desc referencing_object_type
, sed.referenced_schema_name + '.'+ sed.referenced_entity_name AS referenced_object
, sed.referenced_schema_name AS referenced_object_schema
, sed.referenced_entity_name AS referenced_object_name
, obr.type_desc referenced_object_type
FROM sys.all_objects obj
     JOIN sys.sql_expression_dependencies sed 
       ON obj.OBJECT_ID = sed.REFERENCING_ID 
	  JOIN sys.all_objects obr
            ON sed.referenced_id = obr.object_id

-- call based do referenced object 
SELECT *
FROM  admin.vobject_dependencies
WHERE referenced_object_name = 'PurchTable'
ORDER BY referencing_object_name

-- call based on referencing object 
SELECT *
FROM admin.vobject_dependencies
WHERE referencing_object_name = 'PURCHPURCHASEORDERHEADERV2ENTITY'
ORDER BY referenced_object_name

Previous Post <<||>> Next Post

References:
[1] Microsoft Dynamics 365 (2020) Data entities overview [source]
[2] Microsoft Docs (2014) Application tables [source]
[3] Microsoft Docs (2014) Base Enums AX 2012 [source]

Related Posts Plugin for WordPress, Blogger...

About Me

My photo
Koeln, NRW, Germany
IT Professional with more than 25 years experience in IT in the area of full life-cycle of Web/Desktop/Database Applications Development, Software Engineering, Consultancy, Data Management, Data Quality, Data Migrations, Reporting, ERP implementations & support, Team/Project/IT Management, etc.