A Software Engineer and data professional's blog on SQL, data, databases, data architectures, data management, programming, Software Engineering, Project Management, ERP implementation and other IT related topics.
Pages
- 🏠Home
- 🗃️Posts
- 🗃️Definitions
- 🏭Fabric
- ⚡Power BI
- 🔢SQL Server
- 📚Data
- 📚Engineering
- 📚Management
- 📚SQL Server
- 📚Systems Thinking
- ✂...Quotes
- 🧾D365: GL
- 💸D365: AP
- 💰D365: AR
- 👥D365: HR
- ⛓️D365: SCM
- 🔤Acronyms
- 🪢Experts
- 🗃️Quotes
- 🔠Dataviz
- 🔠D365
- 🔠Fabric
- 🔠Engineering
- 🔠Management
- 🔡Glossary
- 🌐Resources
- 🏺Dataviz
- 🗺️Social
- 📅Events
- ℹ️ About
04 August 2020
💼Project Management: Project Execution (Part I: Redefining Projects' Success I)
07 July 2020
🪄SSRS: Graphical Representations II (Sixth Magic Class)
- a level of detail (e.g. Category) which, when grouping the data, leads to a number of manageable series,
-- Sales volume per Product SELECT ITM.Category , ITM.Subcategory , ITM.ProductNumber , ITM.Name , Month(SOH.OrderDate) [Month] , Year(SOH.OrderDate) [Year] , Sum(SOD.OrderQty) OrderQty , Sum(SOD.LineTotal) OrderValue FROM Sales.SalesOrderDetail SOD JOIN Sales.SalesOrderHeader SOH ON SOD.SalesOrderID = SOH.SalesOrderID JOIN Production.vProducts ITM ON SOD.ProductId = ITM.Productid WHERE ITM.ProductNumber IN ('BB-7421', 'BB-9108') GROUP BY ITM.Category , ITM.Subcategory , ITM.ProductNumber , ITM.Name , Month(SOH.OrderDate) , Year(SOH.OrderDate) ORDER BY ITM.Category , ITM.Subcategory , ITM.ProductNumber , [Year] , [Month]
-- Sales volume per Product (extended) SELECT SOD.Category , SOD.Subcategory , SOD.ProductNumber , SOD.Name , SOD.[Month] , SOD.[Year] , SOD.OrderQty , SOD.OrderValue , SUM(SOD.OrderQty) OVER (PARTITION BY SOD.ProductNumber) TotalQty , SUM(SOD.OrderValue) OVER (PARTITION BY SOD.ProductNumber) TotalValue , SUM(SOD.OrderQty) OVER (PARTITION BY SOD.ProductNumber ORDER BY [Year], [Month]) CumulatedQty , SUM(SOD.OrderValue) OVER (PARTITION BY SOD.ProductNumber ORDER BY [Year], [Month]) CumulatedValue FROM (-- cumulated values SELECT ITM.Category , ITM.Subcategory , ITM.ProductNumber , ITM.Name , Month(SOH.OrderDate) [Month] , Year(SOH.OrderDate) [Year] , Sum(SOD.OrderQty) OrderQty , Sum(SOD.LineTotal) OrderValue FROM Sales.SalesOrderDetail SOD JOIN Sales.SalesOrderHeader SOH ON SOD.SalesOrderID = SOH.SalesOrderID JOIN [Production].[vProducts] ITM ON SOD.ProductId = ITM.Productid WHERE ITM.ProductNumber IN ('BB-7421', 'BB-9108') GROUP BY ITM.Category , ITM.Subcategory , ITM.ProductNumber , ITM.Name , Month(SOH.OrderDate) , Year(SOH.OrderDate) ) SOD ORDER BY SOD.Category , SOD.Subcategory , SOD.ProductNumber , SOD.[Year] , SOD.[Month]
Note:When prototyping a report is usually recommended to consider only a small number of records (e.g. only two Products). In addition, do not forget to validate the number or records considered by the logic:
-- checking the view for internal data consistency SELECT count(*) FROM Sales.SalesOrderDetail SOD JOIN Sales.SalesOrderHeader SOH ON SOD.SalesOrderID = SOH.SalesOrderID JOIN Production.vProducts ITM ON SOD.ProductId = ITM.Productid
Using the Report Wizard create a new matrix report called "Sales Volume per Product" based on either of the above queries (I considered the second). Within "Design the Matrix" select the attributes as follows:
Design the Matrix |
First draft in Design mode |
First draft in Preview mode |
Second draft in Design mode |
Horizontal Axis Properties |
Inserting a column within the group |
Repeat the process to add the Subcategory. Eventually you can add also the Product Name, though for it you'll have to select "Inside Group - Right".
Third draft in Design mode |
Third draft in Preview mode |
Sparkline types |
Fourth draft in Design mode |
Fourth draft in Preview mode |
If the prices remained relatively constant over time, there's actually almost no difference between the graphic displays for Order Quantity, respectively for Order Value. Therefore one can remove one of them (e.g. Order Quantity). Being more compact, sparkline-based representations allow to better use the space, therefore you can add more fields into the report.
06 July 2020
🪄SSRS: Graphical Representations I (SQL Server CPU Utilization)
-- cpu utilization for SQL Server and other applications DECLARE @ts_now bigint = (SELECT cpu_ticks/(cpu_ticks/ms_ticks) FROM sys.dm_os_sys_info); SELECT DAT.record_id , DAT.EventTime , DAT.SQLProcessUtilization , DAT.SystemIdle , 100 - (DAT.SystemIdle + DAT.SQLProcessUtilization) OtherUtilization FROM ( SELECT record.value('(./Record/@id)[1]', 'int') record_id , record.value('(./Record/SchedulerMonitorEvent/SystemHealth/SystemIdle)[1]', 'int') SystemIdle , record.value('(./Record/SchedulerMonitorEvent/SystemHealth/ProcessUtilization)[1]', 'int') SQLProcessUtilization , EventTime FROM ( SELECT DATEADD(ms, -1 * (@ts_now - [timestamp]), GETDATE()) EventTime , [timestamp] , CONVERT(xml, record) AS [record] FROM sys.dm_os_ring_buffers WHERE ring_buffer_type = N'RING_BUFFER_SCHEDULER_MONITOR' AND record LIKE N'%<SystemHealth>%') AS x ) AS DAT ORDER BY DAT.record_id DESC;
Preview mode |
Unfortunately the default choice of colors is not really ideal as red is used for warnings, and green for positive trends, which is not necessarily the case for a CPU's utilization.
🪄SSRS (& Paginated Reports): Ranking Rows in Reports
In almost all the reports I built, unless it was explicitly requested no to, I prefer adding a running number (aka ranking) for each record contained into the report, while providing different background colors for consecutive rows. The ranking allows easily identify a record when discussing about it within the report or extracts, while the different background colors allow differentiating between two records while following the values which scrolling horizontally. The logic for the background color can be based on two (or more) colors using the ranking as basis.
Tabular Reports
In a tabular report the RowNumber() function is the straightforward way for providing a ranking. One just needs to add a column into the report before the other columns, giving a meaningful name (e.g. RankingNo) and provide the following formula within its Expression:
= RowNumber(Nothing)
When 'Nothing' is provided as parameter, the ranking is performed across all the report. If is needed to restrict the Ranking only to a grouping (e.g. Category), then group's name needs to be provided as parameter:
= RowNumber("Category")
Matrix Reports
Unfortunately, in a matrix report based on aggregation of raw data the RowNumber() function stops working, the values shown being incorrect. The solution I use to solve this is based on the custom GetRank() VB function:
Dim Rank as Integer = 0 Dim LastValue as String = "" Function GetRank(group as string) as integer if group <> LastValue then Rank = Rank + 1 LastValue = group end if return Rank end function
The function compares the values provided in the call against a global scope LastValue text value. If the values are different, then a global scope Rank value is incremented by1, while the LastValue is initialized to the new value, otherwise the values remaining the same. The logic is basic also for a non-programmer.
The above code needs to be added into the Code section of Report's Properties for the function to be available:
Adding the code in Report Properties |
=Code.GetRank(Fields!ProductNumber.Value)
Note:
As it seems, on the version of Reporting Services Extension I use, the function has only a page scope, the value being reset after each page. However when exporting the data with Excel the ranking is applied to the whole dataset.
Providing Alternate Colors
Independently of the report type, one can provide an alternate color for table's rows by selecting the row with the data and adding the following expression into the BackaroundColor property:
=Iif(ReportItems!RankingNo.Value Mod 2, "White", "LightSteelBlue")
Notes:
1) For a tabular report the cost of calling the RowNumber function instead of referring to the RankingNo cell is relatively small. One can write it also like this:
=llf(RowNumber(Nothing) Mod 2 = 0, "White", "LightSteelBlue")
<Code>Dim Rank as Integer = 0 Dim LastValue as String = "" Dim Concatenation = "" Function GetRank(group as string) as integer if group <> LastValue then Rank = Rank + 1 LastValue = group end if Concatenation = Concatenation & vbCrLf & Rank & "/" & group & "/" & LastValue return Rank end function</Code> </Report>
🪄SSRS (& Paginated Reports): Matrix Report Display (Fifth Magic Class)
-- dropping the vProducts view (cleaning after) --DROP VIEW IF EXISTS [Production].[vProducts] -- creating the vProducts view CREATE VIEW [Production].[vProducts] AS SELECT p.[ProductID] , p.ProductNumber , p.[Name] , IsNull(p.Size, '') + IsNull(' ' + p.SizeUnitMeasureCode, '') Size , p.Color , P.Style , p.ProductModelID , pm.[Name] AS [ProductModel] , p.StandardCost , P.ListPrice , P.SafetyStockLevel , P.ReorderPoint , p.SellStartDate , p.SellEndDate , p.ProductSubcategoryID , PPS.Name Subcategory , PPS.ProductCategoryID , PPC.Name Category , P.MakeFlag , P.FinishedGoodsFlag FROM [Production].[Product] p LEFT JOIN [Production].[ProductModel] pm ON p.[ProductModelID] = pm.[ProductModelID] LEFT JOIN Production.ProductSubcategory PPS ON P.ProductSubcategoryID = PPS.ProductSubcategoryID LEFT JOIN Production.ProductCategory PPC ON PPS.ProductCategoryID = PPC.ProductCategoryID GO -- reviewing the data SELECT * FROM [Production].[vProducts] -- checking the view for internal data consistency SELECT count(*) FROM [Production].[Product] p LEFT JOIN [Production].[ProductModel] pm ON p.[ProductModelID] = pm.[ProductModelID] LEFT JOIN Production.ProductSubcategory PPS ON P.ProductSubcategoryID = PPS.ProductSubcategoryID LEFT JOIN Production.ProductCategory PPC ON PPS.ProductCategoryID = PPC.ProductCategoryID
-- dropping the vProductInventory view (cleaning after) --DROP VIEW IF EXISTS Production.vProductInventory -- creating the view CREATE VIEW Production.vProductInventory AS SELECT PPI.ProductId , PPD.ProductNumber , PPD.Name ProductName , PPD.ProductModel , PPD.Size , PPD.Category , PPD.Subcategory , PPD.Style , PPD.StandardCost , PPD.ListPrice , PPD.StandardCost * PPI.Quantity InventoryValue , PPD.ListPrice * PPI.Quantity SalesValue , PPD.MakeFlag , PPI.Locationid , PPL.Name Location , PPI.Shelf , PPI.Bin , PPI.Quantity FROM [Production].[ProductInventory] PPI JOIN [Production].[vProducts] PPD ON PPI.ProductID = PPD.ProductID JOIN [Production].[Location] PPL ON PPI.LocationID = PPL.LocationID -- reviewing the data SELECT * FROM Production.vProductInventory -- checking the view for internal data consistency SELECT count(*) FROM [Production].[ProductInventory] PPI JOIN [Production].[vProducts] PPD ON PPI.ProductID = PPD.ProductID JOIN [Production].[Location] PPL ON PPI.LocationID = PPL.LocationID
Note:
It's important to check the internal consistency of the views or queries used, on whether the logic removes or duplicates data. For this one can run the query for the uppermost table, and add repeatedly one more join for each run to see whether the number of records remains the same. One can shortcut the validation by checking only the number of records from the base table and for the whole query, and only if there are differences use the previously mentioned approach. (This is how I observed that the Production.vProductDetails view is not usable, because it considers only the Products having a valid Model.)
Creating the Report
We can now use the Production.vProductInventory view to create the Product Inventory by Location report based on the following query:
-- Product Inventory by Location SELECT PPI.Category , PPI.Subcategory , PPI.ProductNumber , PPI.ProductName , PPI.ProductModel , PPI.Size , PPI.Style , PPI.StandardCost , PPI.ListPrice , PPI.Location , PPI.Quantity , PPI.InventoryValue , PPI.SalesValue FROM Production.vProductInventory PPI ORDER BY PPI.Category , PPI.Subcategory , PPI.ProductNumber
Note:
The attributes can be provided in the order in which they should be displayed in the report, reducing thus the overhead in the further steps.
Using the Reporting Wizard via the Add New Report select in the first step the data source, while in the next step provide the above query:
Design the Query |
In the next step select the "Matrix" Report Type:
Select the Report Type |
Within the "Design the Matrix" section assign the fields as follows (all the fields except the ones considered as Columns and Details will be considered as Rows):
Design the Matrix |
Into the last step give the report a meaningful name (e.g. Product Inventory by Location):
Completing the Wizard |
In theory the report is ready to run, however before doing that change the formatting by aligning the headers to the center and eventually change their size from 11 to 10 pixels, respectively rename the dataset (e.g. Inventory). To obtain the same information about the grouping as below change into the "Advanced Mode".
Design View |
And here's the output (I had to scroll through the pages to find meaningful values, therefore part of the Details header is not shown):
Preview |
Restructuring the Grouping
As can be seen into the Design Mode, the wizard created a grouping for each attribute provider into the Details (see matrix1_Category, matrix1_Subcategory, etc.). Therefore, the values will not be repeated for each row, which is typically desirable for visualizations but not when exporting the data to Excel for further processing. I prefer the latter version, therefore to obtain this behavior one can go on and delete all the grouping via "Delete group only" except the matrix1_Category:
This action deleted unfortunately all the Detailed headers except Category. To bring them back into the grouping double click on the and add each field into the Group expressions as below:
Group Properties |
As final change before running the report one can add header names for the Detail attributes. After these changes reports' design looks as follows:
Report Design with one grouping |
And here's the final report with the values repeating for each row:
Preview Report without formatting |
Note:
To avoid removing the groupings, I prefer to add only one Detail field into the query, typically the field which will make the row unique into the output (e.g. Product Number) and add the further fields (actually replace the below query with the one above) after the Wizard created the report. One still needs to add the columns manually into the report. In the end the effort is similar.
-- Product Inventory by Location SELECT PPI.ProductNumber , PPI.Location , PPI.Quantity , PPI.InventoryValue , PPI.SalesValue FROM Production.vProductInventory PPI ORDER BY PPI.Category , PPI.Subcategory , PPI.ProductNumber
Changing the Design
Report's design can be slightly improved by adding various formatting of the cells or values. One can use similar formatting as the ones consider in the previous post. The only thing difficult to implement will be a ranking function (see Ranking Rows in Reports). After the design changes here's the report:
Final Report |
Note:
Of course, together with parameters one can also add totals after each Category or Subcategory to the report if needed, though the latter is more appropriate for design purposes and not for further data consumption.
Happy coding!
💠🛠️🪄SQL Server: Undocumented (Part III: SQL Server CPU Utilization via the Ring Buffer)
-- cpu utilization for SQL Server and other applications DECLARE @ts_now bigint = (SELECT cpu_ticks/(cpu_ticks/ms_ticks) FROM sys.dm_os_sys_info); SELECT DAT.record_id , DAT.EventTime , DAT.SQLProcessUtilization , DAT.SystemIdle , 100 - (DAT.SystemIdle + DAT.SQLProcessUtilization) OtherUtilization FROM ( SELECT record.value('(./Record/@id)[1]', 'int') record_id , record.value('(./Record/SchedulerMonitorEvent/SystemHealth/SystemIdle)[1]', 'int') SystemIdle , record.value('(./Record/SchedulerMonitorEvent/SystemHealth/ProcessUtilization)[1]', 'int') SQLProcessUtilization , EventTime FROM ( SELECT DATEADD(ms, -1 * (@ts_now - [timestamp]), GETDATE()) EventTime , [timestamp] , CONVERT(xml, record) AS [record] FROM sys.dm_os_ring_buffers WHERE ring_buffer_type = N'RING_BUFFER_SCHEDULER_MONITOR' AND record LIKE N'%<SystemHealth>%') AS x ) AS DAT ORDER BY DAT.record_id DESC;
CPU Utilization for my home SQL Server lab |
-- dropping the table -- DROP TABLE IF EXISTS dbo.T_RingBufferReadings -- reinitilizing the history -- TRUNCATE TABLE dbo.T_RingBufferReadings -- creating the table CREATE TABLE dbo.T_RingBufferReadings ( Id bigint IDENTITY (1,1) NOT NULL , RecordId bigint , EventTime datetime2(3) NOT NULL , SQLProcessUtilization int NOT NULL , SystemIdle int NOT NULL , OtherUtilization int NOT NULL ) -- reviewing the data SELECT * FROM dbo.T_RingBufferReadings ORDER BY EventTime DESC
If there are many records, to improve the performance, one can create also an index, which can include the reading points as well:
-- creating a unique index with an include CREATE UNIQUE NONCLUSTERED INDEX [UI_T_RingBufferReadings_EventTime] ON dbo.T_RingBufferReadings ( EventTime ASC, RecordId ASC ) INCLUDE (SQLProcessUtilization, SystemIdle, OtherUtilization) GO
The above query based on the DMV becomes:
-- cpu utilization by SQL Server and other applications DECLARE @ts_now bigint = (SELECT cpu_ticks/(cpu_ticks/ms_ticks) FROM sys.dm_os_sys_info); INSERT INTO dbo.T_RingBufferReadings SELECT record_id , DAT.EventTime , DAT.SQLProcessUtilization , DAT.SystemIdle , 100 - (DAT.SystemIdle + DAT.SQLProcessUtilization) OtherUtilization FROM ( SELECT record.value('(./Record/@id)[1]', 'int') record_id , record.value('(./Record/SchedulerMonitorEvent/SystemHealth/SystemIdle)[1]', 'int') SystemIdle , record.value('(./Record/SchedulerMonitorEvent/SystemHealth/ProcessUtilization)[1]', 'int') SQLProcessUtilization , EventTime FROM ( SELECT DATEADD(ms, -1 * (@ts_now - [timestamp]), GETDATE()) EventTime , [timestamp] , CONVERT(xml, record) AS [record] FROM sys.dm_os_ring_buffers WHERE ring_buffer_type = N'RING_BUFFER_SCHEDULER_MONITOR' AND record LIKE N'%<SystemHealth>%') AS x ) AS DAT LEFT JOIN dbo.T_RingBufferReadings RBR ON DAT.record_id = RBR.Recordid WHERE RBR.Recordid IS NULL ORDER BY DAT.record_id DESC;
Note:
A ServerName column can be added to the table if is needed to store the values for different SQL Servers. Then the LEFT JOIN has to consider the new added column.
Either of the two queries can be used to display the data points within a chart via SSRS, Power BI or any reporting tool available.
Happy coding!
References:
[1] Grant Fritchey (2014) SQL Server Query Performance Tuning: Troubleshoot and Optimize Query Performance in SQL Server 2014, 4th Ed.
[2] Sunil Agarwal et al (2005), Troubleshooting Performance Problems in SQL Server 2005, Source: TShootPerfProbs.docx
[3] Sunil Agarwal et al (2008), Troubleshooting Performance Problems in SQL Server 2008, Source: TShootPerfProbs2008.docx
[4] Microsoft SQL Docs (2018) Related Dynamic Management Views, Source
[5] Microsoft SQL Docs (2017) Use ring buffers to obtain health information about Always On availability groups, Source
About Me
- Adrian
- Koeln, NRW, Germany
- IT Professional with more than 24 years experience in IT in the area of full life-cycle of Web/Desktop/Database Applications Development, Software Engineering, Consultancy, Data Management, Data Quality, Data Migrations, Reporting, ERP implementations & support, Team/Project/IT Management, etc.