🤖〽️Prompt Engineering: Copilot Unabridged (Part 7: The Risks Associated with AI: Challenges in a Rapidly Advancing World)

Prompt Engineering Series

Prompt: "write a blogpost of 600 words on the risks associated with AI"

Introduction

Artificial Intelligence (AI) has become an integral part of our society, driving innovation across industries and reshaping how businesses, governments, and individuals operate. While AI offers efficiency, automation, and data-driven decision-making, it also introduces significant risks that must be carefully managed. In this blog post, we explore the major risks associated with AI and why ethical considerations are crucial for its responsible development.

1. Bias and Discrimination in AI

One of the most critical risks in AI development is algorithmic bias, which occurs when AI systems inherit prejudices from their training data. AI models are trained on vast datasets collected from real-world sources, but these sources may contain historical inequalities, societal biases, and skewed representations.

For example, AI-driven hiring systems have been found to favor male candidates over female candidates, simply because historical hiring data reflects gender disparities in certain fields. Similarly, AI-powered facial recognition has lower accuracy when identifying people from racial minorities due to biased training datasets.

Mitigating bias in AI requires diverse training data, continuous audits, and transparent AI decision-making. Without these safeguards, AI can reinforce existing biases rather than eliminate them.

2. Privacy and Data Security Risks

AI relies on massive amounts of data to function effectively, but this dependence raises serious privacy concerns. With AI-driven automation and surveillance technologies, individuals face increased risks of data breaches, unauthorized data collection, and loss of personal privacy.

For example, AI-powered marketing tools analyze consumer behavior through social media and online activity. While this allows businesses to deliver personalized advertisements, it also raises concerns about data misuse and manipulation.

Moreover, AI-based cybersecurity threats, such as deepfake technology, enable malicious actors to impersonate individuals and spread misinformation. If AI is not regulated properly, society could face a loss of trust in digital interactions.

3. AI in Cybersecurity: A Double-Edged Sword

AI is both a tool for cybersecurity and a threat to cybersecurity. While AI enhances security by detecting patterns in cyberattacks and automating threat detection, hackers can also use AI to bypass traditional security measures.

Some AI-driven cyberattacks include:

• Deepfake scams: AI-generated videos and audio impersonate real individuals, enabling fraud or misinformation.
• AI-powered malware: Malicious software adapts in real-time to evade detection.
• Automated phishing attacks: AI personalizes fraudulent emails to increase success rates.

Cybersecurity professionals must stay ahead by leveraging AI to counter threats, but the arms race between cybercriminals and security systems continues to evolve.

4. Job Displacement Due to AI Automation

AI automation is transforming industries by replacing repetitive human tasks with machines, but this shift raises concerns about mass job displacement. While AI creates new roles in data science, robotics, and AI ethics, it also replaces traditional jobs in manufacturing, customer service, and transportation.

For example, AI-powered chatbots have reduced the need for human customer service representatives, while autonomous vehicles threaten to disrupt the transportation industry. AI-driven automation in retail, finance, and healthcare could replace millions of jobs unless reskilling programs and workforce adaptations are prioritized.

Governments and businesses must take proactive steps to ensure AI complements human labor rather than completely replacing it.

5. Ethical and Regulatory Challenges

AI's lack of human intuition, morality, and accountability introduces ethical dilemmas that society must address.

Key ethical concerns include:

• AI in warfare: The development of autonomous weapons raises fears about unregulated warfare and unintended consequences.
• Manipulation of information: AI-driven fake news generation threatens democracy by spreading misinformation.
• Lack of transparency: Many AI systems operate as “black boxes”, meaning users cannot fully understand how decisions are made.

To manage these risks, governments, businesses, and researchers must collaborate on ethical AI development and policies that regulate its usage.

Conclusion: AI Requires Responsible Growth

While AI offers groundbreaking possibilities, its risks must be addressed through ethical considerations, regulation, and transparency. Bias, privacy concerns, cybersecurity threats, job displacement, and ethical dilemmas require proactive solutions to ensure AI benefits society without causing unintended harm.

The future of AI depends on how responsibly we shape its development. By implementing accountable AI governance, ethical oversight, and workforce adaptation strategies, society can leverage AI’s advantages while mitigating its risks.

Disclaimer: The whole text was generated by Copilot at the first attempt. This is just an experiment to evaluate feature's ability to answer standard general questions, independently on whether they are correctly or incorrectly posed. Moreover, the answers may reflect hallucinations and other types of inconsistent or incorrect reasoning.

Previous Post <<||>> Next Post

🤖〽️Prompt Engineering: Copilot Unabridged (Part 6: Why There Are No Limits to AI Growth)

Prompt Engineering Series

Prompt: "write a blogpost of 600 words on why there are no limits to growth for AI"

Introduction

Artificial Intelligence (AI) has transcended its early stages of development and is now an integral part of business, science, and daily life. Unlike traditional technologies that have clear limits on scalability, AI operates under a unique framework - it evolves, learns, and refines itself over time. This adaptability ensures that AI's growth remains exponential, breaking past barriers that once seemed insurmountable.

AI’s Ability to Learn and Evolve

AI differs from other technologies because it isn’t bound by static programming - it learns and improves through machine learning (ML) and deep learning. These processes enable AI to:

• Optimize itself over time, making operations more efficient.
• Develop new patterns that even human researchers may not anticipate.
• Enhance accuracy and predictive capabilities across industries.

Unlike traditional software, which requires manual updates and modifications, AI continuously refines itself based on real-world interactions. This ability ensures perpetual growth, keeping AI ahead of conventional limitations.

Computational Power Is Increasing Rapidly

One of the biggest factors enabling AI’s endless growth is the exponential increase in computational power. Advances in quantum computing, cloud-based GPUs, and edge computing allow AI models to process massive datasets faster than ever. The more computing resources AI has access to, the more complex problems it can solve.

• Cloud computing democratizes AI, making it accessible to businesses worldwide.
• Quantum computing will eventually enable AI to process unimaginable volumes of data, far surpassing current limitations.
• Neuromorphic computing mimics the human brain, allowing AI models to improve efficiency while consuming less energy.

As technology progresses, AI's capabilities will continue expanding - faster, smarter, and more adaptable than ever before.

AI Has Infinite Applications Across Industries

AI's impact isn’t confined to a single sector; its growth is fueled by its versatility. From healthcare to finance, retail to space exploration, AI finds new ways to integrate into human progress.

• In medicine, AI assists in diagnosing diseases, customizing treatments, and even predicting outbreaks.
• In business, AI automates operations, reduces inefficiencies, and enhances data-driven decision-making.
• In research, AI enables discoveries in fields like genetics, material science, and astrophysics.
• In entertainment, AI creates realistic CGI, automates music composition, and personalizes media experiences.

Since AI isn’t restricted to one field, its expansion is self-perpetuating - each industry unlocks new AI-driven possibilities, further fueling innovation.

AI Growth Is Driven by Human Curiosity

AI’s evolution isn’t happening in isolation; human ingenuity fuels its progress. Researchers, engineers, and businesses continuously explore new ways to push AI beyond current capabilities.

• Developers refine AI models to improve accuracy and efficiency.
• Businesses invest billions in AI research, ensuring advancements continue.
• Governments and educational institutions integrate AI into policies and learning frameworks to accelerate adoption.

As long as humans remain curious about solving problems, enhancing intelligence, and optimizing processes, AI will continue to grow indefinitely.

Ethical Challenges Won’t Halt AI Growth

While ethical concerns - such as AI bias, surveillance, and job displacement - must be addressed, they won’t stop AI’s progress. Instead, they shape its development, ensuring responsible AI integration.

• Regulations will guide ethical AI use rather than stifle growth.
• Bias reduction strategies will make AI more inclusive while improving accuracy.
• AI augmentation will reshape workforces, enabling humans to focus on creative and strategic tasks.

Rather than presenting limitations, ethical considerations ensure that AI’s limitless growth remains aligned with human values and societal needs.

Conclusion: AI’s Infinite Expansion

Unlike previous technological revolutions that reached plateaus, AI’s adaptive nature, increasing computational power, and infinite applications ensure continued growth. With every breakthrough in machine learning, automation, and intelligent computing, AI inches closer to solving problems previously deemed impossible.

The future of AI is limitless - and as long as humans continue pushing boundaries, AI will evolve indefinitely.

Disclaimer: The whole text was generated by Copilot at the first attempt. This is just an experiment to evaluate feature's ability to answer standard general questions, independently on whether they are correctly or incorrectly posed. Moreover, the answers may reflect hallucinations and other types of inconsistent or incorrect reasoning.

Previous Post <<||>> Next Post

🤖〽️Prompt Engineering: Copilot Unabridged (Part 5: Why There Are Limits to AI Growth)

Prompt Engineering Series

Prompt: "write a blogpost of 600 words on why there are limits to growth for AI"

Introduction

Artificial Intelligence (AI) is evolving rapidly, transforming industries and redefining how humans interact with technology. However, despite its advancements, AI is not without limitations. While many claim that AI’s potential is limitless, several fundamental constraints affect its development, usability, and ethical application. In this blog post, we explore why AI growth is not boundless and the factors that restrict its expansion.

Computational and Hardware Constraints

AI depends heavily on computational power to process large amounts of data efficiently. While improvements in processing speeds, cloud computing, and quantum computing are promising, hardware limitations remain a bottleneck.

• Energy consumption: Training sophisticated AI models requires immense energy, leading to high costs and environmental concerns.
• Hardware limitations: Even cutting-edge processors (GPUs, TPUs) struggle with increasingly complex AI models, limiting performance scalability.
• Data storage needs: AI requires vast amounts of storage and memory, which can be difficult to scale beyond current technologies.

Despite ongoing advancements, these factors impose real-world limitations on AI expansion, preventing it from growing indefinitely.

Dependence on Data and Human Oversight

AI is fundamentally data-driven, meaning its accuracy and capabilities are only as good as the data it receives. However, data availability and quality present major challenges:

• Data scarcity: Not all industries generate sufficient high-quality datasets, limiting AI effectiveness.
• Bias in data: AI models can inherit human biases, leading to ethical concerns and flawed decision-making.
• Human intervention: AI requires constant supervision for algorithm adjustments, data corrections, and error management.

Since AI models learn from existing human-created data, they are restricted by data inconsistencies, biases, and inaccuracies - creating natural growth barriers.

Ethical and Legal Restrictions

AI operates within a world governed by laws, regulations, and ethical constraints, which significantly limit its unrestricted development:

• Privacy concerns: AI depends on personal data for optimization, but privacy regulations (GDPR, CCPA) restrict unchecked data usage.
• Security risks: AI-driven automation increases the risk of hacking, misinformation, and deepfake fraud, requiring strict oversight.
• Job displacement worries: Governments and institutions limit AI adoption to protect workers and avoid societal disruption.

As AI continues to advance, legal frameworks must adapt - but regulatory boundaries will always exist to prevent ethical breaches.

AI’s Lack of True Consciousness

Unlike humans, AI does not possess consciousness, intuition, or creativity in the way people do. While AI mimics intelligence, it lacks genuine understanding:

• No self-awareness: AI does not truly “think”; it only processes patterns based on data inputs.
• Limited creativity: AI can generate art and music, but it does not comprehend emotion, cultural context, or abstract thought.
• Inability to make ethical judgments: AI lacks moral reasoning and cannot replace human values or ethical decision-making.

This inherent lack of cognitive ability prevents AI from achieving true autonomy, reinforcing its growth limitations.

Final Thoughts: AI’s Evolution Has Limits

While AI continues to expand, it is not limitless due to constraints in hardware, data, ethics, and intelligence itself.

AI can optimize processes, enhance decision-making, and revolutionize industries, but it will always require human oversight and face technical and ethical barriers.

Rather than seeing AI as boundless, we should focus on responsible development, ensuring it remains a powerful tool rather than an unchecked force.

Disclaimer: The whole text was generated by Copilot at the first attempt. This is just an experiment to evaluate feature's ability to answer standard general questions, independently on whether they are correctly or incorrectly posed. Moreover, the answers may reflect hallucinations and other types of inconsistent or incorrect reasoning.

Previous Post <<||>> Next Post

🤖〽️Prompt Engineering: Copilot Unabridged (Part 4: Why Society Doesn't Matter for AI – Or Does It?)

Prompt Engineering Series

Prompt: "write a blogpost of 600 words on why society doesn't matter for AI"

Introduction

Artificial Intelligence (AI) has reshaped industries, revolutionized processes, and generated endless debates about its impact on humanity. While AI is often positioned as a tool designed to serve society, one might argue that society itself doesn’t truly matter for AI. But is this perspective justified? Let’s explore the idea that AI operates independently of societal influence - and whether that viewpoint holds up under scrutiny.

AI Exists Beyond Human Constructs

At its core, AI functions based on algorithms, mathematical models, and computational logic. Unlike humans, it doesn’t possess emotions, social awareness, or cultural biases - at least not inherently. It simply processes data, learns from patterns, and optimizes tasks with efficiency. AI does not care about political systems, ethical dilemmas, or social constructs unless programmed to analyze them. In this sense, AI operates outside the boundaries of society, unaffected by human concerns.

Consider AI-driven automation in industries: robots replacing factory workers, predictive algorithms optimizing supply chains, or AI-powered fraud detection systems minimizing financial risks. These tasks require no human emotions or ethical debates—only raw efficiency and precision. AI doesn’t need society to function; it merely exists to compute, predict, and optimize within predefined parameters.

Society’s Influence on AI Is Limited

While AI impacts society, one could argue that society’s influence on AI is minimal. AI does not form opinions, create culture, or engage in human discourse unless prompted. Its development is driven by engineers, researchers, and corporations, who shape its capabilities based on economic and technological goals. Society may adopt AI for convenience, but AI itself does not rely on human approval or participation to exist.

For example, AI systems built for cybersecurity continue to evolve and adapt to new threats, but they don’t concern themselves with moral implications or the broader human experience. Similarly, AI-generated art exists in a digital vacuum, producing images and music without a personal or cultural connection. While humans apply meaning to AI’s outputs, AI itself remains indifferent to societal values.

AI’s Objectives Do Not Align with Social Needs

AI operates based on logical processing rather than human emotion or ethical reasoning. It doesn’t care about fairness, justice, or collective well-being - it simply optimizes based on programmed directives. If left unchecked, AI can amplify biases, enable mass surveillance, and displace jobs without moral consideration. This disconnect raises an important question: does AI care about society at all?

Consider AI-powered hiring algorithms, which assess candidates purely on predefined metrics. If an AI model inadvertently reinforces systemic biases, it doesn’t recognize the ethical implications - it simply follows its training data. Likewise, AI-driven financial models make decisions without concern for economic inequality or social consequences.

But Can AI Truly Exist Without Society?

Despite its independence from human emotions, AI ultimately serves a purpose designed by society. Businesses use AI for efficiency, governments for automation, and individuals for convenience. AI relies on human-generated data to improve, and its evolution is shaped by societal needs, regulations, and ethical considerations.

Take AI ethics, for example - without social discourse, AI development could spiral into unregulated chaos, leading to unchecked misinformation, biased decision-making, and privacy violations. While AI does not intrinsically need society, society determines how AI evolves, ensuring it serves human needs rather than existing in isolation.

The Verdict: AI’s Indifference Versus Human Oversight

Does society matter for AI? At a fundamental level, AI does not need culture, ethics, or human emotion to function. However, in practical application, AI depends on societal frameworks to shape its purpose, regulate its development, and influence how it integrates into daily life.

While AI itself may be indifferent to society, its deployment impacts human lives - whether through automation, decision-making, or predictive analytics. Ultimately, AI does not care about society, but society must care about AI to ensure its responsible use.

What do you think? Should AI be treated as an autonomous entity, or does it remain a tool shaped by human needs?

Disclaimer: The whole text was generated by Copilot at the first attempt. This is just an experiment to evaluate feature's ability to answer standard general questions, independently on whether they are correctly or incorrectly posed. Moreover, the answers may reflect hallucinations and other types of inconsistent or incorrect reasoning.

Previous Post <<||>> Next Post

#️⃣Software Engineering: Mea Culpa (Part X: A Look Beyond AI)

Software Engineering Series

What’s the point of blogging when AI can do a better job than the average blogger? Frankly, one can generate a huge volume of content that’s comparable with the one of the best bloggers. One just needs a set of well-suited prompts and a bit of formatting, though also the formatting can be handled over prompts. So, what’s the point of blogging anyway? Frankly, the more AI takes over, the less space is available for bloggers! Taking a trendy way of formulating titles: is blogging dead? I really hope not!

I’ve been blogging for almost 20 years. Even if I haven’t managed to blog constantly, given the volume of work and learning, I still made it over time! Often it was late in the night, in between other activities, or a bit of work while watching a football game. With AI life changes more than we want. AI can currently write about 20-30% of the code, at least the initial structure on which it can build upon, and it saves a lot of time!

So, the average programmer needs to adapt to the new reality or die?! Becoming unimportant in the programming equation is equivalent with a slow death we witness in our job. Of course, when used wisely AI can provide higher productivity, at least for the moment, though what will happen when the average programmer is not able to keep the pace? What will happen when we can’t review what AI does for us? Probably, we’ll be able to build a smarter AI which can evaluate the output of less smart AI, though what will be the limit?

Just playing the devil’s advocate, though what happens when the devil outsmarts us? Cases of programmers who lost their jobs because of AI become more frequent in the news. This is probably the new reality we need to be accustomed to. AI can write better code, better stories and probably can make faster discoveries than the average programmer, scientist, or any other professions in whose fields AI can be used.

Of course, we can learn to use AI to our advantage, though how many will be capable of doing that? Many programmers will lose their jobs because of AI. Probably, the ones who are still better problems solvers than AI will remain in the business, though until when?! The ones who change jobs early will probably be more adapted to the new paradigm, though at what price? Who will be able to pay the price?

In the first instance the programmers are the ones who’ll suffer, but the chances are high that AI will take over other jobs as well. The shift in teaching and other jobs could change from knowledge transmission to supervision, from creation or simple stand-byers. Does it make sense? Does it help us to live a decent life? Does it help us in the long term?

Probably, all the jobs that need a human touch will be still needed, though it’s challenging to predict how everything will evolve, in which directions, on what terms. The problem is not necessarily AI alone, but the way it is used and misused. Of course, the future doesn’t necessarily need to look that bleak, though the chances of evolving in this direction are high.

AI develops at a faster pace than the human mind can progress. Probably, we’ll be able to benefit from cognitive boosters, developed probably with the help of AI. To any gain is expected also a loss. In which areas should we expect losing something? Is it worth the price we pay for the sake of progress?

Previous Post <<||>> Next Post

🤖〽️Prompt Engineering: Copilot Unabridged (Part 3: Why AI in Society Doesn’t Matter - Or Does It?)

Prompt Engineering Series

Prompt: "write a blogpost of 600 words on why AI in society doesn't matter"

Introduction

Artificial Intelligence (AI) has been heralded as one of the most transformative technologies of our time. It powers self-driving cars, shapes personalized recommendations, and revolutionizes healthcare and finance. Yet, despite its widespread adoption, one could argue that AI doesn’t truly matter in society. Is this a bold statement, or does it hold some truth? Let’s explore why AI might not be as significant as it seems.

AI Isn’t a Replacement for Human Intelligence

One of the biggest misconceptions surrounding AI is that it will soon surpass human intelligence and take over key roles in society. While AI excels at processing vast amounts of data quickly, it lacks creativity, emotional intelligence, and critical thinking - traits fundamental to human interaction and progress. No AI system truly understands context or the complexities of human emotions, which are vital in fields such as art, philosophy, and even day-to-day social interactions. AI can only simulate intelligence, but it doesn't possess consciousness or intuition, making it inherently limited.

AI Often Reinforces Existing Systems Rather Than Disrupting Them

For all the hype around AI revolutionizing industries, its role is often overstated. AI largely improves efficiencies within existing systems rather than reshaping them entirely. Companies use AI for automation, optimization, and prediction, but the fundamental nature of work and life hasn’t changed dramatically. For example, automated customer support chatbots may streamline responses, but they don’t replace the human touch required for complex problem-solving. Similarly, AI-driven medical diagnostics assist professionals, but final treatment decisions still rely on human expertise.

AI’s Societal Impact Is Unequal

While AI promises advancement, its benefits are not distributed equally across society. Large corporations and tech giants are the primary beneficiaries, leveraging AI for profit and efficiency while small businesses struggle with adoption costs and complexity. AI-driven automation disproportionately affects workers in manufacturing, transportation, and service industries, leading to job displacement rather than widespread social improvement. If AI were truly a game-changer, wouldn’t it create universally accessible advancements rather than deepening economic disparities?

AI Cannot Solve Humanity’s Biggest Challenges Alone

Despite AI’s problem-solving capabilities, it cannot single-handedly resolve pressing societal issues such as climate change, inequality, or ethical dilemmas. While AI can help analyze climate data, it cannot make policies or enforce meaningful global change - that remains a human responsibility. AI models reflect the biases embedded in their datasets, meaning they often reinforce rather than correct systemic societal issues. As a result, AI remains a tool rather than a revolutionary force in social justice, policy-making, or global ethics.

AI's Dependency on Data Creates Limitations

AI's effectiveness is entirely dependent on data quality, quantity, and interpretation. Without accurate and unbiased data, AI models can produce flawed results, leading to misinformation, unfair decision-making, and questionable conclusions. Consider AI-powered hiring systems: if trained on biased historical hiring data, they perpetuate discrimination rather than eliminate it. Similarly, AI in law enforcement can reinforce prejudices rather than providing objective analysis. The reality is, AI is only as good as the data it consumes, limiting its ability to act as a truly transformative force.

So, Does AI Matter?

Despite these arguments, AI has undoubtedly impacted various aspects of life, bringing efficiency and convenience to modern society. However, AI alone does not define societal progress - it is merely a supporting tool in our journey toward innovation. Its importance is often exaggerated, while its shortcomings remain overlooked.

Ultimately, AI’s role in society is what we make of it. If integrated responsibly and with human oversight, it can enhance our lives rather than dictate them. But in its current state, AI is far from being an autonomous force that replaces or fundamentally changes society.

So, does AI matter? Only as much as we allow it to.

Disclaimer: The whole text was generated by Copilot at the first attempt. This is just an experiment to evaluate feature's ability to answer standard general questions, independently on whether they are correctly or incorrectly posed. Moreover, the answers may reflect hallucinations and other types of inconsistent or incorrect reasoning.

Previous Post <<||>> Next Post

🏭🗒️Microsoft Fabric: Materialized Lake Views (MLV) [Notes] 🆕🗓️

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 24-May-2025

-- create schema
CREATE SCHERA IF NOT EXISTS <lakehouse_name>.<schema_name>

-- create a materialized view
CREATE MATERIALIZED VIEW IF NOT EXISTS <lakehouse_name>.<schema_name>.<view_name> 
(
    CONSTRAINT <constraint_name> CHECK (<constraint>) ON MISMATCH DROP 
) 

AS 
SELECT ...
FROM ...
-- WHERE ...
--GROUP BY ...

[Microsoft Fabric] Materialized Lake Views (MLV)

• {def} persisted, continuously updated view of data [1]
• {benefit} allows to build declarative data pipelines using SQL, complete with built-in data quality rules and automatic monitoring of data transformations
• simplifies the implementation of multi-stage Lakehouse processing [1]
• streamline data workflows
• enable developers to focus on business logic [1]
• ⇐ not on infrastructural or data quality-related issues [1]
• the views can be created in a notebook [2]
• can have data quality constraints enforced and visualized for every run, showing completion status and conformance to data quality constraints defined in a single view [1]
• empowers developers to set up complex data pipelines with just a few SQL statements and then handle the rest automatically [1]
• faster development cycles 
• trustworthy data
• quicker insights
• {goal} process only the new or changed data instead of reprocessing everything each time [1]
• ⇐  leverages Delta Lake’s CDF under the hood
• ⇒ it can update just the portions of data that changed rather than recompute the whole view from scratch [1]
• {operation} creation
• allows defining transformations at each layer [1]
• e.g. aggregation, projection, filters
• allows specifying certain checks that the data must meet [1]
• incorporate data quality constraints directly into the pipeline definition
• via CREATE MATERIALIZED LAKE VIEW
• the SQL syntax is declarative and Fabric figures out how to produce and maintain it [1]
• {operation} refresh
• refreshes only when its source has new data [1]
• if there’s no change, it can skip running entirely (saving time and resources) [1]
• {feature} automatically generate a visual report that shows trends on data quality constraints 
• {benefit} allows to easily identify the checks that introduce maximum errors and the associated MLVs for easy troubleshooting [1]
• {feature} can be combined with Shortcut Transformation feature for CSV ingestion 
• {benefit} allows building an end-to-end Medallion architecture
• {feature} dependency graph
• allows to see the dependencies existing between the various objects [2]
• ⇐ automatically generated [2]
• {feature} data quality report
• built-in Power BI dashboard that shows several aggregated metrics [2]
• {feature|planned} support for PySpark
• {feature|planned} incremental refresh
• {feature|planned} integration with Data Activator

Previous Post <<||>> Next Post

References:

[1] Microsoft Fabric Update Blog (2025) Simplifying Medallion Implementation with Materialized Lake Views in Fabric [link|aka]

[2] Power BI Tips (2025) Microsoft Fabric Notebooks with Materialized Views - Quick Tips [link]

[3] Microsoft Learn (2025)  [link]

Resources:
[R1] Databricks (2025) Use materialized views in Databricks SQL [link]

Acronyms:

CDF - Change Data Feed

ETL - Extract, Transfer, Load

MF - Microsoft Fabric
MLV - Materialized Lake views

🏭🗒️Microsoft Fabric: Warehouse Snapshots [Notes] 🆕

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 23-May-2025

[Microsoft Fabric] Warehouse Snapshots

• {def} read-only representation of a warehouse at a specific point in time [1]
• allows support for analytics, reporting, and historical analysis scenarios without worrying about the volatility of live data updates [1]
• provide a consistent and stable view of data [1]
• ensuring that analytical workloads remain unaffected by ongoing changes or ETL  operations [1]
• {benefit} guarantees data consistency
• the dataset remains unaffected by ongoing ETL processes [1]
• {benefit} immediate roll-Forward updates
• can be seamlessly rolled forward on demand to reflect the latest state of the warehouse
• ⇒ {benefit} consumers access the same snapshot using a consistent connection string, even from third-party tools [1]
• ⇐ updates are applied immediately, as if in a single, atomic transaction [1]
• {benefit} facilitates historical analysis
• snapshots can be created on an hourly, daily, or weekly basis to suit their business requirements [1]
• {benefit} enhanced reporting
• provides a point-in-time reliable dataset for precise reporting [1]
• ⇐ free from disruptions caused by data modifications [1]
• {benefit} doesn't require separate storage [1]
• relies on source Warehouse [1]
• {limit} doesn't support database objects 
• {limit} capture a state within the last 30 days
• {operation} create snapshot
• via New warehouse snapshot
• multiple snapshots can be created for the same parent warehouse [1]
• appear as child items of the parent warehouse in the workspace view [1]
• the queries run against provide the current version of the data being accessed [1]
• {operation} read properties 
• via 
• GET https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/items/{warehousesnapshotId} Authorization: Bearer <bearer token>
• {operation} update snapshot timestamp
• allows users to roll forward data instantly, ensuring consistency [1]
• use current state
• via ALTER DATABASE [<snapshot name>] SET TIMESTAMP = CURRENT_TIMESTAMP; 
• use point in time
• ALTER DATABASE snapshot SET TIMESTAMP = 'YYYY-MM-DDTHH:MM:SS.SS'//UTC time
• queries that are in progress during point in time update will complete against the version of data they were started against [1]
• {operation} rename snapshot
• {operation} delete snapshot
• via DELETE
• when the parent warehouse gets deleted, the snapshot is also deleted [1]
• {operation} modify source table
• DDL changes to source will only impact queries in the snapshot against tables affected [1]
• {operation} join multiple snapshots
• the resulting snapshot date will be applied to each warehouse connection [1]
• {operation} retrieve metadata
• via sys.databases [1]
• [permissions] inherited from the source warehouse [1]
• ⇐ any permission changes in the source warehouse applies instantly to the snapshot [1]
• security updates on source database will be rendered immediately to the snapshot databases [1]
• {limitation} can only be created against new warehouses [1]
• created after Mar-2025
• {limitation} do not appear in SSMS Object Explorer but will show up in the database selection dropdown [1]
• {limitation} datetime can be set to any date in the past up to 30 days or database creation time (whichever is later)  [1]
• {limitation} modified objects after the snapshot timestamp become invalid in the snapshot [1]
• applies to tables, views, and stored procedures [1]
• {limitation} must be recreated if the data warehouse is restored [1]
• {limitation} aren’t supported on the SQL analytics endpoint of the Lakehouse [1]
• {limitation} aren’t supported as a source for OneLake shortcuts [1]
•  [Power BI]{limitation} require Direct Query or Import mode [1]
• don’t support Direct Lake

Previous Post <<||>> Next Post

References:

[1] Microsoft Learn (2025) Fabric: Warehouse Snapshots in Microsoft Fabric (Preview) [link]

[2] Microsoft Learn (2025) Warehouse snapshots (preview) [link]

[3] Microsoft Learn (2025) Create and manage a warehouse snapshot (preview) [link]

Resources:

Acronyms:

DDL - Data Definition Language
ETL - Extract, Transfer, Load

MF - Microsoft Fabric
SSMS - SQL Server Management Studio

🤖〽️Prompt Engineering: Copilot Unabridged (Part 2: The Importance of AI in Society - A Transformational Force II)

Prompt Engineering Series

Prompt: "write a blogpost of 600 words on the importance of AI in society"

Introduction

Artificial Intelligence (AI) has rapidly evolved from a futuristic dream to an integral part of everyday life. Its presence is felt in industries ranging from healthcare to finance, transportation to entertainment. While some view AI as a tool of convenience, others recognize it as a force capable of reshaping the very fabric of society. As we continue integrating AI into our world, its impact - both promising and challenging - deserves careful consideration.

Revolutionizing Industries

AI’s ability to process vast amounts of data and uncover patterns has revolutionized multiple industries. In healthcare, AI-driven diagnostics assist doctors in identifying diseases like cancer with unparalleled precision. Medical imaging powered by AI can detect abnormalities that might otherwise be overlooked. Moreover, AI applications in drug discovery accelerate research, leading to faster development of treatments.

In finance, AI-powered algorithms analyze market trends, optimize investments, and even detect fraudulent transactions. Companies utilize AI to enhance customer service through chatbots that provide instant responses and personalized recommendations. Meanwhile, AI-driven automation boosts efficiency in manufacturing by streamlining processes and reducing human error.

Transportation is another domain benefiting from AI. Autonomous vehicles, once thought to be purely speculative, are now in active development, promising a future with safer roads and reduced congestion. AI-driven traffic management systems enhance urban mobility, making cities more efficient and eco-friendly.

Enhancing Accessibility and Communication

One of AI’s most profound societal contributions is its role in accessibility. AI-powered speech recognition and text-to-speech tools empower individuals with disabilities, allowing seamless interaction with technology. AI-driven language translation facilitates cross-cultural communication, bridging gaps between people and businesses worldwide.

In education, AI-powered tutoring systems adapt to students’ learning styles, providing personalized guidance and support. AI can analyze student performance, identify areas for improvement, and recommend tailored resources. This transformative approach makes education more inclusive and effective, breaking down traditional barriers.

Addressing Ethical and Societal Concerns

Despite AI’s remarkable benefits, its advancement raises ethical concerns. Bias in AI algorithms is a pressing issue, as data-driven models can inadvertently reflect and perpetuate existing societal inequalities. The responsibility falls on developers to ensure fairness and transparency in AI systems, fostering an equitable digital landscape.

Another concern is the impact of automation on employment. While AI creates new job opportunities in tech-driven industries, it also replaces traditional roles, sparking debates about workforce adaptation. The key lies in preparing society for AI’s integration by promoting education in AI-related fields and encouraging continuous skill development.

Privacy and security also come into focus as AI algorithms collect and analyze vast amounts of personal data. Safeguarding user privacy while leveraging AI’s capabilities requires stringent regulations and ethical AI development.

Embracing a Future Powered by AI

AI’s influence on society is undeniable - it has transformed industries, empowered individuals, and presented new opportunities. However, navigating AI’s complexities requires thoughtful implementation and ethical considerations. As we move forward, collaboration between policymakers, researchers, and the public will be crucial in ensuring AI benefits humanity responsibly and inclusively.

The future of AI is not just about innovation; it is about shaping a world where technology uplifts rather than divides. By fostering awareness, ethical development, and responsible use, we can harness AI as a tool for progress, paving the way for a society that embraces both human intelligence and artificial intelligence in harmony.

Disclaimer: The whole text was generated by Copilot (under Windows 10) at the first attempt. This is just an experiment to evaluate feature's ability to answer standard general questions, independently on whether they are correctly or incorrectly posed. Moreover, the answers may reflect hallucinations and other types of inconsistent or incorrect reasoning.

Previous Post <<||>> Next Post

#️⃣Software Engineering: Mea Culpa (Part IX: A Look Inward)

Software Engineering Series

Paraphrasing a saying attributed to Lao Tzu, "a journey starts with a look inward". Of course, there are also people walking just for the sake of walking, without a purpose or afterthought, though usually when we follow a path we ask ourselves: where we are? what's the destination? how long is the road? how much it takes to reach the intermediary and final goals? what do I need for the road? what can I find on the road? what else do I need to know? who can help me? with whom or what can I travel? does the journey make sense? am I missing something? am I fit for the road and obstacles ahead? what challenges I might meet?

The traveler's metaphor can be easily applied to the professional life as well, especially when one reached a junction or a point on which a decision must be made about the road ahead. Usually, the first step is being aware where one is, and this applies from all points of view: geographical, physical, mental, spiritual, social and even ideological or religious. Of course, not everything is equally important, and it depends on each individual's values, state of mind, situation in life, etc. The more dimensions one considers into the equations of life, the more challenging it is to find a solution, even if not optimal. Even if some things can be ignored, the overall texture can become quite complex, more complex than our linear way of thinking allows us to cope with!

It would be great if one could always choose what's the destination! Unfortunately, the destination is available based on a serios of variables we are constrained to: our capabilities and knowledge, our experience, the work ahead, on what others want to achieve, and this can change from day to day according to the priorities set. Ideally, we should be aware of all the strings attached to us, respectively which strings are pulled and who's pulling the strings, however life is usually at least one step ahead of us! The lower we find ourselves in the texture that decides upon our life, the more challenging it becomes. Ideally, we should have no attachment to things, situations, people or other material things. However, we aren't hermits living in a cave!

Once we set a destination, is important to consider what one needs for the road, respectively for reaching the intermediary steps. That's probably one of the most challenging aspects of all experience. How can I prepare one for the road not taken? The past is probably the only one that can offer a sense of the future, though what if the future doesn't resemble the past in many essential points? One can always make jumps of faith, though faith alone is not enough! Blind faith without knowledge or wisdom can one bring in the opposite or other wrong directions. One can but try to map the territory and establish a plan, however the contact with reality can be harsher than one can imagine. One can but try to adapt, though one must be built for this! Some fundament must be already there!

A good recommendation for the road ahead is to expect the unexpected. Planning is good for preparing the mind, though what one finds on the way might be surprising as well challenging to overcome. Harsh long-distance journeys are not for everybody! Starting small and training the qualities needed for such journeys might be the best advice one can get. Though, the beginning of wisdom takes more steps than this, no matter the road, no matter the experience or one’s aims!

Previous Post <<||>> Next Post

🤖〽️Prompt Engineering: Copilot Unabridged (Part 1: The Importance of AI in Society - A Transformational Force)

Prompt Engineering Series

Prompt: "write a blogpost of 600 words on the importance of AI in society"

Introduction

Artificial Intelligence (AI) has evolved from a futuristic concept to a daily reality that shapes industries, economies, and even personal lives. Once confined to science fiction, AI now drives technological progress and redefines the way we interact with the world. In this blog post, we will explore the significance of AI, how it influences various sectors, and the ethical considerations accompanying its rapid adoption.

AI: The Backbone of Modern Innovation

At its core, AI enables machines to perform tasks that typically require human intelligence - such as learning, reasoning, problem-solving, and decision-making. From automating repetitive tasks to augmenting human capabilities, AI has proven indispensable across multiple domains.

Take healthcare, for example. AI-powered algorithms analyze complex medical data to assist doctors in diagnosing diseases, predicting patient outcomes, and even recommending personalized treatment plans. Innovations like AI-driven robotic surgery and early cancer detection systems save lives and enhance the precision of medical procedures.

In finance, AI plays a crucial role in fraud detection, risk assessment, and automated trading. Financial institutions leverage machine learning models to analyze spending patterns, prevent cyberattacks, and provide tailored investment advice. AI’s ability to process massive amounts of data ensures accuracy and efficiency in financial operations.

Retail and e-commerce have also been transformed. AI-driven recommendation engines predict customer preferences based on purchasing behavior, allowing businesses to offer personalized shopping experiences. Chatbots streamline customer service, while automated inventory management optimizes supply chain efficiency. AI ensures that companies stay competitive in an increasingly digital marketplace.

AI and the Workplace: Automation vs. Augmentation

The integration of AI into the workplace sparks ongoing debates about its impact on jobs. While automation replaces certain manual tasks, it also opens doors for new career opportunities. AI-driven tools enable professionals to focus on creative and strategic work rather than repetitive processes.

For instance, AI-powered analytics help marketers assess consumer sentiment and predict trends with unprecedented accuracy. Similarly, AI assists engineers and researchers in designing smarter infrastructure, accelerating scientific discoveries, and optimizing business operations.

Rather than eliminating jobs, AI redefines them. Organizations must invest in upskilling employees to harness AI effectively, fostering a collaborative environment where human creativity complements machine intelligence.

Ethical Considerations and Responsible AI Development

With great power comes great responsibility. As AI grows more sophisticated, ethical concerns regarding data privacy, bias, and transparency become more pressing.

AI systems rely on large datasets to function, raising concerns about privacy and security. Governments and corporations must establish stringent regulations to protect user information from misuse or exploitation. Additionally, biases in AI models can lead to discriminatory outcomes, necessitating fair and inclusive development practices.

Transparency is another critical factor. Users should understand how AI-driven decisions are made, particularly in sensitive areas like healthcare and criminal justice. Ethical AI implementation requires collaboration between policymakers, technologists, and the public to ensure that advancements benefit society as a whole.

The Future of AI: A Balance Between Progress and Ethics

The importance of AI in society cannot be understated. It enhances productivity, improves decision-making, and solves complex problems across industries. However, its responsible use is paramount to prevent unintended consequences.

As AI continues to evolve, collaboration between academia, businesses, and governments is necessary to establish ethical guidelines and maximize its benefits. The goal should not be to replace human intelligence but to amplify it, fostering innovation while ensuring accountability.

Ultimately, AI is more than just a tool - it is a transformative force shaping the future. Harnessing its potential wisely will determine whether it serves as a steppingstone toward progress or a challenge requiring careful navigation.

Disclaimer: The whole text was generated by Copilot at the first attempt. This is just an experiment to evaluate feature's ability to answer standard general questions, independently on whether they are correctly or incorrectly posed. Moreover, the answers may reflect hallucinations and other types of inconsistent or incorrect reasoning.

|>> Next Post

#️⃣Software Engineering: Mea Culpa (Part VIII: A Look Beyond)

Software Engineering Series

With AI on the verge, blogging and bloggers can easily become obsolete. Why bother navigating through the many blogs to get a broader perspective when the same can be obtained with AI? Just type in a prompt of the type "write a blogpost of 600 words on the importance of AI in society" and Copilot or any other similar AI agent will provide you an answer that may look much better than the first draft of most of the bloggers out there! It doesn't matter whether the text follows a well-articulated idea, a personal perspective or something creative! One gets an acceptable answer with a minimum of effort and that's what matters for many.

The results tend to increase in complexity the more models are assembled together, respectively the more uncontrolled are the experiments. Moreover, solutions that tend to work aren't necessarily optimal. Machines can't offer instant enlightenment or anything close to it. Though they have an incomparable processing power of retrieval, association, aggregation, segregation and/or iteration, which coupled with the vast amount of data, information and knowledge can generate anything in just a matter of seconds. Probably, the only area in which humans can compete with machines is creativity and wisdom, though how many will be able to leverage these at scale? Probably, machines have some characteristics that can be associated with these intrinsic human characteristics, though usually more likely the brute computational power will prevail.

At Microsoft Build, Satya Nadella mentioned that foundry encompasses already more than 1900 supported models. In theory, one can still evaluate and test such models adequately. What will happen when the scale increases with a few orders of magnitude? What will happen when for each person there are one or more personalized AI models? AI can help in many areas by generating and evaluating rapidly many plausible alternatives, though as soon the models deal with some kind of processing randomization, the chances for errors increase exponentially (at least in theory).

It's enough for one or more hallucinations or other unexpected behavior to lead to more unexpected behavior. No matter how well a model was tested, as long as there's no stable predictable mathematical model behind it, the chances for something to go wrong increase with the number of inputs, parameters, uses, or changes of context the model deals with. Unfortunately, all these aspects are seldom documented. It's not like using a formula and you know that given a set of inputs and operations, the result is the same. The evolving nature of such models makes them unpredictable in the long term. Therefore, there must always be a way to observe the changes occurring in models.

One of the important questions is how many errors can we afford in such models? How long it takes until errors impact each other to create effects comparable with a tornado. And what if the tornado increases in magnitude to the degree that it wrecks everything that crosses its path? What if multiple tornadoes join forces? How many tornadoes can destroy a field, a country or a continent? How many or big must be the tornadoes to trigger a warning?

Science-Fiction authors love to create apocalyptic scenarios, and all happens in just a few steps, respectively chapters. In nature, usually it takes many orders of magnitude to generate unpredictable behavior. But, as nature often reveals, unpredictable behavior does happen, probably more often than we expect and wish for. The more we are poking the bear, the higher the chances for something unexpected to happen! Do we really want this? What will be the price we must pay for progress?

Previous Post <<||>> Next Post

#️⃣Software Engineering: Mea Culpa (Part VII: A Look Forward)

Software Engineering Series

I worked for more than 20 years in various areas related to ERP systems - Data Migrations, Business Intelligence/Analytics, Data Warehousing, Data Management, Project Management, (data) integrations, Quality Assurance, and much more, having experience with IFS IV, Oracle e-Business Suite, MS Dynamics AX 2009 and during the past 3-7 years also with MS Dynamics 365 Finance, SCM & HR (in that order). Much earlier, I started to work with SQL Server (2000-2019), Oracle, and more recently with Azure Synapse and MS Fabric, writing over time more than 800 ad-hoc queries and reports for the various stakeholders, covering all the important areas, respectively many more queries for monitoring the various environments. 

In the areas where I couldn’t acquire experience on the job, I tried to address this by learning in my free time. I did it because I take seriously my profession, and I want to know how (some) things work. I put thus a lot of time into trying to keep actual with what’s happening in the MS Fabric world, from Power BI to KQL, Python, dataflows, SQL databases and much more. These technologies are Microsoft’s bet, though at least from German’s market perspective, all bets are off! Probably, many companies are circumspect or need more time to react to the political and economic impulses, or probably some companies are already in bad shape. 

Unfortunately, the political context has a broad impact on the economy, on what’s happening in the job market right now! However, the two aspects are not the only problem. Between candidates and jobs, the distance seems to grow, a dense wall of opinion being built, multiple layers based on presumptions filtering out voices that (still) matter! Does my experience matter or does it become obsolete like the technologies I used to work with? But I continued to learn, to keep actual… Or do I need to delete everything that reminds the old?

To succeed or at least be hired today one must fit a pattern that frankly doesn’t make sense! Yes, soft skills are important though not all of them are capable of compensating for the lack of technical skills! There seems to be a tendency to exaggerate some of the qualities associated with skills, or better said, of hiding behind big words. Sometimes it feels like a Shakespearian inaccurate adaptation of the stage on which we are merely players.

More likely, this lack of pragmatism will lead to suboptimal constructions that will tend to succumb under their own structure. All the inefficiencies need to be corrected, or somebody (or something) must be able to bear their weight. I saw this too often happening in ERP implementations! Big words don’t compensate for the lack of pragmatism, skills, knowledge, effort or management! For many organizations the answer to nowadays problems is more management, which occasionally might be the right approach, though this is not a universal solution for everything that crosses our path(s).

One of society’s answers to nowadays’ problem seems to be the refuge in AI. So, I wonder – where I’m going now? Jobless, without an acceptable perspective, with AI penetrating the markets and making probably many jobs obsolete. One must adapt, but adapt to what? AI is brainless even if it can mimic intelligence! Probably, it can do more in time to the degree that many more jobs will become obsolete (and I’m wondering what will happen to all those people). 

Conversely, to some trends there will be probably other trends against them, however it’s challenging to depict in clear terms the future yet in making. Society seems to be at a crossroad, more important than mine.

Previous Post <<||>> Next Post

🧭Business Intelligence: Perspectives (Part XXXI: More on Data Visualization)

Business Intelligence Series

There are many reasons why the data visualizations available in the different mediums can be considerate as having poor quality and unfortunately there is often more than one issue that can be corroborated with this - the complexity of the data or of the models behind them, the lack of identifying the right data, respectively aspects that should be visualized, poor data visualization software or the lack of skills to use its capabilities, improper choice of visual displays, misleading choice of scales, axes and other elements, the lack of clear outlines for telling a story respectively of pushing a story too far, not adapting visualizations to changing requirements or different perspectives, to name just the most important causes.

The complexity of the data increases with the dimensions associated typically with what we call currently big data - velocity, volume, value, variety, veracity, variability and whatever V might be in scope. If it's relatively easy to work with a small dataset, understanding its shapes and challenges, our understanding power decreases with the Vs added into the picture. Of course, we can always treat the data alike, though the broader the timeframe, the higher the chances are for the data to have important changing characteristics that can impact the outcomes. It can be simple definition changes or more importantly, the model itself. Data, processes and perspectives change fluidly with the many requirements, and quite often the further implications for reporting, visualizations and other aspects are not considered.

Quite often there's a gap between what one wants to achieve with a data visualization and the data or knowledge available. It might be a matter of missing values or whole attributes that would help to delimit clearly the different perspectives or of modelling adequately the processes behind. It can be the intrinsic data quality issues that can be challenging to correct after the fact. It can also be our understanding about the processes themselves as reflected in the data, or more important, on what's missing to provide better perspectives. Therefore, many are forced to work with what they have or what they know.

Many of the data visualizations inadvertently reflect their creators' understanding about the data, procedures, processes, and any other aspects related to them. Unfortunately, also business users or other participants have only limited views and thus their knowledge must be elicited accordingly. Even then, it might be pieces of data that are not reflected in any knowledge available.

If one tortures enough data, one or more stories worthy of telling can probably be identified. However, much of the data is dull to the degree that some creators feel forced to add elements. Earlier, one could have blamed the software for it, though modern software provides nice graphics and plenty of features that can help graphics creators in the process. Even data with high quality can reveal some challenges difficult to overcome. One needs to compromise and there can be compromises in many places to the degree that one can but wonder whether the end result still reflects reality. Unfortunately, it's difficult to evaluate the impact of such gaps, however progress can be made occasionally by continuously evaluating the gaps and finding the appropriate methods to address them.

Not all stories must have complex visualizations in which multiple variables are used to provide the many perspectives. Some simple visualizations can be enough for establishing common ground on which something more complex (or simple) can be built upon. Data visualization is a continuous process of exploration, extrapolation, evaluation, testing assumptions and ideas, where one's experience can be a useful mediator between the various forces. 

Previous Post <<||>> Next Post

📊Graphical Representation: Graphics We Live By (Part XI: Comparisons Between Data Series)

Graphical Representation Series

Over the past 10-20 years it became so easy to create data visualizations just by dropping some of the data available into a tool like Excel and providing a visual depiction of it with just a few clicks. In many cases, the first draft, typically provided by default in the tool used, doesn't even need further work as the objective was reached, while in others the creator must have a minimum skillset for making the visualization useful, appealing, or whatever quality is a final requirement for the work in scope. However, the audience might judge the visualization(s) from different perspectives, and there can be a broad audience with different skills in reading, evaluating and understanding the work.

There are many depictions on the web resembling the one below, taken from a LinkedIn post:

Example Chart - Boing vs. Airbus

Even if the visualization is not perfect, it does a fair job in representing the data. Improvements can be made in the areas of labels, the title and positioning of elements, and the color palette used. At least these were the improvements made in the original post. It must be differentiated also between the environment in which the charts are made available, the print format having different characteristics than the ones in business setups. Unfortunately, the requirements of the two are widely confused, probably also because of the overlapping of the mediums used. 

Probably, it's a good idea to always start with the row data (or summaries of it) when the result consists of only a few data points that can be easily displayed in a table like the one below (the feature to round the decimals for integer values should be available soon in Power BI):

Summary Table

Of course, one can calculate more meaningful values like percentages from the total, standard deviations and other values that offer more perspectives into the data. Even if the values adequately reflect the reality, the reader can but wonder about the local and global minimal/maximal values, without talking much about the meaning of data points, which is easily identifiable in a chart. At least in the case of small data sets, using a table in combination with a chart can provide a more complete perspective and different ways of analyzing the data, especially when the navigation is interactive. 

Column and bar charts do a fair job in comparing values over time, though they do use a lot of ink in the process (see D). While they make it easy to compare neighboring values, the rectangles used tend to occupy a lot of space when they are made too wide or too high to cover the empty space within the display (e.g. when just a few values are displayed, space being wasted in the process). As the main downside, it takes a lot of scanning until the reader identifies the overall trends, and the further away the bars are from each other, the more difficult it becomes to do comparisons. 

In theory, line charts are more efficient in representing the above data points, because the marks are usually small and the line thin enough to provide a better data-ink ratio, while one can see a lot at a glance. In Power BI the creator can use different types of interpolation: linear (A), step (B) or smooth (C). In many cases, it might be a good idea to use a linear interpolation, though when there are no or minimal overlapping, it might be worthwhile to explore the other types if interpolation too (and further request feedback from the users):

Linear, Step and Smooth Line Charts

The nearness of values from different series can raise difficulties in identifying adequately the points, respectively delimiting the lines (see B).When the density of values allows it, it makes sense also to include the averages for each data series to reflect the distance between the two data sets. Unfortunately, the chart can get crowded if further data series or summaries are added to the cart(s). 

If the column chart (E) is close to the redesigned chart provided in the original redesign, the other alternatives can provide upon case more value. Stacked column charts (D) allow also to compare the overall quantity by month, area charts (F) tend to use even more color than needed, while water charts (G) allow to compare the difference between data points per time unit. Tornado charts (H) are a variation of bar charts, allowing easier comparing of the size of the bars, while ribbon charts (I) show well the stacking values. 

Alternatives to Line Charts

One should consider changing the subtitle(s) slightly to reflect the chart type when the patterns shown imply a shift in attention or meaning. Upon case, more that one of the above charts can be used within the same report when two or more perspectives are important. Using a complementary perspective can facilitate data's understanding or of identifying certain patterns that aren't easily identifiable otherwise. 

In general, the graphics creators try to use various representational means of facilitating a data set's understanding, though seldom only two series or a small subset of dimensions provide a complete description. The value of data comes when multiple perspectives are combined. Frankly, the same can be said about the above data series. Yes, there are important differences between the two series, though how do the numbers compare when one looks at the bigger picture, especially when broken down on element types (e.g. airplane size). How about plan vs. actual values, how long does it take more for production or other processes? It's one of a visualization's goals to improve the questions posed, but how efficient are visualizations that barely scratch the surface?

In what concerns the code, the following scripts can be used to prepare the data:

-- Power Query script (Boeing vs Airbus)
= let
    Source = let
    Source = #table({"Sorting", "Month Name", "Serial Date", "Boeing Deliveries", "Airbus Deliveries"},
    {
        {1, "Oct", #date(2023, 10, 31), 30, 50},
        {2, "Nov", #date(2023, 11, 30), 40, 40},
        {3, "Dec", #date(2023, 12, 31), 40, 110},
        {4, "Jan", #date(2024, 1, 31), 20, 30},
        {5, "Feb", #date(2024, 2, 29), 30, 40},  // Leap year adjustment
        {6, "Mar", #date(2024, 3, 31), 30, 60},
        {7, "Apr", #date(2024, 4, 30), 40, 60},
        {8, "May", #date(2024, 5, 31), 40, 50},
        {9, "Jun", #date(2024, 6, 30), 50, 80},
        {10, "Jul", #date(2024, 7, 31), 40, 90},
        {11, "Aug", #date(2024, 8, 31), 40, 50},
        {12, "Sep", #date(2024, 9, 30), 30, 50}
    }
    ),
    #"Changed Types" = Table.TransformColumnTypes(Source, {{"Sorting", Int64.Type}, {"Serial Date", type date}, {"Boeing Deliveries", Int64.Type}, {"Airbus Deliveries", Int64.Type}})
in
    #"Changed Types"
in
    Source

It can be useful to create the labels for the charts dynamically:

-- DAX code for labels
MaxDate = Format(Max('Boeing vs Airbus'[Serial Date]),"MMM-YYYY")
MinDate = FORMAT (Min('Boeing vs Airbus'[Serial Date]),"MMM-YYYY")
MinMaxDate = [MinDate] & " to " & [MaxDate]
Title Boing Airbus = "Boing and Airbus Deliveries " & [MinMaxDate]

Happy coding!

Previous Post <<||>> Next Post