🧭Business Intelligence: Mea Culpa (Part III: Problem Solving)

Business Intelligence Series

I've been working for more than 20 years in BI and Data Analytics area, in combination with Software Engineering, ERP implementations, Project Management, IT services and several other areas, which allowed me to look at many recurring problems from different perspectives. One of the things I learnt is that problems are more complex and more dynamic than they seem, respectively that they may require tailored dynamic solutions. Unfortunately, people usually focus on one or two immediate perspectives, ignoring the dynamics and the multilayered character of the problems!

Sometimes, a quick fix and limited perspective is what we need to get started and fix the symptoms, and problem-solvers usually stop there. When left unsupervised, the problems tend to kick back, build up momentum and appear under more complex forms in various places. Moreover, the symptoms can remain hidden until is too late. To this also adds the political agendas and the further limitations existing in organizations (people, money, know-how, etc.).

It seems much easier to involve external people (individual experts, consultancy companies) to solve the problem(s), though unless they get a deep understanding of the business and the issues existing in it, the chances are high that they solve the wrong problems and/or implement the wrong solutions. Therefore, it's more advisable to have internal experts, when feasible, and that's the point where business people with technical expertise and/or IT people with business expertise can help. Ideally, one should have a good mix and the so called competency centers can do a great job in handling the challenges of organizations. 

Between business and IT people there's a gap that can be higher or lower depending on resources know-how or the effort made by organizations to reduce it. To this adds the nature of the issues existing in organizations, which can vary considerable across departments, organizations or any other form of establishment. Conversely, the specific skillset can be transmuted where needed, which might happen naturally, though upon case also considerable effort needs to be involved in the process.

Being involved in similar tasks, one may get the impression that one can do whatever the others can do. This can happen in IT as well on the business side. There can be activities that can be done by parties from the other group, though there are also many exceptions in both directions, especially when one considers that one can’t generalize the applicability and/or transmutation of skillset. 

A more concrete example is the know-how needed by a businessperson to use the BI infrastructure for answering business questions, and ideally for doing all or at least most of the activities a BI professional can do. Ideally, as part of the learning path, it would be helpful to have a pursuable path in between the two points. The mastery of tools helps in the process though there are different mindsets involved.

Unfortunately, the data-related fields are full of overconfident people who get the problem-solving process wrong. Data-based problem-solving resumes in gathering the right facts and data, building the right conceptual model, identifying the right questions to ask, collecting more data, refining methods and solutions, etc. There’s aways an easy wrong way to solve a problem!

The mastery of tools doesn’t imply the mastery of business domains! What people from the business side can bring is deeper insight in the business problems, though getting from there to implementing solutions can prove a long way, especially when problems require different approaches, different levels of approximations, etc. No tool alone can bridge such gaps yet! Frankly, this is the most difficult to learn and unfortunately many data professionals seem to get this wrong!

Previous Post <<||>> Next Post

🧭🏭Business Intelligence: Microsoft Fabric (Part V: One Person Can’t Learn or Do Everything)

Business Intelligence Series

Today’s Explicit Measures webcast [1] considered an article written by Kurt Buhler (The Data Goblins): [Microsoft] "Fabric is a Team Sport: One Person Can’t Learn or Do Everything" [2]. It’s a well-written article that deserves some thought as there are several important points made. I can’t say I agree with the full extent of some statements, even if some disagreements are probably just a matter of semantics.

My main disagreement starts with the title “One Person Can’t Learn or Do Everything”. As clarified in webcast's chat, the author defines “everything" as an umbrella for “all the capabilities and experiences that comprise Fabric including both technical (like Power BI) or non-technical (like adoption data literacy) and everything in between” [1].

For me “everything” is relative and considers a domain's core set of knowledge, while "expertise" (≠ "mastery") refers to the degree to which a person can use the respective knowledge to build back-to-back solutions for a given area. I’d say that it becomes more and more challenging for beginners or average data professionals to cover the core features. Moreover, I’d separate the non-technical skills because then one will also need to consider topics like Data, Project, Information or Knowledge Management.

There are different levels of expertise, and they can vary in depth (specialization) or breadth (covering multiple areas), respectively depend on previous experience (whether one worked with similar technologies). Usually, there’s a minimum of requirements that need to be covered for being considered as expert (e.g. certification, building a solution from beginning to the end, troubleshooting, performance optimization, etc.). It’s also challenging to roughly define when one’s expertise starts (or ends), as there are different perspectives on the topics. 

Conversely, the term expert is in general misused extensively, sometimes even with a mischievous intent. As “expert” is usually considered an external consultant or a person who got certified in an area, even if the person may not be able to build solutions that address a customer’s needs. 

Even data professionals with many years of experience can be overwhelmed by the volume of knowledge, especially when one considers the different experiences available in MF, respectively the volume of new features released monthly. Conversely, expertise can be considered in respect to only one or more MF experiences or for one area within a certain layer. Lot of the knowledge can be transported from other areas – writing SQL and complex database objects, modelling (enterprise) semantic layers, programming in Python, R or Power Query, building data pipelines, managing SQL databases, etc. 

Besides the standard documentation, training sessions, and some reference architectures, Microsoft made available also some labs and other material, which helps discovering the features available, though it doesn’t teach people how to build complete solutions. I find more important than declaring explicitly the role-based audience, the creation of learning paths for the various roles.

During the past 6-7 months I've spent on average 2 days per week learning MF topics. My problem is not the documentation but the lack of maturity of some features, the gaps in functionality, identifying the respective gaps, knowing what and when new features will be made available. The fact that features are made available or changed while learning makes the process more challenging. 

My goal is to be able to provide back-to-back solutions and I believe that’s possible, even if I might not consider all the experiences available. During the past 22 years, at least until MF, I could build complete BI solutions starting from requirements elicitation, data extraction, modeling and processing for data consumption, respectively data consumption for the various purposes. At least this was the journey of a Software Engineer into the world of data. 

References:
[1] Explicit Measures (2024) Power BI tips Ep.328: Microsoft Fabric is a Team Sport (link)
[2] Data Goblins (2024) Fabric is a Team Sport: One Person Can’t Learn or Do Everything (link)

🧭Business Intelligence: New Technologies, Old Challenges (Part I: An Introduction)

Each important technology has the potential of creating divides between the specialists from a given field. This aspect is more suggestive in the data-driven fields like BI/Analytics or Data Warehousing. The data professionals (engineers, scientists, analysts, developers) skilled only in the new wave of technologies tend to disregard the role played by the former technologies and their role in the data landscape. The argumentation for such behavior is rooted in the belief that a new technology is better and can solve any problem better than previous technologies did. It’s a kind of mirage professionals and customers can easily fall under.

Being bigger, faster, having new functionality, doesn’t make a tool the best choice by default. The choice must be rooted in the problem to be solved and the set of requirements it comes with. Just because a vibratory rammer is a new technology, is faster and has more power in applying pressure, this doesn’t mean that it will replace a hammer. Where a certain type of power is needed the vibratory rammer might be the best tool, while for situations in which a minimum of power and probably more precision is needed, like driving in a nail, then an adequately sized hammer will prove to be a better choice.

A technology is to be used in certain (business/technological) contexts, and even if contexts often overlap, the further details (aka requirements) should lead to the proper use of tools. It’s in a professional’s duties to be able to differentiate between contexts, requirements and the capabilities of the tools appropriate for each context. In this resides partially a professional’s mastery over its field of work and of providing adequate solutions for customers’ needs. Especially in IT, it’s not enough to master the new tools but also have an understanding about preceding tools, usage contexts, capabilities and challenges.

From an historical perspective each tool appeared to fill a demand, and even if maybe it didn’t manage to fill it adequately, the experience obtained can prove to be valuable in one way or another. Otherwise, one risks reinventing the wheel, or more dangerously, repeating the failures of the past. Each new technology seems to provide a deja-vu from this perspective.

Moreover, a new technology provides new opportunities and requires maybe to change our way of thinking in respect to how the technology is used and the processes or techniques associated with it. Knowledge of the past technologies help identifying such opportunities easier. How a tool is used is also a matter of skills, while its appropriate use and adoption implies an inherent learning curve. Having previous experience with similar tools tends to reduce the learning curve considerably, though hands-on learning is still necessary, and appropriate learning materials or tutoring is upon case needed for a smoother transition.

In what concerns the implementation of mature technologies, most of the challenges were seldom the technologies themselves but of non-technical nature, ranging from the poor understanding/knowledge about the tools, their role and the implications they have for an organization, to an organization’s maturity in leading projects. Even the most-advanced technology can fail in the hands of non-experts. Experience can’t be judged based only on the years spent in the field or the number of projects one worked on, but on the understanding acquired about implementation and usage’s challenges. These latter aspects seem to be widely ignored, even if it can make the difference between success and failure in a technology’s implementation.

Ultimately, each technology is appropriate in certain contexts and a new technology doesn’t necessarily make another obsolete, at least not until the old contexts become obsolete.

Previous Post <<||>>Next Post

🧭Business Intelligence: Perspectives (Part VI: Data Soup - Reports vs. Data Visualizations)

Business Intelligence Series

Considering visualizations, John Tukey remarked that ‘the greatest value of a picture is when it forces us to notice what we never expected to see’, which is not always the case for many of the graphics and visualizations available in organizations, typically in the form of simple charts and dashboards, quite often with no esthetics or meaning behind.

In general, reports are needed as source for operational activities, in which the details in form of raw or aggregate data are important. As one moves further to the tactical or strategic aspects of a business, visualizations gain in importance especially when they allow encoding data and information, respectively variations, trends or relations in smaller places with minimal loss of information.

There are also different aspects of visualizations that need to be considered. Modern tools allow rapid visualization and interactive navigation of data across different variables which is great as long one knows what is searching for, which is not always the case.

There are junk charts in which the data drowns in graphical elements that bring no value to the reader, in extremis even distorting the message/meaning.

There are graphics/visualizations that attempt bringing together and encoding multiple variables in respect to a theme, and for which a ‘project’ is typically needed as data is not ad-hoc available, don’t have the desired quality or need further transformations to be ready for consumption. Good quality graphics/visualizations require time and a good understanding of the business, which are not necessarily available into the BI/Analytics teams, and unfortunately few organizations do something in that direction, ignoring typically such needs. In this type of environments is stressed the rapid availability of data for decision-making or action-relevant insight, which depends typically on the consumer.

The story-telling capabilities of graphics/visualizations are often exaggerated. Yes, they can tell a story though stories need to be framed into a context/problem, some background and further references need to be provided, while without detailed data the graphics/visualizations are just nice representations in which each consumer understands what he can.

In an ideal world the consumer and the ‘designer’ would work together to identify the important data for the theme considered, to find the appropriate level of detail, respectively the best form of encoding. Such attempts can stop at table-based representations (aka reports), respectively basic or richer forms of graphical representations. One can consider reports as an early stage of the visualization process, with the potential to derive move value when the data allow meaningful graphical representations. Unfortunately, the time, data and knowledge available seldom make this achievable.

In addition, a well-designed report can be used as basis for multiple purposes, while a graphic/visualization can enforce more limitations. Ideal would be when multiple forms of representation (including reports) are combined to harness the value of data. Navigations from visualizations to detailed data can be useful to understand what happens; learning and understanding the various aspects being an iterative process.

It’s also difficult to demonstrate the value of insight derived from visualizations, especially when graphical literacy goes behind the numeracy and statistical literacy - many consumers lacking the skills needed to evaluate numbers and statistics adequately. If for a good artistic movie you need an assistance to enjoy the show and understand the message(s) behind it, the same can be said also about good graphics/visualizations. Moreover, this requires creativity, abstraction-based thinking, and other capabilities to harness the value of representations.

Given the considerable volume of requirements related to the need of basis data, reports will continue to be on high demand in organizations. In exchange visualizations can complement them by providing insights otherwise not available.

Initially published on Medium as answer to a post on Reporting and Visualizations. 

Data Science: Data Strategy (Part II: Generalists vs Specialists in the Field)

Division of labor favorizes the tasks done repeatedly, where knowledge of the broader processes is not needed, where aspects as creativity are needed only at a small scale. Division invaded the IT domains as tools, methodologies and demands increased in complexity, and therefore Data Science and BI/Analytics make no exception from this.

The scale of this development gains sometimes humorous expectations or misbelieves when one hears headhunters asking potential candidates whether they are upfront or backend experts when a good understanding of both aspects is needed for providing adequate results. The development gains tragicomical implications when one is limited in action only to a given area despite the extended expertise, or when a generalist seems to step on the feet of specialists, sometimes from the right entitled reasons. 

Headhunters’ behavior is rooted maybe in the poor understanding of the domain of expertise and implications of the job descriptions. It’s hard to understand how people sustain of having knowledge about a domain just because they heard the words flying around and got some glimpse of the connotations associated with the words. Unfortunately, this is extended to management and further in the business environment, with all the implications deriving from it. 

As Data Science finds itself at the intersection between Artificial Intelligence, Data Mining, Machine Learning, Neurocomputing, Pattern Recognition, Statistics and Data Processing, the center of gravity is hard to determine. One way of dealing with the unknown is requiring candidates to have a few years of trackable experience in the respective fields or in the use of a few tools considered as important in the respective domains. Of course, the usage of tools and techniques is important, though it’s a big difference between using a tool and understanding the how, when, why, where, in which ways and by what means a tool can be used effectively to create value. This can be gained only when one’s exposed to different business scenarios across industries and is a tough thing to demand from a profession found in its baby steps. 

Moreover, being a good data scientist involves having a deep insight into the businesses, being able to understand data and the demands associated with data – the various qualitative and quantitative aspects. Seeing the big picture is important in defining, approaching and solving problems. The more one is exposed to different techniques and business scenarios, with right understanding and some problem-solving skillset one can transpose and solve problems across domains. However, the generalist will find his limitations as soon a certain depth is reached, and the collaboration with a specialist is then required. A good collaboration between generalists and specialists is important in complex projects which overreach the boundaries of one person’s knowledge and skillset. 

Complexity is addressed when one can focus on the important characteristic of the problem, respectively when the models built can reflect the demands. The most important skillset besides the use of technical tools is the ability to model problems and root the respective problems into data, to elaborate theories and check them against reality. 

Complex problems can require specialization in certain fields, though seldom one problem is dependent only on one aspect of the business, as problems occur in overreaching contexts that span sometimes the borders of an organization. In addition, the ability to solve problems seem to be impacted by the diversity of the people involved into the task, sometimes even with backgrounds not directly related to organization’s activity. As in evolution, a team’s diversity is an important factor in achievement and learning, most gain being obtained when knowledge gets shared and harnessed beyond the borders of teams.

Note:
Written as answer to a Medium post on Data Science generalists vs specialists.

Data Science: Data Strategy (Part I: Big Data vs. Business Strategies)

A strategy, independently on whether applied to organizations, chess, and other situations, allows identifying the moves having the most promising results from a range of possible moves that can change as one progresses into the game. Typically, the moves compete for same or similar resources, each move having at the respective time a potential value expressed in quantitative and/or qualitative terms, while the values are dependent on the information available about one’s and partners’ positions into the game. Therefore, a strategy is dependent on the decision-making processes in place, the information available about own business, respective the concurrence, as well about the game.

Big data is not about a technology but an umbrella term for multiple technologies that support in handling data with high volume, veracity, velocity or variety. The technologies attempt helping organizations in harnessing what is known as Big data (data having the before mentioned characteristics), for example by allowing answering to business questions, gaining insight into the business or market, improving decision-making. Through this Big data helps delivering value to businesses, at least in theory.

Big-data technologies can harness all data of an organization though this doesn’t imply that all data can provide value, especially when considered in respect to the investments made. Data bring value when they have the potential of uncovering hidden trends or (special) patterns of behavior, when they can be associated in new meaningful ways. Data that don’t reflect such characteristics are less susceptible of bringing value for an organization no matter how much one tries to process the respective data. However, looking at the data through multiple techniques can help organization get a better understanding of the data, though here is more about the processes of attempting understanding the data than the potential associated directly with the data.

Through active effort in understanding the data one becomes aware of the impact the quality of data have on business decisions, on how the business and processes are reflected in its data, how data can be used to control processes and focus on what matters. These are aspects that can be corroborated with the use of simple BI capabilities and don’t necessarily require more complex capabilities or tools. Therefore allowing employees the time to analyze and play with the data, can in theory have a considerable impact on how data are harnessed within an organization.

If an organization’s decision-making processes is dependent on actual data and insight (e.g. stock market) then the organization is more likely to profit from it. In opposition, organizations whose decision-making processes hand handle hours, days or months of latency in their data, then more likely the technologies will bring little value. Probably can be found similar examples for veracity, variety or similar characteristics consider under Big data.

The Big data technologies can make a difference especially when the extreme aspects of their characteristics can be harnessed. One talks about potential use which is different than the actual use. The use of technologies doesn’t equate with results, as knowledge about the tools and the business is mandatory to harness the respective tools. For example, insight doesn’t necessarily imply improved decision-making because it relies on people’s understanding about the business, about the numbers and models used.

That’s maybe one of the reasons why organization fail in deriving value from Big data. It’s great that companies invest in their Big data, Analytics/BI infrastructures, though without working actively in integrating the new insights/knowledge and upgrading people’s skillset, the effects will be under expectations. Investing in employees’ skillset is maybe one of the important decisions an organization can make as part of its strategy.

Note:
Written as answer to a Medium post on Big data and business strategies. 

🎓Knowledge Management: Keeping Current or the Quest to Lifelong Learning for IT Professionals

Introduction

    The pace with which technologies and the business changes becomes faster and faster. If 5-10 years back a vendor needed 3-5 years before coming with a new edition of a product, nowadays each 1-2 years a new edition is released. The release cycles become shorter and shorter, vendors having to keep up with the changing technological trends. Changing trends allow other vendors to enter the market with new products, increasing thus the competition and the need for responsiveness from other vendors. On one side the new tools/editions bring new functionality which mainly address technical and business requirements. On the other side existing tools functionality gets deprecated and superset by other. Knowledge doesn’t resume only to the use of tools, but also in the methodologies, procedures, best practices or processes used to make most of the respective products. Evermore, the value of some tools increases when mixed, flexible infrastructures relying on the right mix of tools working together.

    For an IT person keeping current with the advances in technologies is a major requirement. First of all because knowing modern technologies is a ticket for a good and/or better paid job. Secondly because many organizations try to incorporate in their IT infrastructure modern tools that would allow them increase the ROI and achieve further benefits. Thirdly because, as I’d like to believe, most of the IT professionals are eager to learn new things, keep up with the novelty. Being an adept of the continuous learning philosophy is also a way to keep the brain challenged, other type of challenge than the one we meet in daily tasks.

Knowledge Sources

    Face-to-face or computer-based trainings (CBTs) are the old-fashioned ways of keeping up-to-date with the advances in technologies though paradoxically not all organizations afford to train their IT employees. Despite of affordable CBTs, face-to-face trainings are quite expensive for the average IT person, therefore the IT professional has to reorient himself to other sources of knowledge. Fortunately many important Vendors like Microsoft or IBM provide in one form or another through Knowledge Bases (KB), tutorials, forums, presentations and Blogs a wide range of resources that could be used for learning. Similar resources exist also from similar parties, directly or indirectly interested in growing the knowledge pool.

    Nowadays reading a book or following a course it isn’t anymore a requirement for learning a subject. Blogs, tutorials, articles and other types of similar material can help more. Through their subject-oriented focus, they can bring some clarity in a small unit of time. Often they come with references to further materials, bring fresh perspectives, and are months or even years ahead books or courses. Important professionals in the field can be followed on blogs, Twitter, LinkedIn, You Tube and other social media platforms. Seeing in what topics they are interested in, how they code, what they think, maybe how they think, some even share their expertize ad-hoc when asked, all of this can help an IT professional considerably if he knows how to take advantage of these modern facilities.

    MOOCs start to approach IT topics, and further topics that can become handy for an IT professional. Most of them are free or a small fee is required for some of them, especially if participants’ identity needs to be verified. Such courses are a valuable resource of information. The participant can see how such a course is structured, what topics are approached, and what’s the minimal knowledge base required; the material is almost the same as in a normal university course, and in the end it’s not the piece of paper with the testimonial that’s important, but the change in perspective we obtained by taking the course. In addition the MOOC participant can interact with people with similar hobbies, collaborate with them on projects, and why not, something useful can come out of it. Through MOOCs or direct Vendor initiatives, free or freeware versions of software is available. Sometimes the whole functionality is available for personal use. The professional is therefore no more dependent on the software he can use only at work. New possibilities open for the person who wants to learn.

Maximizing the Knowledge Value

    Despite the considerable numbers of knowledge resources, for an IT professional the most important part of his experience comes from hand-on experience acquired on the job. If the knowledge is not rooted in hand-on experience, his knowledge remains purely theoretical, with minimal value. Therefore in order to maximize the value of his learning, an IT professional has to attempt using his knowledge as much and soon as possible in praxis. One way to increase the value of experience is to be involved in projects dealing with new technologies or challenges that would allow a professional to further extend his knowledge base. Sometimes we can choose such projects or gain exposure to the technologies, though other times no such opportunities can be sized or identified.

    Probably an IT professional can use in his daily duties 10-30% of what he learned. This percentage can be however increased by involving himself in other types of personal or collective (open source or work) projects. This would allow exploring the subjects from other perspective. Considering that many projects involve overtime, many professionals have also a rich personal life, it looks difficult to do that, though not impossible.

    Even if not on a regular basis achievable, a professional can allocate 1-3 hours on a weekly basis from his working time for learning something new. It can be something that would help directly or indirectly his organization, though sometimes it pays off to learn technologies that have nothing to do with the actual job. Somebody may argue that the respective hours are not “billable”, are a waste of time and other resources, that the technologies are not available, that there’s lot of due tasks, etc. With a little benevolence and with the right argumentation also such criticism can be silenced. The arguments can be for example based on the fact that a skilled professional can be with time more productive, a small investment in knowledge can have later a bigger benefit for both parties – employee and employer. An older study was showing that when IT professionals was given some freedom to approach personal projects at work, and use some time for their own benefit, the value they bring for an organization increased. There are companies like Google who made from this type of work a philosophy.

    A professional can also allocate 1-3 hours from his free time while commuting or other similar activities. Reading something before going to bed or as relaxation after work can prove to be a good shut-down for the brain from the daily problems. Where there’s interest in learning something new a person will find the time, no matter how busy his schedule is. It’s important however to do that on a regular basis, and with time the hours and knowledge accumulate.

    It’s also important to have a focused effort that will bring some kind of benefit. Learning just for the sake of learning brings little value on investment for a person if it’s not adequately focused. For sure it’s interesting and fun to browse through different topics, it’s even recommended to do so occasionally, though on the long run if a person wants to increase the value of his knowledge, he needs somehow to focus the knowledge within a given direction and apply that knowledge.

    Direction we obtain by choosing a career or learning path, and focusing on the direct or indirect related topics that belong to that path. Focusing on the subjects related to a career path allows us to build our knowledge further on existing knowledge, understanding a topic fully. On the other side focusing on other areas of applicability not directly linked with our professional work can broaden our perspective by looking at one topic from another’s topic perspective. This can be achieved for example by joining the knowledge base of a hobby we have with the one of our professional work. In certain configurations new opportunities for joint growth can be identified.

    The value of knowledge increases primarily when it’s used in day-to-day scenarios (a form of learning by doing). It would be useful for example for a professional to start a project that can bring some kind of benefit. It can be something simple like building a web page or a full website, an application that processes data, a solution based on a mix of technologies, etc. Such a project would allow simulating to some degree day-to-day situations, when the professional is forced to used and question some aspects, to deal with some situations that can’t be found in textbook or other learning material. If such a project can bring a material benefit, the value of knowledge increases even more.

    Another way to integrate the accumulated knowledge is through blogging and problem-solving. Topic or problem-oriented blogging can allow externalizing a person’s knowledge (aka tacit knowledge), putting knowledge in new contexts into a small focused unit of work, doing some research and see how other think about the same topic/problem, getting feedback, correcting or improving some aspects. It’s also a way of documenting the various problems identified while learning or performing a task. Blogging helps a person to improve his writing communication skills, his vocabulary and with a little more effort can be also a visit card for his professional experience.

    Trying to apply new knowledge in hand-on trainings, tutorials or by writing a few lines of code to test functionality and its applicability, same as structuring new learned material into notes in the form of text or knowledge maps (e.g. concept maps, mind maps, causal maps, diagrams, etc.) allow learners to actively learn the new concepts, increasing overall material’s retention. Even if notes and knowledge maps don’t apply the learned material directly, they offer a new way of structuring the content and resources for further enrichment and review. Applied individually, but especially when combined, the different types of active learning help as well maximize the value of knowledge with a minimum of effort.

Conclusion

    The bottom line – given the fast pace with which new technologies enter the market and the business environment evolves, an IT professional has to keep himself up-to-date with nowadays technologies. He has now more means than ever to do that – affordable computer-based training, tutorials, blogs, articles, videos, forums, studies, MOOC and other type of learning material allow IT professionals to approach a wide range of topics. Through active, focused, sustainable and hand-on learning we can maximize the value of knowledge, and in the end depends of each of us how we use the available resources to make most of our learning experience.

✨Performance Management: Skills (Just the Quotes)

"By far the most valuable possession is skill. Both war and the chances of fortune destroy other things, but skill is preserved." Hipparchus, Commentaries, 2nd century BC)

"Let a man practice the profession which he best knows." (Cicero, "Tusculanarum Disputationum", cca. 45 BC)

"Numeracy has two facets - reading and writing, or extracting numerical information and presenting it. The skills of data presentation may at first seem ad hoc and judgmental, a matter of style rather than of technology, but certain aspects can be formalized into explicit rules, the equivalent of elementary syntax." (Andrew Ehrenberg, "Rudiments of Numeracy", Journal of Royal Statistical Society, 1977)

"Five coordinating mechanisms seem to explain the fundamental ways in which organizations coordinate their work: mutual adjustment, direct supervision, standardization of work processes, standardization of work outputs, and standardization of worker skills." (Henry Mintzberg, "The Structuring of Organizations", 1979)

"Training is the teaching of specific skills. It should result in the employee having the ability to do something he or she could not do before." (Mary A Allison & Eric Anderson, "Managing Up, Managing Down", 1984)

"The skills that make technical professionals competent in their specialties are not necessarily the same ones that make them successful within their organizations." (Bernard Rosenbaum, "Training", 1986)

"[…] data analysis in the context of basic mathematical concepts and skills. The ability to use and interpret simple graphical and numerical descriptions of data is the foundation of numeracy […] Meaningful data aid in replacing an emphasis on calculation by the exercise of judgement and a stress on interpreting and communicating results." (David S Moore, "Statistics for All: Why, What and How?", 1990)

"[By understanding] I mean simply a sufficient grasp of concepts, principles, or skills so that one can bring them to bear on new problems and situations, deciding in which ways one’s present competencies can suffice and in which ways one may require new skills or knowledge." (Howard Gardner, "The Unschooled Mind", 1991)

"Education is not the piling on of learning, information, data, facts, skills, or abilities - that's training or instruction - but is rather making visible what is hidden as a seed." (Thomas W Moore, "The Education of the Heart", 1996)

"Even when you have skilled, motivated, hard-working people, the wrong team structure can undercut their efforts instead of catapulting them to success. A poor team structure can increase development time, reduce quality, damage morale, increase turnover, and ultimately lead to project cancellation." (Steve McConnell, "Rapid Development", 1996)

"Success or failure of a project depends upon the ability of key personnel to have sufficient data for decision-making. Project management is often considered to be both an art and a science. It is an art because of the strong need for interpersonal skills, and the project planning and control forms attempt to convert part of the 'art' into a science." (Harold Kerzner, "Strategic Planning for Project Management using a Project Management Maturity Model", 2001)

"Even with simple and usable models, most organizations will need to upgrade their analytical skills and literacy. Managers must come to view analytics as central to solving problems and identifying opportunities - to make it part of the fabric of daily operations." (Dominic Barton & David Court, "Making Advanced Analytics Work for You", 2012)

"The biggest thing to know is that data visualization is hard. Really difficult to pull off well. It requires harmonization of several skills sets and ways of thinking: conceptual, analytic, statistical, graphic design, programmatic, interface-design, story-telling, journalism - plus a bit of ‘gut feel.’ The end result is often simple and beautiful, but the process itself is usually challenging and messy." (David McCandless, 2013)

"Finding the right answer is important, of course. But more important is developing the ability to see that problems have multiple solutions, that getting from X to Y demands basic skills and mental agility, imagination, persistence, patience." (Mary H Futrell)

"Productivity is the name of the game, and gains in productivity will only come when better understanding and better relationships exist between management and the work force. [...] Managers have traditionally developed the skills in finance, planning, marketing and production techniques. Too often the relationships with their people have been assigned a secondary role. This is too important a subject not to receive first-line attention." (William Hewlett, "The Human Side of Management", [speech])

"Solving problems is a practical skill like, let us say, swimming. We acquire any practical skill by imitation and practice." (George Polya)

🌡️Performance Management: Competency (Definitions)

"An ability to perform business processes, which are supported by necessary available resources, practices, and activities, allowing the organization to offer products/services." (Jiri Hodík et al, "e-Cat for Partner Profiling and Competency Management Tool", 2008)

"Present or target capacity of a group or an individual to perform a cognitive, affective, social or psychomotor skill with regard to certain area of knowledge and in a specific context. The context consists in defining whether the skill can be attributed to the knowledge in a guided or autonomous way, in simple or complex, familiar or new situations, in a global or partial, persistent or sporadic manner." (Gilbert Paquette et al, "Principled Construction and Reuse of Learning Designs", Handbook of Research on Learning Design and Learning Objects: Issues, Applications, and Technologies, 2009)

"The ability to do something successfully or efficiently, often broken down into skills, knowledge, and attitude." (Alfonso Urquiza, "Competency Management Information Systems", 2009)

"The underlying characteristics of an individual (a motive, trait, skill, aspect of one’s self image or social role, or a body of knowledge) which underlie performance or behavior at work." (Jorge Valdés-Conca & Lourdes Canós-Darós, "B2E Relationships, Intranets, and Competency Management", 2009)

"A cluster of knowledges, understandings, skills, attitudes, values, and interests that are required for the performance of a function. In this case the function would be to be competent in counseling adult learners." (John A Henschke, "Counseling in an Andragogical Approach", 2012)

"A specific, identifiable, definable, and measurable knowledge, skill, ability, and/or other deployment-related characteristic (e.g., attitude, behavior, physical ability) which a human resource may possess and which is necessary for, or material to, the performance of an activity within a specific business context." Nancy B Hastings & Karen L Rasmussen, "Designing and Developing Competency-Based Education Courses Using Standards", 2017)

"Competency is the ability to demonstrate a specified level of knowledge or skill." (Christine K S Irvine & Jonathan M Kevan, "Competency-Based Education in Higher Education", 2017)

"Expected capacity the learner should build to be successful in his/her career. Competency is written in broader terms and are not directly measurable." (Devrim Ozdemir & Carla Stebbins, "A Framework for the Evaluation of Competency-Based Curriculum", 2017)

"Competencies are specific knowledge-based skills, abilities, or expertise in a subject area. When these skillsets are shared across a profession, they are said to have core competencies." (Valerie A Storey et al, "Developing a Clinical Leadership Pipeline: Planning, Operation, and Sustainability", 2019)

"Multidimensional construct which represents what a person is capable of doing. It includes knowledge, skills, experience, abilities, values, attitudes, personality traits, among others." (Geraldina Silveyra et al, "Proposal of a Comprehensive Model of Teachable Entrepreneurship Competencies (M-TEC): Literature Review and Theoretical Foundations", 2019)

"The ability to act successfully on the basis of practical experience, skill, and knowledge in solving professional problems. Is understood as a formal system characteristic, which is described as a set of requirements for the knowledge, skills and qualities of the employee for a function, position or role in the organization." (Vitaly V Martynov et al, "CSRP: System Design Technology of Training Information Support of Competent Professionals", 2019)

"A guiding tool including knowledge, abilities, distinguished personal attributes, and behaviours for higher performance contributing to achieving strategic goals of the company." (Mustafa K Topcu, "Competency Framework for the Fourth Industrial Revolution", 2020)

"Proficiency or mastery of identified knowledge, skills or abilities." (Ernst Jan van Weperen et al, "Sustainable Entrepreneurial Thinking: Developing Pro-Active, Globally Aware Citizens", 2020)

"Capacity to perform something in an effective manner. Involves individual attitudes, knowledge, and skills necessary, and behaviors." (Christiane Molina, "Management Education for a Sustainable World: Aiming for More Than Business as Usual", 2021)

"Competency refers to observable and measurable skills that integrate the knowledge, skills, values, and attitudes required of a professional in the practice of his specialty." (Maria M P Calimag, "The ePortfolio: Technology-Enhanced Authentic Assessment in the Continuum of Medical Education", 2021)

"The sum of knowledge, skills, values, attitudes, and individual characteristics that enable a person to perform actions successfully." (Almudena Eizaguirre et al, "A Methodological Proposal to Analyse the Process for Implementing Competency-Based Learning (CBL) in a Business School", 2021)

🌡️Performance Management: Training (Definitions)

"Formal and informal learning options, which may include in-class training, informal mentoring, Web-based training, guided self-study, and formalized on-the-job training programs. The learning options selected for each situation are based on an assessment of the need for training and the performance gap to be addressed." (Sandy Shrum et al, "CMMI®: Guidelines for Process Integration and Product Improvement", 2003)

[cross-training:] "When an employee in one primary job task is trained in another or other tasks." (Robert McCrie, "Security Operations Management" 2nd Ed., 2006)

"An umbrella term to include training, development, and education, where training is learning that pertains to the job, development is learning for the growth of the individual that is not related to a specific job, and education is learning to prepare the individual but not related to a specific job." (Richard Caladine, "Taxonomies for Technology", 2008)

"Learning is a personal construction of knowledge. In order to learn a particular concept or skill, the learner needs to consider how new information relates to the existing understandings that the learner has. The process of sifting through available information in order to select the most appropriate information to use in knowledge construction requires the skills of information literacy. Good information literacy skills are a prerequisite for effective learning." (Carmel McNaught, "Information Literacy in the 21st Century", 2008)

"Activities undertaken to ensure that all individuals have the knowledge and skills required to perform their assignments." (Sally A Miller et al, "People CMM: A Framework for Human Capital Management" 2nd Ed., 2009)

"It is the process of fixing meaning to stimulus. It is the process of constructing new knowledge. Learning should proceed from learner’s sense of vocation, occur in settings or activity systems where the function and purposes of the learning are clear and explicit, focus primarily on developing the capacity to do and where learners seek to accomplish goals. In addition, learning should involve sharing meaning and building connection among meanings and different renditions of the meaning." (Kisilu M Kitainge, "Challenges of Training Motor Vehicle Mechanics for Changing World Contexts and Emergent Working Conditions: Cases of Kenya and Australia", 2009)

"Learning occurs through a cognitive process that occurs in the mind of the individual or, in contrast, learning occurs through a process of socialization and increasing participation rather than formal inquiry." (Mary F Ziegler, "Three Theoretical Perspectives on Informal Learning at Work", 2009)

"The process to obtain or transfer knowledge, skills, and abilities needed to carry out a specific activity or task" (Bettina M Davis & Wendy L Combsand, "Demystifying Technical Training: Partnership, Strategy, and Execution", 2009)

[business training: "Training on concepts that teach skills to understand and work effectively within a company." (Bettina M Davis & Wendy L Combsand, "Demystifying Technical Training: Partnership, Strategy, and Execution", 2009)

[IT training:] "Training on content involving the development, maintenance, and use of computer systems, software, and networks." (Bettina M Davis & Wendy L Combsand, "Demystifying Technical Training: Partnership, Strategy, and Execution", 2009)

[non-technical training:] "Training that is not technical training, for example, personal effectiveness or business training." (Bettina M Davis & Wendy L Combsand, "Demystifying Technical Training: Partnership, Strategy, and Execution", 2009)

[cross-training:] "Enables personnel to learn tasks associated with more than one job." (Barry Berman & Joel R Evans, "Retail Management: A Strategic Approach" 12th Ed., 2013)

"Programs used to teach new (and existing) personnel how best to perform their jobs or how to improve themselves." (Barry Berman & Joel R Evans, "Retail Management: A Strategic Approach" 12th Ed., 2013)

"Is a multidimensional process that results in a relatively enduring change in a person or persons, and consequently how that person or persons will perceive the world and reciprocally respond to its affordances physically, psychologically, and socially. The process of learning has as its foundation the systemic, dynamic, and interactive relation between the nature of the leaner and the objective of the learning as ecologically situated in a given time and place as well as over time." (Francisco Cua, "Authentic Education: Affording, Engaging, and Reflecting", 2014)

[on-the-job training:] "Training from an experienced employee to a new employee while working on the job. This is a form of one-on-one training." (Darril Gibson, "Effective Help Desk Specialist Skills", 2014)

"It can be defined as a mental activities by means of which knowledge, skill attitude are acquired, retained and utilized. It is defined it as changes in the particular form, change in behaviour tendency, resulting in relatively permanent practice. It involves that the changes, which occurs as a result of reinforced practice that gives new meaning and orientation. This leads to acquisition of new skills, behaviour tendency that is permanent." (Monsuru B Muraina, "Relevance of the Use of Instructional Materials in Teaching and Pedagogical Delivery: An Overview", 2015)

"Learning is a dynamic concept; it refers to the various processes by which skills and knowledge are acquired by individuals and, through them by organizations. Learning encompasses processes and outcomes as well as both, individual and organizational levels; it´s use in theory emphasizes the continually changing nature of organizations, and that goes beyond the view of organizations as bundles of resources. Learning includes the capacity to create new capabilities both internally and by acquiring knowledge from sources external to the firm. It also includes the methods for the diffusion of the new knowledge throughout the firm organization." (Arturo T Vargas & Javier J Villazul, "Learning and Innovation in Multinational Companies from Emerging Economies: The Case of CEMEX", 2016)

"The process of improving performance in one or more aspects of an employee’s work output through additional knowledge and or skill." (Fred MacKenzie, "7 Paths to Managerial Leadership", 2016)

"Learning is the act of gaining new knowledge, behaviors, skills, or ability. It may be regarded as a process, rather than a collection of factual and procedural knowledge. Human learning may occur as part of education, professional development, or training." (Chunfang Zhou, "Developing Creativity and Learning Design by Information and Communication Technology (ICT) in Developing Contexts", 2018)

[technical training:] "covers the acquisition of knowledge, skills and competencies leading to overall individual or company performance in the use and application of technology." (BCS Learning & Development Limited, "CEdMA Europe", 2019)

"Learning involves any process that in living organisms leads to permanent capacity change. Learning develops knowledge, abilities, understandings, emotions, attitudes, and sociality, which are important elements of the conditions and raw material of society." (Chunfang Zhou & Zhiliang Zhu, "Fostering Problem-Based Learning (PBL) in Chinese Universities for a Creative Society", 2019)

"The capacity of an individual and an organization to explore new challenges and contexts. It is an opportunity to unlearn which is a dynamic way of learning. It is through unlearning that people shape their brain, to readjust and continue learning. It is essential condition for transformation, creativity and innovation." (Ana Martins et al, "Unravelling Hurdles to Organizational Sustainability by Virtue of Sharing and Creating Knowledge", 2019)

"A shift of mind and what goes on inside learners as they undertake to gain or acquire new knowledge, understanding, skill, attitudes, values, and interests. The ‘what goes on’ could be described as perceiving - sensing and feeling concrete reality, thinking or reasoning abstractly; and internalizing or processing - making it a part of ourselves by actively jumping in and trying it, or reflecting on and watching what is happening; thus, the learner - anywhere along his/her life path, at any age - would have going on inside of him/her the perceiving and internalizing of new knowledges, understandings, skills, attitudes, values, and interests." (John A Henschke,"Leadership Ethics in Higher Education Administration: An Andragogical Perspective", 2020)

🌡Performance Management: Alfred Thompson's “Over-Educated, Yet Under-Qualified?” [1] (Answer)

Performance Management Series

In schools the accent is on theory and algorithms, the small projects target the learning of a technology, their complexity and difficulties involved being quite small when compared with real life applications. Taking an application from design to production and later during support phase requires time and the mix of knowledge from different fields, thing quite difficult to do in a school project, while the structural context in an organization, the requirements and work in a team, is again quite different. Going above the basic features of a programming language takes time, it depends on the learning curve of the programming language and the capacity of the learner, on the complexity of the tasks approached and on the knowledge (made) available. 

I can’t say that schools can do much in this direction because it’s quite difficult to cover all the aspects in just 8-20 classes, in which the students are introduced into the concepts and some basic applications. What the schools could do in order to support their students is to provide the required infrastructure (mainly computers), bring the technologies and learning material up to date, direct gradually the focus from theory to applicability, and eventually support users getting some additional experience in organizations. It’s in students’ attribution to make most of the learning experience in schools, though often even if the want, need and infrastructure is there, fighting with the lack of time is quite hard.

One of the tough realities in IT is that it takes time to link the dots, and as you already highlighted, it takes about a year before a college/university graduate to become really productive. Now I have to say that this depends also on the organization’s culture/environment, on how it supports the learning process, how it helps the new comer to become part of the team and become productive. I’m saying that because I’ve seen companies doing minimum in this direction, just expecting the new comer to catch everything on the fly and be productive in a matter of weeks. 

Those working in IT for a longer time know that is not entirely possible, though there are also some exceptions. There are also organizations that train the new comers, introduce them into tasks evolving in complexity based on each person’s skills, provide resources (software tools, books, courses and other type of learning material) and an environment that facilitates learning. Having time allocated for learning new things, participating in activities that allow the distribution of knowledge within a team, having professionals whom you could ask questions or who could mentor you through the learning process, I consider all these as being essential for a modern IT organization.

The theory learned in schools need to be supported by hand-on experience in order to make most of the learning process, IT organizations are maybe the best places to do that, though I’m not sure how much that is possible. There are schools, organizations and governments that support this type of learning, though, unfortunately is not everywhere possible to do that or at least not for everybody. I think it’s in everybody’s interest to make most of the learning process, for schools to have highly skilled graduates, for organizations to have productive employees, a pool of college graduates resources from where they could select potential employees, for students to be skilled, and thus have higher chances of finding a job, while for governments this could lead in theory to a smaller unemployment rate. I find important the constructive involvement of all parties; now, I wonder how many schools, organizations or governments are trying to do something, change something into this direction.

References:
[1] Alfred Thompson (2010) “Over-Educated, Yet Under-Qualified?”

🏗️Software Engineering: Problem Solving (Just the Quotes)

"We can scarcely imagine a problem absolutely new, unlike and unrelated to any formerly solved problem; but if such a problem could exist, it would be insoluble. In fact, when solving a problem, we should always profit from previously solved problems, using their result or their method, or the experience acquired in solving them." (George Polya, 1945)

"The problems are solved, not by giving new information, but by arranging what we have known since long." (Ludwig Wittgenstein, "Philosophical Investigations", 1953)

"A great many problems are easier to solve rigorously if you know in advance what the answer is." (Ian Stewart, "From Here to Infinity", 1987)

"An important symptom of an emerging understanding is the capacity to represent a problem in a number of different ways and to approach its solution from varied vantage points; a single, rigid representation is unlikely to suffice." (Howard Gardner, "The Unschooled Mind", 1991)

"[By understanding] I mean simply a sufficient grasp of concepts, principles, or skills so that one can bring them to bear on new problems and situations, deciding in which ways one’s present competencies can suffice and in which ways one may require new skills or knowledge." (Howard Gardner, "The Unschooled Mind", 1991) 

"Solving a problem for which you know there’s an answer is like climbing a mountain with a guide, along a trail someone else has laid. In mathematics, the truth is somewhere out there in a place no one knows, beyond all the beaten paths. And it’s not always at the top of the mountain. It might be in a crack on the smoothest cliff or somewhere deep in the valley." (Yōko Ogawa, "The Housekeeper and the Professor", 2003)

"Framing the right problem is equally or even more important than solving it." (Pearl Zhu, "Change, Creativity and Problem-Solving", 2017)

"A great discovery solves a great problem but there is a grain of discovery in the solution of any problem. Your problem may be modest; but if it challenges your curiosity and brings into play your inventive faculties, and if you solve it by your own means, you may experience the tension and enjoy the triumph of discovery." (George Polya)

"A great many problems are easier to solve rigorously if you know in advance what the answer is." (Ian Stewart, "From Here to Infinity", 1987)"Every problem has a solution; it may sometimes just need another perspective.” (Rebecca Mallery et al, “NLP for Rookies”, 2009)

"Mostly we rely on stories to put our ideas into context and give them meaning. It should be no surprise, then, that the human capacity for storytelling plays an important role in the intrinsically human-centered approach to problem solving, design thinking." (Tim Brown, "Change by Design: How Design Thinking Transforms Organizations and Inspires Innovation", 2009)

"Design patterns make design easier but not easy. Their application still requires a modest amount of reasoning and problem solving." (Eddie Burris, "Programming in the Large with Design Patterns", 2012)

"Knowledge of design patterns simplifies software design by reducing the number of design problems that have to be solved from first principles. Design problems that match documented design patterns have ready-made solutions. The remaining. problems that don't match documented design patterns must be solved from first principles. Even here, knowledge of design patterns can potentially help with original design. Design patterns are paragons of good design. Studying design patterns helps to develop the intellectual concepts and principles needed to solve unique design problems from first principles." (Eddie Burris, "Programming in the Large with Design Patterns", 2012)

"What makes a design pattern unique is its intent. The intent of a pattern is the problem solved or reason for using it. […] What distinguishes one pattern from another is the problem solved. You can infer the solution structure and problem solved from the pattern name but you can’t infer the pattern name from the solution structure alone." (Eddie Burris, "Programming in the Large with Design Patterns", 2012) 

"One of the broad truths we’ve seen to be true is the idea that finding problems earlier in the developer workflow usually reduces costs." (Titus Winters, "Software Engineering at Google: Lessons Learned from Programming Over Time", 2020)

"A problem thoroughly understood is always fairly simple." (Charles Kettering)

"A problem well-defined is a problem half solved." (John Dewey)

"An expert problem solver must be endowed with two incompatible qualities, a restless imagination and a patient pertinacity." (Howard W Eves)

"Finding the right answer is important, of course. But more important is developing the ability to see that problems have multiple solutions, that getting from X to Y demands basic skills and mental agility, imagination, persistence, patience." (Mary H Futrell)

"I have not seen any problem, however complicated, which, when you looked at it in the right way, did not become still more complicated." (Paul Anderson)

"I knew nothing, except how to think, how to grapple with a problem and then go on grappling with it until you had solved it." (Sir Barnes Wallis)

"Man is not born to solve the problems of the universe, but to find out where the problems begin, and then to take his stand within the limits of the intelligible." (Johann Wolfgang von Goethe)

"One is always a long way from solving a problem until one actually has the answer." (Stephen Hawking) 

"One measure of our understanding is the number of independent ways we are able to get to the same result." (Richard P Feynman) 

“Solving problems is a practical skill like, let us say, swimming. We acquire any practical skill by imitation and practice.” (George Polya)

"Some problems are just too complicated for rational logical solutions. They admit of insights, not answers." (Jerome B Wiesner)

"The best way to escape from a problem is to solve it." (Brendan Francis)

"The greatest challenge to any thinker is stating the problem in a way that will allow a solution." (Bertrand Russell)

"The measure of our intellectual capacity is the capacity to feel less and less satisfied with our answers to better and better problems." (Charles W Churchman)

"The mere formulation of a problem is often far more essential than its solution. To raise new questions, new possibilities, to regard old problems from a new angle, requires creative imagination and marks real advances in science." (Albert Einstein)

"The worst thing you can do to a problem is solve it completely." (Daniel Kleitman)

"There is no such thing as a problem without a gift. We seek problems because we need their gifts." (Richard Bach)

"To ask the right question is harder than to answer it." (Georg Cantor)

"When the answer to a mathematical problem cannot be found, then the reason is frequently that we have not recognized the general idea from which the given problem only appears as a link in a chain of related problems." (David Hilbert) 

"You are never sure whether or not a problem is good unless you actually solve it." (Mikhail Gromov)