💻IT: Grid Computing (Definitions)

"A grid is an architecture for distributed computing and resource sharing. A grid system is composed of a heterogeneous collection of resources connected by local-area and/or wide-area networks (often the Internet). These individual resources are general and include compute servers, storage, application servers, information services, or even scientific instruments. Grids are often implemented in terms of Web services and integrated middleware components that provide a consistent interface to the grid. A grid is different from a cluster in that the resources in a grid are not controlled through a single point of administration; the grid middleware manages the system so control of resources on the grid and the policies governing use of the resources remain with the resource owners." (Beverly A Sanders, "Patterns for Parallel Programming", 2004)

"Clusters of cheap computers, perhaps distributed on a global basis, connected using even something as loosely connected as the Internet." (Gavin Powell, "Beginning Database Design", 2006)

"A step beyond distributed processing. Grid computing involves large numbers of networked computers, often geographically dispersed and possibly of different types and capabilities, that are harnessed together to solve a common problem." (Judith Hurwitz et al, "Service Oriented Architecture For Dummies" 2nd Ed., 2009)

"A web-based operation allowing companies to share computing resources on demand." (DAMA International, "The DAMA Dictionary of Data Management", 2011)

"The use of networks to harness the unused processing cycles of all computers in a given network to create powerful computing capabilities." (Linda Volonino & Efraim Turban, "Information Technology for Management" 8th Ed., 2011)

"A distributed set of computers that can be allocated dynamically and accessed remotely. A grid is distinguished from a cloud in that a grid may be supported by multiple organizations and is usually more heterogeneous and physically distributed." (Michael McCool et al, "Structured Parallel Programming", 2012)

"the use of multiple computing resources to leverage combined processing power. Usually associated with scientific applications." (Bill Holtsnider & Brian D Jaffe, "IT Manager's Handbook" 3rd Ed., 2012)

"A step beyond distributed processing, involving large numbers of networked computers (often geographically dispersed and possibly of different types and capabilities) that are harnessed to solve a common problem. A grid computing model can be used instead of virtualization in situations that require real time where latency is unacceptable." (Marcia Kaufman et al, "Big Data For Dummies", 2013)

"A named set of interconnected replication servers for propagating commands from an authorized server to the rest of the servers in the set." (IBM, "Informix Servers 12.1", 2014)

"A type of computing in which large computing tasks are distributed among multiple computers on a network." (Jim Davis & Aiman Zeid, "Business Transformation: A Roadmap for Maximizing Organizational Insights", 2014)

"Connecting many computer system locations, often via the cloud, working together for the same purpose." (Jason Williamson, "Getting a Big Data Job For Dummies", 2015)

"A computer network that enables distributed resource management and on-demand services." (Forrester)

"A computing architecture that coordinates large numbers of servers and storage to act as a single large computer." (Oracle, "Oracle Database Concepts")

"connecting different computer systems from various location, often via the cloud, to reach a common goal." (Analytics Insight)

💻IT: Latency (Definitions)

"The fixed cost of servicing a request, such as sending a message or accessing information from a disk. In parallel computing, the term most often is used to refer to the time it takes to send an empty message over the communication medium, from the time the send routine is called to the time the empty message is received by the recipient. Programs that generate large numbers of small messages are sensitive to the latency and are called latency-bound programs." (Beverly A Sanders, "Patterns for Parallel Programming", 2004)

"The amount of time it takes a system to deliver data in response to a request. For mass storage devices, it is the time it takes to place the read or write heads over the desired spot on the media. In networks, it is a function of the electrical and software properties of the network connection." (Tom Petrocelli, "Data Protection and Information Lifecycle Management", 2005)

"The time delay it takes for a network packet to travel from one destination to another." (John Goodson & Robert A Steward, "The Data Access Handbook", 2009)

"The time it takes for a system to respond to an input." (W Roy Schulte & K Chandy, "Event Processing: Designing IT Systems for Agile Companies", 2009)

"A period of time that the computer must wait while a disk drive is positioning itself to read a particular block of data." (Rod Stephens, "Start Here!™ Fundamentals of Microsoft® .NET Programming", 2011)

"The measure of time between two events, such as the initiation and completion of an event, or the read on one system and the write to another system." (DAMA International, "The DAMA Dictionary of Data Management", 2011)

"The time period from start to completion of a unit of work." (Max Domeika, "Software Development for Embedded Multi-core Systems", 2011)

"The time it takes to complete a task - that is, the time between when the task begins and when it ends. Latency has units of time. The scale can be anywhere from nanoseconds to days. Lower latency is better in general." (Michael McCool et al, "Structured Parallel Programming", 2012)

"The amount of time lag before a service executes in an environment. Some applications require less latency and need to respond in near real time, whereas other applications are less time-sensitive." (Marcia Kaufman et al, "Big Data For Dummies", 2013)

"A delay. Can apply to the sending, processing, transmission, storage, or receiving of information." (Mike Harwood, "Internet Security: How to Defend Against Attackers on the Web" 2nd Ed., 2015)

"A period of waiting for another component to deliver data needed to proceed." (Faithe Wempen, "Computing Fundamentals: Introduction to Computers", 2015)

"The time it takes for the specified sector to be in position under the read/write head" (Nell Dale & John Lewis, "Computer Science Illuminated" 6th Ed., 2015)

"The delay between when an action such as transmitting data is taken and when it has an effect." (O Sami Saydjari, "Engineering Trustworthy Systems: Get Cybersecurity Design Right the First Time", 2018)

💻IT: Automation (Definitions)

"The act of replacing control of a manual process with computer or electronic controls." (DAMA International, "The DAMA Dictionary of Data Management", 2011)

[soft automation:] "automation that is configurable through software without requiring changes to the underlying code of the software itself." (Meredith Zozus, "The Data Book: Collection and Management of Research Data", 2017)

[hard automation:] "automation that requires computer programming to be altered if changes are required." (Meredith Zozus, "The Data Book: Collection and Management of Research Data", 2017)

[Decision Automation:] "This broad term refers to computerized systems that make decisions and have some capability to independently act upon them. Decision automation refers to using technologies, including computer processing, to make decisions and implement programmed decision processes." (Ciara Heavin & Daniel J Power, "Decision Support, Analytics, and Business Intelligence" 3rd Ed., 2017)

"Automation is machine-controlled execution of actions, based on artificial intelligence and machine learning that do not require human intervention. It enables speed to action to help reduce time taken by human operators." (Heru Susanto et al, "Data Security for Connected Governments and Organisations: Managing Automation and Artificial Intelligence", 2021)

"refers to the technology where procedures or processes are performed with minimal human intervention. Machines can be configured based on an explicit set of rules or algorithms." (Accenture)

"performing all or part of a set of tasks with a machine rather than through human effort (NRC 1998)

[Intelligent Automation:] "refers to an automation solution that is enhanced with cognitive capabilities that enable programs and machines to learn, interpret and respond." (Accenture)

💻IT: Gateway (Definitions)

"A network software product that allows computers or networks running dissimilar protocols to communicate, providing transparent access to a variety of foreign database management systems (DBMSs). A gateway moves specific database connectivity and conversion processing from individual client computers to a single server computer. Communication is enabled by translating up one protocol stack and down the other. Gateways usually operate at the session layer." (Microsoft Corporation, "SQL Server 7.0 System Administration Training Kit", 1999)

"Connectivity software that allows two or more computer systems with different network architectures to communicate." (Sybase, "Glossary", 2005)

"A generic term referring to a computer system that routes data or merges two dissimilar services together." (Paulraj Ponniah, "Data Warehousing Fundamentals for IT Professionals", 2010)

"A software product that allows SQL-based applications to access relational and non-relational data sources." (DAMA International, "The DAMA Dictionary of Data Management", 2011)

"An entrance point that allows users to connect from one network to another." (Linda Volonino & Efraim Turban, "Information Technology for Management" 8th Ed., 2011)

[database gateway:] "Software required to allow clients to access data stored on database servers over a network connection." (Craig S Mullins, "Database Administration: The Complete Guide to DBA Practices and Procedures" 2nd Ed., 2012)

"A connector box that enables you to connect two dissimilar networks." (Faithe Wempen, "Computing Fundamentals: Introduction to Computers", 2015)

"A node that handles communication between its LAN and other networks" (Nell Dale & John Lewis, "Computer Science Illuminated, 6th Ed.", 2015)

"A system or device that connects two unlike environments or systems. The gateway is usually required to translate between different types of applications or protocols." (Shon Harris & Fernando Maymi, "CISSP All-in-One Exam Guide" 8th Ed., 2018)

"An application that acts as an intermediary for clients and servers that cannot communicate directly. Acting as both client and server, a gateway application passes requests from a client to a server and returns results from the server to the client." (Sybase, "Open Server Server-Library/C Reference Manual", 2019)

💻IT: Interprocess Communication (Definitions)

"A method of letting threads and processes transfer data and messages among themselves; used to offer services to and receive services from other programs (for example, named pipes)." (Owen Williams, "MCSE TestPrep: SQL Server 6.5 Design and Implementation", 1998)

"A system by which threads and processes can transfer data and messages among themselves. Interprocess communication (IPC) is used to offer and receive services from other programs." (Microsoft Corporation, "SQL Server 7.0 System Administration Training Kit", 1999)

"A mechanism through which operating system processes and threads exchange data and messages. IPCs include local mechanisms, such as Windows shared memory, or network mechanisms, such as Windows Sockets." (Anthony Sequeira & Brian Alderman, "The SQL Server 2000 Book", 2003)

"A mechanism of an operating system that allows processes to communicate with each other within the same computer or over a network." (Sybase, "Open Server Server-Library/C Reference Manual", 2019)

"The ability of one task or process to communicate with another in a multitasking operating system. Common methods include pipes, semaphores, shared memory, queues, signals, and mailboxes." (Microsoft, "SQL Server 2012 Glossary", 2012)

"A tool designed for developers to allow communication and sharing of data between applications." (Mike Harwood, "Internet Security: How to Defend Against Attackers on the Web 2nd Ed.", 2015)

ITIL: Change Request (Definitions)

"Any request submitted by a customer (buyer or users) for a change to alter the system. These appear in various forms and include software trouble reports (STRs) and baseline change requests (BCRs). BCRs request changes to the system’s specification." (Richard D Stutzke, "Estimating Software-Intensive Systems: Projects, Products, and Processes", 2005)

"Requests to expand or reduce the project scope, modify policies, processes, plans, or procedures, modify costs or budgets, or revise schedules." (Project Management Institute, "Practice Standard for Project Estimating", 2010)

"A document requesting a change for a project." (Bonnie Biafore, "Successful Project Management: Applying Best Practices and Real-World Techniques with Microsoft® Project", 2011)

"An official document requesting modification of existing features, requirements or functions or new ones. Change Request should contain description of the current solution, justification for a change and suggested (desired) solution." (IQBBA, "Standard glossary of terms used in Software Engineering", 2011)

"An appeal to the change review board for a modification to the project by any stakeholder. The request typically includes the justification, relationship of the change to the project goal and objectives, description of the change and deliverables, and effect on project risk." (Bonnie Biafore & Teresa Stover, "Your Project Management Coach: Best Practices for Managing Projects in the Real World", 2012)

"Change management notification for change in current process/environment." (Bill Holtsnider & Brian D Jaffe, "IT Manager's Handbook" 3rd Ed, 2012)

"A formal proposal to modify any document, deliverable, or baseline." (For Dummies, "PMP Certification All-in-One For Dummies" 2nd Ed., 2013)

"1. Written request or proposal to perform a specific change for a development product or to allow it to be implemented. 2. A request to change some software artifact due to a change in requirements." (Tilo Linz et al, "Software Testing Foundations" 4th Ed, 2014)

"Change management notification for a change in current process/environment." (Bill Holtsnider & Brian D Jaffe, "IT Manager's Handbook" 3rd Ed", 2012)

#️⃣Software Engineering: Programming (Part XV: Rapid Prototyping - Introduction)

Software Engineering Series

Rapid (software) prototyping (RSP) is a group of techniques applied in Software Engineering to quickly build a prototype (aka mockup, wireframe) to verify the technical or factual realization and feasibility of an application architecture, process or business model. A similar notion is the one of Proof-of-Concept (PoC), which attempts to demonstrate by building a prototype, starting an experiment or a pilot project that a technical concept, business proposal or theory has practical potential. In other words in Software Engineering a RSP encompasses the techniques by which a PoC is lead.

In industries that consider physical products a prototype is typically a small-scale object made from inexpensive material that resembles the final product to a certain degree, some characteristics, details or features being completely ignored (e.g. the inner design, some components, the finishing, etc.). Building several prototypes is much easier and cheaper than building the end product, they allowing to play with a concept or idea until it gets close to the final product. Moreover, this approach reduces the risk of ending up with a product nobody wants.

A similar approach and reasoning is used in Software Engineering as well. Building a prototype allows focusing at the beginning on the essential characteristics or aspects of the application, process or (business) model under consideration. Upon case one can focus on the user interface (UI) , database access, integration mechanism or any other feature that involves a challenge. As in the case of the UI one can build several prototypes that demonstrate different designs or architectures. The initial prototype can go through a series of transformations until it reaches the desired form, following then to integrate more functionality and refine the end product gradually. This iterative and incremental approach is known as rapid evolutional prototyping.

A prototype is useful especially when dealing with the uncertainty, e.g. when adopting (new) technologies or methodologies, when mixing technologies within an architecture, when the details of the implementation are not known, when exploring an idea, when the requirements are expected to change often, etc. Building rapidly a prototype allows validating the requirements, responding agilely to change, getting customers’ feedback and sign-off as early as possible, showing them what’s possible, how the future application can look like, and this without investing too much effort. It’s easier to change a design or an architecture in the concept and design phases than later.

In BI prototyping resumes usually in building queries to identify the source of the data, reengineer the logic from the business application, prove whether the logic is technically feasible, feasibility being translate in robustness, performance, flexibility. In projects that have a broader scope one can attempt building the needed infrastructure for several reports, to make sure that the main requirements are met. Similarly, one can use prototyping to build a data warehouse or a data migration layer. Thus, one can build all or most of the logic for one or two entities, resolving the challenges for them, and once the challenges solved one can go ahead and integrate gradually the other entities.

Rapid prototyping can be used also in the implementation of a strategy or management system to prove the concepts behind. One can start thus with a narrow focus and integrate more functions, processes and business segments gradually in iterative and incremental steps, each step allowing to integrate the lesson learned, address the risks and opportunities, check the progress and change the direction as needed.

Rapid prototyping can prove to be a useful tool when given the chance to prove its benefits. Through its iterative and incremental approaches it allows to reach the targets efficiently

Previous Post <<||>>  Next Post

#️⃣Software Engineering: Programming (Part XIV: Good Programmer, Bad Programmer)

Software Engineering Series

The use of denominations like 'good' or 'bad' related to programmers and programming carries with it a thin separation between these two perceptional poles that represent the end results of the programming process, reflecting the quality of the code delivered, respectively the quality of a programmer’s effort and  behavior as a whole. This means that the usage of the two denominations is often contextual, 'good' and 'bad' being moving points on a imaginary value scale with a wide range of values within and outside the interval determined by the two.

The 'good programmer' label is a idealization of the traits associated with being a programmer – analyzing and understanding the requirements, filling the gaps when necessary, translating the requirements in robust designs, developing quality code with a minimum of overwork, delivering on-time, being able to help others, to work as part of a (self-organizing) team and alone, when the project requires it, to follow methodologies, processes or best practices, etc. The problem with such a definition is that there's no fix limit, considering that programmer’s job description can include an extensive range of requirements.

The 'bad programmer' label is used in general when programmers (repeatedly) fail to reach others’ expectations, occasionally the labeling being done independently of one’s experience in the field. The volume of bugs and mistakes, the fuzziness of designs and of the code written, the lack of comments and documentation, the lack of adherence to methodologies, processes, best practices and naming conventions are often considered as indicators for such labels. Sometimes even the smallest mistakes or the wrong perceptions of one’s effort and abilities can trigger such labels.

Labeling people as 'good' or 'bad' has the tendency of reinforcing one's initial perception, in extremis leading to self-fulfilling prophecies - predictions that directly or indirectly cause themselves to become true, by the very terms on how the predictions came into being. Thus, when somebody labels another as 'good' or 'bad' he more likely will look for signs that reinforce his previous believes. This leads to situations in which "good" programmers’ mistakes are easier overlooked than 'bad' programmers' mistakes, even if the mistakes are similar.

A good label can in theory motivate, while a bad label can easily demotivate, though their effects depend from person to person. Such labels can easily become a problem for beginners, because they can easily affect beginners' perception about themselves. It’s so easy to forget that programming is a continuous learning process in which knowledge is relative and highly contextual, each person having strengths and weaknesses.

Each programmer has a particular set of skills that differentiate him from other programmers. Each programmer is unique, aspect reflected in the code one writes. Expecting programmers to fit an ideal pattern is unrealistic. Instead of using labels one should attempt to strengthen the weaknesses and make adequate use of a person’s strengths. In this approach resides the seeds for personal growth and excellence.

There are also programmers who excel in certain areas - conceptual creativity, ability in problem identification, analysis and solving, speed, ingenuity of design and of making best use of the available tools, etc. Such programmers, as Randall Stross formulates it, “are an order of magnitude better” than others. The experience and skills harnessed with intelligence have this transformational power that is achievable by each programmer in time.

Even if we can’t always avoid such labeling, it’s important to become aware of the latent force the labels carry with them, the effect they have on our colleagues and teammates. A label can easily act as a boomerang, hitting us back long after it was thrown away.

Previous Post <<||>>  Next Post

#️⃣Software Engineering: Programming (Part XIII: Misconceptions about Programming II)

Continuation

One of the organizational stereotypes is having a big room full of cubicles filled with employees. Even if programmers can work in such settings, improperly designed environments restrict to a certain degree the creativity and productivity, making more difficult employees' collaboration and socialization. Despite having dedicated meeting rooms, an important part of the communication occurs ad-hoc. In open spaces each transient interruption can easily lead inadvertently to loss of concentration, which leads to wasted time, as one needs retaking thoughts’ thread and reviewing the last written code, and occasionally to bugs.

Programming is expected to be a 9 to 5 job with the effective working time of 8 hours. Subtracting the interruptions, the pauses one needs to take, the effective working time decreases to about 6 hours. In other words, to reach 8 hours of effective productivity one needs to work about 10 hours or so. Therefore, unless adequately planned, each project starts with a 20% of overtime. Moreover, even if a task is planned to take 8 hours, given the need of information the allocated time is split over multiple days. The higher the need for further clarifications the higher the chances for effort to expand. In extremis, the effort can double itself.

Spending extensive time in front of the computer can have adverse effects on programmers’ physical and psychical health. Same effect has the time pressure and some of the negative behavior that occurs in working environments. Also, the communication skills can suffer when they are not properly addressed. Unfortunately, few organizations give importance to these aspects, few offer a work free time balance, even if a programmer’s job best fits and requires such approach. What’s even more unfortunate is when organizations ignore the overtime, taking it as part of job’s description. It’s also one of the main reasons why programmers leave, why competent workforce is lost. In the end everyone’s replaceable, however what’s the price one must pay for it?

Trainings are offered typically within running projects as they can be easily billed. Besides the fact that this behavior takes time unnecessarily from a project’s schedule, it can easily make trainings ineffective when the programmers can’t immediately use the new knowledge. Moreover, considering resources that come and go, the unwillingness to invest in programmers can have incalculable effects on an organization performance, respectively on their personal development.

Organizations typically look for self-motivated resources, this request often encompassing organization’s whole motivational strategy. Long projects feel like a marathon in which is difficult to sustain the same rhythm for the whole duration of the project. Managers and team leaders need to work on programmers’ motivation if they need sustained performance. They must act as mentors and leaders altogether, not only to control tasks’ status and rave and storm each time deviations occur. It’s easy to complain about the status quo without doing anything to address the existing issues (challenges).

Especially in dysfunctional teams, programmers believe that management can’t contribute much to project’s technical aspects, while management sees little or no benefit in making developers integrant part of project's decisional process. Moreover, the lack of transparence and communication lead to a wide range of frictions between the various parties.

Probably the most difficult to understand is people’s stubbornness in expecting different behavior by following the same methods and of ignoring the common sense. It’s bewildering the easiness with which people ignore technological and Project Management principles and best practices. It resides in human nature the stubbornness of learning on the hard way despite the warnings of the experienced, however, despite the negative effects there’s often minimal learning in the process...

To be eventually continued…

Previous Post <<||>>  Next Post

#️⃣Software Engineering: Programming (Part XII: Misconceptions about Programming - Part I)

Software Engineering Series

Besides equating the programming process with a programmer’s capabilities, minimizing the importance of programming and programmers’ skills in the whole process (see previous post), there are several other misconceptions about programming that influence process' outcomes.

Having a deep knowledge of a programming language allows programmers to easily approach other programming languages, however each language has its own learning curve ranging from a few weeks to half of year or more. The learning curve is dependent on the complexity of the languages known and the language to be learned, same applying to frameworks and architectures, the scenarios in which the languages are used, etc. One unrealistic expectation is that the programmers are capablle of learning a new programming language or framework overnight, this expectation pushing more pressure on programmers’ shoulders as they need to compensate in a short time for the knowledge gap. No, the programming languages are not the same even if there’s high resemblance between them!

There’s lot of code available online, many of the programming tasks involve writing similar code. This makes people assume that programming can resume to copy-paste activities and, in extremis, that there’s no creativity into the act of programming. Beside the fact that using others’ code comes with certain copyright limitations, copy-pasting code is in general a way of introducing bugs in software. One can learn a lot from others’ code, though programmers' challenge resides in writing better code, in reusing code while finding the right the level of abstraction.  

 

There’s the tendency on the market to build whole applications using wizard-like functionality and of generating source-code based on data or ontological models. Such approaches work in a range of (limited) scenarios, and even if the trend is to automate as much in the process, is not what programming is about. Each such tool comes with its own limitations that sooner or later will push back. Changing the code in order to build new functionality or to optimize the code is often not a feasible solution as it imposes further limitations.

Programming is not only about writing code. It involves also problem-solving abilities, having a certain understanding about the business processes, in which the conceptual creativity and ingenuity of design can prove to be a good asset. Modelling and implementing processes help programmers gain a unique perspective within a business.

For a programmer the learning process never stops. The release cycle for the known tools becomes smaller, each release bringing a new set of functionalities. Moreover, there are always new frameworks, environments, architectures and methodologies to learn. There’s a considerable amount of effort in expanding one's (necessary) knowledge, effort usually not planned in projects or outside of them. Trainings help in the process, though they hardly scratch the surface. Often the programmer is forced to fill the knowledge gap in his free time. This adds up to the volume of overtime one must do on projects. On the long run it becomes challenging to find the needed time for learning.

In resource planning there’s the tendency to add or replace resources on projects, while neglecting the influence this might have on a project and its timeline. Each new resource needs some time to accommodate himself on the role, to understand project requirements, to take over the work of another. Moreover, resources are replaced on project with a minimal or even without the knowledge transfer necessary for the job ahead. Unfortunately, same behavior occurs in consultancy as well, consultants being moved from one known functional area into another unknown area, changing the resources like the engines of different types of car, expecting that everything will work as magic.

Continued…

Previous Post <<||>>  Next Post

#️⃣Software Engineering: Programming (Part XI: The Dark Side)

Software Engineering Series

As member of programmers' extended community, it’s hard to accept some of the views that inconsiderate programmers and their work. In some contexts, maybe the critics reveal some truths. It’s in human nature to generalize some of the bad experiences people have or to oversimplify some of programmers’ traits in stereotypes, however the generalizations and simplifications with pejorative connotations bring no service to the group criticized, as well to the critics.

The programmer finds himself at the end of the chain of command, and he’s therefore the easiest to blame for the problems existing in software development (SD). Some of the reasoning fallacies are equating the process of programming with programmers' capabilities, when the problems reside in the organization itself – the way it handles each step of the processes involved, the way it manages projects, the way it’s organized, the way it addresses cultural challenges, etc.

The meaningful part of the SD starts with requirements’ elicitation, the process of researching and discovering the requirements based on which a piece of software is built upon. The results of the programming process are as good as the inputs provided – the level of detail, accuracy and completeness with which the requirements were defined. It’s the known GIGO (garbage in, garbage out) principle. Even if he questions some of the requirements, for example, when they are contradictory or incomplete, each question adds more delays in the process because getting clarifying the open issues involves often several iterations. Thus, one must choose between being on time and delivering the expected quality. Another problem is that the pay-off and perception for the two is different from managerial and customers’ perspective.

A programmer’s work, the piece of software he developed, it’s seen late in the process, when it’s maybe too late to change something in utile time. This happens especially in waterfall methodology, this aspect being addressed by more modern technologies by involving the customers and getting constructive feedback early in the process, and by developing the software in iterations.

Being at the end of the chain command, programming is seen often as a low endeavor, minimizing its importance, maybe because it seems so obvious. Some even consider that anybody can program, and it’s true that, as each activity, anyone can learn to program, same as anyone can learn another craft, however as any craft it takes time and skills to master. The simple act of programming doesn’t make one a programmer, same as the act of singing doesn’t make one a singer. A programmer needs on average several years to achieve an acceptable level of mastery and profoundness. This can be done only by mastering one or more programming languages and frameworks, getting a good understanding of the SD processes and what the customers want, getting hand-on experience on a range of projects that allow programmers to learn and grow.

There are also affirmations that contain some degrees of truth. Overconfidence in one’s skills results in programmers not testing adequately their own work. Programmers attempt using the minimum of effort in achieving a task, the development environments and frameworks, the methodologies and other tools playing an important part. In extremis, through the hobbies, philosophies, behaviors and quirks they have, not necessarily good or bad, the programmers seem to isolate themselves.

In the end the various misconceptions about programmers have influence only to the degree they can pervade a community or an organization’s culture. The bottom line is, as Bjarne Stroustrup formulated it, “an organization that treats its programmers as morons will soon have programmers that are willing and able to act like morons only” [1].

Previous Post <<||>>  Next Post

References:
[1] "The C++ Programming Language" 2nd Ed., by Bjarne Stroustrup, 1991

🧊💫Data Warehousing: Architecture (Part II: Data Warehousing and Microsoft Dynamics 365)

With Dynamics 365 (D365) Online Microsoft made an important strategical move on the ERP market, however in what concerns the BI & Data Warehousing (BI/DW) area Microsoft changed the rules of the game by allowing no direct SQL access to the production environment. This primarily means that will become challenging for organizations to use the existing DW infrastructure to access the D365 data, and for Vendors and Service Providers to provide BI/DW solutions integrated within the D365 platform.

D365 includes its own data warehouse (actually data mart) designed for financial reporting however as per now it can’t be extended to support other business areas. The solution favorited by Microsoft for DW seems to be the use of an Azure SQL Database aka BYOD (Bring Your Own Database) to which entity-based data can be exported incrementally (aka incremental push) or fully (aka full push) via the Data Management Framework (DMF) packages.

Because many of the D365 tables (e.g. Inventory Transactions, Products, Customers, Vendors) were overnormalized over the years and other tables were added as part of new functionality, to hide this complexity, Microsoft introduced a new layer of abstraction formed from data entities organized within an entity store. Data entities are view-like encapsulations of the underlying D365 table schema, the data import/export from and D365 being performed extensively over these data entities via the DMF, which extends the Data Import/Export Framework (DIXF).

One can use thus a BYOD as a direct source for other reporting tools as long they support a connection to Azure, otherwise the data can be further loaded into a database into the cloud, which seems to be the best option until now, as long the organization has other data that need to be consolidated for reporting. From here on, one deals with the traditional way of reporting and the available infrastructure can be extended to use an additional data source.

The BYOD solution comes with several restrictions: a package needs to be created for each business unit, no composite data entities can be exported, data entities that don’t have a unique key can’t be exported via an incremental push, data entities can change over times (new versions being available), while during synchronization no active locks should be on the database. In addition, organizations which followed this path report also some bugs that needed to be addressed via the Microsoft support. Even if the about 1700 available data entities facilitate to some degree data consumption, they seem to be more appropriate for data migrations and data integrations than for DW workloads.

In absence of direct SQL connectivity, in theory organizations can still use SSIS or similar integration tools to connect to D365 production databases and consume data entities via the Open Data Protocol (OData), a standard that defines a set of best practices for building and consuming RESTful APIs. Besides some architectural challenges, loading big tables with transactional data is reportedly slow and impracticable for loading a data warehouse. Therefore, the usability of such an architecture becomes limited in time.

Microsoft imposed a hard limitation upon its D365 architecture by making its production database inaccessible. Of course, there’s still time for Microsoft to do some magic and pull new solutions from the technology stack hat. Unfortunately, the constraints imposed to the production environments limit organizations’ choices of building a modern and flexible data warehouse. For the future it would be great if the DMF could be used directly with standard SQL Server databases, avoiding thus the need for the intermediary Azure database, or if a real-time operational solution could be provided out-of-the-box. We’ll see what the future brings...

🧭Business Intelligence: Perspectives (Part IV: How Big Is Your Report?)

How big are your reports? How big reports needs to be? Do your reports really reflect your needs? Have they become too cluttered with data? Do you have too many reports on the same topic? How many is too many? These are the few of the questions BI developers and users should ask themselves altogether from time to time.

A report is any document with textual and/or graphical formatted output of data from one or more data sources, (previously) designed to convey a basis for decision making or operational activities. A report is characterized by the amount of data it holds (the datasets), the amount of data is based on (the source data), the number and complexity of the queries on which the report is based, the number of data sources, the manner in which data are structured (tabular, matrix, graphical), the filtering and sorting possibilities, as well by the navigability possibilities (drilldown, drill-through, slice-and-dice, etc.). 

On the other side for users are important characteristics like reports’ performance, the amount of useful information it conveys, the degree to which a report helps address a business need, the quality of data, the degree to which it satisfies the various policies, the look and feel, the possibility of exporting the data to standard file formats.

A report’s size is defined typically by the product of columns and records the report displays plus the formatting and various types of graphical content, however this depends on the filter criteria used by the user. Usually is considered the average size of a report based on the typical filters used. Nowadays networks and database specific techniques allow displaying fairly big reports (20-50 Mb) in a fairly amount of time (10-20 seconds) without affecting network, respectively database’s performance, which for most of the requests should be enough. When the users need bigger volumes of data then a direct data dump (extract) from the database should be considered, when possible. (A data export is not a report and they should be differentiated as such.)

The number of records that could be shown in a report is dependent on reporting framework’s capabilities, e.g. there are reporting tools that cope well with showing a few thousands records but have difficulties in showing or exporting tens of thousands of records. The best example into this respect is Excel and its well-known limitation of 65536 records (2^16)  and 256 columns (2^8) that in the meantime has been addressed in Excel 2007 and enlarged to 1 million records (2^20), respectively 16k (2^14). Even so the reporting tools that use older drivers can fail exporting all the data to Excel when the former limitation is reached.

In general, reports with too many columns tend to obfuscate data’s understanding and are more difficult to navigate. The more the user needs to scroll horizontally the higher in general the obfuscation. If the users really need 50 columns then they should be provided, however in general 20-25 should be enough for an operational report. Tactical and strategic reports need a restrained focus and the information should be provided in a screen without the need of scrolling.

When reports get too big is recommended to split the reports in two or more reports to address specific requirements, however this can lead to too many distinct reports, and further to duplication of effort for creating and documenting them, and the duplication of logic and data. Therefore, the challenge is to find the right balance between the volume of reports, their usability and the effort needed to manage them. In certain scenarios it makes even sense to consolidate similar reports.

Previous Post <<||>> Next Post

𖣯Strategic Management: Strategic Perspectives (Part I: Agile vs. Lean Organizations)

Agile and lean are two important concepts that pervaded the organizations in the past 20-30 years, though they continue to have little effect on organizations’ operations.

Agile is rooted in the need to respond promptly to the changing needs of an organization. The agile philosophy was primarily groomed in Software Development to reconcile the changing customer requirements with disciplined project execution, however it can be applied to an organization’s processes as well. An agile process is in general a process designed to deliver the intended results in an effective and efficient manner by addressing promptly the changing requirements in customers’ needs.

Lean is a systematic method for the minimization of waste, rooted as philosophy in manufacturing. The lean mindset attempts removing the non-value-added activities from processes because they bring no value for the customers. Thus a lean process is a process designed to deliver the intended results in an effective and efficient manner by focusing on the immediate needs of the customers, what customers want and value (when they want it). 

Effective means being successful in producing a desired or intended result, while efficient means achieving maximum productivity with minimum wasted effort or expense. The requirement for a process to be effective and efficient is translated in delivering what’s intended by using a minimum of steps designed in such a way that the quality of the end results is not affected, at least not for the essential characteristics. Efficiency is translated also in the fact that the information, material and resources’ flow suffer minimal delays.

Agile focuses in answering promptly the changing requirements in customers’ needs, while lean focuses on what customers wants and value while eliminating waste. Both mindsets seem to imply iterative and adaptive approaches in which the improvement happens gradually. Through their nature the two mindsets seem to complete each other. Some even equate agile with lean however an agile process is not necessarily lean and vice-versa.

To improve the effectiveness and efficiency of its operations an organization should aim developing processes that are agile and lean to optimize the information and material flows, while focusing on its users’ changing needs, and while eliminating continuously the activities that lead to waste. And waste can take so many forms – the unnecessary bureaucracy reflected in multiple and repetitive sign-offs and approvals, the lack of empowerment, not knowing what to do, etc.

There’s important time wasted just because the users don’t know or don’t understand an organization’s processes. If an organization can’t find rules that everyone understands then a process is doomed, independently of the key area the process belongs to. There’s also the tendency of attempting to address each exception within a process to the degree that multiple processes result. There’s no perfect process, however one can define the basic flow and document the main exceptions, while providing users some guidelines in navigating the unknown and unpredictable.

As part of same tendency it makes sense to move requests that respect a standard procedure on the list of standard requests instead of following futile steps just for the sake of it. It’s the case of requests that can be fulfilled with internal resources, e.g. the development of reports or extraction of data, provisioning of SharePoint websites, some performance optimizations, etc. In addition, one can unify processes that seem to be disconnected, e.g. the handling of changes as part of the Change Management respectively Project Management as they involve almost the same steps.

Probably it's in each organization’s interest to discover and explore the benefits of applying the agile and lean mindsets to its operation and integrate them in its culture

Previous Post <<||>> Next Post

💼Project Management: Methodologies (Part IV: Agility under Eyeglasses II)

Continuation

Employees are used to follow procedures and processes, and when they aren’t available insecurity rules - each day there’s another idea advanced how things are supposed to work. Practically, the Agile approaches (incl. Agile Prince2) focus on certain aspects and ignore specific Project Management activities that need to be performed inside of a project – releasing resources for the project, getting users on-board, getting management’s buy-in, etc. Therefore, they need to be used with a methodology that offers the lacking processes. Problematic is when is considered that the Agile approaches are self-consistent and the Project Management practices and principles don’t apply anymore.

It’s true that the Agile methods attempt reconciling disciplined project execution with creativity and innovation, however innovation is needed typically in design (incl. prototyping) , while in programing there isn’t lot of room for creativity per se. The real innovation appears when the customer lists the functionality it needs from a system and the vendor, after analyzing all the related requirements, is capable to evaluate and propose a solution from the industry trending technologies. That’s innovation and not changing controls in user interfaces!

User stories are good for situations in which an organization doesn’t know what’s doing or the tasks have a deep segmentation and specialization. Starting from user stories and building upwards to processes can prove to be a waste of time the customer pays for, while the approach leaves few room for innovation. In big projects it’s also difficult to sense the contradictions from user stories or their duplication. Even if the user stories allow maybe (but not necessarily) a better effort estimate the level of detail can become overwhelming for any skilled solution architect.

It’s also true that an agile approach needs a culture with certain characteristics. A culture can’t be changed with one project or several projects running in parallel. Typically, is recommended to start with a pilot test, assert organization’s readiness, disseminate knowledge, start several small to medium projects and build from there. For sure starting a big project with an agile methodology  will involve more challenges to the extent the challenges will push back.

One sign of agility is when self-organizing teams emerge within projects, however it takes time and training to build such teams. The seeds must be planted long before, for such teams to emerge. The key is being able of working in such teams. In extremis, conflicts appear when multiple self-organizing teams appear, each with its own political agenda, agendas that don’t necessarily match project manager or stakeholders’ agendas, and from here a large range of potential conflicts.

The psychological effect of tight sprints (iterations) and daily status meetings for the whole duration of a project is not to neglect. It builds unnecessary stress and, unless the planning reaches perfection, the programmer or consultant will often find himself in the position to be in defensive. The frequent meetings can easily become a source for nuisance and in extremis can lead to extreme behavior that can easily affect the productivity and involved persons’ health.

Personally, I wouldn’t recommend using an Agile methodology for a big project like an ERP implementation unless it was adequately adapted to organization’s needs. This doesn’t necessarily mean that the Agile methods aren’t suitable for big projects, it means that the risks are high because in big projects there’s the chance for all these mentioned issues to occur.

Despite the weak points of the Agile methods, when adequately applied, they have the chance of better performing than the “traditional” approaches. Even if people tend to see more the negative sides there’s lot of potential in being agile.

Previous Post <<||>>Next Post