QSM: Volumn 2.1: How to Observe Software Systems

Part I. 받아들이기 (Intake)

Chapter 1. 관찰이 왜 중요한가 (Why Observation is Important)

Summary

1. Almost any time software engineering management fails, it fails because of quality problems in the process as well as the product.
2. Quality problems "surprise" management because management is not making effective observations.
3. One of the things managers fail to observe is their own software engineering culture, which is not surprising, because culture is "what you know that you don't know you know."
4. Crosby applied the idea of cultural patterns to the study of industrial processes and observed five patterns based largely on the management attitudes to be found in each.
5. The Software Engineering Institute (SEI) related Crosby's work to software and identified five levels of "process maturity" related to the types of processes found in each pattern.
6. Classifying organizations by what people in them say they're doing, we can match them to the numbers of other systems of patterns as follows,
   1. Oblivious: "We don't even know that we're performing a process."
   2. Variable: "We do whatever we feel like at the moment."
   3. Routine: "We follow our routines (except when we panic)."
   4. Steering: "We choose among our routines by the results they produce."
   5. Anticipating: "We establish routines based on our past experience with them."
   6. Congruent: "We all are involved in improving everything all of the time."
7. The SEI method of determining software cultural pattern starts by conducting a survey and following it up with interviews and observations. If the interviews and observations are omitted, the survey can prove worse than useless.
8. Each culture has its own characteristic observation style.
9. The Oblivious (Pattern 0) and Variable (Pattern 1) cultures typically observe only after there is some trouble, which greatly increases their costs and risks.
10. Routine (Pattern 2) organizations try to be much more systematic in their approach to software and thus to observation as well. They tend to focus on failure information. The Routine pattern handles a lot of things well, until they are surprised by a crisis.
11. Steering (Pattern 3) organizations often transform the fledgling "metrics" programs of the Routine organization and try to observe human behavior more directly.
12. Steering managers know how to follow through appropriately, not by blaming but by gathering information.
13. Multiple sources of failure detection means greater likelihood of detection at any particular moment in the life cycle. Early detection means reduced development and usage costs.
14. Anticipating (Pattern 4) organizations use measurement to focus on prevention of problems, and are continually improving the sources of information available to managers.
15. Congruent (Pattern 5) organizations learn to observe earlier, by taking note of more subtle things. They also work on meta-observation—the observation of their own ways of observing. And everything they learn is applied throughout their organization.

Chapter 2: 무엇을 관찰할지 고르기 (Selecting What to Observe)

Summary

1. You can characterize management as well as cultural patterns by what they systematically observe and fail to observe.
2. The first major step in the Interaction Model is intake. I decide to obtain data from the world, I select which data to obtain, and I decide when I have enough.
3. In the input step, we need to be aware of the possibility that we aren't getting exactly what is out there in the world, even what is intentionally sent.
4. Data coming from the world doesn't make anyone react in a certain way.
5. Every situation offers possibilities for observation, but I must choose which possibilities to accept. Observing everything is neither humanly possible nor economically feasible. To observe one thing, I must omit observing something else.
6. Even without explicit rewards and punishments, the mere fact of observing something implicitly reinforces it.
7. Measurement is dangerous without a personal model of what the measurements might mean. A personal model of observation guides our observation efforts so they will be meaningful, efficient, and minimally disturbing to the very project I'm trying to control. The measurement model tells me what to measure, how to interpret those data, why I'm measuring, how much precision I need to measure.
8. Many examples imply that using a wrong model will also cause problems, perhaps even worse than no model at all.
9. Only when I know the state of the system can I make sensible interventions to control its behavior.
10. The ability to put different meanings on the same observation probably accounts for many of the software disasters that are so characteristic of Pattern 2 (Routine) organizations.
11. Observations that we can't interpret in terms of possible control interventions are not useful. Observations that we can interpret in this way are useful, even if they don't seem to be "measurements."
12. One important measure of an organization is how preoccupied the managers are with overly precise measurements for which they have no working models.
13. Two major management mistakes are lack of systems thinking and failure to generate meaningful observations. To some extent, doing better on one helps compensate for doing badly on the other.

Chapter 3: 제품을 시각화하기 (Visualizing the Product)

Summary

1. Both Deming's and Crosby's ideas on quality are derived from experience with manufacturing organizations. Software development, however, is not primarily a manufacturing operation, but an engineering project management operation.
2. Both manufacturing management and project management share the cybernetic model of control. For the cybernetic model to be applicable, outputs must be visible—a condition that is not automatically met by software.
3. When acting as observers, people ordinarily have these sensory modalities to draw upon—visual, auditory, kinesthetic, olfactory, and gustatory. We each have our favorite modalities, but when deprived of the use of our favorite modality, we can adapt by using the others.
4. We can only experience software indirectly, though some translation process (although programmers often talk as if they are in direct contact with software). An enormous number of familiar software tools have been designed to make software more visible. These observability tools are to software what radar was to warfare: they make it possible to steer "in the dark."
5. The more sub-modalities that carry the same information, the stronger the message. On other hand, multiple modalities may distort the information for certain observers. Such distortion may be perfectly justified for managing, though not for science.
6. Because our brain capacity is limited, we need more than one picture of the same measurement. The distortions in those pictures are the price we pay for overcoming our limited mental capacity.
7. Every picture of software is a map, and the map is not the territory. For effective software management, many maps—many visualizations as well as representations in other modalities—are needed of the same territory.
8. Anything that prevents observation of the product also prevents the organization from achieving higher levels of productivity and quality. Without visibility, we lose the ability to guarantee control over any project.
9. The finished product is the first thing that people should be able to visualize. Deming observed that people who can't visualize what they're working on will certainly have a hard time coming up with real contributions to its quality.
10. Visibility is so important we can make a quick assessment of whether or not an organization has reached a Pattern 3, or Steering, culture by asking three questions: Is everything available to view? visualizable? undistorted?

Chapter 4: 절차(프로세스)를 시각화하기 (Visualizing the Process)

Summary

1. Anticipating (Pattern 4) managers are not just steering the quality of each product, they are steering the quality of all products at the same time by steering the process.
2. When processes are your products, you must be able to visualize your processes, using the same criteria you used when software was the product: available to view, visualizable, and undistorted.
3. Process information must be presented in formats that the human sensory system can use. Deming's work has strongly emphasized the importance of a variety of pictures to make process data available to the visual system.
4. Feedback of process information is prey to all the same distortions as product information—and a few more.
5. We can make a quick assessment of whether or not an organization has reached a Pattern 4, or Anticipating, culture, with questions about process visibility rather than product visibility. Pattern 4 managers take the view that no matter what else it is, everything is information.
6. If everyone is taught a shared set of visualization techniques, they will share a common vocabulary when discussing process improvement. This is an essential part of the Deming approach.
7. Visualizations can be used to describe flow, cause and effect, distributions, and trends.
8. A project control panel combines a number of visualizations in one small area to show all the vital signs of the project. By monitoring the project control panel, a manager can know quickly when some aspect of the project requires attention.

Part II. 의미 (Meaning)

Chapter 5: 해석에 대한 한 케이스 스터디 (A Case Study of Interpretation)

Summary

• The Satir Interaction Model says that people always attempt to make meaning from the data they take in.
• Software culture makes its presence known through unconscious patterns that persist over time. Their patterns can be seen in many ways. The slip chart is a visualization of data that helps us interpret the differences between an organization's promise and delivery—a good characterization of its culture.
• The slip chart depicts a measurement that can be obtained cheaply on any project using information already available and public. The information consists of scheduled delivery dates, changes in those delivery dates, and the dates on which those changes are announced.
• A slip is a change in the scheduled delivery date. The announcement date is the date on which a new scheduled delivery date is made public. The slip chart reveals cultural patterns by plotting the new delivery date versus the announcement date for that next new delivery date.
• The difference between the announcement date and the old delivery date is the lead. The slip-lead chart is a scatter diagram relating slips and leads.
• In a healthy project, the closer the project is to the scheduled date before announcing a slip, the smaller that slip will be.
• Each slip adds an overhead delay, which is one reason for the advice, "Take no small slips."
• Consistent patterns of measurement reveal many aspects of an organization's culture; the meaning can be extracted from the data with a variety of visualizations, such as slip charts, slip-lead charts, and ramp charts.
• People's reactions to visualizations also provide data that suggest meaning.
• The most important cultural meanings to unearth are attitudes that block interpretations of data because they block the learning that could lead to improvement in software engineering practices.
• The key to uncovering these deeper meanings is to ask yourself, "What are the models that generated the behavior that produced those measurements?"
• A ramp chart is simply a slip chart with lines to the baseline filled in. This chart may reveal that an organization ignores the schedules it announces under pressure.

Chapter 6: 관찰로부터 의미를 만들 때 자주 하는 실수들 (Pitfalls When Making Meaning from Observations)

Summary

1. The Satir Model says that after receiving data, my next internal step is to interpret what I have taken in. I interpret according to my past experience, which may be different from the past experience of others, so we may have different interpretations of the same observation.
2. It's always important to remember that more than one interpretation is possible. If you can't think of at least three different interpretations of what you received, you haven't thought enough about what it might mean.
3. Interpretation is not observation. To keep the difference clear, we ask the Data Question: "What did you see or hear that led you to that conclusion?"
4. Here is a somewhat systematic process for making interpretations:
   1. Take in data. Use the Data Question.
   2. Form hypotheses about meaning.
   3. Reduce hypotheses to one. Choose the most likely.
   4. Decide if it's important.
   5. Do something about it. Perform a test that could invalidate the hypothesis.
   6. Take in data to see if the hypothesis is invalidated. If it is, return to step 1, or 2, or 3.
5. There are many cheap and easy measures you can try before embarking on an expensive measurement program. Start with something simple and apply the interpretation process.
6. Meta-observations are observations about observations, and often tell us more than the observations themselves. Even if you can't get the data you want, you always get data about the problems of getting and keeping data in an organization.
7. The attempt to invalidate hypotheses is important because of the following pitfalls:
8. Something may not be what it seems, if there is no verification.
9. Data may have been unconsciously selected.
10. Data may have been consciously selected.
11. Measurements may be wrong because of misunderstanding.
12. Measurements may be wrong because of falsification.

Chapter 7: 품질에 대한 직접적인 관찰 (Direct Observation of Quality)

Summary

1. To produce quality software you need to know: "What is the quality of this product, right now?" There are two different approaches to answering this question—the direct approach and the indirect approach.
2. For many people, "quality" is such an ambiguous term that nobody could be against it. Ultimately, everyone has the same definition of quality, and it goes something like this: "Quality is whatever I like."
3. Quality is relative. Quality is value to some person. Value is what are people willing to pay to have their requirements met. Perfectionists whose strongest desire is to find the one right way will not be satisfied with this relative definition of quality. But, then, perfectionists won't be satisfied with anything, so forget them.
4. Many software organizations today are so overloaded with quality problems that they are no longer coping effectively with their business of developing software. Yet many managers fail to recognize the relationship between this overload and quality problems.
5. Managers also fail to recognize the relationship between their own actions and the results they're getting.
6. Every software problem is a quality problem. The Zeroth Law of Quality says, If you don't care about quality then you can meet any other requirement. If you don't have to control quality, you can control anything else.
7. Quality is the most direct software measure we can find, and the only direct way to measure quality is with the people whose ideas count. The ultimate political question is, therefore, Who gets to control the definition of quality?
8. Power corrupts. And what power corrupts most thoroughly is the ability to make meaning of observations. When software developers control the definition of quality, they may do well in the short run, but they will ultimately drive their customers to the first competitor they can find.
9. The measurement of the absence of errors is not the same as direct measurement of quality. Real quality improvement always starts with knowing what your customers want. That's the only direct measure of quality.

Chapter 8: 비용과 가치 측정하기 (Measuring Cost and Value)

Summary

1. The motivation for improving quality always starts with a study of the value of quality, but many managers seem to confuse cost and value.
2. When under pressure to justify its existence, an organization has to decide whether to emphasize the cost side or the value side. The switch from cost observation to value observation is the strongest indication that an organization has made the transition from Pattern 2 to Pattern 3.
3. "Effort moves to what is counted." Cost counting leads to cost reduction. Value counting leads to value enhancement. Cost reduction is limited by the annual budget. Value enhancement is unlimited.
4. When someone says, "Software costs too much," it's always a coded message meaning, "Software isn't worth enough (to me)." Value is always perceived value, so we must know who is doing the perceiving.
5. When a software project simply collapses, the quality question is easy to answer—quality is zero. Underlying all quality collapses is the simple fact that in software we are attempting to achieve quality through precision higher than human beings have ever attempted before.
6. Software is a young, maturing business. What was an acceptable process for producing quality in the earlier system becomes unacceptable in its larger, more complex successors. Thus, even when software value doesn't collapse, it decays.
7. The Second Law of Thermodynamics says that you have to pay to get quality, and the higher quality you want, the more you have to pay. The First Law of Human Nature says that nobody wants to believe the Second Law of Thermodynamics applies to them.
8. The investment in quality must be more than money and hard work. To produce quality consistently, managers must learn new ways of thinking.
9. Systems thinking tells us that to produce quality, we must monitor requirements as they change, or as our understanding of them changes. Then we must make adjustments in the process on the basis of deviations between what is required and what is produced. That's the manager's job.
10. Measurement of quality through secondary requirements is indirect measurement, because it requires an extra step to relate the measurements to quality, and that extra step could go wrong. The possibility of misinterpretation means that the more indirect the observation, the more carefully it must be managed.
11. Discontent over standards arises when people who must conform to the standards cannot make the cause-effect connection between the standard and the value of the standard. The detailed impact case method and the single greatest benefit method allow us to make this connection and thus measure quality.
12. The detailed impact case method is an exhaustive study of the value impacts of a change. It is based on the idea of tracing requirements through a diagram of effects.
13. The single greatest benefit method is one approach to putting a floor estimate on value, while keeping the cost and time of the estimating method down to practical levels. The basic question of the greatest single benefit method is "What is the one greatest benefit that you can attribute to this change?"

Chapter 9: 메타 측정 (Meta-Measurement)

Summary

1. Meta-measurement is the measurement of the measurement system. Meta-measurement is important because in control system failures, the measurement system is one of the first things to break down.
2. Lack of awareness is the surest way to observe a control system failure. During a quality crisis, nobody knows what's really happening to them.
3. Software organizations in quality crises cannot produce accurate information about how many problems they are experiencing, let alone the specific nature of those problems.
4. Another reliable meta-measurement symptom of a quality crisis is failure to update the project management reports.
5. When essential work items such as specifications can't be ignored, they are often simply forgotten. Thus, you can diagnose the level of quality problems quite easily by asking people to find things that they need to do their work—like the specifications—then observing how long it takes to remember where they are, or if they even exist.
6. When our intellectual control system is breaking down, we cannot even give you an accurate description or name of what we are looking for. Such imprecise language is a tell-tale sign of loss of intellectual control.
7. When we're in crisis, it makes sense to simplify the situation. One way we do this is by trying to ignore difficult facts. We don't handle problems as they arise, we fail to record problems so that others can handle them, and in the worst case, we intentionally fail to record problems.
8. A successful control system must not only have information on the behavior of the system being controlled, but also external information about what standards are to be used for measuring that performance. Lack of knowledge of external standards of performance is thus a sure sign we're in quality trouble.
9. The troubled organization doesn't have standards of overall performance, but at least it should have standards for the present piece of work—in other words, specifications. Letting go of specifications is a tempting "solution" when our organization is experiencing serious quality problems.
10. Written specifications are a danger to the organization that's trying to pretend that they're meeting specifications, so you hear many references to undocumented verbal agreements.
11. Even when our organization is submerged in problems, it can recover if we are able to step back and get a look at ourselves so they can apply their creativity to the right things.
12. Sometimes we believe we have a grip on what's happening, but are too overloaded to actually check. Our overconfidence can cause even more trouble when we're eventually proved wrong. Many of our impressive "measurements" are actually a smoke screen that keeps us from getting a true picture of our communication patterns.
13. Other smoke screens are pseudo-reviews. Any indication that our technical reviews have become pseudo-reviews should raise a red flag.
14. When things grow bad enough, we start cutting communication lines to help us keep things within our control. This cutting, however, worsens the situation by creating a self-reinforcing feedback dynamic in which all lines are eventually cut. When some communication lines start to shut down, those lines that do remain open may deteriorate in quality.
15. Our communication channels deteriorate when we don't or can't talk directly to one another about problems, but remain silent, gossip, pass rumors, or act indirectly.
16. Duplicated work is typical of the quality crisis. We often duplicate measurements, to try to make them look better, or to make ourselves feel better.
17. More and more meetings accomplish less and less. Meetings grow larger when we feel that the only way to find out anything is to be at a meeting, so we have to go in an attempt to do the best job we can, under the circumstances.
18. We often try to reduce our overload by isolating ourselves from our co-workers. Whether the isolation is physical or emotional, we actually hurt our ability to do a good job by removing our greatest source of assistance.
19. If things become really bad, we may cut ourselves off from any source of information that might make us more aware of how bad the crisis really is. The instant reaction to any new piece of information is to deny it, saying there are no substantiating facts. And, frequently, no facts can be produced.
20. All of us think twice before we willingly become part of the system for delivering bad news. When we don't build or maintain information systems that could produce useful measurements, we then have to try to manage without measurements—and as a result fan the flames of crisis.