Using Design of Experiments to Sort Through Randomness (and the perception of “best”)

The Annual Fishing Trip is an unlikely place for Design of Experiments but proves the technique does not have to be overly formal to sort through randomness and bias.

Design of Experiments is a tool for gaining deeper insights into how input factors influence outcomes. Confounding factors can create a false impression of causation or exaggerate or mask the true effect of the primary variables being studied. Asset managers, reliability engineers, data analysts, and other technologists are well served by understanding this technique. Sorting through randomness and not taking “best” on face value are important outcomes.

The Great Fishing Competition

Andy and I had been fishing together for nearly 20 years, including an annual weeklong fishing trip. The debate over who was the better fisherman had surfaced countless times in bars and over beers. We designed a plan based on my experience with the Design of Experiments. We would fish twice per day for a week on the annual fishing trip and isolate one or more confounding factors each day.

What is the Purpose of Design of Experiments?

Design of Experiments (DOE) aims to systematically plan, conduct, and analyze experiments to understand the relationships between inputs and outputs. It helps researchers and practitioners optimize processes, products, and systems.

The goal of the DOE is to:

1. Identify Cause-and-Effect Relationships

2. Optimize Processes and Products

3. Control Variation

4. Test Multiple Factors Simultaneously

5. Minimize Confounding Factors

6. Improve Decision Making

The Full Story of the Great Fishing Competition

 DOE achieves isolation of factors through:

1. Randomization

2. Blocking

3. Factorial Designs

4. Replication

5. Control Groups

   
   

By systematically planning and analyzing experiments, DOE can identify and mitigate the impact of confounding factors, leading to more reliable and valid conclusions.

What Professions Use Design of Experiments The Most?

Engineering, food science, pharmaceuticals, marketing & consumer research, and data analytics are areas where you will commonly find forms of Design of Experiments.

It Is Not Always “Statistical” In Practice

The Great Fishing Competition is a good example of how the Design of Experiments does not have to be statistical to provide definitive insights. In practice, isolating different variables through a few tests can provide enough insight that one thing is not necessarily better than the other.

Andy and I are both pretty good fishermen.

We Are Both Good Fisherman

In the end, there was no clear winner of the Great Fishing Competition. After a tough seven-day battle, a single fish on the last cast of the last day proved to be the difference maker. Andy won the competition by the most weight (pounds) of fish, but I caught a few more and the biggest fish.

Our informal Design of Experiments provided enough information to sort through the randomness and our biases. We are both pretty good fishermen.

With all its unpredictability, the ocean has its own way of deciding who the better fisherman is on any given day.

References:

Wikipedia

The Reliability Handbook

The Great Fishing Competition

---

JD Solomon Inc. provides solutions for program development, asset management, and facilitation at the nexus of facilities, infrastructure, and the environment. Sign up for monthly updates related to our firm.

---

JD Solomon is the founder of JD Solomon, Inc., the creator of the FINESSE fishbone diagram®, and the co-creator of the SOAP criticality method©. He is the author of Communicating Reliability, Risk & Resiliency to Decision Makers: How to Get Your Boss’s Boss to Understand and Facilitating with FINESSE: A Guide to Successful Business Solutions.

Helpful Tips on 5 Aspects of Work Management

JD Solomon presented “Five Aspects of Work Management that Impact Maintenance and Reliability” at this year’s WEASC Operator Conference on Tuesday October 29th, 2024.

The five topics that were discussed were Planning & Scheduling, Preventative Maintenance, Work Documentation, Maintaining Asset Inventory & Drawings, and Trend Analysis & PM Optimization.

The most commonly asked questions on each topic from Solomon’s thirty-year career were answered. At least one practical approach for implementing each topic was covered.

Look for an article on each topic to appear in Asset Management Insights in the upcoming months.

---

JD Solomon Inc. provides solutions for program development, asset management, and facilitation at the nexus of facilities, infrastructure, and the environment. Sign up for monthly updates related to our firm.

---

 JD Solomon is the founder of JD Solomon, Inc., the creator of the FINESSE fishbone diagram®, and the co-creator of the SOAP criticality method©.

Should Project Managers Rely on a Guesstimate?

Project managers can develop quick estimates with confidence.

The term “guesstimate” is an estimate made with a combination of guesswork and calculation. It’s an approximate calculation when you don’t have all the facts or precise data. Project managers find themselves needing to rely on guesstimates as projects move through implementation. Even more, project developers live on guesstimates in the conceptual development phase. The central question is whether there is a structured way to do a guesstimate.

 

Brief Examples

Guesstimate blend of the words “guess” and “estimate.”

As a project manager, you receive an estimate that site cleanup after a storm will take some time. When pressed, the contractor says, "About a week.”

A change order request includes a unique process controller. The equipment vendor tells the designer and contractor that the base equipment is available and easy to quote. However, the special controller is back-ordered due to a version update. When asked for a price, the vendor replies, “The supplier will not even quote it, but I would budget $10,000.”

Both are simple examples of guesstimates that project managers deal with every day. When developing an entire project, the magnitude of the guesstimate is bigger and contains a lot more uncertainty.

Measurement

Measurement is a process of assigning numbers to objects so that the unique relationships of the objects and the unique properties of the numbers themselves are reflected. Two fundamental measurement theories are representativeness, meaning that objects can actually be assigned to a number, and uniqueness, meaning that a number cannot be assigned to the same object by different measurers.

As project developers or managers, we would like to have a single scale of measurement for all of our variables. That's why we often try to reduce everything to a dollar or time value.

Quantitative versus Qualitative

Quantitative means of, relating to, or involving the measurement of quantity or amount. Think inches, feet, meters, pounds, or kilograms. Measurement theory applies to significant decimal places.

Qualitative means of, relating to, or involving quality or kind of things. Think beauty, fun, heat or cold, or color. We can often develop a scale, 1 to 5 or 1 to 10, to help describe the weight of evidence. The quality of some things is not easily measured with a unique scale.

As project developers or managers, we want all quantitative values to be developed from a singular scale. Unfortunately, that's usually not the case. Now, we are moving into the world of guesstimates.

Objective versus Subjective

Objective means expressing or dealing with facts or conditions as perceived without distortion by personal feelings, prejudices, or interpretations. Unless you are in a laboratory where input variables can be isolated, it is almost impossible for an estimate to be absent of personal feelings, prejudices, or interpretations.

The challenge is really how to make something less subjective.

We normally do this by adding some degree of quantification and scale.

Truncated (or trimmed) Mean: An Approach for Reliable Guesstimates

A truncated or trimmed mean is a statistical measure of central tendency, much like the mean and median. It involves the calculation of the mean after discarding given parts of a probability distribution or sample at the high and low end.

In practice, our resources are usually limited to 5 to 10 people providing their best guesses. We drop the lowest and highest values. Then, we average what's left.

A Structured Way to Do Guesstimates

In practice, guesstimates are improved by asking more than one party and sorting through a common value.

In the example of cleanup after a storm, a project manager would call five or six other contractors with similar experience or in the same situation. Throw out the low and high values. Average the rest of the values. Feel good about the guesstimate if the average is close to the contractor's estimate.

It’s a little harder with specialty parts and backorders, but the same process applies. Talk to five or six other people with the same experience, throw out the high and low estimates, average the rest, and decide whether to take the guesstimate.

On bigger issues like how long it takes to get something permitted, assemble a panel of subject matter experts and formally facilitate the process.

Should Project Managers Rely on a Guesstimate?

Project managers are forced to rely on guesstimates every day. There's nothing wrong with that. This brief article provides some insights and a method for making guesstimates better. Look for more on how calibration of the estimators and Monte Carlo simulations can improve guesstimates more if the context requires it. For now, project managers should do their best every day to improve the guesstimates we live by.

References

• Wikipedia

• "A Revisit of the Scales, Measurement Theory, and Statistical Analysis Controversy"

---

JD Solomon Inc. provides solutions for program development, asset management, and facilitation at the nexus of facilities, infrastructure, and the environment. Sign up for monthly updates related to our firm.

---

JD Solomon is the founder of JD Solomon, Inc., the creator of the FINESSE fishbone diagram®, and the co-creator of the SOAP criticality method©. He is the author of Communicating Reliability, Risk & Resiliency to Decision Makers: How to Get Your Boss’s Boss to Understand and Facilitating with FINESSE: A Guide to Successful Business Solutions.