< back

A primer on metrics.

Metrics are fast becoming the means by which businesses measure and manage performance. What are they? What do different kinds of metrics tell you? Here's a practical guide.

By Gary T. Smith

Performance management is the focus of tremendous attention today. Whether it's called business, corporate, or enterprise performance management, metrics and their use in dashboards and other reporting formats form a key part of how organizations are to convey concise and timely information. Unfortunately, amid the buzz, very few words have been written focused on what metrics are, how they should be defined, and how you can determine their relevance and validity. You might find such discussions in academic writing, but not in the trade press.

In this article and the next installments, my goal is to focus on metrics themselves, beginning with a basic discussion of what they are. With accepted definitions, organizations can make true progress with performance management.

Metrics Defined
Metrics are measurements. For this discussion, they are measurements of process elements (inputs, activities, and outputs) in relation to an explicit benchmark — a specified level of performance. Therefore, metrics are process-based. These processes can be in research, manufacturing, finance, operations, marketing, services, or any other process-based activity. No doubt, the more discretely you've defined a process, the more accurate your results will be. Generally, however, you can define metrics for all but the most amorphous of processes.

Metrics by themselves merely state process parameters. In other words, when used to measure performance, users view them relative to a stated policy or benchmark — something I will discuss in more detail in the second part of this article. Ideally, organizations establish benchmarks using statistical methods. In a few cases (for example, the introduction of a totally new process) the management team, perhaps in concert with consultants and subject-matter experts, will arbitrarily establish the initial benchmark. Once established in this manner, however, actual measurements and statistics will refine the benchmark over time.

Metrics must be selected to give the greatest amount of information while using the least amount of resources required to report them in the most effective manner.

Processes Defined
What is a process, exactly? Merriam-Webster defines a process as: "a series of actions or operations conducing to an end." For our purposes, we will also include defined input and output attributes for the process in the definition.

Processes have interface points and workflow. Where processes exchange inputs/outputs — that is, where touchpoints exist — there will be an interface. Some methodologies refer to this interface as material flow, though the connotation that the interface involves something concrete does not always apply. In relationships from customer to company, regulatory agency to company, company to markets, and company to vendors you will see examples of external process interfaces. Other examples of relationships involving internal process interfaces include accounting to finance, operations to accounting, shop floor control to master production scheduling, and manufacturing to distribution. With a little thought, it becomes obvious that myriad processes and interfaces are relevant to any enterprise.
Within and between processes, there is workflow. Workflow is a series of discrete tasks connected by dependencies; workflow consumes resources. Each task consists of operations performed by resources upon components to create a discrete product.

It is possible to measure each operation, task input, task output, dependency chain, interface point, and resource. These measures are the base components for metrics that will provide business intelligence regarding enterprise performance. However, this is not to say that every identifiable operation, task input, task output, dependency chain, interface point, or resource should be measured and monitored. The effort required to collect, compile, and report on metrics must be cost effective. Realistically, you can't report on every measure.

Types of Measures
There can be as many measures as there are things to measure. It may be stating the obvious that a measure must be quantifiable, but you might be surprised at how many subjective "measures" you will find in a given environment.

You can describe quantifiable measures using discrete unit descriptors. For example, a measure of distance could be "inches" or "feet." A temperature measure could be degrees Celsius or Fahrenheit. It is the unit of measure that defines a measure, not what is being measured or the quantity of the measure. For example, "inches" is a unit of measure: "the distance traveled by an object," or the quantity of units traveled by an object in a given distance, are not typically units of measure.

If used only once, measures are of very little use. However, if they are part of a collection taken from a statistical sample population, measures can be extremely useful. If I say that the measure of the current temperature is 60 degrees, I'm not giving you very much information. You might ask where the temperature reading was taken, if the temperature is taken indoors or out, if it were taken in July or January, and whether this is the "normal" temperature. (In this context, normal implies some type of time-series analysis and comparison to a baseline or benchmark, and any variation from that baseline.) However, if I qualified my statement by saying that it was taken on the beach on the Mediterranean coast of Spain in June, you would know that this measure is unusual and warrants further investigation.

Days of sales outstanding (DSO) is a good example of a binary measure commonly found in manufacturing, usually accompanied by a set of limits, such as 30, 60, or 90 days. The statement "DSO is between 10 and 30" requires a determination of either yes or no, exclusively, even though it is stated as a range of values. DSO is either within the range, or it isn't.

Additive metrics are characterized by arithmetic counts, differences, intervals, or products. These metrics are best suited for discrete integer-value data. Some examples would include the number of (whole) units produced per hour/shift/day; sales volume (dollars, units) for a given period; and activity-based costs. (The subject of Activity-Based Costing is beyond the scope of this article.)

Simple ratios (proportions) and averages determine the mean value of a population, or the ratio of one measure to another measure. Some examples would include:

Descriptive statistics, which include methods to describe data, are often used with the previous types of measures. Measures and measure types are the tools and raw material for constructing metrics and process indicators.

Metric Categories
You can categorize metrics into simple or compound types. By "simple," I mean singular and direct. These metrics stand alone; they are not combined with other metrics and are for measuring individual attributes of the entity of interest. An example would be the time required for a particular loan processor to process a bank loan. The entity (processing a bank loan) is measured individually (without regard to other loan processors) for the attribute "time." When averaged over time, you could see the performance standard for that loan processor, and his or her variance from that standard. The metric is direct: that is, not derived from other metrics nor evaluated relative to the behavior of other loan processors.
Compound metrics are more complicated. These can be derived (that is, indirect), composite, and/or layered (hierarchical). Compound metrics are not mutually exclusive; overlap is possible. However, overlap is generally discouraged; instances of overlapping metrics need to be uncovered and the offending metrics combined or eliminated as part of a metrics maintenance process. Different and overlapping metric types should be converted to a common unit of measure (such as dollars, units, occurrences, or time granularity), with an appropriate conversion factor (dollars/hour, units/hour, dollars/unit, or units/dollar).
Compound metrics are, of course, the most complex type. They must be constructed carefully and understood clearly to maintain their validity and relevance. A poorly constructed metric of this type can do irreparable damage to the business — not to mention the credibility of the business analyst(s) responsible for the metrics portfolio effort.

Compound metrics can be further categorized: Weighted and composite averages. Although these averages are easy to use (within certain rules), I categorize them as complex because they are compound. Weighted Averages have myriad uses and are developed easily. However, it is also easy to devise weighted averages that, while appearing to be valid when initially developed, produce invalid results. If the underlying assumptions upon which weight assignments are based are incorrect, then they can't produce correct results.

Composite averages represent the mean of a group of averages. Composite averages can also be assigned weights. Where this is done, you can distribute the weight of the composite average back down to the composite average's components as percentages of the total of the component average values. In this way, you can determine each component's weight contribution and its sensitivity impact. This method will work at any level of hierarchy, although in short order it can become extremely complex.

Statistical analysis. Beyond the area of descriptive statistics lies the rich field of inferential statistics. Included here are regression and forecasting, correlation and variance, and other analytical tools. The breadth and depth of this topic lies beyond this article, but the importance of these tools to the application of Compound Metrics can't be overstated. In fact, for applications such as Six Sigma, these tools are essential.

Layered metrics. An associated class of composite metrics is layered (or "consolidation") metrics. These are characterized by a hierarchical relationship among the metric components. For a more detailed discussion, I would refer readers to the available literature on the Analytical Hierarchical Process (AHP) as an application of hierarchical metrics. The AHP illustrates how subordinate metrics (or sets of metrics) can influence super-ordinate metrics.

Thresholds and triggers. With these metrics, you can initiate some action based upon an out-of-bounds value or group of values. Perhaps the best example is statistical process control (SPC), most commonly found in manufacturing, where high volume, long-running processes produce large numbers of identical units.

In SPC, measurements are taken periodically (such as hourly), or quantitatively (for example, every 100 units), and plotted on a series chart. This chart would define upper control limits (UCL), lower control limits (LCL), and the target (baseline) value. When a measurement exceeds the UCL or LCL, the system can initiate an exception activity to correct the problem that generated the exception. Additionally, a variety of series types might indicate a marked trend in either direction (for example, four increasingly positive, or increasingly negative, measurements) that can trigger a corrective action, even though the UCL or LCL measures are not yet exceeded.

Gary T. Smith is a consultant with 25 years of experience in all areas if IT, including positions as IT directorship, project management, and consultant to support global enterprises working with BI, data warehousing, and Oracle database management.

< back