Testing Proves Testing Works

Big Lots, the Columbus, Ohio-based closeout chain, credits sophisticated testing and analysis with helping cut costs for advertising, marketing, utilities, staffing, store operations, real estate and inventory.

While it has nurtured a culture of testing for some time, the 1,361-store company has refined its processes with the help of the “Test and Learn” discipline of Arlington, Va.-based software vendor Applied Predictive Technologies (APT).

Big Lots undertook a 16-week pilot of APT’s software and consulting services in October 2008. Most companies conduct two tests before deciding whether to sign on full-time, says APT CEO Anthony Bruce. Big Lots conducted three tests and also used the APT software tool to review approximately 20 previous tests for a comparison of the results, says Dan Yokum, the retailer’s director of strategic planning. “We wanted to get a good feel for the value the tool would offer, versus what we were doing internally.”

In one test, Big Lots determined the correct number of print circulars to distribute to each store. The test took into consideration the fact that Big Lots was increasing the number of e-mail circulars it sends to its Buzz Club Rewards members at the same time it was reducing print-circular circulation, Yokum says. The retailer has achieved several million dollars in annualized savings as a result of employing the lessons learned in the test, he says.

Another pilot helped Big Lots test the labor-management tool the chain was planning to use. In addition to the effectiveness of the labor-scheduling software, it wanted to learn how sales volume was affected by altering payroll hours. APT helped determine which stores to include in the “hold-out” or control group that did not receive the new scheduler.

That second pilot laid the groundwork for Big Lots to “flex down” on payroll hours in response to the economic downturn, confident that the reductions would not harm sales volume, says Tim Johnson, Big Lots vice president of strategic planning and investor relations.

The second pilot also helped determine which of several hundred store characteristics lent themselves to reducing payroll hours. Attributes the chain tested included the age of the store, sales volume, payroll as a percentage of sales, full-time versus part-time associate mix and management turnover.

Merchandising, the subject of the third test, helped Big Lots choose new products and find the best inventory level for established products, Yokum says.

The nearly two dozen previous tests Big Lots reviewed using the APT tool covered subjects that included energy management systems, furniture departments, television advertising, new fixtures, surveillance systems and new-store cannibalization. “We wanted to look at as many different tests as we could across all areas of the business and make sure we were hitting on marketing tests, store ops, real estate, loss prevention and merchandising,” Yokum says.

What Big Lots found, he says, was that they were “doing a pretty good job overall in measuring the results, but the value of APT was they really give us a better understanding of the disparity in the results.” The tool helped the chain understand why some stores over-performed or under-performed on an initiative, he says.

Prior to using APT, Big Lots reviewed the overall results of a test and averaged pluses and minuses to come up with a net positive or net negative result, Yokum says. With the APT tool, the chain learns why some stores were affected by an initiative and some were unaffected. Attributes for the test of circulars, for example, included the number of Buzz Club members at each store and the population of the trade area around the store.

The parts of the test varied in duration, Yokum says. Testing an ad promotion took just a week, but the chain spent a year reviewing the results of a store remodeling, he says. “It depends on the consistency of the results and when we start seeing the stabilization of the trend,” he says.
By the end of the pilots, Big Lots began seeing the value of APT in three areas — test design, test analysis and roll-out recommendations.

Creating the perfect test
A properly designed test uses the right number of stores to provide a representative sample of the company across key attributes so as “to be able to extrapolate across the chain,” Yokum says. Good test design also requires an effective control-group matching strategy. With the algorithms of the APT tool, Big Lots can select a control group of stores that mirrors the characteristics of the test store in geography, clientele, the attributes being tested and financial patterns during the base-line period.

“We can get more sophisticated, and each control group is unique to the test store,” Yokum says. “Before APT, we were using a similar methodology, but when we were picking control groups it was usually balance of the chain or balance of the region. Every test store had the same control group.”

Choosing the right control group of stores with similar attributes minimizes “noise,” those irrelevant factors that falsely color results, Bruce says. Overlapping initiatives create noise, so Big Lots works to isolate the impact of each initiative.

Other types of noise include seasonality, weather, recent store remodeling, nearby store openings or closings, changes in competition and management turnover. Big Lots also takes care to avoid falling victim to bias caused by upward or downward sales trends that began prior to, and continued through, the test period.

Full engagement
APT provides Big Lots with a three-member team comprised of Bruce, an engagement manager and a business analyst. The team responds quickly when issues arise, including on evenings and weekends. In weekly conference calls Big Lots consults with its APT team on how the retailer is setting and analyzing tests. At least once a quarter, the APT team attends one of the monthly meetings of the Big Lots testing committee.

Big Lots formed the committee in part to make sure management agreed to the cost of taking on APT and would abide by the rules APT and the retailer worked out. “We wanted to make sure they were engaged in what we were doing here,” Johnson says. “The last thing we wanted was every part of the business doing tests that would create all kinds of noise. Putting some structure around it was really important.”

Yokum has attended APT’s annual user conference the last two years, and Johnson says Big Lots has embraced “a culture of Test and Learn.

“We’ve been fortunate enough to have 14 straight record quarters in earnings per share, even though the retail environment hasn’t been the best,” Johnson says. “Test and Learn has been a big part of that, and APT has been a very good partner in that regard.”