Suzy's MaxDiff Experimental Design Deep Dive

MaxDiff Experimental design plays a key role in producing accurate and reliable results. A strong MaxDiff design will ensure that there is little to no bias in attribute exposure. Luckily, most survey platforms, including Suzy, provide an optimized experimental design for you! The optimal experimental design is based on key study inputs and core experimental design principles, which we will explain below. 

Key Study Inputs: 

  • Number of attributes to be tested
  • Number of desired responses/sample size
  • Attribute complexity 

All of the above are specified by the user. In certain cases, the number of attributes will dictate the optimal sample size for the study. Generally, the larger number of attributes being tested, the higher your sample size should be. For example:

  • If you are testing 5 attributes, you will need a minimum of 120 respondents
  • If you are testing 6 attributes, you will need a minimum of 720 respondents
  • If you are testing 7+ attributes, you will need a minimum of 1,000 respondents

Please be mindful when adding additional targeting criteria to ensure that you are able to reach the minimum sample needed.   

Core Experimental Design Principles: 

  • Frequency balance: Each item appears an equal number of times to each respondent and across all respondents
  • Orthogonality: Each attribute is shown in the same sets with each other attribute an equal number of times
  • Across-set positional balance: Each attribute appears an equal number of times in each of the subsets
  • Within-set positional balance: Each attribute appears an equal number of times in the top, middle, and bottom positions in the sets. All items are either directly or indirectly compared to all other items. 
  • Connectivity: The above four principles can be followed, and even still, two attributes could be considered connected, which means they appear near one another more often than any other combination. Minimizing this connectivity ensures that no two attributes are shown in the same set (first, second, third) and same position (top, middle, bottom) more often than any other combinations. This allows each attribute to truly get a fair chance to be chosen as best or worst. 

By ensuring the above criteria are met as best as possible, all attributes can be placed on a common utility scale by calculating the utility score for each attribute. 

Together, the key study inputs and the core design principles will determine the design of your MaxDiff. Suzy's MaxDiff action was designed to ensure all key study inputs and experimental design principles work together to give you a balanced and experimental rigor when running your research. You will take care of determining the number of attributes and desired sample size, and Suzy will take care of the rest!

In order to ensure valid results, it is critically important that your MaxDiff action fills to the desired number of respondents. If your MaxDiff action is closed early or does not fill to the specified number of respondents, there is a high likelihood that your study is not balanced, potentially causing exposure biases towards certain attributes.