11 Designing a Research Study

From Variables to Research Design

Variables and Operational Definitions

Part of generating a hypothesis involves identifying the variables that you want to study and operationally defining those variables so that they can be measured. Like we discussed in previous chapters, once we have established a set of propositions on the theoretical plane, we need to establish how these translate to hypotheses on the empirical plane. This involves deciding which variables to use, such as the previous example where intelligence could be measured using a person’s intelligence quotient (IQ).

A variable is a quantity or quality that varies across people or situations. For example, the height of the students enrolled in a university course is a variable because it varies from student to student. The chosen major of the students is also a variable as long as not everyone in the class has declared the same major. Variables can be either quantitative or categorical. A quantitative variable is a quantity, such as height, that is typically measured by assigning a number to each individual. Other examples of quantitative variables include people’s level of talkativeness, how depressed they are, and the number of siblings they have. A categorical variable is a quality, such as chosen major, and is typically measured by assigning a category label to each individual (e.g., Marketing, Change Management, Innovation Management, etc.). Other examples include people’s nationality or their occupation.

After the researcher generates their hypothesis and selects the variables they want to manipulate and measure, the researcher needs to find ways to actually measure the variables of interest, which requires operational definitions. Like discussed in a previous chapter, this involves the definition of a variable on the empirical plane. The definition should specify precisely how it is to be measured. Most variables that researchers are interested in studying cannot be directly observed or measured and this poses a problem because empiricism (observation) is at the heart of the scientific method. Most variables can be operationally defined in many different ways. Researchers are wise to choose an operational definition that has been used extensively in the research literature.

Qualtitative vs. Quantiative Research

One important choice that a researcher has to make in their research design is whether to use quantitative research or qualitative research. There are many methods in both designs, and which method a researcher chooses entirely depends on their research goals. Moreover, sometimes both go hand-in-hand in cases of so-called mixed designs.

If the goal is to test a clearly defined hypothesis between a clear and relatively small set of variables, quantitative methods may be more advantageous. Quantitative methods are advantageous in that they give precise answers to well-defined questions, which generalize to larger populations. However, qualitative methods perform better in other areas. For example, a researcher might not know where to begin: many research fields start off with qualitative methods to gain an initial overview of which variables might be relevant to study. Another areas where qualitative methods can do well are in their sensitivity to context. Whereas quantitative need to specify exactly which variables they’re interested in in advance, qualitative research may be more open to finding other contextual factors that are relevant. In a few chapters, we will discuss specific quantitative research designs (like experiments) and qualitative research designs (like interviews) in greater detail.

Sampling and Measurement

In addition to identifying which variables to manipulate and measure, operationally defining those variables and which exact research design to use, researchers need to identify the population of interest. Researchers in business are usually interested in drawing conclusions about some very large group of people. This is called the population. It could be customers, managers, employees, CEO’s, or other groups or subjects. Actual studies however usually involves studying only a small subset or sample of the population, often due to time- or budget constraints. For example, a researcher might measure CEO pay for a small set of CEO’s, or observe managerial effectiveness for a small set of managers. It is important, therefore, for researchers to use a representative sample—one that is similar to the population in important respects.

One method of obtaining a sample is simple random sampling, in which every member of the population has an equal chance of being selected for the sample. For example, a pollster could start with a list of all the registered voters in a city (the population), randomly select 100 of them from the list (the sample), and ask those 100 whom they intend to vote for. Unfortunately, random sampling is difficult in most business research because the populations are less clearly defined than the registered voters in a city. How could a researcher give all managers an equal chance of being selected for a sample? The most common alternative to random sampling is convenience sampling, in which the sample consists of individuals who happen to be nearby and willing to participate (such as freshman business administration students). Of course, the obvious problem with convenience sampling is that the sample might not be representative of the population and therefore it may be less appropriate to generalize the results from the sample to that population.

Experimental Research

The next step a researcher must take is to decide which type of approach they will use to collect the data. As you will learn in this course, there are many different approaches to research that can be divided in many different ways. Where quantitative research designs are concerned, one of the most fundamental distinctions is between experimental and non-experimental approaches.

Researchers who want to test hypotheses about causal relationships between variables (i.e., their goal is to explain) should typically use an experimental method where possible. This is because the experimental method is usually the most obvious methods for determining causal relationships. Using the experimental approach, researchers first manipulate one or more variables while attempting to control extraneous variables, and then they measure how the manipulated variables affect participants’ responses.

The terms independent variable and dependent variable are used in the context of experimental research. The independent variable is the variable the experimenter manipulates (it is the presumed cause) and the dependent variable is the variable the experimenter measures (it is the presumed effect).

Extraneous variables are any variable other than the dependent variable. Confounds are a specific type of extraneous variable that systematically varies along with the variables under investigation and therefore provides an alternative explanation for the results. When researchers design an experiment they need to ensure that they control for confounds; they need to ensure that extraneous variables don’t become confounding variables because in order to make a causal conclusion they need to make sure alternative explanations for the results have been ruled out.

As an example, if we manipulate the lighting in the room and examine the effects of that manipulation on workers’ productivity, then the lighting conditions (bright lights vs. dim lights) would be considered the independent variable and the workers’ productivity would be considered the dependent variable. If the bright lights are noisy then that noise would be a confound since the noise would be present whenever the lights are bright and the noise would be absent when the lights are dim. If noise is varying systematically with light then we wouldn’t know if a difference in worker productivity across the two lighting conditions is due to noise or light. So confounds are bad, they disrupt our ability to make causal conclusions about the nature of the relationship between variables. However, if there is noise in the room both when the lights are on and when the lights are off then noise is merely an extraneous variable (it is a variable other than the independent or dependent variable) and we don’t worry much about extraneous variables. This is because unless a variable varies systematically with the manipulated independent variable it cannot be a competing explanation for the results.

Other Research Designs: Quasi-experimental and Observational research

Besides experiments, there are many other (quantitative) research designs. Some of these methods are more descriptive in nature. Researchers who are simply interested in describing characteristics of people, describing relationships between variables, and using those relationships to make predictions can use non-experimental research methods. For example, an HRM/OB researcher interested in relating transformational leadership to employee commitment might simply ask a set of managers about their management style to assess whether they can be classified as using transformational leadership. Next, they might ask employees working in said manager’s team to fill out a survey that aims to measure their commitment. With these pieces of information, the researcher can assess whether a (statistical) relationship exists between both variables. In contrast to experiments however, such findings are not causal: correlation does not equal causation. If they find a relationship between transformational leadership and employee commitment, there might be other factors at play. For example, maybe both the manager and the employee work at a firm which tends to hire excellent, dedicated and high quality workers, causing the firm to both hire transformational leaders and committed workers. If this is the case, you would see a relationship between both of these variables without an actual relationship being there.

However, there are also research designs/methods that aim to make observational (that is: nonexperimental) data useful for causal inference, though this usually requires a stricter set of assumptions. The key challenge facing many researchers is that in many situations, it is simply unethical or completely impossible to perform experiments. Psychologists cannot start torturing respondents, neither can economists start manipulating the economy. Luckily, there are many so-called quasi-experimental research methods out there that aim to make observational data useful for causal inferences. While it goes beyond  the scope of this book, methods such as regression discontinuity designs, difference-in-difference approaches, and many, many more hold such promise. In fact, causal inference is so central to many areas of science that developing methodology to do causal inference on such observational data is still a very active area of research.

Laboratory vs. Field Research

The next major distinction between research methods is between laboratory and field studies. A laboratory study is a study that is conducted in the laboratory environment. In contrast, a field study is a study that is conducted in the real-world, in a natural environment.

Laboratory experiments typically have high internal validity. Internal validity refers to the degree to which we can confidently infer a causal relationship between variables. When we conduct an experimental study in a laboratory environment we have very high internal validity because we manipulate one variable while controlling all other outside extraneous variables. When we manipulate an independent variable and observe an effect on a dependent variable and we control for everything else so that the only difference between our experimental groups or conditions is the one manipulated variable then we can be quite confident that it is the independent variable that is causing the change in the dependent variable. In contrast, because field studies are conducted in the real-world, the experimenter typically has less control over the environment and potential extraneous variables, and this decreases internal validity, making it less appropriate to arrive at causal conclusions.

But there is typically a trade-off between internal and external validity. External validity simply refers to the degree to which we can generalize the findings to other circumstances or settings, like the real-world environment. When internal validity is high, external validity tends to be low; and when internal validity is low, external validity tends to be high. So laboratory studies are typically low in external validity, while field studies are typically high in external validity. Since field studies are conducted in the real-world environment it is far more appropriate to generalize the findings to that real-world environment than when the research is conducted in the more artificial sterile laboratory.

Finally, there are field studies which are non-experimental in nature because nothing is manipulated. But there are also field experiments where an independent variable is manipulated in a natural setting and extraneous variables are controlled. Depending on their overall quality and the level of control of extraneous variables, such field experiments can have high external and high internal validity.