When people take the sleeping drug zolpidem (Ambien), they sometimes end up with embarrassing stories—raiding the refrigerator while sleepwalking, going on Amazon.com shopping sprees they don’t remember, or texting incoherent messages to friends and family.

Of course, not all the side effects are funny. For example, in 2010, twice as many women went to the emergency room because of adverse reactions to zolpidem as did men.

By 2013, the Food & Drug Administration told makers of medications containing the drug to cut the zolpidem dose in half for women. FDA based its decision in part on data demonstrating that women clear the drug from their bodies more slowly than men do, meaning that more women are still drowsy the day after taking it.

“That is a great real-life example where a drug was shown to have a big difference between men and women,” says Monica P. Mallampalli, the vice president of scientific affairs at the Society for Women’s Health Research.

Zolpidem isn’t the only drug that wields a difference in side effects among the sexes, Mallampalli says. Between 1997 and 2000, eight of the 10 drugs withdrawn from the U.S. market posed a greater health risk to women than to men, according to a 2001 report from the Government Accounting Office (now named the Government Accountability Office).

Men and women are also prone to some diseases at different rates, Mallampalli says. For example, women are twice as likely to have depression as men. And autism spectrum disorder is 4.5 times as common in boys as in girls.

None of this is surprising to scientists who study the biology of sex differences. They often point to the succinct conclusion of a 2001 report from the U.S. Institute of Medicine (IOM; now named the Health & Medicine Division): Sex matters. The report explained that every cell in our bodies has a sex and that sex differences start in the womb and continue throughout our lives. As a result, the report recommended that scientists study sex differences “from womb to tomb.”

But some scientists warn that researchers are still not adequately considering sex when studying the biological systems that underlie human health. In fact, in many fields of biomedical research, scientists continue to carry out basic and preclinical research predominantly in one sex of animals, most often males. Until this sex bias gets corrected, some scientists worry biomedical research will continue to build its knowledge of health and disease on an incomplete foundation.

To repair that foundation and ensure basic and preclinical research is as rigorous as possible, the U.S. National Institutes of Health implemented a policy last year that requires all researchers applying for funding to consider sex as a variable in their proposed research. NIH hopes the policy pushes biomedical researchers to include more female animals and cells in their studies, paving the way for new discoveries of how sex affects our biology.

Not all scientists agree with the policy. Some critics worry that the policy’s implementation will lead to scientists ignoring the role of social and psychological variables in health differences between men and women, overburden already tight research budgets with additional experiments, and flood the literature with subpar data produced by researchers going through the motions to fulfill the new requirement.

The problem and a solution

In 1993, the U.S. Congress addressed a similar sex bias in clinical research in humans. Until then, clinical trials often lacked women subjects. The NIH Revitalization Act that year required that NIH-funded trials start including them.

Sex-difference researchers think a similar policy change is now needed in basic and preclinical research. “Basic research on animals is at the foundation for much of the biological knowledge on which medicine rests,” says Arthur Arnold, who studies sex differences at the University of California, Los Angeles, and is editor-in-chief of Biology of Sex Differences. “We study animals to learn basic biological principles.”

When early-stage research doesn’t include both male and female animals or cells, scientists can’t determine whether those principles apply to both sexes, Arnold says.

“There is a huge implication for drug development,” says Jeffrey S. Mogil, a pain researcher at McGill University. If all the basic and preclinical experiments on a particular disease pathway are carried out in male animals, then researchers will not know if a given drug target behaves the same way in females.

A handful of studies in the past decade have highlighted the extent of the sex bias in early-stage research. Mogil reviewed papers published in Pain from 1996 to 2005 and found that 79% of them studied only male animals, while less than 10% studied both sexes (Pain 2005, DOI: 10.1016/j.pain.2005.06.020).

Annaliese K. Beery and Irving Zucker of the University of California, Berkeley, performed a larger analysis of sex bias in biomedical research. The duo read through 2,000 or so papers from four of the top journals in each of 10 biomedical fields in 2009. They found biases toward male animals in eight of the 10 fields. The fields with the highest male-to-female ratios were neuroscience at 5.5:1, pharmacology at 5:1, and physiology at 3.7:1. Reproduction and immunology were the only two female-skewed fields with female-to-male ratios of 1.6:1 and 2.2:1, respectively (Neurosci. Biobehav. Rev. 2011, DOI: 10.1016/j.neubiorev.2010.07.002).

So why do so many researchers study only male animals?

Some sex-difference researchers claim there has long been an unspoken assumption that males and females don’t differ much. “The prevailing idea was that if you study males you’re going to get the basic biology, and it will probably apply to females,” Arnold says.

But, as the 2001 IOM report pointed out, every cell has a sex. Cells in many organs and tissues that have nothing to do with reproduction still contain receptors for sex hormones, says Virginia M. Miller, a sex-difference researcher at Mayo Clinic. When the hormones bind to these receptors, they change a cell’s biology by regulating protein synthesis and cell signaling pathways. And then there are the sex chromosomes in each cell’s nucleus. Arnold’s lab and others have found that the presence of a Y chromosome, or two X chromosomes, can significantly affect how a cell behaves.

Another assumption that has possibly contributed to the male sex bias is that data from female animals are more variable than data from males, Mogil says. Scientists have worried that the fluctuating hormone levels that are part of female animals’ estrous cycle cause greater variability in traits when compared with males.

“That’s perfectly reasonable as an assumption, but it’s empirically wrong,” Mogil says. In his review of Pain papers, Mogil also analyzed raw data from standard pain tests in mice and found no difference in the variability between males and females. UC Berkeley’s Zucker performed a broader literature review in 2014 by analyzing the variability of 30 categories of mouse traits in 293 papers. He too found no difference between males and females (Neurosci. Biobehav. Rev. 2014, DOI: 10.1016/j.neubiorev.2014.01.001).

No matter why people started working mainly with male animals, the practice has become the convention in many fields. And it’s hard to break away from convention, says Beery, who is now at Smith College. When much of the data on a certain biological system have been collected in male animals, it’s easier to continue to use males to test a new hypothesis instead of going back to establish the same baseline data in females.

This limitation made it hard in some fields to convince researchers that experiments with females were warranted, Beery says. She and others in neuroscience have had peer reviewers ask them to justify why they studied only females, or females in addition to males. “You’d rarely have the same scrutiny if you proposed an all-male study,” Beery says. “It was just the default.”

Such conventions now face a challenge from an NIH policy that went into effect at the beginning of 2016.

NIH Director Francis S. Collins and Janine A. Clayton, who is the director of the NIH Office of Research on Women’s Health, announced plans to develop a policy to balance sex in biomedical research in 2014 (Nature, DOI: 10.1038/509282a). In that article and elsewhere, Clayton explains that the policy is part of NIH’s drive to improve reproducibility in the research it funds. “The tenets of the scientific method are to ensure robust and unbiased design, analyses, and reporting,” Clayton says. “If you are including only one sex from the beginning for no compelling reason, then by definition your research is biased.”

In the new policy, NIH asks scientists to consider how sex may play a role in the systems they study and then propose experiments accordingly. “The policy in a nutshell is that NIH expects investigators to account for sex as a biological variable in their research designs, analyses, and reporting,” Clayton says.

Under the new policy, NIH grant applications don’t have a specific box to check or new section to complete. Applicants just describe how they will consider sex in their planned research strategy. Clayton stresses that there is no one way to comply with the policy; the specifics of a biological system and the questions being asked will determine how a scientist might approach experimental design with sex in mind.

And the policy doesn’t prohibit scientists from studying just one sex. It requires that the applicants justify why running experiments in one sex is necessary, for example, when studying sex-specific diseases such as prostate cancer or when resources are scarce, such as with experiments in nonhuman primates.

The people charged with assessing these sex considerations are the grant reviewers in NIH study sections. “They are the best scientists in that field,” Clayton says. “They are in the best position to assess whether an application is appropriately accounting for sex as a biological variable in the context of what is known in that field.”

The critics

After Collins and Clayton announced plans to develop this new policy in 2014, NIH sought comment from the scientific community and public, receiving responses from 222 scientific societies, patient advocacy groups, and individuals. Eighty-seven percent of respondents agreed that sex was a critical variable in studying biological systems. But 86% had at least one concern about implementing such a policy. The biggest concerns were about the research costs and complexity of experimental designs that would be necessary to comply with the policy.

The policy announcement also sparked comments from critics. R. Douglas Fields, a neuroscientist in Bethesda, Md., has written a couple of critiques of the policy, including a letter to the editor of Nature in response to Collins and Clayton’s original announcement (2014, DOI: 10.1038/510340a). As with other critics of the policy, Fields acknowledges sex differences exist, are significant, and should be studied.

“There is no debate about the need to study sex differences in biology,” he says. “The question is, what is the right way to go about it?”

Criticisms of the policy differ in specifics, but most come down to questioning the effectiveness of asking all scientists to consider biological sex in their research. By creating a blanket policy, NIH has overlooked scientific and practical concerns, critics say.

In a perspective critiquing the policy, Lise Eliot, a neuroscientist at the Chicago Medical School of Rosalind Franklin University of Medicine & Science, and Sarah S. Richardson, a science historian at Harvard University, argue that focusing on sex differences in animal models will be less fruitful than understanding the impact of cultural, social, and economic factors in shaping health differences between men and women (J. Neurosci. 2016, DOI: 10.1523/jneurosci.1391-16.2016).

For example, Eliot says diagnostic criteria, especially in mental health, can be laden with gender stereotypes that skew the prevalence of disease. Depression diagnostic criteria don’t include anger, aggression, substance abuse, and risk taking, which men tend to report more often than do women. Researchers led by Lisa A. Martin at the University of Michigan, Dearborn, added such symptoms to traditional depression criteria and found that the prevalence of depression in men and women equalized (JAMA Psychiatry 2013, DOI: 10.1001/jamapsychiatry.2013.1985).

Proponents of the NIH policy do not disagree that these social and cultural factors contribute to health differences between men and women. But they argue that biological sex differences still exist and should be studied.

One of the most persistent criticisms of the NIH policy is that it would require researchers to effectively double the number of animals they study, leading to increased research costs. Researchers in this camp claim that to effectively take sex into account in most experiments, they would have to run their experiments in parallel in males and females.

The policy doesn’t require this, Clayton argues. “It is not a requirement to double a sample size,” she says. “And it is not a requirement for every experiment to be powered to detect sex differences.”

In a perspective last year in Nature, McGill University’s Mogil proposed that researchers keep their usual number of animals but work with half males and half females to satisfy the NIH policy (Nature 2016, DOI: 10.1038/535S7a). He admits that such an approach will not have sufficient statistical power to detect small sex differences, but it could still find big ones. Clayton also thinks that for basic research, reporting data collected from both sexes is valuable, even if a researcher isn’t analyzing the data for sex differences. “That will continue to build the evidence base,” she says. Down the road, another person may build on these data and design an experiment to uncover a sex difference.

But running half-male/half-female experiments often won’t work, Fields contends. First, he argues, scientists sometimes use one sex over the other for a reason. For example, the mouse that best models multiple sclerosis is a female mouse, he says. So when researchers want to understand some aspect of the disease mechanism or test possible treatments, they reach for a female, not a male, mouse.

Also, Fields says, mixing both sexes in an experiment increases variability in the data, which can hinder a researcher’s ability to test a hypothesis. Often, a trait a researcher is measuring in an experiment will differ between male and female animals; for example, maybe male mice finish a maze on average faster than females do. So, Fields says, pooling the sexes’ measurements leads to a larger spread in those data, making it harder to spot any difference between control and experimental groups.

As Fields’s argument goes, pooling males and females would lower the statistical power of scientists’ experiments. David Elashoff, a biostatistician at UCLA, explains statistical power like this: “Assuming that your starting hypothesis is correct, what is the chance that your experiment will prove that statistically?”

Elashoff agrees that the variance in the pooled data would increase because both males and females are present. But he points out that there are experimental designs, such as 2 × 2 factorial designs, and statistical analyses, such as analysis of variance, that would allow scientists to account for this added variance so researchers could still test their hypotheses with adequate power.

Elashoff says these methods still won’t allow a researcher to demonstrate that an effect or phenomenon occurs in both males and females. The methods just allow the researcher to compare control and experimental groups while providing equal representations of each sex. To demonstrate both sexes are behaving similarly, researchers would have to run the experiment separately in males and females.

And, Elashoff says, some researchers worry that simply providing equal representation of males and females will not meet some grant reviewers’ expectations for adequate accounting for sex. The UCLA biostatistician helps colleagues write about 30 NIH grants each year. He says some of his colleagues want to avoid any criticism that could hurt the chances of their grants being funded, so they err on the side of thoroughness and propose studying both sexes, doubling their sample sizes.

Concerns about the numbers of animals used in experiments eventually become budget concerns. It costs a lot to buy, breed, and care for lab animals.

NIH doesn’t report how much it spends on lab animals in the U.S., and most researchers are reluctant to disclose such details about their animal research. But at the low end of the cost scale, a three-week-old C57BL/6 mouse—which is one of the most popular inbred strains—costs about $20. Other more specialized strains can cost hundreds of dollars.

Housing and caring for animals also carry costs. Animal housing facilities charge researchers per cage per day to cover feeding the animals, keeping their cages clean, and observing the animals’ health. Depending on the facility and specific needs, such as biohazard protocols, most rates for mice are between 50 cents and a couple dollars per cage per day.

As a result, purchasing and caring for a few hundred C57BL/6 mice for a year can cost tens of thousands of dollars.

Proponents of the NIH policy don’t dismiss concerns about additional animal costs. “There may be increased costs in the beginning for some investigators, under some circumstances, to do the experiments in the most rigorous ways,” Clayton says. “Those investigators should make the case for that design and prepare a budget that appropriately addresses the increased need. That is addressed in peer review.”

But Fields says that beyond the cost issue, requiring researchers to consider sex in their experimental designs adds another way for reviewers to reject a proposal, further increasing competition for funding. He worries that to stay competitive, many researchers will treat this requirement as a box to check off, proposing experiments that include females but are underpowered to make any meaningful conclusions about how sex affects the given system.

So instead of asking everyone to think about sex, Fields thinks NIH should make specific calls for sex-difference research in areas it deems important. That approach will attract scientists who are interested in and experienced with studying sex differences, leading to higher-quality research.

Sex-difference research is a specialized field. “People who don’t know what they’re doing may do it wrong,” says Louise D. McCullough, the department chair of neurology at the University of Texas Health Science Center at Houston. She agrees in part with Fields’s suggestion that NIH put out specific calls for sex-difference research. “And actually NIH does.”

McCullough is referring to additional funding that NIH has given to researchers who already have grants from the institutes. These administrative supplements allow the scientists to go back and determine if a sex difference exists in the system they’ve already been studying. “We’ve actually spent over $16 million in the last three years to help investigators to start to look at these issues,” Clayton says.

But many proponents of the NIH policy think it’s important to distinguish between designing experiments to possibly detect sex differences and studying how those differences arise. They see the NIH policy as encouraging the former.

“I don’t think everyone should be intensely studying the causes of sex differences,” Mayo Clinic’s Miller says. “But researchers ought to do at least some modicum of work to answer the question of whether their main findings are true in both sexes.”

Looking forward

Because the policy has been in effect for just over a year, Clayton thinks it’s too early to measure its success. “We’re looking forward to the new insights that are going to be gained by applying considerations of sex as a biological variable,” she says. “That ultimately will be the true value of the policy. But it’s too early to assess that.”

When Canada implemented a similar policy in 2010, the country saw some changes early on. In that policy, the Canadian Institutes of Health Research requires scientists applying for funding to answer two questions: Does biological sex play a role in the proposed research? And does gender—the social and cultural aspects of sex—play a role? Applicants then have to explain why or why not.

An analysis of application data from three grant competition cycles—December 2010, June 2011, and December 2011—revealed an increase in the number of applicants answering yes to at least one of the two questions. The rate was 26% in December 2010 and 48% a year later (PLOS One 2014, DOI: 10.1371/journal.pone.0099900).

For the NIH policy to have lasting effects, proponents say it must have teeth: Study sections must consider the new requirement earnestly. McCullough worries that if reviewers are reluctant to penalize applicants for not addressing sex as a variable in their research, then scientists will not take the policy seriously.

Clayton says her office has been supporting grant reviewers by providing resources on considering sex as a biological variable. The office is also developing ways to evaluate applications of the policy. She points out that some institutes in NIH have published review articles and other materials to address how sex may play a role in their specific disciplines. For example, the director of the National Institute on Alcohol Abuse & Alcoholism published a review article last year on ways to think about studying sex differences in animal models of addiction.

Clayton has also reached out to scientific societies and journal editors. When Beery and Zucker reviewed the biomedical literature in their 2011 paper, they found that the percentage of articles that entirely omitted the sex of the animals studied varied depending on the field, from fewer than 8% to more than 60%. Clayton and NIH have been working to lower those rates by encouraging editors to adopt policies that require authors to report the sex of the animals used and to present data broken down by sex.

Journals have made some progress on this front, Clayton says. Proceedings of the National Academy of Sciences, Nature Publishing Group journals, and Cell Press journals, for example, require authors to describe the sex of animals used in experiments.

PLOS journals encourage authors to follow a set of international guidelines on animal research reporting that include reporting the sexes of animals.

But the Science family of journals and American Chemical Society journals don’t have such a requirement. ACS publishes C&EN.

A Science spokesperson says the journals would be open to considering such a policy in the future. And an ACS Publications spokesperson tells C&EN: “Given the limited number of articles ACS publishes where this NIH policy may be relevant and noting that our editorial approach is already broadly aligned to this directive, we don’t expect to introduce any additional specific policy at this time.”

Overall, advocates of sex-difference research see NIH’s recent actions as a step in the right direction, in particular because it is raising awareness about how sex plays an important role in biology. “If we go looking for sex differences in basic biological processes and we don’t find them, and then we go back to using whatever sex happens to be handy, that is a fine outcome for me,” Smith’s Beery says. “But I think we need to do some real basic-science exploration first because we’re only scratching the surface of some very important differences.”