Volume 87 Issue 32 | pp. 36-37
Issue Date: August 10, 2009

MS Proteomics On Firmer Footing

Multilab studies assess performance in mass spectrometry-based proteomics experiments
Department: Science & Technology
Keywords: proteomics, HUPO, mass spectrometry, multiple-reaction monitoring, CPTAC, National Cancer Institute
Susan Abbatiello (left) and Hasmik Keshishian of Carr’s lab at the Broad Institute perform MRM-MS analyses on a triple-quadrupole mass spectrometer.
Credit: Maria Nemchuk/Broad Institute
Susan Abbatiello (left) and Hasmik Keshishian of Carr’s lab at the Broad Institute perform MRM-MS analyses on a triple-quadrupole mass spectrometer.
Credit: Maria Nemchuk/Broad Institute

Despite the increasing popularity of mass spectrometry for both discovery-driven and quantitative proteomics, doubts have persisted about the reproducibility of the measurements. Those doubts must be put to rest before such methods will be widely accepted for identification, verification, and validation of protein biomarkers of disease.

Two recent multilaboratory studies suggest that the outlook for mass spectrometry-based proteomics is promising, although trouble spots remain.

In the first study, sponsored by the Human Proteome Organization (HUPO), 24 academic and three vendor laboratories analyzed a sample consisting of 20 human proteins by mass spectrometry (Nat. Methods 2009, 6, 423). Each protein contained at least one unique 1,250-dalton peptide, and participants were asked to analyze these peptides as well.

Other than providing the participants a database to search, the organizers gave the participants free rein to carry out the analysis, says lead author John J. M. Bergeron of McGill University, in Montreal. They asked the participants to characterize the proteins using whatever protocols and instrumentation they typically use in discovery-based proteomics experiments. Because the researchers don’t know what proteins are present, they must acquire full MS and MS/MS (tandem) mass spectra and look for all possibilities.

At first blush, the results weren’t encouraging. Only seven laboratories correctly identified all 20 proteins on the first go-around. The results were even worse for the 1,250-Da peptides—only one laboratory correctly identified all the peptides. With coaching, all laboratories correctly identified all 20 proteins. But even with coaching, only one other lab identified the peptides.

If the study had stopped there, the outcome would be disheartening. But the HUPO study organizers took the extra step of collecting the raw mass spectral data from all participants to try to pinpoint the sources of error. They found that everybody had sufficient data to identify the proteins and peptides, Bergeron says. Many of the labs reported fewer tandem mass spectra, which are needed to identify both the proteins and the peptides, than they had actually acquired, Bergeron says.

The 1,250-Da peptides exercise served to test the ability of the mass spectrometers and their operators to acquire enough tandem mass spectra to distinguish 22 peptides with the same nominal mass. “Most folks were not spending enough time to ensure they have enough tandem mass spectra to assure themselves that they could get the 1,250-Da peptides,” Bergeron says. But it’s also fair to say that even when they had the necessary spectra, they didn’t know how to find the peptides, he says.

One of the main concerns in discovery-driven proteomics is whether mass spectrometers collect enough tandem mass spectra. Such collection is a stochastic process that depends on the instrument selecting enough precursor ions for subsequent fragmentation. All the labs had enough data to identify all the proteins and most of the peptides, regardless of the type of mass spectrometer, Bergeron emphasizes. “The technology is fabulous,” he says.

Bergeron is less sanguine about the software and databases people use. “There’s a false assurance that the search engines and databases are able to handle high-quality tandem mass spectra,” he says. “There may be a false sense of security that if you have high-quality tandem mass spectra, then you’ve overcome the big hurdle and everything else should be automatic. It’s not.”

Some caveats about the study are in order, however. Because the samples were equimolar mixtures of 20 proteins in buffer solution, they did not mimic a true proteomics experiment either in the dynamic range of the protein concentrations or in the complexity of the matrix.

“The HUPO study, while useful for testing system performance and database searching for a protein test mixture, is not a suitable analog for proteomics experiments,” says Steven A. Carr, director of proteomics at the Broad Institute of Massachusetts Institute of Technology and Harvard University, in Cambridge, Mass., and a participant in the HUPO study. “Few in the proteomics community would call this a proteomics experiment by today’s standards. Nevertheless, this study clearly demonstrates that there are problems yet to be resolved in the use of discovery proteomics technologies, even in the case of the analysis of simple protein mixtures.” Those problems appear to be software and data analysis issues, although issues with sample preparation cannot be ruled out, he adds.

The test sample represents a standard against which other laboratories can assess their own performance. “We clearly need standards, and we clearly need training,” says William S. Hancock, a chemistry professor at Northeastern University, editor of the Journal of Proteome Research, and a participant in the HUPO study. “I don’t think there’s ever going to be just one standard. There will be protein mixes, and there’ll be strong suggestions for someone new in the field to use a certain mix,” he adds.

The second study focused on benchmarking an MS method that is used to verify and quantify the presence of particular proteins in complex biological samples such as human plasma. Organized under the aegis of the National Cancer Institute’s (NCI’s) Clinical Proteomic Technology Assessment for Cancer (CPTAC) program, the study assessed the precision and reproducibility of multiple-reaction monitoring (MRM) MS measurements of proteins in plasma (Nat. Biotechnol. 2009, 27, 633). In MRM-MS, the presence of selected proteins is verified by monitoring specific transitions from precursor to fragment ions in tandem mass spectra of peptides derived from the targeted proteins. The levels of protein present are quantified by stable-isotope dilution using synthetically labeled versions of the peptide analytes. Such measurements could become important for verification and eventual clinical validation of biomarkers.

“We carried out the first large-scale evaluation of multiplexed MRM-MS for quantitative measurement of biomarker candidates in plasma,” says Carr, who was the lead author of the CPTAC study. “What gives MRM-MS its level of specificity is the fact that you have multiple criteria that are being used to judge that you’re measuring the right thing.” These criteria are the chromatographic retention time, coelution of the labeled standard, the mass of the labeled and unlabeled peptides, the fragmentation pattern, and the relative abundance of fragments to each other and to the internal reference standard. “These factors give us very high confidence that we are detecting what we think we are and also enable us to detect and eliminate interferences, if present,” Carr says.

In the three-tier study, eight labs quantified 11 peptides from seven proteins spiked at nine different concentrations. In the first two phases, peptides and digested proteins were spiked into digested plasma before being sent to participants. In the third phase, the participating laboratories received samples of undigested plasma spiked with the seven target proteins at each of the nine concentrations.

Even in the third phase, which most closely simulates a biomarker verification experiment, the variation in measurements within and between labs was as good as 20% across the entire concentration range. Such values are still higher than but within striking distance of the reproducibility acceptable for clinical assays, which is in the 10 to 15% range.

The raw data from both the HUPO and CPTAC studies have been deposited in Tranche, a publicly available proteomics database at the University of Michigan. The reagents from the CPTAC study are available from NCI.

“Other labs that want to get into this game and benchmark their performance in measuring proteins in plasma now have a reasonable starting point,” Carr says. These data sets, for which the levels and identities of the proteins are known, “provide a rich source for software developers to refine and develop new algorithms for analysis of highly multiplexed MRM-MS data,” he notes. The demonstrated reproducibility should help speed the acceptance of MRM-MS by both the proteomics and clinical communities, Carr says. “This isn’t just about biomarkers. It’s really about targeted measurement of any protein in any system.”

Chemical & Engineering News
ISSN 0009-2347
Copyright © American Chemical Society

Leave A Comment

*Required to comment