Estimating relative abundances of proteins from shotgun proteomics data

McIlwain et al. BMC Bioinformatics 2012, 13:308http://www.biomedcentral.com/1471-2105/13/308

SOFTWARE Open Access

Estimating relative abundances of proteinsfrom shotgun proteomics dataSean McIlwain1, Michael Mathews1, Michael S Bereman1, Edwin W Rubel2,3, Michael J MacCoss1

and William Stafford Noble1,4*

Abstract

Background: Spectral counting methods provide an easy means of identifying proteins with differing abundancesbetween complex mixtures using shotgun proteomics data. The crux spectral-counts command,implemented as part of the Crux software toolkit, implements four previously reported spectral counting methods,the spectral index (SIN), the exponentially modified protein abundance index (emPAI), the normalized spectralabundance factor (NSAF), and the distributed normalized spectral abundance factor (dNSAF).

Results: We compared the reproducibility and the linearity relative to each protein’s abundance of the four spectralcounting metrics. Our analysis suggests that NSAF yields the most reproducible counts across technical and biologicalreplicates, and both SIN and NSAF achieve the best linearity.

Conclusions: With the crux spectral-counts command, Crux provides open-source modular methods toanalyze mass spectrometry data for identifying and now quantifying peptides and proteins. The C++ source code,compiled binaries, spectra and sequence databases are available at http://noble.gs.washington.edu/proj/crux-spectral-counts.

BackgroundExisting methods for differential proteomics (reviewed by[1]) fall into two categories: spectral counting methodsthat rely on counting the number of spectra that map toa given protein across multiple experiments, and peptidechromatographic peak intensity methods that use the areaunder the peptide precursor ion peak as a measure ofpeptide abundance. In principle, methods based on massspectrometry peak areas are potentially much more accu-rate, but these methods require highly reproducible liquidchromatography as well as accurate methods for chro-matographic alignment and identification of peaks withinthe profile spectra. In contrast, spectral counting meth-ods are straightforward to employ and have been shown tocorrectly detect known differences between samples [2],which contributes to their wide use.

The command line tool crux spectral-countsimplements four popular spectral counting methods: the

*Correspondence: [email protected] of Genome Sciences, University of Washington, Seattle, WA, USA4Department of Computer Science and Engineering, University ofWashington, Seattle, WA, USAFull list of author information is available at the end of the article

spectral index (SIN ) [3], the exponentially modified pro-tein abundance index (emPAI) [4], the normalized spectralabundance factor (NSAF) [5], and the distributed normal-ized spectral abundance factor (dNSAF) [6]. The cruxspectral-counts command is integrated within theCrux software toolkit, which provides actively main-tained open-source methods to identify and now quantifypeptides and proteins from shotgun mass spectrometrydatasets. Crux supports a variety of input spectra formats,and the tools can easily be incorporated into proteomicanalysis pipelines, such as the Trans-Proteomic Pipeline(TPP) [7]. Finally, the modular design of Crux allowsimprovements to one part of the toolkit to be propagatedthrough downstream analyses.

Currently, several software packages offer spectralcounting protein quantification methods [8]. ProteoIQ(http://www.bioinquire.com) and Scaffold [9] are com-mercial software products that post-process results froma variety of database search programs. Freely availabletools such as APEX [10], emPAI calc [11], and PepC [12]each offer a single spectral counting method. Table 1 com-pares the features of six software spectral counting tools.Crux offers more spectral counting methods than other

© 2012 McIlwain et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the CreativeCommons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, andreproduction in any medium, provided the original work is properly cited.

McIlwain et al. BMC Bioinformatics 2012, 13:308 Page 2 of 6http://www.biomedcentral.com/1471-2105/13/308

Table 1 Spectral counting software

Crux APEX emPAICalc

PepC ProteoQ Scaffold

Metrics

Provided

SIN X

emPAI X X

NSAF X

dNSAF X

Raw X X X

Other X X X

Other

Features

ParsimonyAnalysis

X X X

Peptide-LevelCounting

X

Free X X X X

Opensource

X X X

WebInterface

X X

Graphicaluserinterface

X X X

Scriptable X X X

This table summarizes the features of various spectral counting softwaremethods.

tools and is the only method to provide peptide-level inaddition to protein-level counts.

Using crux spectral-counts, we compared andcontrasted the reproducibility and linearity of the fourspectral counting methods. Our experiments suggest thatthe NSAF metric provides the most reproducible proteinquantification. In contrast, our linearity experiments showthat SIN and NSAF provide the best performance, withdNSAF providing intermediate performance and emPAIyielding the worst linearity.

The contributions of this paper are thus two-fold: wedescribe a performance comparison of the reproducibil-ity and linearity of the SIN , emPAI, NSAF, and dNSAFprotein quantification methods, and we provide to theproteomics community a flexible, open source spectralcounting software tool.

ImplementationSoftwareThe crux spectral-counts command is imple-mented as part of the Crux proteomics software toolkit[13]. The toolkit is implemented in C++ as a single binary

that supports commands for database searching and avariety of downstream analyses [14-18].

The crux spectral-counts command takes asinput a protein database in FASTA format and a collec-tion of peptide-spectrum matches (PSMs) produced by adatabase search procedure. The PSMs may be in Crux’stab-delimited text format, PeptideProphet’s PepXML ormzIdentML [19]. To compute the SIN score, a set of spec-tra must also be provided as input in MS2, mzXML, ormgf format. By default, crux spectral-counts willselect the PSMs in the input by a user modifiable thresholdof q-value ≤ 0.01.

For each protein with at least one spectral count, theprogram then computes the NSAF, dNSAF, emPAI, or theSIN score. The NSAF metric is defined as

NSAFN = sN/LN∑ni=1(si/Li)

where N is the protein index, sN is the number of spectramatched to protein N, LN is the length of protein N, and nis the total number of proteins in the input database.

The dNSAF metric is given by

dNSAFN =suN +∑k

j=1 dj,N ssj,N

LN

∑ni=1

sui +∑k

j=1 dj,issj,i

Li

where sun is the spectral count for the peptides uniquely

mapping to protein N, ssj,N is the spectral count of degen-

erate peptide j (out of the protein’s k degenerate peptides)mapped to protein N, and dj,N is the distribution factor ofpeptide shared counts, defined by the equation

dj,N = suN∑n

i=1 sui

The metric emPAI is defined as

emPAIN =

⎛⎝10

pobservedN

pobservableN − 1

⎞⎠

∑ni=1

⎛⎝10

pobservedi

pobservablei − 1

⎞⎠

where pobservableN is the number of unique peptides observ-

able for protein N and pobservedN is the number of unique

peptides observed for protein N.Finally, the SIN score is calculated using

SIN =∑pN

j=1

(∑sjk=1 ik

)

LN(∑n

j=1 SIj)

where pN is number of unique peptides in protein N, sjis the number of spectra assigned to peptide j, and ik is

McIlwain et al. BMC Bioinformatics 2012, 13:308 Page 3 of 6http://www.biomedcentral.com/1471-2105/13/308

the total fragment ion intensity of spectrum k. Analogousscores can also be computed for each peptide, rather thanfor each protein. A detailed description of the peptide-level scoring metrics is available in the on-line documen-tation. As output, crux spectral-counts producesa tab-delimited file listing proteins and their correspond-ing counts, in reverse sorted order.

The crux spectral-counts command also com-putes a parsimonious set of proteins, using the greedy setcover approach used by IDPicker [20]. Users thus havethe option of considering spectral counts only for proteinswithin the parsimonious set.

Data CollectionFor the reproducibility experiments, proteins wereextracted from the cochlear nucleus of the developingmouse brain at postnatal day 7 and postnatal day 21. Twobiological replicates were generated for each age by dis-secting out the cochlear nuclei from two separate mice ateach age. One of the 21-day mice was used to generate twosamples, thereby providing a technical replicate in addi-tion to a biological replicate. The samples prepared fromthe chicken brain were derived from nucleus laminaris,an auditory region in the brain stem. Samples were takenfrom the dorsal (D) and ventral (V) regions of this area.For each region, two biological replicates were generated,and one of those replicates was also subjected to techni-cal replication. Each sample was digested with trypsin andsubjected to liquid chromatography followed by tandemmass spectrometry.

For the linearity experiments, we used eight samplesthat represent a dilution curve of 48 known proteinssynthesized by Sigma (UPS1, http://www.sigmaaldrich.com). These data sets are mixtures (Std1–Std8) of theC. elegans lysate at equal concentrations and the 48 pro-teins, diluted by a two-fold in each successive standard.Std 8 has the lowest concentration of the known pro-teins (6 fmol) and Std 1 has the highest concentration(870 fmol).

All three data sets are publicly available at http://noble.gs.washington.edu/proj/crux-spectral-counts.

Data analysisThe fragmentation spectra from the experiments weresearched against their respective mouse, chicken, orthe C. elegans+UPS1 protein database using cruxsequest-search followed by crux q-ranker, withthe default parameters. crux spectral-counts wasapplied to the peptide-spectrum matches (PSMs) thatreceived q-values ≤ 0.01. The resulting data sets for themouse and chicken replicates are summarized in Addi-tional file 1: Table S1, and the UPS1 dilution curve datasets are summarized in Additional file 1: Table S2.

ResultsTesting reproducibility between replicatesTo investigate the reproducibility of the four spectralcount methods, we analyzed mass spectrometry datafrom technical and biological replicates from chicken andmouse samples. We then produced a scatter plot for eachpair of biological or technical replicates and computed thecorresponding Spearman correlation. For these compar-isons, proteins identified in only one of the two datasetswere ignored. Figure 1 shows sixteen such plots, corre-sponding to one biological and one technical replicatefor chicken and mouse, respectively. The complete collec-tion of 76 plots is provided as Additional file 1: FiguresS1–S2. From these analyses, as summarized in Table 2,we draw two primary conclusions. First, the spectralcounts are generally reproducible: the mean correlationvalue across all 76 pairs is 0.867, and the minimumcorrelation is 0.719. Second, reassuringly, the technicalreplicates produce higher correlations than the biologicalreplicates: the mean correlation among 24 pairs of tech-nical replicates is 0.885, whereas the corresponding valuefor the 52 pairs of biological replicates is 0.859 (two-tailedWilcoxon rank-sum test p-value=0.026).

To test whether the observed differences in correla-tions among the four metrics are significant, we applied aWilcoxon signed-rank test to paired sets of correlations.With four metrics, there are six possible paired compar-isons. Figure 2 shows the results of this analysis, whereone metric attaining a significant increase (using a Bon-ferroni p-value of 0.05/6 = 0.008333) over another isindicated by a directed edge. From this graph we concludethat, for the biological and technical replicates, NSAFyields significantly more reproducible quantification val-ues than SIN , dNSAF and emPAI. Our reproducibilityresults agree with Colaert et al., who claim that NSAFis more reproducible than SIN and emPAI [21]. How-ever, in contrast to our results, Griffen et al. report bet-ter reproducibility across replicates for SIN compared toNSAF [3].

Testing linear response for protein abundance acrosssamplesTo determine the linear response of each of the spectralcount metrics, we analyzed mass spectra from a dataset ofsamples that form a dilution curve of forty-eight proteinswith known amounts spiked into a C. elegans lysate. Weperformed linear regression between each protein spec-tral count and the associated amounts across the dilutioncurve samples. For this analysis, we only included pro-teins that obtain a positive spectral count in three or moreof the data sets, which results in a comparison of forty-two proteins across the four metrics. We then carriedout a Wilcoxon signed rank test analysis separately onthe average correlation, R2, and the mean percent error

McIlwain et al. BMC Bioinformatics 2012, 13:308 Page 4 of 6http://www.biomedcentral.com/1471-2105/13/308

SIN EMPAI NSAF dNSAF

Tech

nica

lre

plic

ates

Bio

logi

cal

repl

icat

es

Mouse experiments

Tech

nica

lre

plic

ates

Bio

logi

cal

repl

icat

es

Chicken experiments

Figure 1 Reproducibility of spectral counts across biological and technical replicate experiments. Each plot compares either the SIN , emPAI,NSAF or dNSAF measure for proteins that were reproducibly identified across two replicate experiments. For visualization purposes, the counts areplotted on a logarithmic scale. The number in the lower right corner of each panel is the corresponding Spearman correlation and the number inthe upper left is the number of datapoints compared.

(MPE). The results of these tests (Figure 3) are fairly con-sistent with one another: NSAF significantly outperformsdNSAF, and dNSAF and SIN significantly outperformemPAI.

Colaert et al. (2011) claim that SIN is more accurate thanboth NSAF and emPAI [21], but we find evidence only tosupport the former claim, even though our experimentsemploy a wider dynamic range of protein abundance(6.7–20 fmol versus 6–870 fmol) and more data sets (twoversus eight). Based on our experiments, we conclude thatNSAF or SIN are the methods of choice for ensuring anaccurate linear response between a protein’s change inabundance across different samples.

It is worth noting that Griffin et al. (2010) observe agood linear fit between SIN and protein quantification.

However, their evaluation methodology fits a single line toall of the SIN values from many proteins, whereas we havefit a separate line for each protein. This difference reflectsour belief that spectral counting methods are most usefulas measures of the relative abundance of a single pro-tein between two experiments. We did not test the claimthat SIN provides an accurate absolute protein abundancemetric.

DiscussionOverall, our experiments suggest a relative ordering ofspectral counting methods according to their repro-ducibility and the linearity of their response, but we canonly speculate as to the reasons for the ranking that weobserve. For example, we note that NSAF outperforms

McIlwain et al. BMC Bioinformatics 2012, 13:308 Page 5 of 6http://www.biomedcentral.com/1471-2105/13/308

Table 2 Spectral-counting reproducibility performance onmouse and chicken replicates

Metric Technical Biological All Replicates

SIN 0.885 0.848 0.859

emPAI 0.870 0.858 0.862

NSAF 0.899 0.876 0.884

dNSAF 0.886 0.852 0.863

All Metrics 0.885 0.859 0.867

This table summarizes the average correlation of the spectral-counting metricsacross the technical and biological replicates.

the emPAI metric in both of our experiments. The emPAImeasure takes into account the least information—notonly does it ignore fragment ion intensities, but emPAIalso fails to account for the length of the protein. Appar-ently, this relatively simple approach is insufficient toaccurately estimate protein abundance. The relative per-formance of NSAF and SIN , on the other hand, is lessclear: NSAF yields more reproducible results than SIN butthe two methods are statistically indistinguishable withrespect to linearity. The main difference between SIN andthe other three metrics is that SIN is the only metricthat takes into account the intensities of the fragment ionpeaks. In this sense, SIN goes a bit beyond the strict def-inition of “spectral counting.” Our experiments do notsupport the claim that such intensity information is valu-able for quantification. However, the conflicting results ofour study and Collaert et al., on the one hand, versus Grif-fin et al. on the other hand, suggests perhaps that furthercomparison of these methods is warranted.

An additional direction for future work involves quan-tifying the linearity and reproducibility of proteins in asegregated fashion according to protein abundance. Forexample, visual inspection of Figure 1 suggests that per-haps the SIN measure yields more reproducible counts forhigh abundance proteins, with a corresponding decreasein reproducibility as the abundance drops. Arguably, in

SIn emPAI

NSAF

dNSAF

Figure 2 Comparison of spectral counts across replicates. Thisgraph summarizes the statistical analysis of the reproducibilitymeasurements. An edge leading out from node A to node B indicatesa statistically significant improvement in reproducibility for method Arelative to method B.

SIn

emPAI

NSAF

dNSAF

Figure 3 Comparison of spectral counts across UPS1 dilutioncurve. This graph summarizes the statistical analysis of the linearitymeasurements. Two types of analysis were performed, using thelinear regression correlation, R2 and mean percent error (MPE) for theC. elegans + UPS1 dilution curve dataset. An edge leading out fromnode A to node B indicates a statistically significant improvement inlinearity for method A relative to method B.

many studies, such low abundance proteins are of thegreatest interest; hence, it may be worthwhile to investi-gate in a systematic fashion the extent to which either thelinearity or the reproducibility of a given spectral countingmeasure varies as a function of protein abundance.

ConclusionsQuantifying protein amounts in mass spectrometry byspectral counting is a simple and robust method formeasuring the relative change of protein amounts acrossdifferent samples; however, many different algorithmsexist for assigning a score to each identified protein.Using crux spectral-counts, we compared andcontrasted four spectral counting methods with respectto their reproducibility across replicates and their linearresponse relative to protein abundance. Crux providesa flexible, easy to use open source tool for performingprotein quantification using spectral counting.

Availability and requirementsProject name: Crux tandem mass spectrometry analysissoftwareProject home page: http://noble.gs.washington.edu/proj/cruxOperating systems: Linux, MacOS, Windows + CygwinProgramming language: C++Other requirements: Crux has no requirements to installthe binary version under Linux or MacOS. On Windows,Crux requires Cygwin. To compile Crux requires a c++compiler, cmake, and Subversion.

McIlwain et al. BMC Bioinformatics 2012, 13:308 Page 6 of 6http://www.biomedcentral.com/1471-2105/13/308

License: ApacheAny restrictions to use by non-academics: None

Additional file

Additional file 1: Supplementary Information. Supplementary Tables 1and 2 and Suplementary Figures 1 and 2 are provided as quantify-supplement.pdf.

Competing interestsThe authors declare that they have no competing interests.

Authors’ contributionsThe chicken and mouse samples were provided by ER’s lab, and the LC-MS/MSdata were collected by members of the MM lab. MB prepared and collectedthe UPS1 + C. elegans dilution sample datasets. MM wrote the initial code forcrux spectral-counts and the initial draft of the manuscript. SMfinished the coding of crux spectral-counts and the final draft withWSN. All authors revised and approved the final manuscript.

AcknowledgementsNIH awards R01 EB007057, P41 GM103533 and R01 DC03829. The authorsacknowledge Karl Schweighofer for his input on the cruxspectral-counts tool and the anonymous reviewers for many helpfulsuggestions.

Author details1Department of Genome Sciences, University of Washington, Seattle, WA,USA. 2Department of Otalaryngology–HNS, University of Washington, Seattle,WA, USA. 3Department of Physiology & Biophysics, University of Washington,Seattle, WA, USA. 4Department of Computer Science and Engineering,University of Washington, Seattle, WA, USA.

Received: 13 April 2012 Accepted: 31 October 2012Published: 19 November 2012

References1. Wang M, You J, Bemis KG, Tegeler TJ, Brown DP: Label-free mass

spectrometry-based protein quantification technologies inproteomic analysis. Brief Funct Genomic Proteomic 2008, 7(5):329–339.

2. Searle BC, Tabb DL, Falkner JA, Kowalak JA, Meyer-Arendt K, Rudnick PA,Seymour SL, Lane WS: iPRG2009 study: testing for differencesbetween complex samples in proteomics datasets. Poster at ABRF20092009 , 28(1):83–89.

3. Griffin NM, Yu J, Long F, Oh P, Shore S, Li Y, Koziol JA, Schnitzer JE: Label-free, normalized quantification of complex mass spectrometry datafor proteomic analysis. Nat Biotechnol 2010, 28:83–89.

4. Ishihama Y, Oda Y, Tabata T, Sato T, Nagasu T, Rappsilber J, Mann M:Exponentially Modified Protein Abundance Index (emPAI) forEstimation of Absolute Protein Amount in Proteomics by theNumber of Sequenced Peptides per Protein. Mol Cell Proteomics 2005,4(9):1265–1272.

5. Paoletti AC, Parmely TJ, Tomomori-Sato C, Sato S, Zhu D, Conaway RC,Conaway JW, Florens L, Washburn MP: Quantitative proteomic analysisof distinct mammalian Mediator complexes using normalizedspectral abundance factors. Proc Nat Acad Sci USA 2006,103(50):18928–18933.

6. Zhang Y, Wen Z, Washburn MP, Florens L: Refinements to Label FreeProteome Quantitation: How to Deal with Peptides Shared byMultiple Proteins. Anal Chem 2010, 82(6):2272–2281.

7. Keller A, Eng J, Zhang N, Li X, Aebersold R: A uniform proteomicsMS/MS analysis platform utilizing open XML file formats. Mol Syst Biol2005, 1:2005.0017.

8. Neilson KA, Ali NA, Muralidharan S, Mirzaei M, Mariani M, Assadourian G,Lee A, van Sluyter SC, Haynes PA: Less label, more free: Approaches inlabel-free quantitative mass spectrometry. Proteomics 2011,11(4):535–553.

9. Searle BC: Scaffold: A bioinformatic tool for validating MS/MS-basedproteomic studies. Proteomics 2010, 10(6):1265–1269.

10. Braisted J, Kuntumalla S, Vogel C, Marcotte E, Rodrigues A, Wang R, HuangST, Ferlanti E, Saeed A, Fleischmann R, Peterson S, Pieper R: The APEXQuantitative Proteomics Tool: Generating protein quantitationestimates from LC-MS/MS proteomics results. BMC Bioinformatics2008, 9:529.

11. Shinoda K, Tomita M, Ishihama Y: emPAI Calc-for the estimation ofprotein abundance from large-scale identification data by liquidchromatography-tandem mass spectrometry. Bioinformatics 2010,26(4):576–577.

12. Heinecke NL, Pratt BS, Vaisar T, Becker L: PepC: proteomics software foridentifying differentially expressed proteins based on spectralcounting. Bioinformatics 2010, 26(12):1574–1575.

13. Park CY, Klammer AA, Kall L, MacCoss MP, Noble WS: Rapid and accuratepeptide identification from tandem mass spectra. J Proteome Res2008, 7(7):3022–3027.

14. Kall L, Storey JD, MacCoss MJ, Noble WS: Assigning significance topeptides identified by tandem mass spectrometry using decoydatabases. J Proteome Res 2008, 7:29–34.

15. Spivak M, Weston J, Tomazela D, MacCoss MJ, Noble WS: Directmaximization of protein identifications from tandem mass spectra.Mol Cell Proteomics 2012, 11(2):M111.012161. [PMC3277760].

16. Klammer AA, Park CY, Noble WS: Statistical calibration of the SEQUEST

XCorr function. J Proteome Res 2009, 8(4):2106–2113.17. Hsieh E, Hoopmann M, Maclean B, Maccoss M: Comparison of database

search strategies for high precursor mass accuracy MS/MS data.J Proteome Res 2009.

18. McIlwain S, Draghicescu P, Singh P, Goodlett DR, Noble WS: Detectingcross-linked peptides by searching against a database ofcross-linked peptide pairs. J Proteome Res 2010, 9(5):2488–2495.[PMC20349954].

19. Jones AR, Eisenacher M, Mayer G, Kohlbacher O, Siepen J, Hubbard SJ,Selley JN, Searle BC, Shofstahl J, Seymour SL, Julian R, Binz PA, DeutschEW, Hermjakob H, Reisinger F, Griss J, Vizcaıno JA, Chambers M, Pizarro A,Creasy D: The mzIdentML data standard for massspectrometry-based proteomics results. Mol Cell Proteomics 2012,11(7):M111.014381.

20. Zhang B, Chambers MC, Tabb DL: Proteomic parsimony throughbipartite graph analysis improves accuracy and transparency.J Proteome Res 2007, 6(9):3549–3557.

21. Colaert N, Vandekerckhove J, Gavaert K, Martens L: A comparison ofMS2-based label-free quantitative proteomic techniques withregards to accuracy and precision. Proteomics 2011, 11(6):1110–1113.

doi:10.1186/1471-2105-13-308Cite this article as: McIlwain et al.: Estimating relative abundances ofproteins from shotgun proteomics data. BMC Bioinformatics 2012 13:308.

Submit your next manuscript to BioMed Centraland take full advantage of:

• Convenient online submission

• Thorough peer review

• No space constraints or color figure charges

• Immediate publication on acceptance

• Inclusion in PubMed, CAS, Scopus and Google Scholar

• Research which is freely available for redistribution

Submit your manuscript at www.biomedcentral.com/submit