Estimating the Diagnostic Accuracy

Nusrat Shafiq1, Samir Malhotra1, Deepak K Bhasin2, Surinder Rana2, Shabir Siddhu1,
Promila Pandhi1
Departments of 1Pharmacology and 2Gastroenterology, Post Graduate Institute of Medical
Education and Research. Chandigarh, India
ABSTRACT
Context Approximately 15-20% of cases of
acute pancreatitis are categorized as severe.
There is a lack of accurate predictors of
disease severity. Several studies have
evaluated the usefulness of procalcitonin as a
marker of severe disease. Reports regarding
the diagnostic accuracy of procalcitonin are
conflicting.
Objective The present meta-analysis was
carried out to evaluate the relevance of
procalcitonin as a predictor of disease
severity.
Methods
Two investigators working
independently attempted to locate eligible
studies by electronic and manual means.
Studies in which at least one of the markers of
disease severity was procalcitonin were
included for analysis. For all the studies
included, the following parameters were
calculated: true positive, false negative, false
positive and true negative. A summary
receiver operating characteristic (SROC)
curve was generated from these parameters.
Results Four studies were finally included in
the analysis. The unweighted regression line
parameters and were 3.633 and 1.399,
respectively. The values for and for
weighted regression line were 3.637 and
1.428. The SROC curve generated
demonstrated that procalcitonin is not a good
predictor of the severity of acute pancreatitis.
Conclusion The available data indicates that
procalcitonin cannot be considered a good
marker for assessing the severity of
pancreatitis.
INTRODUCTION
Acute pancreatitis is usually a mild disease
with minimal organ dysfunction. However,
15-20% of all cases demonstrate severe acute
pancreatitis [1, 2]. In acute pancreatitis, early
assessment of the patient which can lead to an
accurate prediction of the severity is useful
for several reasons. The first well-established
step is the need to categorize patients at risk
for complications for appropriate stratification
in clinical trials. Furthermore, it is important
to identify the patients who are at risk for
developing complications in order to be able
to initiate effective management before those
complications develop.
The lack of accurate predictors of disease
severity makes such categorization difficult.
Several biochemical parameters [3], contrast-
enhanced computed tomography [4, 5], and
multiple clinico-biochemical scores [6, 7]
have been used to assess the severity of acute
pancreatitis. An ideal prognostic method
should be simple, inexpensive, routinely

Page 2
JOP. J Pancreas (Online) 2005; 6(2):231-237.
JOP. Journal of the Pancreas – http://www.joplink.net – Vol. 6, No. 3 – May 2005. [ISSN 1590-8577]
232
available and highly accurate. Such a method
is however not yet available.
Procalcitonin is a 116-amino acid propeptide
of calcitonin with a molecular weight of 13
kDa [8]. It has been introduced as an early
marker of severe infection and inflammation
[9, 10]. Several studies [11, 12, 13, 14, 15, 16,
17, 18, 19, 20] have evaluated the usefulness
of procalcitonin as a predictor of severity and
the development of infected necrosis in acute
pancreatitis.
In view of the conflicting reports of the
diagnostic accuracy of procalcitonin in
predicting the severity of pancreatitis, we
aimed at adopting a meta-analytic approach
for arriving at a conclusion regarding this test
as a predictor of severity.
METHODS
We systematically searched MEDLINE and
EMBASE for all relevant articles until
November 2004. We first researched medical
subject heading (MeSH) terms and textwords
for "Markers" AND "Acute Pancreatitis".
Secondly, we researched MeSH terms and
textwords for "Procalcitonin" AND "Acute
Pancreatitis" AND "Severity". We then
combined the two searches and retrieved all
the relevant articles found by either search.
The manual search was carried out by looking
at the reference lists of the retrieved articles
and the Index Medicus. All the relevant
articles thus obtained were combined with
those obtained from the electronic search.
Data Extraction
Two investigators conducted the search
independently. Studies in which at least one
of the markers for predicting the severity of
pancreatitis using procalcitonin were
included. Studies had to present the sensitivity
and specificity of the procalcitonin test as a
predictor of the severity of pancreatitis to be
considered for inclusion. Otherwise, the study
had to present enough data to allow them to
be calculated. Studies conducted exclusively
in patients with post-ERCP pancreatitis, and
those evaluating outcomes other than the
severity of pancreatitis, such as the
development of infected necrosis or multiple
organ failure only, were excluded from the
evaluation.
Analysis
A summary receiver operating curve (SROC)
as described previously [21] was generated.
Briefly, in tests of diagnostic accuracy,
sensitivity and specificity are calculated using
a particular threshold. For the determination
of sensitivity and specificity, a threshold
value is generally decided a priori. Several
studies done to evaluate the effectiveness of a
particular diagnostic test in predicting some
outcomes may show different sensitivity and
specificity values because of different
thresholds. SROC is a method of pooling the
results of different studies to judge the
predictive value of a test. The following
parameters were calculated from the studies:
true positive (TP), false negative (FN), false
positive (FP) and true negative (TN). The true
positive rates (TPR) and false positive rates
(FPR) were converted to their logistic
transformations by using the formulae given
below:
Logit (TPR) = ln (TPR / (1 - TPR))
Logit (FPR) = ln (FPR / (1 - FPR))
The sum (S) and difference (D) of the two
transformations were then calculated:
= Logit (TPR) + Logit (FPR)
= Logit (TPR) - Logit (FPR)
is equivalent to the diagnostic log-odds
ratio (ln(OR)), which conveys the accuracy of
the test in discriminating cases from non-
cases. can be interpreted as a measure of the
diagnostic threshold, with high values
corresponding to liberal inclusion criteria for
cases. = 0 when TPR= 1 - FPR, that is, on
the anti-diagonal from the top-left to bottom-
right corners of the SROC space [22].
Next, in order to estimate the relationship
between and S, the two were adapted to a
linear model:
b S i

Page 3
JOP. J Pancreas (Online) 2005; 6(2):231-237.
JOP. Journal of the Pancreas – http://www.joplink.net – Vol. 6, No. 3 – May 2005. [ISSN 1590-8577]
233
The coefficient represents the dependence
of the test accuracy on the threshold; if is
near 0, then the studies are homogeneous and
can by summarized by an overall OR, noting
that = ln(OR). If is not equal to 0, then the
studies are heterogeneous with respect to OR.
In this case, can be thought of as the value of
ln(OR) when = 0 [22].
Weights were employed to reflect inter-study
heterogeneity with respect to the sample
variance of D [22]. The weighting parameter
w’ was defined as:
w=1/(1/(TP+0.5)+1/(FN+0.5)+1/(TN+0.5)+1/(FP+0.5))
Both weighted and unweighted regression line
parameters and were obtained.
These values were then utilized to return to
the transformed values for TPR as given by
the following formula:
TPR=1/(1+(1/(ei/(1-b).(FPR/(1-FPR))(1+b)/(1-b))))
A plot between TPR thus obtained and FPR
gave the SROC.
STATISTICS
Estimates were tested vs. 0 by the t-test and
were reported together with their standard
errors (SE) and 95% confidence intervals
(CI). The chi-squared test and linear
regression were applied. The statistical
analyses were performed by running the SPSS
(version 8.0 for Windows) using a personal
computer.
RESULTS
Ten studies were identified [11, 12, 13, 14,
15, 16, 17, 18, 19, 20] out of which four [11,
12, 13, 14] were finally included in the
analysis (Figure 1). The details of the studies
included are given in Table 1. These studies
included a total of 313 patients with acute
pancreatitis. These patients had been
categorized into mild and severe cases
irrespective of the etiology of the pancreatitis.
The TP, FP, FN, TN values, their logit
transformed values and the 95% confidence
intervals are given in Table 2.
The parameters and (±SE) obtained from
these data by unweighted regression analysis
(r=0.968; P=0.016) were 3.584±0.381 (95%
CI: 1.946-5.222; P=0.011) and 1.297±0.166
(95% CI: 0.580-2.013; P=0.016), respectively.
The values for and for weighted regression
line (r=0.964; P<0.001) were 3.601±0.142
(95% CI: 3.280-3.922; P<0.001) and
1.332±0.085 (95% CI: 1.139-1.524; P<0.001),
respectively. Both the estimated values were
significantly different from 0 and, therefore,
the studies resulted heterogeneous with
respect to OR.
The unweighted SROC generated from the
reverse extrapolated values of TPR plotted
against FPR demonstrates that procalcitonin is
Figure 1. Flowchart of studies evaluated for inclusion
in the meta-analysis.
Table 1. Details of the 4 studies included in the analysis.
Study
Number
of patients
Basis for the classification
of severity
Mild
disease
Severe
disease
Kylänpää-Bäck et al. [11]
124
38
Atlanta criteria
Frasquet et al. [12]
36
15
Atlanta criteria
Ammori et al. [13]
55
14
Atlanta criteria
Melzi d’Eril et al. [14]
19
12
Atlanta criteria
Total
234
79

Page 4
JOP. J Pancreas (Online) 2005; 6(2):231-237.
JOP. Journal of the Pancreas – http://www.joplink.net – Vol. 6, No. 3 – May 2005. [ISSN 1590-8577]
234
not a good predictor of the severity of acute
pancreatitis (Figure 2). Out of the 4 studies
included in our analysis, two showed very
low sensitivities [12, 14], and paradoxically,
higher false positive rates than the other two
studies [11, 13]. To rule out random error as
being a cause of such a finding [23], TPR
values were confirmed to be non-
homogeneously distributed among the 4
studies (P<0.001), while no significant
differences were obtained for FPR values
(P=0.750).
Moreover, the regression coefficient for TPR
and FPR was negative (r = -0.894) when all
the studies were considered, suggesting that,
basically, a valid SROC curve cannot be
found using the present data, and therefore, a
single summary curve should not be
constructed [23].
DISCUSSION
In our study, we have used a relatively new
approach of meta-analyzing diagnostic
studies, namely the SROC method for
assessing the utility of procalcitonin as a
predictor of the severity of acute pancreatitis.
Reports on diagnostic tests when being
evaluated in the initial stages, may show a
large discrepancy. Such a situation was also
observed for ‘procalcitonin’ as a marker for
the severity of acute pancreatitis. A plot
between TPR and FPR at various thresholds,
the ROC plot, is commonly used in presenting
the report of a single study. SROC curves
give a good overview of pooled results of
several studies [24].
The results of our meta-analysis show that
procalcitonin may not be a useful marker for
estimating the severity of acute pancreatitis.
Several elements of this analysis merit further
discussion.
Table 2. Parameters for construction of the summary receiver operating (SROC) curve.
Kylänpää-Bäck
et al. [11]
Frasquet
et al. [12]
Ammori
et al. [13]
Melzi d’Eril
et al. [14]
True positive: TP
27
4
9
1
False positive: FP
20
8
8
4
False negative: FN
11
11
5
11
True negative: TN
104
28
47
15
Odds ratio: OR
(95% CI)
12.76
(5.46-29.83)
1.27
(0.32-5.10)
10.58
(2.81-39.81)
0.34
(0.03-3.49)
True positive rate: TPR
(95% CI)
0.711
(0.644-0.776)
0.267
(0.141-0.391)
0.643
(0.547-0.737)
0.083
(-0.088-0.254)
False positive rate: FPR
(95% CI)
0.161
(0.161-0.226)
0.222
(0.097-0.347)
0.145
(0.050-0.240)
0.211
(-0.009-0.429)
Logit (TPR)
0.898
-1.012
0.588
-2.3979
Logit (FPR)
-1.649
-1.253
-1.771
-1.322
Weight
5.504
2.165
2.349
0.961
S
-0.751
-2.264
-1.183
-3.720
D
2.547
0.241
2.358
-1.076
Figure 2. Summary receiver operating (SROC) curve
for procalcitonin as a marker of the severity of acute
pancreatitis.

Page 5
JOP. J Pancreas (Online) 2005; 6(2):231-237.
JOP. Journal of the Pancreas – http://www.joplink.net – Vol. 6, No. 3 – May 2005. [ISSN 1590-8577]
235
First, two studies [12, 14] showed very low
sensitivity, and paradoxically, higher false
positive rates than the other two studies [11,
13]. In view of this lack of a monotonically
increasing relationship between TPR and FPR
in two of the studies included in our meta-
analysis, we needed to rule out random error
as being a cause of such a finding [23]. This
was done showing that the TPRs were not
found to be homogenous. Because of this lack
of homogeneity among the studies, pooling of
such data may be problematic. However,
exclusion of negative studies from systematic
reviews and meta-analyses may not be the
best approach [24] since they already suffer
the drawback of publication bias (negative
studies are less likely to be published).
Therefore, we decided to include all eligible
studies in our meta-analysis.
There may be several reasons for the
heterogeneity observed. First, studies showing
procalcitonin to be a poor marker of the
severity of pancreatitis had a greater
percentage (70% versus 35%) of patients with
pancreatitis of biliary origin [12, 14] and it
has been observed that procalcitonin may not
be a sensitive marker for this type of
pancreatitis [14, 18]. This could have
contributed to the heterogeneity observed.
Secondly, different assay techniques (manual
methods or kits) were used in different
studies. Also, the coefficient of variation was
not reported by any of the studies. These
factors may also have contributed to the
heterogeneity. Thirdly, the time of blood
sampling with respect to the onset of
symptoms may also lead to some discrepancy
in test results. For example, in one study [11]
showing good correlation of procalcitonin
with severity, the procalcitonin was measured
even after 4 days of the onset of the
symptoms and in another [12] showing poor
correlation, the measurement was done early
(within the first 24 hours of the onset of
symptoms). Since the markers of severity may
be highly time-sensitive [12], such differences
can also account for the heterogeneity among
the studies.
It is well-known that the closer the ROC and
the SROC curves are to the upper left-hand
quadrant, the more accurate they are, because
the TPR is 1 and the FPR is 0. The SROC
curve obtained by us does not lie in the upper
left quadrant as would be desired. Other than
the explanations given above, a small sample
size (with respect to both the number of
studies available for analysis as well as the
number of patients included in each study)
may be another possible explanation for our
SROC curve not occupying the upper left-
hand corner [22]. The total sample size in our
study was 313 patients, with two studies [12,
14] having sample sizes less than fifty.
However, the position of the graphical points
obtained in the subgroup analysis shown in
Figure 2 may be expected to result in an
optimal SROC curve provided a larger
number of studies with similar results are
available.
Other studies [15, 18] have evaluated
procalcitonin as a predictor of the
development of complications such as
pancreatic necrosis, multiple dysfunction
syndrome, but not for the severity of the
disease per se. Our exclusion criteria did not
permit us to incorporate these studies in our
analysis.
In a review on biological markers for
assessing the severity of acute pancreatitis [3],
procalcitonin was considered to be a good
marker in predicting disease severity early on
and it was assigned to category ‘A’. However,
in this review, studies [12, 14] showing low
sensitivity of procalcitonin as a marker for
severity were not mentioned.
In conclusion, a valid SROC curve cannot be
found in our data and this result indicates that,
from the data available so far, procalcitonin
cannot be considered a good marker for
assessing the severity of acute pancreatitis.
Studies with larger numbers of patients, with
more homogenous patient populations, and
better correlation between the onset of
symptoms and blood sampling and more
similarity in the assay techniques are required
in order to resolve the issue.
Received January 26th, 2005 - Accepted
March 24th, 2005

Page 6
JOP. J Pancreas (Online) 2005; 6(2):231-237.
JOP. Journal of the Pancreas – http://www.joplink.net – Vol. 6, No. 3 – May 2005. [ISSN 1590-8577]
236
Keywords Biological Markers; Calcitonin;
Meta-Analysis; ROC Curve; Pancreatitis,
Acute Necrotizing; Sensitivity and Specificity
Abbreviations FN: false negative; FP: false
positive; FPR: false positive rate MeSH:
medical subjects heading; SROC: summary
receiver operating curve; TN: true negative
TP: true positive; TPR: true positive rate
Acknowledgements Ashok Saluja, UMASS,
and the Editor of Hepatogastroenterology for
providing us some of the articles for our
meta-analysis
Correspondence
Samir Malhotra
Department of Pharmacology
Post Graduate Institute of Medical Education
and Research
Chandigarh
India-160012
 
References
1. Steinberg W, Tenner S. Acute pancreatitis. N Engl
J Med 1994; 330:1198-210. [PMID 7811319]
2. Barie PS. A critical review of antibiotic
prophylaxis in severe acute pancreatitis. Am J Surg
1996; 172:38S-43S. [PMID 9003689]
3. Werner J, Hartwig W, Uhl W, Muller C, Buchler
MW. Useful markers for predicting severity and
monitoring progression of acute pancreatitis.
Pancreatology 2003; 3:115-27. [PMID 12748420]
4. Kivisaari L, Somer K, Standertskjold-Nordenstam
CG, Schroder T, Kivilaakso E, Lempinen M. Early
detection of acute fulminant pancreatitis by contrast-
enhanced computed tomography. Scand J
Gastroenterol 1983; 18:39-41 [PMID 6675177]
5. London NJ, Neoptolemos JP, Lavelle J, Bailey I,
James D. Contrast-enhanced abdominal computed
tomography scanning and prediction of severity of
acute pancreatitis: a prospective study. Br J Surg 1989;
76:268-72. [PMID 2720324]
6. Ranson JH, Rifkind KM, Roses DF, Fink SD, Eng
K, Spencer FC. Prognostic signs and the role of
operative management in acute pancreatitis. Surg
Gynecol Obstet 1974; 139:69-81. [PMID 4834279]
7. Wilson C, Heath DI, Imrie CW. Prediction of
outcome in acute pancreatitis: a comparative study of
APACHE II, clinical assessment and multiple factor
scoring systems. Br J Surg 1990; 77:1260-4. [PMID
2253005]
8. Oczenski W, Fitzgerald RD, Schwarz S.
Procalcitonin: a new parameter for the diagnosis of
bacterial infection in the peri-operative period. Eur J
Anaesthesiol 1998; 15:202-9. [PMID 9587727]
9. Oberhoffer M, Vogelsang H, Russwurm S,
Hartung T, Reinhart K. Outcome prediction
bytraditional and new markers of inflammation in
patients with sepsis. Clin Chem Lab Med 1999;
37:363-8. [PMID 10353484]
10. Hensel M, Volk T, Docke WD, Kern F, Tschirna
D, Egerer K, et al. Hyperprocalcitonemia in patients
withnoninfectious SIRS and pulmonary dysfunction
associated
with
cardiopulmonary
bypass.
Anaesthesiology 1998; 89:93-104 [PMID 9667299]
11. Kylanpaa-Back ML, Takala A, Kemppainen E,
Puolakkainen P, Haapiainen R, Repo H. Procalcitonin
streip test in the early prediction of severe acute
pancreatitis. Br J Surg 2001; 88:222-7. [PMID
11167871]
12. Frasquet J, Saez J, Trigo C, Martinez J, Such J,
Perez-Mateo M. Early measurement of procalcitonin
does not predict severity in patients with acute
pancreatitis. Br J Surg 2003; 90:1129-30. [PMID
12945081]
13. Ammori BJ, Becker KL, Kite P, Snider RH, Nylen
ES, White JC, et al. Calcitonin precursors in the
prediction of severity of acute pancreatitis on the day
of admission. Br J Surg 2003; 90:197-204. [PMID
12555296]
14. Melzi D'Eril GV, Merlini G, Finazzi S, Bosoni T,
Barakat B, Pezzilli R. Procalcitonin is not a reliable
marker for the assessment of severity in acute
pancreatitis without infectious complications. Clin
Chem 2000; 46:428-430 [PMID 10702536]
15. Kylanpaa-Back ML, Takala A, Kemppainen EA,
Puolakkainen PA, Leppaniemi AK, Karonen SL, et al.
Procalcitonin, soluble interleukin-2receptor, and
soluble E-selectin in predicting the severity of acute
pancreatitis. Crit Care Med 2001; 29:63-9. [PMID
11176162]
16. Muller CA, Uhl W, Printzen G, Gloor B,
Bischofberger H, Tcholakov O, Buchler MW. Role of
procalcitonin and granulocyte colony stimulating factor
in the early prediction of infected necrosis in severe
acute pancreatitis. Gut 2000; 46:233-8. [PMID
10644318]
17. Oezcueruemez-Porsch M, Kunz D, Hardt PD,
Fadgyas T, Kress O, Schulz HU, et al. Diagnostic
relevance of interleukin pattern, acute-phase proteins,
and procalcitonin in early phase of post-ERCP

Page 7
JOP. J Pancreas (Online) 2005; 6(2):231-237.
JOP. Journal of the Pancreas – http://www.joplink.net – Vol. 6, No. 3 – May 2005. [ISSN 1590-8577]
237
pancreatitis. Dig Dis Sci 1998; 43:1763-9. [PMID
9724166]
18. Rau B, Steinbach G, Gansauge F, Mayer JM,
Grunert A, Beger HG. The potential role of
procalcitonin and interleukin-8 in the prediction of
infected necrosis in acute pancreatitis. Gut 1997;
41:832-40. [PMID 9462219]
19. Pindak D, Parrak V, Pechan J, Vavrecka A, Kuzela
L, Fuchs D, Irsakova J. The clinical value of
procalcitonin in prediction of severity and outcome in
acute pancreatitis. Hepatogastroenterology 2003;
50(Suppl II):28-9. [PMID 15244180]
20. Pezzilli R, Melzi d'Eril GV, Morselli-Labate AM,
Merlini G, Barakat B, Bosoni T. Serum amyloid A,
procalcitonin, and C-Reactive protein in early
assessment of severity of acute pancreatitis. Dig Dis
Sci 2000; 45:1072-8. [PMID 10877218]
21. Littenberg B, Moses LE. Estimating diagnostic
accuracy from multiple conflicting reports: A new
meta-analysis method. Med Decis Making 1993;
13:313-21. [PMID 8246704]
22. Walter SD. Properties of the summary receiver
operating characteristic (SROC) curve for diagnostic
test data. Stat Med 2002; 21:1237-56. [PMID
12111876]
23. Midgette AS, Stukel TA, Littenberg B. A meta-
analytic method for summarizing diagnostic test
performances:
receiver-operating-characteristic-
summary point estimates. Med Decis Making 1993;
13:253-7. [PMID 8412556]
24. Sutton AJ, Abrams KR, Jones DR, Sheldon TA,
Song F. In Methods for meta-analysis in medical
research. John Wiley & Sons Ltd. New York. 2000 ;
205-28]

There are no products listed under this category.