Outlier Detection Procedures in a Sample from a Gumbel Distribution with Unknown Scale Parameter

I. INTRODUCTION

Lalitha and Tripathi (2018) have discussed detection of a single upper and lower outlier when the location parameter and the scale parameter of Gumbel distribution were assumed to be known. But if the scale parameter is not known then in such situation, previously discussed procedures should be suitably modified. Thus, when the scale parameter is not known, an estimator of the scale parameter is used in the test statistic. Then it's critical values and the corresponding performance study were done by simulation technique. Since the study is about outlying observation, the entire sample should not be considered for the estimation of the scale parameter. Hence, the best linear unbiased estimate suggested by Balakrishnan and Cohen (1991) for a type II censored sample was used as an estimator for the scale parameter. But with this estimator, the values of the tabulated coefficients were available only for samples of size at most 10. Hence, a modified form of the moment estimator is considered and the statistics were studied. But on using an estimator for the scale parameter, derivation of the theoretical probability distribution of the test statistic is extremely tedious. Hence, the critical values as well as the performance of the test statistic were obtained by using simulation technique.

II. TEST STATISTIC USING THE BEST LINEAR UNBIASED ESTIMATOR FOR THE SCALE PARAMETER

Three test statistic , and have been suggested to detect an upper outlier, lower outlier and a pair of outlying observations respectively to test the null hypothesis against the slippage alternative i.e. there is one or a pair of observation(s) from another Gumbel with a shifted scale parameter , where the scale parameter was unknown. Hence for , the best linear unbiased estimator suggested by Balakrishnan and Cohen (1991) given as , where r and s denotes number of trimmed observations from lower and upper side respectively and 's are known constants obtained by Balakrishnan and Chan (1992), was used. Since values of 's obtained by Balakrishnan and Chan (1992) up to sample size 30 but only up to 10 values are available, therefore the use of the test statistic, suggested in this case is restricted up to sample size 10. The test statistics for an upper, lower and a pair of observations obtained respectively for this case are as follows.

where , , and are first, and order statistics respectively arranged in an ascending order of magnitude. However, because of the limitations in obtaining the values of 's, these statistics have only limited usage and hence a modified form of these statistics is suggested in this paper.

2.1 Modified Test Statistics using a moment estimator for the scale parameter

The moment estimator of was suggested by Johnson et.al. (1994) and is given as , where is sample standard deviation, can be used as an efficient estimator of . This is because Johnson et.al. (1994) have shown that the moment estimator is about more efficient than Cramer-Rao lower bound estimator of scale parameter. Further, as our work is concerned with the extreme events and the sample standard deviation is affected by the extreme observations, therefore to make the test statistic more efficient, the modified sample standard deviation obtained from a trimmed sample (i.e. a sample obtained after deleting the extreme observations and ) was used instead of a complete sample standard deviation. Thus, in this case, scale parameter was replaced by its modified moment estimator, given as , where obtained from a trimmed sample (i.e. a sample obtained after deleting the extreme observations and ). The test statistics so obtained are as follows

, where , and are first, (n-1)th and nth order statistics respectively arranged in an ascending order of magnitude. These three test statistics can be applied to a sample of size n=3, 4,..., 50. Performances of these three statistics were studied by simulation technique using 10,000 replications.

III. CRITICAL VALUES FOR THE TEST STATISTIC Z1 TO DETECT AN UPPER OUTLIER

The test statistic was used to detect an upper outlying observation in a sample from Gumbel distribution. In the detection of an upper outlying observation, the null hypothesis would state that there is no outlying observation in the sample. As the statistic is based on the difference of the largest and second largest observations, this test statistic should reject the null hypothesis for large values of . Thus an -level critical region will be given as where can be obtained from , where . Critical values of the test statistic for case-I and case-II of the Gumbel distribution were obtained using simulation technique with 10,000 replications which are tabulated in Table 3.1. and Table.3.2. respectively for at 1%, 5% and 10% significance levels. Just as the critical values obtained for a single upper, single lower and a pair of outliers with known scale parameter are free from the scale parameter, calculation of these critical values are also free from the scale parameters.

Table 3.1: Critical values of for case-I at different levels of significance.

Table 3.2. Critical values of for case-II at different levels of significance

IV. CRITICAL VALUES FOR THE TEST STATISTIC TO DETECT A LOWER OUTLIER

It was observed in Lalitha and Tripathi (2018) that, for Gumbel distribution, the density functions on the real line of case-I and case-II of lower outlier are the mirror images of the density functions of the case-II and case-I of upper outlier respectively. Thus for the test statistic , critical values obtained in section 4 for the upper outlier of case-I and case-II, can be used for detection of the lower outlier of case-II and case-I respectively.

V. THE CRITICAL VALUES FOR THE TEST STATISTIC TO DETECT A PAIR OF OUTLIERS

The critical values of the test statistic were obtained by using the simulation technique with 10,000 replications and are given in Table 5.1. The critical values of the test statistic (unknown scale parameter) for case-I and case-II of the Gumbel are close to the critical values obtained by Lalitha and Tripathi (2018) for the Gumbel distribution case-I and case-II (known scale parameter).

Table 5.1: Critical values of for case-I at different levels of significance.

VI. WHEN THE OBSERVATION OF A SAMPLE IS ON EITHER SIDE OF THE LOCATION PARAMETER

In this case it is assumed that some of the observations are below and some are above the location parameter. In this section the moment estimator of , suggested by Johnson et. al. (1994) is used in place of the scale parameter . The detection of an upper and a lower outlying observation can be done as discussed in section 4 and section 5 respectively. But while dealing with a pair of outlying observations the procedure would be slightly different and it is given as follows.

Let be a random sample from a Gumbel distribution with location parameter and scale parameter (unknown), in which some, say m, observations, are less than the location parameter and rest of the observations of the sample are greater than . Then considering the m observations lying below the location parameter, a modified test statistic can be defined for detecting the smallest observation by considering largest observation as , i.e.

As before, the observation corresponding to cannot be declared as an outlying one, being the observation lying closest to the location parameter, when the test statistic falls in the critical region. Thus, in the event of rejection of the null hypothesis, only the smallest observation i.e. should be considered as outlying observation. In this case, the critical values given in case II of section 4 should be used for testing the null hypothesis with a sample size as m. For the rest, observations above the location parameter, a modified test statistic can be defined for detecting the largest observation by considering largest observation as , i.e.

Here again, the observation corresponding to would be lying closest to the location parameter and therefore this observation cannot be declared as an outlying one. In this case the critical values obtained in case I of section 4 should be used for testing the null hypothesis with a suitable modification of the sample size as . As before, in the event of rejection of the null hypothesis, the largest observation i.e. should be declared as outlying observation.

a. Case when only one observation is on either side of the location parameter and the location parameter is also known (i). When only one observation is lying below the location parameter which is assumed to be known, while all other observations are above the location parameter, then the statistic Z can be modified as

Here, if it is assumed that and follows a Gumbel distribution as defined in case II with location and scale parameters and respectively. The critical values can be obtained by using simulation technique with 10,000 replicates. These critical values were tabulated in Table 6.1 for different values of sample size n and different levels of significance, given as follows.

Table 6.1: The critical values of the test statistic when only one observation is lying below the location parameter (known)

(ii). When only one observation is lying above the location parameter, while all other observations are below the location parameter, then the statistic Z is modified as

Here, as it is assumed that , follows a Gumbel distribution as defined in case I with location and scale parameters and respectively. The critical values can be obtained by using simulation technique with 10,000 replications and were tabulated in Table 6.2., as given below

Table 6.2: The critical values of the test statistic when only one observation is lying above the location parameter (known)

b. Case when the location parameter is unknown

When the location parameter is unknown, it can be estimated with a trimmed sample, as suggested in section 3 and is denoted by . As the two extreme observations are suspected outliers therefore the sample should be trimmed at both the ends. The location parameter used in the test statistics given in equations (6.1) and (6.2), is replaced by this estimate . With this estimate, the number of observations on its left and right, i.e. and can be decided. Then for and/ , the above said procedures can be used, as their critical values are independent of both the location and scale parameter. Also when and/(n - m) = 1, the test statistic will be as given above in section 6(a), with the value of the location parameter replaced by its estimator . The critical values so obtained were tabulate in Table 6.3. and Table 6.4. for the two cases of the distribution i.e. when only one observation is below the location parameter and when only one observation is above the location parameter respectively, are as given below.

Table 6.3: The critical values of the test statistic when only one observation is lying below the location parameter (unknown)

Table 6.4: The critical values of the test statistic when only one observation is lying above the location parameter (unknown) Here the critical values were obtained up to sample size 50, therefore the suggested test statistic, based on the trimmed estimate of the scale parameter can be used for large samples as well.

VII. PERFORMANCE STUDY FOR A SINGLE UPPER OUTLIER

Consider a set of observations which contains a single contaminant observation and which comes from a population that has a different distribution from the rest of the observations. The power of the test can be defined as the sample contains one contaminant observation], i.e. the probability of the largest observation of the sample is being identified as discordant. But it is not necessary that the test always identifies as a discordant or defining only power of a test statistic is not sufficient to describe performance of a test statistic. Barnett and Lewis (1994) have suggested probabilities, to as test performance criteria. To improve the performance criteria of a statistic, David and Nagaraja (2003) discussed the properties of five such probabilities, labeled as to as a reasonable measure of the performance of . These probabilities are defined as

, is the probability the observation tested by the test statistic is identified as the outlying one, when it is known that there is one outlier is present.

, is the probability the contaminant observation tested by the test statistic is identified as the outlying one, when it is known that there is one outlier is present.

, is the probability the contaminant observation is the extreme observation tested by the test statistic which is identified as the outlying one, when it is known that there is one outlier is present.

is the probability that the largest observation is the contaminant observation, which is being detected as discordant while the second largest is not a discordant observation; and

, is the conditional probability that when it is given that the contaminant is an outlier and it is also identified as discordant by the test, where is a general test statistic and is the corresponding value for the contaminant observation. David and Nagaraja (2003) have observed that and the information given by and are seen to be limited. Thus, only and are sufficient for defining performance of any test statistic. Further, Hayes and Kinsella (2003) discussed the performance of the discordancy test on the basis of six performance criteria and called them as non spurious power, spurious power, swamping effect, spurious Type II error, partially spurious Type II error and nonspurious Type II error. Also, it was suggested that a good discordancy test should have high value of non spurious power, low value of spurious power and low value of swamping effect. They recommended that out of six performance criteria, probability of the non-spurious power , probability of the spurious power and the probability of the non-spurious type-II error (which gives the probability that the test wrongly identifies a good observation as discordant), are required to specify the test completely. In accordance with the a...

In this section, study of the performance for detection of a lower outlier for the both the cases of Gumbel distribution with location parameter and scale parameter (unknown) are discussed. Case-I: When all observations are greater than the location parameter . The performance of the test statistic to detect an upper outlier in a sample from a Gumbel distribution, after introducing a contaminant observation from another sample from the same distribution with different scale parameter, was observed. This was replicated 10,000 times. From this, the values of the probabilities and for the test statistic were obtained and are given in Table 7.1. It can be observed that at 1% level of significance, the power , the non-spurious power and the conditional power of the test statistic are showing almost negligible changes between sample size 5 and 10, between sample sizes 10 and 15 a rapid increase is observed while beyond 15 the rate of increase is comparatively low. The non-spurious type-II error is also showing almost no change between sample size 5 and 10, between sample sizes 10 and 15 it is increasing while beyond 15 it decreases. The ratio i.e. the probability that the contaminant shows up as an outlier, is almost constant between sample sizes 5 and 10, between sample sizes 10 and 15 it increases with very high rate of increase and beyond 15 rate of increase is comparatively low. Thus, it can be interpreted that the performance of the test statistic is increasing with sample size but beyond sample size 20 there is hardly any variation.

Table 7.1: Performance of the test statistic for case-I at different levels of significance

It can be seen that at 5% level of significance the power, the non spurious power, the conditional power and the probability that the contaminant is identified as an outlier all are increasing rapidly between sample sizes 5 and 10 and beyond 10 its rate of increase is comparatively low. The values of all these probabilities are very high for sample sizes up to 15 while beyond 15 it shows almost no fluctuation. The non spurious type-II error is decreasing very fast between sample sizes 5 and 10 while beyond that it decreases with comparatively low rate of decrease. Hence from inferences discussed above it can be concluded that the performance of the test statistic is good at 5% level of significance. It can be noticed from. at 10% level of significance the power, the non spurious power, the conditional power and the probability of showing up a contaminant as an outlier are very high and increasing rapidly between sample sizes 5 and 10 and beyond that its rate of increase is very low. The non spurious type-II error is increasing with a very low rate of increase with the increase of the sample size up to 15 and beyond sample size 15 it decreases slowly. Hence it can be seen that the calculated probabilities and are high and i.e. non spurious error is low, as desired. Thus, on the basis of above results it can be interpreted that the performance is increasing with sample size up to a certain level and after that the variations are almost uniform, also the performance of the statistic is found to be good for the detection of an upper outlier in a Gumbel sample at 5% and 10% levels of significance, while at 1% level it is comparatively low.

Case-II: When all observations were smaller than the location parameter . The performance of the test statistic to detect an upper outlier in a sample from a Gumbel distribution, after introducing a contaminant observation from another sample of the same distribution with different scale parameter was obtained. These performance probabilities were obtained by simulation technique with 10,000 replicates and are given in

Table 7.2: Performance of the test statistic for case-II at different levels of significance

It was observed from Table 7.2. that the performance of the test statistic under consideration was found to be good (as the performance is increasing with the increase of the sample size and the calculated probabilities and are high and is low, as desired). Also, it can be seen that at 1% level of significance the power, the nonspurious power, the conditional power and the probability of identifying a contaminant as an outlier, are increasing rapidly between sample sizes 5 and 10, beyond that the changes are almost negligible. The non-spurious type-II error is increasing with a small rate of increase at 1% and 5% levels while at 10% level of significance it is decreasing with a very small rate of decrease. Thus, it can be interpreted that the performance of the statistic is increasing with the sample size up to 15 and beyond that almost negligible changes are observed.

VII. PERFORMANCE STUDY FOR A SINGLE LOWER OUTLIER

In this section the performance of was studied for both the cases of the Gumbel distribution. Case-I: When all observations were greater than the location parameter . The performance of the test statistic to detect a lower outlier in a Gumbel sample, after introducing a contaminant observation from another sample of the same distribution with different scale parameter, was studied by simulation technique with 10,000 replicates. These performance probabilities are given in Table 8.1. From this table, it can be observed that the performance is increasing with the increase of the sample size and it is found to be satisfactory beyond sample size 15.

Table 8.1: Performance of the test statistic for case-I at different levels of significanc

It can be seen that the power , the non spurious power and the conditional power are increasing rapidly between sample sizes 5 and 10 at 1% and 10% levels of significance while are increasing rapidly between sample sizes 5 and 15 at 5% level of significance. The non spurious type-II is found to be very low and it shows very small changes throughout. The probability of showing the contaminant as an outlier, is high for high values (greater than 10) of the sample size, it is increasing rapidly between sample sizes 5 and 10 at 1% level of significance. While at 5% and 10% levels of significance it is showing negligible changes between sample sizes 5 and 10, beyond 10 it increases very slowly. On the basis of above inference, it can be interpreted that the power , the non spurious power , the conditional power and the probability that the contaminant is showing up as an outlier, are found to be very high and the non spurious type-II error is low, as desired. Thus, it can be said that the test statistic is performing very well in this case for sample sizes greater than 10.

Case-II: When all observations were smaller than the location parameter . The performance of the test statistic for detection of a lower outlying observation from a Gumbel sample with a contaminant observation which was taken from a sample of the same distribution with different scale parameter was studied. All the performance probabilities were calculated by simulation technique with 10,000 replications, and are given in Table 8.2. From Table 8.2, it can be seen that the performance is increasing with the sample size and it is good enough for sample sizes greater than 10.

Table 8.2: Performance of the test statistic for case-II at different levels of significance

From this, it can be interpreted that the power, the non spurious power and the conditional power of the test statistic are high for large sample sizes; these are increasing rapidly between sample sizes 5 and 10 while beyond sample size 10 it increases with a comparatively low rate of increase. The non spurious type-II error is low and decreasing slowly with a very small rate of decrease. The probability of identifying the contaminant as an outlying observation is also very high and increasing with the increase of the sample size, as desired for a good discordancy test according to Barnett and Lewis (1994) and Hayes and Kinsella (2003). Thus it can be concluded that the test statistic under consideration is performing very well for sample sizes greater than 5

IX. PERFORMANCE STUDY FOR A PAIR OF OUTLIERS

Case-I: When all observations were greater than the location parameter . The performance of the test statistic to detect a pair of outliers in a sample from Gumbel distribution after introducing a pair of contaminants from another sample from the same distribution with a shifted scale parameter was obtained by simulation technique with 10,000 replicates. These performance probabilities are given in Table 9.1.

It can be observed from the above table that the power, the non-spurious power and the conditional power of the test statistic are increasing rapidly throughout. It can also be seen that the power, the non spurious power, the conditional power and the probability of identifying the contaminant as an outlier is increasing with the increase of the sample size and the non spurious type-II error is very low. Since the probabilities are significantly high for sample sizes greater than 10 and is low, as desired for a good discordancy test. Thus, it can be interpreted that the performance the test statistic is good for large samples.

Case-II: When all observations were smaller than the location parameter . The performance of the test statistic to detect a pair of outliers in a sample from a Gumbel distribution after introducing a pair of contaminants from another sample from the same distribution with different scale parameter was studied by simulation technique with 10,000 replications. These performance probabilities are given in Table 9.2.

Table 9.2: Performance of the test statistic for case-II at different levels of significance.

From Table 9.2. it can be observed that since the power, the non spurious power, the conditional power, are very low thus the test statistic is not found to be satisfactory for the detection of a pair of outliers in a sample from case-II of a Gumbel distribution. It can be seen that the power of the statistic is increasing throughout with almost constant rate of increase. The non spurious power, is increasing with a good rate of increase between sample sizes 5 and 15 while beyond that almost no changes are observed. The non spurious type II error of the test statistic at 1% level of significance is found to be very low. The conditional power of the statistic at 1% level of significance shows that the conditional power increases between sample sizes 5 and 10, between sample sizes 10 and 15 almost no changes are observed while beyond sample size 15 again it increases uniformly. The probability of detecting an outlying pair at 1% level of significance is found to be low. It increases rapidly between sample sizes 5 and 15, while beyond 15 the rate of increase is low.

It can also be seen that the power increases with the increase of sample size throughout almost uniformly but small changes are occurring between sample sizes 10 and 15. The non-spurious power of the test statistic at 5% level of significance increases rapidly from sample size 5 to 15 and beyond sample size15, its rate of increase is comparatively low. The spurious power of the test statistic at 5% level of significance increases rapidly between sample sizes 5 and 10, the rate of increase is comparatively low between sample sizes 10 and 25 while beyond sample size 25, almost no changes are seen. The conditional power increases with good rate of increase between sample sizes 5 and 10 but gives relatively low rate of increase between sample sizes 10 and 20, beyond sample size 20 it increases uniformly. The probability of detecting contaminant(s) as an outlying pair at 5% level of significance is found to be good and is increasing with the increase of the sample size.

It can be noticed at 10% level of significance that the power of the test statistic under consideration increases with the increase of sample size up to 15 and beyond sample size 15 it gives relatively small changes. The non-spurious power of the test statistic at 10% level of significance increases rapidly with the increase of sample size from sample size 5 to 10 and beyond sample size 10 it increases uniformly with comparatively low rate of increase. The spurious power of the test statistic at 10% level of significance is found to be low. It decreases between sample sizes 5 and 10 and then starts to increase beyond 10 with a good rate of increase. The conditional power of the statistic at 10% level of significance shows that the conditional power increases rapidly between sample sizes 5 and 20. The probability of detecting contaminant(s) as an outlying pair at 10% level of significance is found to be high and increases with the increase of the sample size up to 10, beyond 10 the rate of increase is comparatively low. It can be concluded from the above inference that the test statistic is not found to be suitable for the detection of a pair of outliers in a sample from a Gumbel distribution. Therefore, it cannot be recommended for detection of a pair of outliers in a sample from a Gumbel distribution for case-II.

Conclusion: It can be concluded from the above study that the two suggested test statistics and (for detection of the single upper and lower outlier) are performing very well, while the performance of the test statistic suggested for the detection of a pair of outliers is very poor, especially for case-II. Thus, use of the test statistics and for detection of an upper and lower outlying observation respectively, can be used for the Gumbel distribution with unknown scale parameter but the test statistic cannot be suggested for efficient results in a Gumbel distribution.