OpenEuphoria

2. Re: Mathematicians !! Percentile/Quartile function

• Posted by "Juergen Luethje" <j.lue at gmx.de> Sep 07, 2005
• 666 views

Pete Stoner wrote:

> I need some mathematical help, I want a function that calculates the
> 1st, 2nd and 3rd Quartiles (otherwise known as 25th, 50th and 75th
> percentiles) from a sequence of values. I can get the 2nd quartile as
> that is also the median and I can get the 1st and 3rd quartiles to
> agree with Excel and 123 if the qty of values is odd, but when I have
> an even quantity the 1st and 3rd quartiles are wrong.
> i.e. for values of {2, 6, 7, 10, 14, 15, 16} the quartiles are 6.5, 10
> and 14.5

IMHO it must be 6, 10, 15.

> but for a range of {2, 6, 7, 10, 14, 15} the correct quartiles seem to
> be 6.25, 8.5 & 13

<snip>

It depends whether we have qualitative or quantitative data.
If you don't know what that means, see e.g. here for an explanation:
http://davidmlane.com/hyperstat/intro.html

>From what you wrote it looks as if you have quantitative data.
Then after my algorithm the wanted quantiles are 6, 8.5, 14.

Following you'll find some code for calculating an arbitrary number of
arbitrary quantiles (e.g. the 25th percentile is the 0.25 quantile).
Because calculating the mean is only allowed for quantitative data, you
must pass TRUE or FALSE to the function as second parameter, depending
on whether or not your data are quantitative.

The algorithm is according to the information given in a German
statistical textbook. AFAIK there are other ways to calculate quantiles,
which might return different results in some cases.

include sort.e

global constant
   FALSE = 0, TRUE = not FALSE

global type boolean (object x)
   if integer(x) then
      return x = FALSE or x = TRUE
   end if
   return FALSE
end type

global type sequence_of_atom (object x)
   if atom(x) then
      return FALSE
   end if
   for i = 1 to length(x) do
      if not atom(x[i]) then
         return FALSE
      end if
   end for
   return TRUE
end type

global function quantile (sequence_of_atom values, boolean quantitative,
sequence_of_atom quant)
   -- # Example #
   -- sequence measured
   -- measured = {32,34,35,37,38,39,41,42,43,46}
   -- ? quantile(measured, TRUE, {0.25,0.50,0.75})
   --
   -- [after Lorenz, Rolf J.:
   --  Grundbegriffe der Biometrie.
   --  Fischer, Stuttgart - Jena - New Yorck
   --  3rd ed. 1992, p. 43]
   sequence ret
   atom temp, i
   integer n

   values = sort(values)
   n = length(values)
   quant = sort(quant)
   ret = repeat(0, length(quant))

   for q = 1 to length(quant) do
      temp = n*quant[q]
      i = floor(temp)
      if temp != i or values[i] = values[i+1] then
         ret[q] = values[i+1]
      elsif quantitative then
         ret[q] = (values[i]+values[i+1])/2
      else
         ret[q] = values[i..i+1]
      end if
   end for

   return ret
end function


sequence measured

measured = {2, 6, 7, 10, 14, 15, 16}
? quantile(measured, TRUE, {0.25, 0.50, 0.75})
measured = {2, 6, 7, 10, 14, 15}
? quantile(measured, TRUE, {0.25, 0.50, 0.75})

Regards,
   Juergen

-- 
Have you read a good program lately?

4. Re: Mathematicians !! Percentile/Quartile function

• Posted by Pete Stoner <stoner.pete at gmail.com> Sep 07, 2005
• 649 views
• Last edited Sep 08, 2005

Juergen Luethje wrote:
> 
> Pete Stoner wrote:
> 
> > I need some mathematical help, I want a function that calculates the
> > 1st, 2nd and 3rd Quartiles (otherwise known as 25th, 50th and 75th
> > percentiles) from a sequence of values. I can get the 2nd quartile as
> > that is also the median and I can get the 1st and 3rd quartiles to
> > agree with Excel and 123 if the qty of values is odd, but when I have
> > an even quantity the 1st and 3rd quartiles are wrong.
> > i.e. for values of {2, 6, 7, 10, 14, 15, 16} the quartiles are 6.5, 10
> > and 14.5
> 
> IMHO it must be 6, 10, 15.
Thanks for the reply Juergen, I understand why you say that, I've read that
there are different ways to calculate this but both Lotus 123 and Excel seem to
include the median value when calculating the 1st and 3rd quartiles which gives
6.5, 10, 14.4
> 
> > but for a range of {2, 6, 7, 10, 14, 15} the correct quartiles seem to
> > be 6.25, 8.5 & 13
> 
> <snip>
> 
> It depends whether we have qualitative or quantitative data.
> If you don't know what that means, see e.g. here for an explanation:
> <a
> href="http://davidmlane.com/hyperstat/intro.html">http://davidmlane.com/hyperstat/intro.html</a>

I don't know which type 123 and Excel use (neither say which in the help) but I
would like my calculation to match the output of those as they both give the same
results...
> 
> >From what you wrote it looks as if you have quantitative data.
> Then after my algorithm the wanted quantiles are 6, 8.5, 14.
> 
> Following you'll find some code for calculating an arbitrary number of
> arbitrary quantiles (e.g. the 25th percentile is the 0.25 quantile).
> Because calculating the mean is only allowed for quantitative data, you
> must pass TRUE or FALSE to the function as second parameter, depending
> on whether or not your data are quantitative.
> 
> The algorithm is according to the information given in a German
> statistical textbook. AFAIK there are other ways to calculate quantiles,
> which might return different results in some cases.
> 
> <snip>

Your routine is more adaptable than mine and the results for the even quantity
 are the same for both (but do not match 123 and Excel), but mine matches the
 spreadsheets for an odd qty whereas yours is different again. I'm curious as to
 how they are calculating it.. It would seem to be the 'accepted' norm since 2
 products from different companies give the same results..

Regards Pete.

6. Re: Mathematicians !! Percentile/Quartile function

• Posted by Pete Stoner <stoner.pete at gmail.com> Sep 07, 2005
• 588 views
• Last edited Sep 08, 2005

Lucius L. Hilley III wrote:
> 
> <a
> href="http://www.vias.org/tmdatanaleng/cc_quartile.html">http://www.vias.org/tmdatanaleng/cc_quartile.html</a>
> 
> Depends on how you want to do it.
> I would think that the quartile would be sort of a median between medians
> but...
> It looks as though the above links definition differs slightly.
> It defines the median as the average of the 2 middle numbers if you have an
> even number of values.
> However, it defines the first and third quartile as a specific value that is
> in the list of values.
> 
> }}}
<eucode>
> --Warning: Code should be close enough to rounding for positive numbers.
> function positive_round(object x)
>   return floor(x+0.5)
> end function
> 
> --Warning: Code should be close enough to rounding for negative numbers.
> function negative_round(object x)
>   return floor(x-0.5)
> end function
> 
> function round(object x)
>   object px, nx
> 
>   px = x * (x > 0)
>   nx = x * (0 > x)
> 
>   return positvie_round(px) + negative_round(nx)
> end function
> 
> function quartile(sequence list)
>   integer n, q1_index, q2_index, q3_index
>   atom median
> 
>   n = length(list)
>   q1_index = round((n+1)/4) -- This could easily be optimized
>   q2_index = n/2
>   q3_index = round((n+1)*3/4) -- This could be optimized as well
>   
>   median = list[q2_index]
>   if (n = floor(n)) then -- if even(n) then
>     median += list[q2_index+1]
>     median /= 2
>   end if
> 
>   sequence {list[q1_index], median, list[q3_index]}
> end function
> 
> print(1, quartile({2,6,7,10,14,15}))
> puts(1, 10) -- new line
> machine_proc(26, 0) -- wait_key() cheat !!!
> <font color="#330033"></eucode>
{{{
</font>
> 
>   This code was proudly presented by:
>     Lucius L. Hilley III - Unkmar
> 

Nice try Lucius, but that gives 2, 8.5 and 10.. Correct median but 1st and 3rd
 quartiles are way off..

Regards Pete.

8. Re: Mathematicians !! Percentile/Quartile function

• Posted by Pete Stoner <stoner.pete at gmail.com> Sep 07, 2005
• 577 views
• Last edited Sep 08, 2005

Pete Lomax wrote:
> 
> On Wed, 07 Sep 2005 09:45:41 -0700, Pete Stoner
> <guest at RapidEuphoria.com> wrote:
> 
> >i.e. for values of {2, 6, 7, 10, 14, 15, 16} the quartiles are 6.5, 10 and
> >14.5
> >but for a range of {2, 6, 7, 10, 14, 15} the correct quartiles seem to be
> >6.25, 8.5 & 13
> FWIW, (and strictly from observation only!) the first is (or might be)
>  (6+7)/2, 10, (14+15)/2, while ths second is (or might be)
> (2+6+7+10)/2, (7+10)/2, (10+14+15)/2
> 
> As I don't have either Excel or 123 to hand, I can't experiment more,
> but I wonder what happens with sets of longer numbers.
> 

I just tried it with a larger set..
{2,2,2,3,6,6,8,8,10,12,12,14,16,19,22,25,28,28,32,33}
results are Lucius - 3, 12, 22
Juergen and I got 6, 12, 23.5
123 and Excel give 6, 12, 22.75

Regards Pete

10. Re: Mathematicians !! Percentile/Quartile function

• Posted by Pete Stoner <stoner.pete at gmail.com> Sep 07, 2005
• 617 views
• Last edited Sep 08, 2005

Pete Stoner wrote:
> 
> Pete Stoner wrote:
> > 
> > Pete Lomax wrote:
> > > 
> > > On Wed, 07 Sep 2005 09:45:41 -0700, Pete Stoner
> > > <guest at RapidEuphoria.com> wrote:
> > > 
> > > >i.e. for values of {2, 6, 7, 10, 14, 15, 16} the quartiles are 6.5, 10
> > > >and 14.5
> > > >but for a range of {2, 6, 7, 10, 14, 15} the correct quartiles seem to be
> > > >6.25, 8.5 & 13
> > > FWIW, (and strictly from observation only!) the first is (or might be)
> > >  (6+7)/2, 10, (14+15)/2, while ths second is (or might be)
> > > (2+6+7+10)/2, (7+10)/2, (10+14+15)/2
> > > 
> > > As I don't have either Excel or 123 to hand, I can't experiment more,
> > > but I wonder what happens with sets of longer numbers.
<snip>

I found this interesting article
http://www.gslis.utexas.edu/~wyllys/IRLISMaterials/quartiles.pdf
which explains how Excel (and I presume 123 'copied' them  ) calculates its
quartiles and suggests it is 'wrong' or "the algorithm clearly
yields results, even in certain simple cases, that fail to accord with what most
people who use quartiles expect to see.".... "This [makes] the algorithm . . .
neater from an IT syntax perspective, but less accurate mathematically.".
For the small example they give (even qty of values)  Juergen and I get the
  same results as they do, whereas Excel gives different numbers

so I think I'll stick with what I've got..

Thanks all
Regards Pete.

12. Re: Mathematicians !! Percentile/Quartile function

• Posted by Pete Lomax <petelomax at blueyonder.co.uk> Sep 08, 2005
• 601 views

On Thu, 08 Sep 2005 02:47:46 -0700, CChris <guest at RapidEuphoria.com>
wrote:

>}}}
<eucode>
<snip>
You obviously had the right idea, but it only worked for the first
quartile. The main change I made was:
--   interPoint=(3+length(data))/4
   interPoint=(4-whichOne+length(data)*whichOne)/4

Based on your code, I wrote a complete test harness:

include sort.e

type qtile(object n)
    return integer(n) and n>=0 and n<=4
end type

function quartile(sequence data, qtile whichOne)
-- matches what Excel and 123 return
-- (which does not necessarily mean results are correct)

integer lowerObs
atom interPoint

    interPoint=(4-whichOne+length(data)*whichOne)/4
    lowerObs=floor(interPoint)
    interPoint-=lowerObs
    data=sort(data)&0   -- extra 0 to avoid bounds check
    return (1-interPoint)*data[lowerObs]+interPoint*data[lowerObs+1]
end function

procedure test(sequence testset, sequence expected)
integer testsetshown
    if length(expected)!=5 then ?9/0 end if
    testsetshown=0
    for i=1 to 5 do
        if quartile(testset,i-1)!=expected[i] then
            if not testsetshown then
                ?testset
                testsetshown=1
            end if
            printf(1,"quartile(%d) is %f, expected:%f\n",
			{i-1,quartile(testset,i-1),expected[i]})
        end if
    end for
end procedure
--test({2},			-- data
--     {2, 2, 2, 2, 2})	-- expected quartiles 0..4
test({2, 6, 7, 10, 14, 15, 16},
     {2, 6.5, 10, 14.5, 16})
test({2, 6, 7, 10, 14, 15},
     {2, 6.25, 8.5, 13, 15})
test({2,2,2,3,6,6,8,8,10,12,12,14,16,19,22,25,28,28,32,33},
     {2, 6, 12, 22.75, 33})
test({2,2,2,3,6,6,8,8,10,12,12,14,16,19,22,25,28,28,32,33,34,35,36,38,40,41,45,48,49,51,53,56,57,57,57,59,61,61,62,62},
     {2, 12, 33.5, 51.5, 62})

puts(1,"All Done\n")
if getc(0) then end if

Regards,
Pete (L)

14. Re: Mathematicians !! Percentile/Quartile function

• Posted by "Juergen Luethje" <j.lue at gmx.de> Sep 09, 2005
• 603 views

Pete Stoner wrote:

<snip>

> I found this interesting article
> http://www.gslis.utexas.edu/~wyllys/IRLISMaterials/quartiles.pdf

Also interesting for me, thanks for the link.

> which explains how Excel (and I presume 123 'copied' them  )

It looks as if e.g. OpenOffice and SpreadCE (for Pocket PC) also copied
Excel's algorithms for calculating quartiles and quantiles (and probably
many other algorithms, too).

> calculates its quartiles and suggests it is 'wrong' or "the algorithm
> clearly yields results, even in certain simple cases, that fail to
> accord with what most people who use quartiles expect to see."....

<snip>

I agree to this. Here is another interesting article about problems when
using Excel for serious statistics
http://www.practicalstats.com/Pages/excelstats.html

Following is a slightly improved version of the function that I posted
the day before yesterday ( quantiles1() ), together with a new function
(which is also not in accordance with Excel):

global constant
   FALSE = 0, TRUE = not FALSE

global type boolean (object x)
   if integer(x) then
      return x = FALSE or x = TRUE
   end if
   return FALSE
end type

global type sequence_of_atom (object x)
   if atom(x) then
      return FALSE
   end if
   for i = 1 to length(x) do
      if sequence(x[i]) then
         return FALSE
      end if
   end for
   return TRUE
end type

global type sequence_of_qtile (object x)
   object t

   if atom(x) then
      return FALSE
   end if
   for i = 1 to length(x) do
      t = x[i]
      if sequence(t) or t <= 0 or 1 <= t then
         return FALSE
      end if
   end for
   return TRUE
end type

global function min (sequence s)
   object x

   x = s[1]
   for i = 2 to length(s) do
      if x > s[i] then
         x = s[i]
      end if
   end for
   return x
end function

global function interpol_lin (atom x1, atom y1, atom x2, atom y2, atom x)
   -- Linear interpolation (or extrapolation).
   atom dx

   dx = x2 - x1
   if dx != 0 then
      return (y2-y1)/dx * (x-x1) + y1
   else
      return {}                  -- error
   end if
end function

------------------------------------------------------------------------

global function quantiles1 (sequence_of_atom values, boolean quantitative,
sequence_of_qtile quant)
   -- in: values      : list of measured data, sorted ascending
--     quantitative: TRUE if and only if the values are measured on an
   interval or ratio scale
   --     quant       : list of wanted quantiles
   --
   -- [after
   --  Lorenz, Rolf J.:
   --  Grundbegriffe der Biometrie.
   --  Fischer, Stuttgart - Jena - New Yorck
   --  3rd ed. 1992, p. 43]

   sequence ret
   atom x
   integer n, i

   ret = repeat(0, length(quant))
   n = length(values)

   for k = 1 to length(quant) do
      if (n+1)*min({quant[k],1-quant[k]}) >= 1 then
         x = n*quant[k]
         i = floor(x)
         if x != i or values[i] = values[i+1] then
            ret[k] = values[i+1]
         elsif quantitative then
            ret[k] = (values[i]+values[i+1])/2
         else
            ret[k] = values[i..i+1]
         end if
      else
         ret[k] = {}     -- not enough values to calculate this quantil
      end if
   end for

   return ret
end function

------------------------------------------------------------------------

global function quantiles2 (sequence_of_atom values, boolean quantitative,
sequence_of_qtile quant)
   -- in: values      : list of measured data, sorted ascending
--     quantitative: TRUE if and only if the values are measured on an
   interval or ratio scale
   --     quant       : list of wanted quantiles
   --
   -- [after
   --  Ackermann, Hanns:
   --  Biometrie.
   --  epsilon-Verlag, Hochheim Darmstadt
   --  4th ed. as eBook 2004, p. 34
   --             and
   --  BIAS 8.1
   --  http://www.bias-online.de]

   sequence ret
   atom x
   integer blocks, i

   ret = repeat(0, length(quant))
   blocks = length(values) + 1

   for k = 1 to length(quant) do
      if blocks*min({quant[k],1-quant[k]}) >= 1 then
         x = blocks*quant[k]
         i = floor(x)
         if x = i then
            ret[k] = values[i]
         elsif quantitative then
            ret[k] = interpol_lin(i,values[i],i+1,values[i+1], x)
         else
            ret[k] = values[i..i+1]
         end if
      else
         ret[k] = {}     -- not enough values to calculate this quantil
      end if
   end for

   return ret
end function

------------------------------------------------------------------------

-- Demo
sequence values

values = {2, 6, 7, 10, 14, 15, 16}
-- values = {2, 6, 7, 10, 14, 15}
-- values = {1,2,4,7,8,9,10,12}
-- values = {1,2,3,4}
-- values = {20,30,40,50}

puts(1, "values = ") ? values
puts(1, "---------------------------------------------\n")
puts(1, "quantiles1 = ") ? quantiles1(values, TRUE, {0.25,0.50,0.75})
puts(1, "quantiles2 = ") ? quantiles2(values, TRUE, {0.25,0.50,0.75})

Regards,
   Juergen

-- 
Have you read a good program lately?