Sample size vs. Error A tutorial By Bill Thomas, Colby-Sawyer College.

Sample size vs. Error

A tutorialBy Bill Thomas, Colby-Sawyer College

IntroductionIn the pipetting tutorial, you explored the utility of the mean, the standard deviation and the relative error in describing the reproducibility and accuracy of a sample. You also learned a few tricks for working more efficiently within Excel.

In this tutorial, we are going to explore the relationship between sample size and variability. How well do our descriptors (mean, st. dev.) work to tell us if our sample is representative of the population we are trying to describe, or evaluate. For this we will introduce a new descriptor, the Standard Error of the Mean (SEM), and we will see how it varies with the sample size.

Along the way, we will also gain a bit more experience using Excel.

First, what do we mean by the “population” and the “sample”?

Let’s suppose that our population were the numbers from 1 to 10. There would be 10 members of the population, and it would not be difficult to sample (e.g., consider, do an experiment on, take data on, or do our calculations on) every member of the population.

However, what if our population had 10,000 elements (or even more)? It would be impractical, verging on impossible, to treat every member of the population separately. Thus we frequently select a sample population to represent the full population. This approach makes the work easier, but it raises the question of how representative the sample population is of the entire, or full population.

Consider the images on the following slide.

Full population

Sample population= The sample population

IS the full population; this is ideal

Full population

Sample population

The sample population is much smaller than the full population, but the full population is uniform, so the sample is representative of the full population. This is rarely the case, unfortunately.

Full population

Sample population

Here the full population is non-uniform, so a small sample cannot be truly representative of the full population. This is most often the case we face in reality, and the question is, how to sample appropriately under these circumstances.

Let’s begin by getting a sense for the kind of variability we might encounter. For this we will use a new tool in Excel, a random number generator, that will allow us to create randomly generated populations of numbers of any size and within any limits. The function looks like this:

=RANDBETWEEN(x,y)

It will generate in the cell in which it is written a random number between the set limits x and y. Thus, if you specify 2 and 4, it will produce a number in the chosen cell between 2 and 4.

Let’s use this function.

Open a new Excel file, and in cell A1 type in the function =RANDBETWEEN(9,11). When you click “enter”, there will appear a number between 9 and 11. Drag click this cell down to fill 9 more cells. Each of the 10 cells should now contain a number between 9 and 11. Highlight the 10 vertical cells and drag click them over to column P. You should have an array 16 columns by 10 numbers, each number randomly generated, that looks like this:

Note that the number in each position of your set will be different that that shown here.

Now, there is a little quirk about the function that you just used. Each time you attempt to copy and paste a cell containing it, the number in the cell changes. While this feature is useful for some purposes, it makes what you are about to do a bit more challenging.

We need to have numbers that do not change as you work with them, so here is what to do.

Select all 160 numbers and copy them. Then put the cursor on cell C13. If you have a PC, right click on the cell and select the option “paste special”. If you have a Mac, highlight cell C13 and under the edit option on the menu bar, select “paste special”.

In each case, you will be given a menu, the first choice on which is “paste”. Under the paste submenu, select “values” and click OK.

The numbers that appear will now be fixed; they will not vary when you manipulate them.

Next, as preparation for the steps to come, color the cells and add in the other details shown below.

Remember: A cell with a yellow background color is a cell into which you type a value, a number (like all the values above). A cell with a salmon background color is a cell in which you must write an equation to generate the number shown (which will be the case with most of the steps to follow).

Now, let’s think for a minute about the numbers that you have generated. The range allowed was from 9 to 11 (or 10 +/- 1), so you can see that the average of these numbers ought to be midway between 9 and 11, or 10. Let’s use Excel to calculate the mean of the first 3 numbers in each vertical column of 10 numbers to see how close it is to 10. Set your spread sheet up as shown below, being sure to calculate the mean and (below it) the standard deviation for the first three numbers in each column.

Now look at your 16 means. How similar are they? How close to the expected value of 10 are they? Are you surprised?

Let’s visualize the distribution. Create a scatter plot (without a line connecting the data points) of the means with the axes shown below:

0 2 4 6 8 10 12 14 16 188

9

10

11

12

Distribution of sample means

# of samples

sam

ple

mea

n

There is a number that describes the variation within each data set, and that number is the standard deviation. Calculate the standard deviation for each of your sets of 3 as shown below.

Generate a scatter plot of the standard deviations, as well. What does it tell you about the variation within your data?

The standard deviation expresses the variation within the sample set, but it does not really tell us how well the sample represents (estimates the values for) the full population that we are trying to evaluate. For that we need a new term, the standard error of the mean.

S.E.M. = standard deviation of the sample/square root of the sample size

Use the standard deviations that you just calculated to calculate the standard error for each sample of 3.

Use a scatter plot to visualize the distribution of these values as well.

It should be clear that there is quite a bit of variation in the data when they are sampled three at a time. Are samples of this size good at representing the full population from which they are taken?

What would happen, if we took larger samples from the full population? Begin by doing the same calculations for each full column of 10 (not just the first three values in the column).

Add these new data to each of your three scatter plots.

Now take the data twenty values at a time; calculate the mean, st. dev. and SEM for two columns of data. You will have half as many calculated values as before.

Again, add these new data to each of your three scatter plots.

Complete this exercise in three more successive steps for the data taken 40 values at a time, 80 values at a time, and 160 values at a time, as shown below. Note that 160 values constitutes the entire population.

Conclude by adding these new data to each of your three scatter plots. Do your plots show you a pattern? What is that pattern?

20 20 20 20 20 20 2020

40

80

40 40 40

80

160

In the scatter plots that you have generated thus far, you have illustrated the distributions of your various calculations, showing each descriptor for each population evaluated. It is possible, and often very helpful, to visualize the trends in the data with summary graphs. To this end, calculate to the far right as shown, the means for all the values grouped horizontally across each line.

Thus the value to the right of the first line will be the mean of all 16 means of the data taken three values at a time. The second line will be the mean of all the st. devs. of the the data taken three values at a time, and on down the column to the last value, which will be just the values calculated for the entire population

Graph the mean standard errors calculated against the sample size as indicted on the axes below.

0 20 40 60 80 100 120 140 160 1800

2

4

6

8

10

12

Mean SEM vs sample size

# data points per sample set

Mea

n st

anda

rd e

rror

Do you see a trend? How big does the data set have to be to fulfill the trend? Let’s explore that a bit.

Let’s create a really big data set. Use the random number generator to create a column of 1000 values. Then calculate the mean, st. dev., and SEM for the sample sets having 3, 10, 20, 50, 100, 200, 500 and 1000 values (the last being the entire population). Then make two graphs, one of the mean, the other of the SEM, plotted against sample size.

Do you remember that we assumed at the beginning that the mean for a population distributed randomly about 10 should have a mean of 10? What do the data show you?

Similarly, how big does the sample have to be before the SEM of the sample approximates the SEm of the entire population?

1000

Some things to try:

Now that you have the calculator and graphs all set up, you can “play” pretty easily. What happens to your values and graphs from the previous slide if you change the spread of the data?. For example, what if you used RANDBETWEEN(7,13)? How about RANDBETWEEN(5,15)?

Does changing the size of the data (2000, 5000, 10,000 values) set have any interesting effect on these outcomes? (Don’t be intimidated. The numbers look huge, but you can create such sets in seconds). How big does each population have to be for the mean to resolve to the expected value?

Finally, what have you learned about sampling a population with inherent variability?