Sign In | Subscribe
Start learning today, and be successful in your academic & professional career. Start Today!
Loading video...
This is a quick preview of the lesson. For full access, please Log In or Sign up.
For more information, please see full course syllabus of Statistics
  • Discussion

  • Download Lecture Slides

  • Table of Contents

  • Transcription

  • Related Books

Bookmark and Share
Lecture Comments (6)

0 answers

Post by sepehr zarrin on October 18, 2013

You should've used more examples.....

0 answers

Post by Manoj Joseph on May 1, 2013

do i need to learn basic of alograthim to understand the transformation formula?

0 answers

Post by Manoj Joseph on May 1, 2013

its bit more complex.On top of that the video is taking time in buffering and I am suffering

0 answers

Post by Kambiz Khosrowshahi on March 27, 2013

My apologies about previous comment (frustrated). You actually explained everything quite well, I still wish you had more examples...

0 answers

Post by Jeff Keith on January 22, 2013

You should have more examples these are hard to understand.

0 answers

Post by Tomer Eiges on March 27, 2012

At 9:24 you spelled wear as "where"


Lecture Slides are screen-captured images of important points in the lecture. Students can download and print out these lecture slide images to do practice problems as well as take notes while watching the lecture.

  • Intro 0:00
  • Roadmap 0:05
    • Roadmap
  • Variability (or Spread) 0:45
    • Variability (or Spread)
  • Things to Think About 5:45
    • Things to Think About
  • Range, Quartiles and Interquartile Range 6:37
    • Range
    • Interquartile Range
  • Interquartile Range Example 10:58
    • Interquartile Range Example
  • Variance and Standard Deviation 12:27
    • Deviations
    • Sum of Squares
    • Variance
    • Standard Deviation
  • Sum of Squares (SS) 18:34
    • Sum of Squares (SS)
  • Population vs. Sample SD 22:00
    • Population vs. Sample SD
  • Population vs. Sample 23:20
    • Mean
    • SD
  • Example 1: Find the Mean and Standard Deviation of the Variable Friends in the Excel File 27:21
  • Example 2: Find the Mean and Standard Deviation of the Tagged Photos in the Excel File 35:25
  • Example 3: Sum of Squares 38:58
  • Example 4: Standard Deviation 41:48

Transcription: Variability

Hi and welcome to

Today we are going to be talking about variability.0002

We are going to start off with just a conceptual introduction to the different kinds of ways that you could measure variability.0008

Then we are going to be talking about range, cortex, and inter quartile range.0014

We are going to be talking about variance and standard deviation.0019

In particular, we are going to focus a little bit the concept of sum of squares.0023

We are going to be talking about population, standard deviation versus sample standard deviation and talk about the differences in their formulas.0031

We are going to calculate standard deviation in Excel.0041

Let us get started.0044

Let us think about out conceptual way of thinking about variability.0048

There is lot of different ways that you could actually think about variability.0055

For instance, let me give you this example.0059

Let us say this x right here shown in each of these is the president Barrack Obama.0061

Let us say that this is the president and these are different groups of people that are standing within a formal event.0074

Here we see the secret service and this is how far each of them are from him.0088

Here we see the supreme court justices and they are scattered around him.0093

Here are his cabinet members that he has appointed and they are scattered around him.0100

Here the tea party senators.0105

Let us just that they are the senators that do not like the president as much.0108

There are seem to be hurdled over here.0114

Which of these groups of people are most spread out from the president?0119

Which of these groups of people are closest to him?0126

Who is closest to the president?0129

Can we describe that with a number?0133

There is a couple of ways that you might want to think about.0137

One we might be just look at the farthest person away from the president in each of these sets?0139

Maybe for this it is this guy or this guy and get that distance, maybe that is the distance that you need.0151

For this, it is maybe this guy or this guy.0158

Maybe here it is that guy over there.0162

Maybe here it is this guy, maybe that guy, they seem pretty distant.0166

I knew that guy is a little bit farther.0172

Just looking at the farthest person in the group, that is one way of looking at it.0174

In that case, it does not matter how many people in the group you have.0179

This group has less fewer people that this group but it would not matter if we are just looking at just the one farthest guy in the group.0184

That is one way of looking at it.0193

Another way of looking at it is creating a little boundary and saying how many people are in that boundary.0194

Maybe we have this little square around the president and we just look at how many people are in that square.0203

Maybe for here if we draw a square like that, how many people fall in that square?0208

If that was our measure we would say this group is the closest to the president. Right?0226

Here we have 1 person in this square and none other groups have any people in this square.0236

Maybe we could look at different types of squares and see if that changes anything.0239

That maybe one way of doing it.0245

Another way of doing it might be to find the area of the border.0247

That is another way of doing it.0260

That one does not seems to be a very good model because that one mean that these people are the closest to the president but this is an odd group.0270

They are close to each other but not necessarily close to the president.0280

Should that matter in a measure of variability?0285

That is another thing to think about.0289

The probably one that comes to your mind is this idea that maybe the average distance of all these guys away from the president.0291

Who has the closest average distance?0303

We also would not need to worry about how many are in the group because we divide by the number of people in the group.0309

It actually would not matter if they are close to each other or not, we just care about the distance to the president.0316

These are different ways that you could think about variability.0325

Notice that they are all ways of sticking a number on this concept of variability but you might come up with different numbers.0328

You might come up with different definition for what it means to be spread out versus very close.0337

There are some things to think about, should we be measuring how far they are from the center or how far they are from each other?0347

Center is going to be an important concept in variability so shall we measure it from the median, mode, mean?0357

Does it matter if this group has few and many members?0366

Should that be taking into account?0369

Does it matter what direction away from the president or from that center point if it is to the right or to the left, up or down?0372

What about consistent clustering?0380

Should that matter?0382

Does are some things to think about when we think about a measure of variability.0383

There are lots of different kinds of measures in variability.0388

We are going to talking about two classes of them that are going to address these questions in different ways.0391

The first class of measure that we want to think about are range, cortex, and inter quartile range.0400

This is the idea of just taking the one farthest guy or the one closest guy by looking at that person.0406

Usually, these measures of variability are used with median.0416

It is usually measuring the spread around the median.0422

One of the reason that this is going to be the case is that when we look at range, cortex, and inter quartile range, what we are doing is taking our 0716.8 distribution and cutting it up.0426

Either cutting it up in a half which would be the median, the middle point.0439

Or cutting it up into quartiles, right?0444

Which would be cutting it into ¼ instead of ½.0447

That is the idea.0452

That is why we are going to be using median as their measure of central tendency.0454

When we think about range, you do not need a central tendency at all.0461

What you need is the minimum value and the maximum value and the distance in between.0466

You could think of it as the maximum value in the set of x then subtract the minimum value in the set of x.0473

If you have 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 as your distribution, you take 10 – 1 and your range is 9.0485

The problem with that measure of variability even though it is very simple and intuitive, it is highly susceptible to outliers.0493

If we change our set to something like 1, 2, 3, 4, 5, 6, 7, 8, 9, 100, all of a sudden it will be 100 – 1 and our range will be 99.0504

Just by changing one of our numbers we could drastically change the range.0516

Inter quartile range is going to be less susceptible to those outliers but before we get into 0523

how to calculate inter quartile range, we have to divide that data into quartiles.0529

Let us just look at a simple example.0537

Here what we would need to do is divide this data into quartiles first.0549

Since it is an even number, the median would fall in between 5.5 and to divide it further to the quartiles we divide it by 3 and divide it up to 8.0556

Here is the first quartile, second quartile, third, and fourth.0570

Because of that, this borders actually has special little names.0575

These borders are called Q1, Q2, and Q3, just to indicate that they are the borders of the quartiles.0579

First you divide the data in quartiles and then basically in order to get the interquartile range, you are lapping off these guys on the ends.0590

It is like the end of bread or cucumber, just like chopping it off, casting it aside.0602

Just in case that there are some extreme outliers.0608

Here what we do is we take Q3 – Q1.0612

In this case, it would be 8 – 3 and the inter quartile range would be 5.0620

Here the inter quartile range gives you the idea 50% of the numbers fall into this range because that is two quartiles.0626

That is 50% right there.0638

That is why it is a nice measure.0640

It is more best than actual range because it is less susceptible to outliers.0642

It is still intuitive and you can see that nice 50% of all the numbers falls in this range.0648

That is inter quartile range, pretty easy.0656

Let us do an example.0661

Here let us say that there are these ages and we want to know what are the inter quartile range of these cells.0663

First, it helps to separate them by quartiles.0671

There are 3, 6, 10 numbers here, because of that here is the mid point.0675

The median also called Q2 that is 30.0687

Here is Q1 and here is Q3.0696

In order to find inter quartile range, sometimes called iQr, it is Q3 – Q1.0709

In this case it would be 38 – 20.0721

Inter quartile range is 18.0727

Within 18 here we could just draw that distance of 18.0730

In that distance, 50% of our numbers fall in there, between 20 and 38.0737

We are going to be talking about variance in standard deviation.0749

When we talk about variance in standard deviation, it is more like in that conceptual example, 0753

that distance away from the president, where we are looking at the actual distance.0759

In statistics, what we call distance away from the mean, the president in this case, is a deviation.0766

What we might want to do is get the average deviation but there is going to be a little bit of issue.0776

When we get the deviations from the mean, remember the mean is the value at the middle.0784

The amount is actually in the middle of all the other values.0793

Some of the values are going to be greater than the mean and some of the values are less than the mean.0798

When we add all of those up, the formula looks like this, the summation sign and we take each value in our distribution x sub I, take out the mean.0804

Get that distance away from the mean, that deviation from the mean.0820

When we add all those up, where I goes from 1 all the way to n, however many we have in our sample.0824

We basically get 0 because sometimes the value is greater than the mean and sometimes the value is less than the mean.0832

When it is greater the number is greater than 0.0841

When it is less, the number is less than 0.0845

We add up a whole bunch of positive and negative numbers, you end up getting something very close to 0.0848

That is the problem because when you get 0 as your sum and you divide whatever your n is, 0854

no matter what n is it is going be 0 because 0 divided by anything is 0.0861

This is not going to work for us.0868

That is not going to be good to have every single average deviation being 0.0870

That is not useful.0877

What do we do?0878

Here we are going to sum the squared deviation.0880

Instead of just summing up all the deviations, we are going to square the deviation and them sum those up.0883

Whenever you square it, you get a positive number.0890

The sum of squares is always going to be positive.0895

You will get many advantages out of doing this squaring business and we will learn more about some of those advantages later.0899

Let us talk about how to write this in notation.0905

Here we have that same idea, that same deviation idea where looking at distances away from the mean, 0909

but we are going to square each of those distances.0919

I = 1 to n.0925

Just a word about this summing notation, basically when you have the summing notation whatever is here, 0929

you need to do this first and them sum up everything in here.0939

Sometimes what people do is they sum up all of x sub I first, they sum up all of them up and then subtract out the x.0945

But we are not summing the values, we are summing the squared deviation.0956

You got to get the squared deviation first.0964

Each values is going to have a distance and each of those distance needs to be squared and then you need to add them up.0967

This would not be equal to 0 unless all your values are 0 and your mean is 0.0977

In that case, they would not usually equal to 0.0985

This is going to be called sum of squares and that is often shown by using the term ss.0989

If it is sum of squares are the samples, sometimes you will see this notation where it has a little x down there.0998

If it is the sum of squares of the population which you probably ever have, it will be ss sub X.1006

We could look at the average squared distance from the mean, average squared deviation.1017

You will do that simply by dividing by the number of values you have.1026

When we have the variance of the sample, that is going to be called s2, that is going to be the variance.1030

I will write it in blue, right?1040

That is the variance of a sample.1041

That is just going to be ss ÷ n.1044

The problem with variance is that it is not in the same units as you mean because we have squared all the distances.1051

In order to bring it back to the same unit as the mean, it is easier for comparison, 1060

what we are going to do is get the stan dard deviation by just square rooting each side.1065

Standard deviation is just s and that is going to be just the square root of variance.1071

Standard deviation is now just the average distance from the mean, instead of average squared distance away from the mean.1085

This is going to be for samples, but in order to get variance for the population they use the lower case sigma.1094

For variance it will be lower case Σ2 and for standard deviation it will be just lower case Σ.1105

I will show you in a little bit how to do that.1111

Let us take a little bit of time to talk about sum of squares in depth.1117

Before that, there is a little typo on this page, I’m just going to correct that so that it will be smooth when we get down here.1123

Let us start from the beginning, sum of squares is always this sum of squared distances away from the mean of the sample.1136

The mean of the sample is x bar, that is how we denote it.1145

That is the symbol for it.1150

The sum of squared distances away from the mean is going to be the smallest sum of squares and from any other point.1153

You can pick any other number this will give you the smallest sum of squares.1160

Any other number will give you a bigger sum of squares.1167

Here is the problem, the sample mean is rarely ever the actual population mean.1171

Because of that, the population mean is this any other point.1179

If we have the real some of squares from the population mean, we would actually get a bigger sum of squares than we actually have.1185

That is the problem.1193

Here is why, because then that means because we have a sum of squares that is a little bit to small, 1195

our sample standard deviation is going to be actually a little bit smaller than our population standard deviation all the time.1201

That is an issue.1210

We are always under shooting the population standard deviation.1211

To correct for this, we are going to divide the sum of squares from our sample by a slightly smaller number than we actually do.1215

Right now, to get s or standard deviation, we take sum of squares ÷ n.1227

That is what we do right now.1237

This will help us approximate the actual population.1239

Here we are going to need divide by a slightly smaller number 1246

because when we divide by a smaller number, then our resulting answer is slightly bigger.1252

Dividing by 5 we are going to get a bigger answer than if you divide by 8.1259

Because of that we are going to use that.1268

Instead, in order of approximate the population standard deviation what we are going to do is use ss ÷ n – 1.1272

This is going to be a slightly smaller number giving us a slightly bigger population standard deviation.1293

Why n – 1? Why not n - .5 or n – 2?1301

There is a proof that you could look at up on line called Pessel’s Correction Proof and it is a really elegant proof if you have time to look it up.1307

That is my spill on sum of squares but we will come back to this because it is a pretty important idea.1315

Let us talk about the difference between population standard deviation and sample standard deviation.1323

We always want to make inferences from the sample to the population, that is what we would like to do.1330

Our sample distribution is denoted by lower case x and our population distribution is denoted by upper case X.1337

In order to make that leap, we are going from sample statistics to population parameter.1346

We are going to be estimating things like estimating mu from x bar, that is estimating the mean of the population from the mean of the sample.1364

We are going to estimate the Σ or the standard deviation of a population from s, which is the standard deviation of the sample.1375

Sigma is our new notation, notice that for population we are using parameters with Greek letters and here we are using regular Roman letters.1388

Let us talk about the formulas for these.1403

When we talk about mean, mu in this case, an x bar, in this case.1407

We talk about adding up all of the lower case x and dividing by lower case n.1414

Here we add it up all at once in our upper case X and dividing by upper case N, just superficial changes.1421

When we talk about standard deviation, here we are going to be talking about lower case Σ or talking about s.1433

Let us actually write down this formula.1445

You could write it as √sum of squares ÷n, that is one way to do it.1448

One thing you could do is think about double clicking on this.1455

Just double click on it.1463

Then what we would get is you would see the whole she bang inside.1466

Hopefully I could try.1472

Sum of squares means give me all the squared deviations, distances, away from x bar, square all of those.1474

If you want you could put in I = 1 all the way up to n ÷ n.1485

If we want to actually use this to estimate that, we will divide by n – 1.1505

This is upper case S and I’m going to denote that by using a little bar there.1513

In order to have this estimation, we would use lower case s.1520

In this case, what we would do is divide our sum of squares by n – 1.1534

That is our way of estimating from s to Σ.1540

That is our estimate.1544

When we talk about the population standard deviation, it is still ss ÷ n but it is upper case S this time.1547

When we double click on ss and see what is inside of it, we unpack that, here is what it looks like.1559

It is (X sub I – mu2) ÷ N.1569

Here are all of these formulas.1581

We have formulas for standard deviation of the sample, standard deviation of the population, but we also have this new idea.1592

This is in between this one and this one.1595

It is a way of going from sample information to estimating a population standard deviation.1600

Usually, we do not calculate sigma directly because we do not have every single value for the population.1611

Usually, we calculate small s which is going to be the estimated standard deviation and 1619

we hardly use this one as well because we do not really care about the standard deviation in just our sample.1628

We want to know the standard deviation for the population.1635

Let us go on to our examples.1644

Here is example 1.1646

It says find the mean in standard deviation of the variable friends in the Excel file.1648

If you get the Excel file that you can download, go ahead and click on friends.1655

We are going to be finding the standard deviation for the variable friends.1662

What would be nice is if we could do everything in Excel but before we do that I jut want to make sure you understand how standard deviation works.1671

Because of that I’m going to have you do it manually first.1680

In order to do that, go ahead and go to data, find the variable friends, click on that column 1684

and I’m just going to copy that whole column and paste that right in here.1693

Here I have my entire distribution of friends.1702

I’m going to say Excel calculate the mean for us.1707

I’m going to use the function average and select all this nice data right here, click enter.1712

That is our mean.1725

That mean is not going to change for anybody because mean is just the mean of the entire distribution.1729

I’m just going to put our pointer there and I’m going to say whatever the mean is on top of me, 1737

that is the mean and I’m just going to paste that all the way down.1742

This whole column should have the same mean.1749

The reason I’m doing that is because that is going to make it easier for us to calculate square of deviation.1754

We could just use the locked version of mean too.1762

Let us get our squared deviation.1767

Deviation just means the distance from each value to the mean x bar2.1771

In order to do the square we put in the count and 2.1782

We hit enter and here is our squared deviation.1789

I’m just going to drag that formula all the way down.1794

Here we have a whole bunch of squared deviations.1800

We have to sum up all those squared deviations.1804

Here I’m just going to put in ss because that is what we are going to get and in order to get ss, we just add up this whole column.1809

In order to get variance, where S2 what we need to do is take ss ÷ n.1829

I’m going to take ss ÷ n.1844

I know here that my n is 100 but if you did not know for some reason, you could use the function count 1849

and just ask it count how many values there are.1855

Not count it, just count, count how many values there are.1858

It should be 100.1864

Indeed it is a hundred because it moved the decimal point 2 over.1868

Now we could get standard deviation or S.1873

In order to get that, we just square root our variance.1879

Excel has a function called square root (sqrt) and I’m just going to square root my variance.1883

Here I get a standard deviation of 428.64.1892

I need to do all that just so that you would understand how to calculate standard deviation.1898

Excel has a nice handy way for you to do it.1906

Here I’m going to calculate s automatically.1908

Here we are looking at just s, in order to calculate s we would do stdevp because that is the one where you divide by n.1916

I’m finding the standard deviation of all my squared values, that is wrong.1953

I should be finding the standard deviation of my actual data, right?1956

In this method, you actually do not need any of this.1965

I will just make you go through it so you would learn.1969

When we calculate s automatically, using stdevp you will see that we get the exact same standard deviation 1971

and we do have to do any of that mean calculating or calculating sum of squares of variance or anything like that.1981

There is even a way Excel will calculate for you little s, the estimate of the population standard deviation from the sample.1989

That is the one that you will be most likely using.2004

Because of that, I think that might be a good one for us to do.2007

Sum of squares is going to be the same thing.2010

I’m just going to copy all of this.2017

The sum of squares is going to be the same thing but variance is going to be a little bit different now.2019

Instead, I will be dividing by n, we are going to be dividing by n – 1.2029

I’m going to put in 99 instead of 100.2036

Square rooting, that works the same way, square root of my variance.2043

I noticed that when we divide by n -1, my standard deviation is slightly bigger than it would have been when we just divided up by n.2053

Let us calculate little s automatically.2074

Excel always assumes that is probably what you will be wanting to do.2077

It made stdev that default formula is going to divide by n -1.2084

We see that those two are the same values, a shortcut.2102

You see when you automatically calculate it with Excel, you are not going to need to calculate mean 2107

or the sum of squares but it is nice to know where those things come from.2117

We did that already.2124

Let us find the mean and standard deviation of the tagged photos in the Excel file.2129

If you click over on data, let us go ahead and grab the tagged photos values in that variable column and paste it right in here.2137

It is just easier than going back and forth.2151

Let us find the mean in this sample.2154

I typed in average and I wanted to average all of this then I’m just going to say whatever is above me that is the same mean.2161

Copy and paste it all the way down, everybody else has the same mean.2182

I’m just going to get my squared deviation.2188

It is my first value – the mean2.2193

I’m going to copy and paste that all the way down.2203

Let us get the sum of squares.2213

In order to do that we just find the sum of all these squared deviation.2216

In order to find variance or S2, that is just s2 because that is the one you will be using for most part, right?2229

Our little s2, we take this sum of squares and we divide it by n -1.2240

We could use count, count all of that – 1.2249

All of this is in my denominator and hit enter.2269

That is my variance.2279

What is my standard deviation?2282

My little s, my estimated standard deviation.2286

All I have to do is square root my variance and that is what I got.2289

Let us check our answers by using the automatic Excel version.2296

Here we will put in stdev, I want to put in our actual data, our actual values.2305

This is our real distributions that we are working with here.2318

Excel does it nice and quickly for us.2325

We do not need all of these stuff.2328

In the future, we will just be using this automatic version but I do want you to know where that comes from.2330

Let us go on to example 3.2340

The average number of calories in a frozen yogurt is 250, with an estimated population standard deviation of 30.2342

If 24 frozen yogurts from popular chains where sampled, what would be their ss or sum of squares?2349

Here we know that we do not need the actual values and the means in order to find sum of squares.2358

Because we have some of the other pieces and we could just fill out what is missing and figure out what is missing.2365

We know that they have estimated population and standard deviation.2373

That is little s.2378

In order to get little s, we know that they added up all of the x sub I – the mean2 ÷ n -1 and took the square root of that.2382

We know that is what they did.2404

Another way of writing that is square root of ss / n – 1.2405

Let us fill in what we have.2414

They know that the standard deviation eventually is 30, this s is 30.2417

What we are trying to find out is this.2428

We do not have that ss.2431

But we do have n – 1 because n is 24.2439

24 – 1 is 23.2444

From that, and only that information we could figure out ss and in they have given us this mean 250.2448

It is sort of red airing, you do not actually need it in this problem.2458

I’m going to use a little piece of my Excel as a calculator and here I know I need to square 30, 302.2464

I could just multiply 23 to that.2486

I will get 20,700.2491

My ss is 20, 700.2496

I did not actually need all my values from the distribution nor my mean.2504

Last question, example 4.2512

This is a conceptual question, hopefully this will test you on concepts.2515

When we divide by n – 1, rather than by n, what effect does this have on the resulting standard deviation?2521

N -1 is a smaller number than n, right?2529

Dividing by a smaller number will result in a bigger answer.2532

The resulting standard deviation s will be a little bit greater than this s.2536

This one divides by n and this one divides by n -1.2544

That is it for variability.2556

Thanks for using