WEBVTT mathematics/linear-algebra/hovasapian
00:00:00.000 --> 00:00:04.000
Welcome back to Educator.com and welcome back to linear algebra.
00:00:04.000 --> 00:00:13.000
In our last lesson, we introduced the notion of an orthogonal complement, and this time we are going to continue talking about orthogonal complements.
00:00:13.000 --> 00:00:23.000
We are going to be talking about these 4 fundamental subspaces that are actually associated with any random matrix, and we are going to talk about the relationships that exist between these spaces.
00:00:23.000 --> 00:00:33.000
Then we are going to talk about something called a projection. The projection is a profoundly, profoundly important concept.
00:00:33.000 --> 00:00:42.000
It shows up in almost every area of physics and engineering and mathematics in ways that you would not believe.
00:00:42.000 --> 00:00:52.000
As it turns out, those of you who are engineers and physicists... one of the tools in your tool box that is going to be almost the primary tool for many years to come is going to be the idea of something called Fourier series.
00:00:52.000 --> 00:01:01.000
If you have not been introduced to it yet, you will more than likely be introduced to it sometime this year... and Fourier series actually is an application of projection.
00:01:01.000 --> 00:01:20.000
Essentially what you are doing is you are taking a function and what you are doing is you are projecting that function onto -- how shall I say this -- you are projecting it onto an infinite dimensional vector space on the individual axes which are the trigonometric functions.
00:01:20.000 --> 00:01:36.000
Let us say, for example, I have a function 5x. I can actually project that function onto the cos(x) axis, onto the sin(x) axis, onto the cos(2x) axis, onto the sin(2x) axis, so on and so forth.
00:01:36.000 --> 00:01:40.000
I can actually represent that function in terms of cosine functions.
00:01:40.000 --> 00:01:45.000
Now, you will learn it algebraically, but really what you are doing is you are actually doing a projection.
00:01:45.000 --> 00:01:51.000
You are projecting a function onto other functions, and it is really quite extraordinary.
00:01:51.000 --> 00:02:00.000
When you see it that way, the entire theory of Fourier series becomes open to you, and more than that, the entire theory of orthogonal polynomials becomes open to you.
00:02:00.000 --> 00:02:06.000
That is going to connect to our topic that we discuss in the next lesson, which is Eigenvectors and Eigenvalues.
00:02:06.000 --> 00:02:15.000
So, linear algebra really brings together all areas of mathematics. Very, very central. Okay. Let us get started.
00:02:15.000 --> 00:02:23.000
Let us see. So, let us go ahead and talk about our four fundamental vector spaces associated with a matrix a.
00:02:23.000 --> 00:02:53.000
So, if a is an m by m matrix, then, there are 4 spaces associated with a.
00:02:53.000 --> 00:02:58.000
You actually know of all of these spaces. We have talked about them individually, now we are going to bring them together.
00:02:58.000 --> 00:03:09.000
One is the null space of a, which is, if you remember, is the solution space for the equation ax = 0.
00:03:09.000 --> 00:03:20.000
Let me put that in parentheses here. It is the solution space, the set of all vectors x, such that the matrix a × x = 0. Just a homogeneous system.
00:03:20.000 --> 00:03:41.000
Two, we have something called the row space of a. Well, if you remember, the row space of a if I take the rows of a, just some random m by m matrix, they actually form a series of vectors... m vectors in RN.
00:03:41.000 --> 00:03:49.000
The space that is spanned by those vectors, that is the row space.
00:03:49.000 --> 00:04:06.000
Then we have the null space of a transpose. So, if I take a and just flip it along its main diagonal and then I solve this equation for this set of vectors, x such that a transpose × x = 0, I get its null space.
00:04:06.000 --> 00:04:11.000
It is also a subspace, and the row space is a subspace. All of these are subspaces.
00:04:11.000 --> 00:04:17.000
Oops -- it would be nice if I could actually count properly... 1, 2, 3.
00:04:17.000 --> 00:04:24.000
Now, our fourth space is going to be, well, you can imagine... it is going to be the column space.
00:04:24.000 --> 00:04:38.000
Again, the column space if I take the individual columns of the matrix, they are vectors in RM, and they form a space... the span of those vectors form a space.
00:04:38.000 --> 00:04:50.000
Now, they do not all have to be linearly independent. I can take those vectors, remember from an old discussion and I can find a basis... so the basis might be fewer vectors but they still span the same space.
00:04:50.000 --> 00:04:59.000
Okay. So, let us start with a theorem here, which is an incredibly beautiful theorem.
00:04:59.000 --> 00:05:06.000
As you can figure it out, linear algebra is full of unbelievably beautiful theorems... beautiful and very, very practical.
00:05:06.000 --> 00:05:37.000
If a is m by n, then, this is kind of extraordinary, the null space of a is the orthogonal complement of the row space of a.
00:05:37.000 --> 00:05:44.000
That is kind of amazing. Think about what that means for a second.
00:05:44.000 --> 00:05:59.000
If I just have this rectangular array of numbers, 5 by 6 and I just throw some numbers in there... when I solve the equation ax = 0, the homogeneous system associated with that matrix, I am going to get a subspace, the null space.
00:05:59.000 --> 00:06:10.000
As it turns out, if I take that matrix and I turn it into reduced row echelon, the non-zero rows form a basis for the column space. That is how we found the basis for the column space.
00:06:10.000 --> 00:06:20.000
Those two subspaces, they are orthogonal to each other. That is extraordinary. There is no reason to believe why that should be the case, and yet there it is.
00:06:20.000 --> 00:06:39.000
B, the complement of that, the null space of a transpose is the orthogonal complement to the column space.
00:06:39.000 --> 00:06:47.000
It is the orthogonal complement of the column space of a.
00:06:47.000 --> 00:06:56.000
So, that is the relationship. The null space of a given matrix a, its null space and its column space are orthogonal complements.
00:06:56.000 --> 00:07:13.000
If I take the transpose of a, the null space of the transpose and the column space of the original a, which ends up being the row space of a transpose because I have transposed it... those two are orthogonal complements.
00:07:13.000 --> 00:07:21.000
Let us do an example and see if this... if we can make sense of some of it just by seeing some numbers here.
00:07:21.000 --> 00:07:30.000
So, let us go... let us let a equal, it is going to be a big matrix here, and again we do not worry about big matrices because we have our math software.
00:07:30.000 --> 00:07:37.000
1, -2, 1, 0, 2... now I am not going to go through all of the steps.
00:07:37.000 --> 00:07:43.000
You know, this reduced row echelon, solving homogeneous systems, all of this stuff, I am going to give you the final results.
00:07:43.000 --> 00:07:53.000
At this point, I would like to think that you are reasonably comfortable either with the computational procedure manually, or you are using mathematical software yourself. I just do this in math software, myself.
00:07:53.000 --> 00:08:08.000
1, -1, 4, 1, 3, -1, 3, 2, 1, -1, 2, -3, 5, 1, 5... Okay.
00:08:08.000 --> 00:08:36.000
So, this is our matrix a. Our task is to find the four fundamental spaces associated with this matrix and confirm the theorem.
00:08:36.000 --> 00:08:47.000
So, this random rectangular array of numbers, something really, really amazing emerges from this. There are spaces that are deeply, deeply interconnected.
00:08:47.000 --> 00:08:56.000
So, let us see what happens here. Okay. When we take a, so the first thing we want to do is we want to find the row space of a.
00:08:56.000 --> 00:09:00.000
Let us go ahead and do that. So, row space of a.
00:09:00.000 --> 00:09:20.000
When I take a, and I reduce it to reduced row echelon form, I get the following matrix, 1, 0, 7, 2, 4, 0, 1, 3, 1, 1... and I get 0's in the other 2 rows.
00:09:20.000 --> 00:09:37.000
Well, what that means, basically what that tells me is that my row space, if I take these -- let me go red -- if I take that vector and that vector, they form a basis for the row space.
00:09:37.000 --> 00:10:00.000
So, my row space... basis for the row space and I am going to write these as column vectors, is 1, 0, 7, 2, 4... and 0, 1, 3, 1, 1...
00:10:00.000 --> 00:10:10.000
So, the dimension is 2. There you go. Also, I have my row space is 2-dimensional. Okay.
00:10:10.000 --> 00:10:20.000
Now, I need to find the null space. Well, I can almost guess this. The theorem says that the row space and the null space are going to be orthogonal complements.
00:10:20.000 --> 00:10:30.000
Well, I know that the orthogonal complement, or some subspace + the direct some of some subspace plus its orthogonal complement gives me the actual space itself.
00:10:30.000 --> 00:10:37.000
In this case, I am talking about 1, 2, 3, 4, 5... I am talking about R5.
00:10:37.000 --> 00:10:48.000
Well, if I already have a dimension 2, I know that the dimension of my orthogonal complement is going to be 3 and so I am hoping that when I actually do the calculation I end up with 3 vectors.
00:10:48.000 --> 00:11:02.000
Let us see what happens. The null space, well the null space is the set of all vectors x such that a(x) = 0.
00:11:02.000 --> 00:11:09.000
I solve a homogeneous system and I get my basis... I am not going to actually show this one.
00:11:09.000 --> 00:11:26.000
So, my basis for null of a... I tend to symbolize it like that... is equal... set notation... I have the vector (-7, -3, 1, 0, 0).
00:11:26.000 --> 00:11:43.000
I end up with (-2, -1, 0, 1, 0) and the presumption here is that you are comfortable doing this, the reduced row echelon, solving the homogeneous system, putting it into a form that you can actually read off your vectors.
00:11:43.000 --> 00:11:51.000
(-4, -1, 0, 0, 1). Well, there you go. We have a dimension equals 3.
00:11:51.000 --> 00:12:01.000
So, the dimension 2 + the dimension 3 = 5. The row space was a series of vectors in R5, so our dimensions match.
00:12:01.000 --> 00:12:11.000
Now the question is, I need to basically check that this is the case... I need to check that each of these vectors is orthogonal to the 2 vectors that I found.
00:12:11.000 --> 00:12:19.000
As it turns out, they are orthogonal. When you actually take the dot product of each of these with each of the other ones, you are going to end up with 0.
00:12:19.000 --> 00:12:28.000
So, this confirms our theorem. The first part of the theorem. The row space of a and the null space of a.
00:12:28.000 --> 00:12:40.000
Okay. So, now let us take our column space. So, we are going to take a transpose.
00:12:40.000 --> 00:12:56.000
Now, let me actually write out a transpose, I would like you to see it. It is (1,1,-1,2)... this is going to be R4, okay? -2.
00:12:56.000 --> 00:13:03.000
Now, the column space of a, what I have done here is I have actually transpose a. I have turns the rows in columns and the columns into rows.
00:13:03.000 --> 00:13:10.000
So, now the columns of a are written as rows. That is why I am doing it this way. Okay?
00:13:10.000 --> 00:13:34.000
-2,-1,3,-3,1,4,2,5,0,1,1,1... and I have 2,3,-1, and 5. Okay.
00:13:34.000 --> 00:13:43.000
So, I have 5 vectors in R4. So, here we are talking about R4.
00:13:43.000 --> 00:13:53.000
Alright. Now, when I subject this to reduced row echelon form, I am going to end up with some non-zero columns.
00:13:53.000 --> 00:13:58.000
That is going to be a basis for my column space.
00:13:58.000 --> 00:14:10.000
I get 1, 0, -2, 1, 0, 1, 1, 1, and 0's everywhere else, 0, 0, 0, 0... 0, 0, 0, 0.
00:14:10.000 --> 00:14:16.000
These first 2 actually form a basis for my column space.
00:14:16.000 --> 00:14:29.000
So, let me write that down... basis for my column space equals the set of vector 1, 0, -2, 1, 1, 0, -2, 1... I think it is always best to write them in vertical form, and 0, 1, 1, 1.
00:14:29.000 --> 00:14:50.000
Not good -- we would like them to be clear... 1, 1, 1... there you go. That forms a basis for our column space.
00:14:50.000 --> 00:14:57.000
Well, the dimension is 2. You know your spaces are 4... 4 - 2 is 2.
00:14:57.000 --> 00:15:05.000
We are going to expect that are homogeneous system, our null space of a transpose is going to be 2-dimensional. We should have 2 vectors.
00:15:05.000 --> 00:15:38.000
Well, let us confirm. As it turns out, when we solve a transpose × x = the 0 vector... basis for null -- love this stuff, it is great -- a transpose equals... again, I am just going to give the final answer... 2, -1, 1, 0.
00:15:38.000 --> 00:15:45.000
That is one vector, and the second vector is... these are basis vectors for our subspace... 0, 1.
00:15:45.000 --> 00:15:56.000
Sure enough, we end up with a dimension 2. So, our dimensions match. Now we just need to check that any vector in here × any vector in what we just got ,the column space are orthogonal.
00:15:56.000 --> 00:16:01.000
It turns out that they are. If you do the dot product of those, you are going to end up with 0.
00:16:01.000 --> 00:16:22.000
So, sure enough, once again, row space a, okay? is going to be the orthogonal complement of the null space of a.
00:16:22.000 --> 00:16:35.000
The column space of a is the orthocomplement of null space of a transpose.
00:16:35.000 --> 00:16:41.000
Simply by virtue of rectangular array of numbers, you have this relationship where these spaces are deeply interconnected.
00:16:41.000 --> 00:16:51.000
It is really rather extraordinary... and they add up to the actual dimension of the space. Okay.
00:16:51.000 --> 00:16:59.000
So, let us talk about projections and some applications. So, projections are very, very, very important.
00:16:59.000 --> 00:17:22.000
Recall, if you will that w is a subspace, I am just going to write ss for subspace of RN, then, w direct sum plus w perp is equal to RN.
00:17:22.000 --> 00:17:28.000
We sort of have been hammering that point. That is some subspace and some orthogonal complement, the dimensions add up to n.
00:17:28.000 --> 00:17:33.000
When you add them you actually get this space, RN. Okay.
00:17:33.000 --> 00:17:39.000
That was one of the proofs that we discussed... one of the theorems that we had in the last lesson.
00:17:39.000 --> 00:17:50.000
Now, we did not go through a proof of that, and I certainly urge you, with each of these theorems, to at least look at the proofs, because a lot of the proofs are constructive in nature and they will give you a clue as to why things are the way that they are.
00:17:50.000 --> 00:18:24.000
So, in the proof of the theorem, it is shown that if w, if that subspace has an orthonormal basis, remember an orthonormal basis is well, all of the vector are of length 1, and they are all mutually orthogonal.
00:18:24.000 --> 00:18:36.000
We had that Gram Schmidt orthonormalization process where we take a basis and we can actually turn it into an orthonormal basis by first making it orthogonal and then dividing by the norms of each of those vectors.
00:18:36.000 --> 00:18:52.000
So, if it has an orthonormal basis, let us say w1, w2, all the way to w < font size="-6" > k < /font > , we do not know how many dimensions it is.
00:18:52.000 --> 00:19:31.000
v is any vector in RN, then there exists a unique... there exists unique vectors w from the subspace w and u from the subspace w perp, such that the vector v can be written as w + u.
00:19:31.000 --> 00:19:45.000
Well, we know this already. Essentially what we are saying is that if we take any vector in RN, I can represent it uniquely as some vector from the subspace w + some vector in its orthogonal complement.
00:19:45.000 --> 00:19:56.000
Okay. Here is the interesting part. Also... we will write it as an also... this particular w, let me actually circle it in blue, there is a way to find it.
00:19:56.000 --> 00:20:30.000
Here is how we find it... w = the vector v ⋅ w1 × w1 + the vector v ⋅ w2 × w2 + the vector v ⋅ wk, as many vectors as there are in the basis, × wk.
00:20:30.000 --> 00:20:50.000
This is called the projection. This is called the projection of v onto the vector space w.
00:20:50.000 --> 00:21:02.000
It is symbolized as proj... as a subscript we write the w... that is the subspace w... and this is the vector v. Okay.
00:21:02.000 --> 00:21:10.000
We definitely need to investigate what it is that this looks like. When we do a projection -- let me draw this out so that you see what this looks like.
00:21:10.000 --> 00:21:27.000
So, we are going to be working in R3. So, let me draw a plane here.
00:21:27.000 --> 00:21:40.000
Let me draw a vector... this is going to be our w vector, then this is going to be our u vector.
00:21:40.000 --> 00:21:55.000
Let me make v... let me make it blue. So, v, once again, let us remind ourselves... v is any vector in this particular case, it will just be vector RN.
00:21:55.000 --> 00:22:03.000
You know what, since we are dealing in 3, let me be specific. R3.
00:22:03.000 --> 00:22:27.000
w is our vector in the subspace, which is a 2-dimensional subspace in 2, so this plane here... that is the subspace w, and the u is in the subspace w perp.
00:22:27.000 --> 00:22:48.000
So, here is what we are doing. Well, we said that this particular... so v can be written as something from w + something from u, because we know that RN is equal to the direct sum of w and w perp.
00:22:48.000 --> 00:22:54.000
So some vector from w, some vector from here, so that is a vector in w, that is a vector in w perp.
00:22:54.000 --> 00:23:00.000
Well, when we add them together, we get v. This is just standard vector addition.
00:23:00.000 --> 00:23:17.000
Here is what is really interesting. If we have a basis for this subspace, if we actually project v, project means shine a light on v so that you have a shadow, a shadow of v on this subspace... that is what the projection means.
00:23:17.000 --> 00:23:32.000
That is where you get that v ⋅ w1 × w1 + v ⋅ w2... when you do that, what we just wrote down for the projection, you actually end up finding w.
00:23:32.000 --> 00:23:45.000
Okay. Now, we had also written since v is equal to w + u... as it turns out if I wanted to find u, well, just move that over.
00:23:45.000 --> 00:23:57.000
Equals v - w. That is it. This is really, really great. So, let us do a problem and I think all of this will start to make sense.
00:23:57.000 --> 00:24:19.000
So, let us go... example here... we will let w be a subspace of R3 with an orthonormal basis... I often just write ortho for orthonormal basis.
00:24:19.000 --> 00:24:25.000
Again, orthonormal bases, they tend to have fractions in them because they are of length 1.
00:24:25.000 --> 00:24:33.000
We do not always want to use (0,0,1), we want something to be reasonably exciting.
00:24:33.000 --> 00:24:53.000
Let us go... oops -- these lines are making me crazy -- 2/sqrt(14), 3/sqrt(14), 1/sqrt(14)... one vector.
00:24:53.000 --> 00:25:01.000
The other vector is 1/sqrt(2),0,-1/sqrt(2)... so these are orthonormal.
00:25:01.000 --> 00:25:16.000
It is an orthonormal basis for the subspace R3, there is 2 vectors in it, so our subspace w has dimension 2, which means that our orthogonal complement w perp has dimension 1. 2 + 1 has to equal 3.
00:25:16.000 --> 00:25:29.000
Okay. We will also let v, the vector in R3 equal to some random 4, 2, 7... this is what we want to find.
00:25:29.000 --> 00:25:49.000
We want to find the projection of v onto w, and the vector... and we want to find -- this has to stop, why does this keep happening?
00:25:49.000 --> 00:26:12.000
So, we want to find the projection of v into this subspace, and we want to find the vector u that is orthogonal to every vector in w.
00:26:12.000 --> 00:26:24.000
In other words, we want to find w perp. Okay. So, how can we do that. Switch the page here...
00:26:24.000 --> 00:26:37.000
Well we know that from our formula, from our theorem, that our w is equal to the projection onto w of v, which is what we wanted.
00:26:37.000 --> 00:26:57.000
That is going to equal v ⋅ w1, one of the vectors in the basis × one of the vectors in the basis... plus v ⋅ w2, the second vector in the basis × that vector in the basis.
00:26:57.000 --> 00:27:06.000
Again, I am going to let you work these out. So take v ⋅ w1, you are going to get a scalar, multiply it by w1, it is going to give you a vector.
00:27:06.000 --> 00:27:14.000
You are going to add to that v ⋅ w2, which is a scalar × w2, which is a vector. When you add two vectors together you are going to get a vector.
00:27:14.000 --> 00:27:46.000
So, you will end up with something like this. 21/sqrt(14) × 2/sqrt(14), 3/sqrt(14), 1/sqrt(14), + -3/sqrt(2) ×, well 1/sqrt(2), 0, -1/sqrt(2). That is what this is.
00:27:46.000 --> 00:28:04.000
Then when you put those together, you are going to end up with 21/14, 63/14, 42/14, if I have done my arithmetic correctly.
00:28:04.000 --> 00:28:12.000
So, the projection of v onto the subspace w is this vector right here.
00:28:12.000 --> 00:28:21.000
That is what that means. If I take v and if I take the shadow of v on that subspace, I am going to get a vector.
00:28:21.000 --> 00:28:34.000
It is nice. Okay. Now, well we know that v is equal to w, this is w by the way. That is the projection.
00:28:34.000 --> 00:28:40.000
plus u, well we have v, we just found w, so now we want to find u.
00:28:40.000 --> 00:29:06.000
Well, u is just equal to v - w, when I take v - w, that is equal to, well, (4,2,7) - what I just found... 21/14, 63/14, 42/14.
00:29:06.000 --> 00:29:32.000
I am going to get 35/14, - 35/14, and 56/14. So, my vector v in R3 is a sum of that vector + that vector.
00:29:32.000 --> 00:29:47.000
It is pretty extraordinary, yeah? Again, this idea of a projection. All you are doing is you are taking a random vector and onto another space you are just shining the light. you are just taking the shadow.
00:29:47.000 --> 00:30:00.000
The shadow means you are taking the perpendicular... you are dropping a perpendicular from the end of that vector onto there, and this vector that you get, whatever it is, that is the projection.
00:30:00.000 --> 00:30:05.000
That is all the projection means. Perpendicular.
00:30:05.000 --> 00:30:15.000
As we know, the perpendicular from a point down to something else is the shortest distance from that object to that something else.
00:30:15.000 --> 00:30:25.000
So, let us draw the picture one more time, so that we are clear about what it is we are doing.
00:30:25.000 --> 00:30:41.000
We had w, we had u, I will put v here, that is v, this is -- oops, we wanted this in red.
00:30:41.000 --> 00:31:03.000
This is u, this is w, vector v is equal to w + u, u is equal to the vector v - the vector w. That is what this picture says.
00:31:03.000 --> 00:31:38.000
So, the distance from v, from the vector v to the subspace w... this is a capital W... is, well, the distance from v to the subspace w, the perpendicular distance, well it is equal to the norm of u.
00:31:38.000 --> 00:32:03.000
Well, the norm of u equals the norm of the projection of v onto the subspace w, which is equal to the norm of v - ... no, I am sorry, that is not correct, getting a little ahead of myself here.
00:32:03.000 --> 00:32:18.000
Vector u, the norm of u is the norm of this thing, which is v - the projection of v onto the subspace w.
00:32:18.000 --> 00:32:24.000
In 3-space, it makes sense because you are used to seeing 3-space and distance.
00:32:24.000 --> 00:32:40.000
Well, the distance from this point to this point is just the distance of the vector u, which you can calculate here. That is just the norm, and you know you found w by that formula that we just used which is the projection of v onto the subspace w.
00:32:40.000 --> 00:32:50.000
This is the subspace w, so we project it on here, we get w, here is what may not make sense. What if you are dealing with a 14 dimensional space?
00:32:50.000 --> 00:32:57.000
Let us say that your vector in R-14, you project it onto a subspace which is say 3 dimensional.
00:32:57.000 --> 00:33:02.000
How do you talk about a distance in that case? Well, again, distance is just an algebraic property.
00:33:02.000 --> 00:33:14.000
So, in some sense, you have this distance of a vector in a 14-dimensional space to its 3-dimensional subspace.
00:33:14.000 --> 00:33:27.000
There is a distance "defined," and that distance is precisely the projection of that 14-dimensional vector, of that vector in 14-dimensional space onto the 3-dimensional subspace.
00:33:27.000 --> 00:33:43.000
Again, this is the power of mathematics. We are not limited by reality. We are not limited by our senses. We are, in fact, not limited at all as long as the math supports it. As long as the algebra is correct. We are not limited by time or space.
00:33:43.000 --> 00:33:49.000
Okay. Thank you very much for joining us here at Educator.com to finish up our discussion of orthogonal complements. We will see you next time.