Docsity
Docsity

Prepare for your exams
Prepare for your exams

Study with the several resources on Docsity


Earn points to download
Earn points to download

Earn points by helping other students or get them with a premium plan


Guidelines and tips
Guidelines and tips

Advanced Linear Algebra: Transformations, Geometry, and Inverses, Lecture notes of Machine Learning

lecture slides for introduction to machine leraning

Typology: Lecture notes

2018/2019

Uploaded on 11/01/2019

chetan-reddy
chetan-reddy ๐Ÿ‡บ๐Ÿ‡ธ

9 documents

1 / 14

Toggle sidebar

This page cannot be seen from the preview

Don't miss anything!

bg1
EE 230 Advanced Linear Algebra
Linear Transformation
๐‘‡ ๐‘ข + ๐‘ฃ = ๐‘‡ ๐‘ข + ๐‘‡(๐‘ฃ)
๐‘‡๐‘๐‘ข =๐‘๐‘‡ ๐‘ข , โˆ€ c
๐‘‡๐‘Ž๐‘ฅ +๐‘๐‘ฆ = ๐‘Ž๐‘‡ ๐‘ฅ + ๐‘๐‘‡(๐‘ฆ), โˆ€ ๐‘Ž & ๐‘
Restricted to LINEAR: ๐‘‡ ๐‘ฃ = ๐‘ฃ = โˆ‘2๐‘ฃ2
3is NOT linear
๐‘‡ 0 = 0 cannot move the origin
Linear transformation is not restricted to vector space:
Matrix ๐‘‡can act on functions, polynomials, matrices, etc.
For matrix ๐‘‡such that ๐‘‡๐‘ฃ = ๐‘ข with ๐‘ฃ โˆˆ ๐‘น7and ๐‘ข โˆˆ ๐‘น8,
T8ร—7:<
Rn
โ†’
Rm
is<a<linear<transformation<if
pf3
pf4
pf5
pf8
pf9
pfa
pfd
pfe

Partial preview of the text

Download Advanced Linear Algebra: Transformations, Geometry, and Inverses and more Lecture notes Machine Learning in PDF only on Docsity!

Linear Transformation

๐‘‡ ๐‘๐‘ข = ๐‘๐‘‡ ๐‘ข , โˆ€ c

Restricted to LINEAR:

2

2

3 is NOT linear

๐‘‡ 0 = 0 cannot move the origin

Linear transformation is not restricted to vector space:

Matrix ๐‘‡ can act on functions, polynomials, matrices, etc.

For matrix ๐‘‡ such that ๐‘‡๐‘ฃ = ๐‘ข with ๐‘ฃ โˆˆ ๐‘น

7 and ๐‘ข โˆˆ ๐‘น

8 ,

T 8 ร— 7 : R

n โ†’ R

m is a linear transformation if

Geometrical Operations in 2D: R

2 ร  R

2

๐‘ƒQ =

๐‘…S =

cos ๐œƒ โˆ’ sin ๐œƒ

sin ๐œƒ cos ๐œƒ

๐‘…S๐‘…U =

cos(๐œƒ + ๐œ™) โˆ’ sin(๐œƒ + ๐œ™)

sin(๐œƒ + ๐œ™) cos(๐œƒ + ๐œ™)

= ๐‘…SWU

(Abelian) Lie Group

๐‘…S๐‘…XS = ๐ผ [๐‘…[,^ ๐‘…]^ =^0

2D rotation

๐‘…^ =

Inverse

รผ ๐‘ƒ

3 = ๐‘ƒ

รผ โˆ„ ๐‘ƒ

Xg

If โˆƒ ๐‘ƒ

Xg , then ๐‘ƒ

Xg ๐‘ƒ

3 = ๐‘ƒ

Xg ๐‘ƒ

Xg trivial!

โ˜ž Projection in 2D:

cos

3 ๐œƒ cos ๐œƒ sin ๐œƒ

cos ๐œƒ sin ๐œƒ sin

3 ๐œƒ

๐ถ ๐‘ƒ or ๐ถ(๐‘ƒ

k )

Null space ๐‘(๐‘ƒ)? Column space ๐ถ(๐‘ƒ)?

( Hint: ๐‘ƒ = ๐‘ƒ

k )

cos

3 ๐œƒ cos ๐œƒ sin ๐œƒ

rank = 1

AAACEXicbVDLTsJAFL3FF+ILdeHCzURC4oq0aKJLohuXmMgjoYRMhymdMJ02M7caQvgKP8GtfoA749YvcO2PWAoLAU9yk5Nz7r0z93ixFAZt+9vKra1vbG7ltws7u3v7B8XDo6aJEs14g0Uy0m2PGi6F4g0UKHk71pyGnuQtb3g79VuPXBsRqQccxbwb0oESvmAUU6lXPDGuFoMAqdbRE3GNUC4GHGmvWLIrdgaySpw5KdXykKHeK/64/YglIVfIJDWm49gxdsdUo2CSTwpuYnhM2ZAOeCeliobcdMfZARNSTpU+8SOdlkKSqX8nxjQ0ZhR6aWdIMTDL3lT8z+sk6F93x0LFCXLFZg/5iSQYkWkapC80ZyhHCwu9cFIo9wVlWqSfJyygmjJMQyykqTjLGaySZrXiXFSq95el2s0sHsjDKZzBOThwBTW4gzo0gMEEXuAV3qxn6936sD5nrTlrPnMMC7C+fgEUBZ3o^ s^!^ sin^ โœ“

AAACEXicbVDLTsJAFL3FF+ILdeHCzURC4oq0aKJLohuXmMgjoYRMhymdMO00M7caQvgKP8GtfoA749YvcO2PWAoLAU9yk5Nz7r0z93ixFAZt+9vKra1vbG7ltws7u3v7B8XDo6ZRiWa8wZRUuu1Rw6WIeAMFSt6ONaehJ3nLG95O/dYj10ao6AFHMe+GdBAJXzCKqdQrnjBXi0GAVGv1RFymjIsBR9orluyKnYGsEmdOSrU8ZKj3ij9uX7Ek5BEySY3pOHaM3THVKJjkk4KbGB5TNqQD3klpRENuuuPsgAkpp0qf+EqnFSHJ1L8TYxoaMwq9tDOkGJhlbyr+53US9K+7YxHFCfKIzR7yE0lQkWkapC80ZyhHCwu9cFIo9wVlWqSfJyygmjJMQyykqTjLGaySZrXiXFSq95el2s0sHsjDKZzBOThwBTW4gzo0gMEEXuAV3qxn6936sD5nrTlrPnMMC7C+fgHxJZ3T^ c^!^ cos^ โœ“

๐‘ ๐‘ƒ or ๐‘(๐‘ƒ

k )

โˆ’ tan ๐œƒ

โ˜ž (Mirror) reflection in 2D: Householder transformation

cos 2 ๐œƒ sin 2๐œƒ

sin 2 ๐œƒ โˆ’ cos 2 ๐œƒ

Parallelogram Mirror

Column 1

Column 2

AAAB/nicbVDLSgNBEOz1GddX1KOXwRDwFHZF0GPQi8cI5gHJEmZnZ5Mxsw9meoWwBPwEr/oB3sSrv+LZH3GyycEkFjQUVd0z3eWnUmh0nG9rbX1jc2u7tGPv7u0fHJaPjls6yRTjTZbIRHV8qrkUMW+iQMk7qeI08iVv+6Pbqd9+4kqLJH7Accq9iA5iEQpG0UitHg450n654tScAmSVuHNSqZegQKNf/ukFCcsiHiOTVOuu66To5VShYJJP7F6meUrZiA5419CYRlx7ebHthFSNEpAwUaZiJIX6dyKnkdbjyDedEcWhXvam4n9eN8Pw2stFnGbIYzb7KMwkwYRMTyeBUJyhHC886EcTuxoIypQwyxM2pIoyNInZJhV3OYNV0rqouYbfX1bqN7N4oASncAbn4MIV1OEOGtAEBo/wAq/wZj1b79aH9TlrXbPmMyewAOvrF+LTlis=AAAB/nicbVDLSgNBEOz1GddX1KOXwRDwFHZF0GPQi8cI5gHJEmZnZ5Mxsw9meoWwBPwEr/oB3sSrv+LZH3GyycEkFjQUVd0z3eWnUmh0nG9rbX1jc2u7tGPv7u0fHJaPjls6yRTjTZbIRHV8qrkUMW+iQMk7qeI08iVv+6Pbqd9+4kqLJH7Accq9iA5iEQpG0UitHg450n654tScAmSVuHNSqZegQKNf/ukFCcsiHiOTVOuu66To5VShYJJP7F6meUrZiA5419CYRlx7ebHthFSNEpAwUaZiJIX6dyKnkdbjyDedEcWhXvam4n9eN8Pw2stFnGbIYzb7KMwkwYRMTyeBUJyhHC886EcTuxoIypQwyxM2pIoyNInZJhV3OYNV0rqouYbfX1bqN7N4oASncAbn4MIV1OEOGtAEBo/wAq/wZj1b79aH9TlrXbPmMyewAOvrF+LTlis=AAAB/nicbVDLSgNBEOz1GddX1KOXwRDwFHZF0GPQi8cI5gHJEmZnZ5Mxsw9meoWwBPwEr/oB3sSrv+LZH3GyycEkFjQUVd0z3eWnUmh0nG9rbX1jc2u7tGPv7u0fHJaPjls6yRTjTZbIRHV8qrkUMW+iQMk7qeI08iVv+6Pbqd9+4kqLJH7Accq9iA5iEQpG0UitHg450n654tScAmSVuHNSqZegQKNf/ukFCcsiHiOTVOuu66To5VShYJJP7F6meUrZiA5419CYRlx7ebHthFSNEpAwUaZiJIX6dyKnkdbjyDedEcWhXvam4n9eN8Pw2stFnGbIYzb7KMwkwYRMTyeBUJyhHC886EcTuxoIypQwyxM2pIoyNInZJhV3OYNV0rqouYbfX1bqN7N4oASncAbn4MIV1OEOGtAEBo/wAq/wZj1b79aH9TlrXbPmMyewAOvrF+LTlis=AAAB/nicbVDLSgMxFM34rOOr6tJNsBRclRkRdFl047KCfUA7lEwm08YmmSG5I5Sh4Ce41Q9wJ279Fdf+iGk7C9t64MLhnHuTe0+YCm7A876dtfWNza3t0o67u7d/cFg+Om6ZJNOUNWkiEt0JiWGCK9YEDoJ1Us2IDAVrh6Pbqd9+YtrwRD3AOGWBJAPFY04JWKnVgyED0i9XvJo3A14lfkEqqECjX/7pRQnNJFNABTGm63spBDnRwKlgE7eXGZYSOiID1rVUEclMkM+2neCqVSIcJ9qWAjxT/07kRBozlqHtlASGZtmbiv953Qzi6yDnKs2AKTr/KM4EhgRPT8cR14yCGC88GMqJW404oZrb5TEdEk0o2MRcm4q/nMEqaV3UfMvvLyv1myKfEjpFZ+gc+egK1dEdaqAmougRvaBX9OY8O+/Oh/M5b11zipkTtADn6xd8M5Xj^ โœ“

AAAB/nicbVDLSgMxFL3js46vqks3wVJwY5mpgm6Eopu6q2Af0A4lk8m0sUlmSDJCKQU/wa1+gDtx66+49kecTruwrQcuHM65N7n3+DFn2jjOt7Wyura+sZnbsrd3dvf28weHDR0litA6iXikWj7WlDNJ64YZTluxolj4nDb9we3Ebz5RpVkkH8wwpp7APclCRrBJpUb1ulw7u+vmC07JyYCWiTsjhUoOMtS6+Z9OEJFEUGkIx1q3XSc23ggrwwinY7uTaBpjMsA92k6pxIJqb5RtO0bFVAlQGKm0pEGZ+ndihIXWQ+GnnQKbvl70JuJ/Xjsx4ZU3YjJODJVk+lGYcGQiNDkdBUxRYvhw7kFfjO1iwDBRLF0ekT5WmJg0MTtNxV3MYJk0yiX3vFS+vyhUbqbxQA6O4QROwYVLqEAValAHAo/wAq/wZj1b79aH9TltXbFmM0cwB+vrF14klTo=^ H^ = 2P^ ^ I

Reverse problem: Constructing the matrix of linear transformation

go

2o

8o

โ‹ฏ 8 ร— 7

T 8 ร— 7 : R

n โ†’ R

m

๐‘— = 1 , 2 , โ‹ฏn

Example 1:

Consider a linear operator L acting on R

2 as

Find the matrix of L with respect to the basis:

So we obtain, L =

T T

g

3

โˆ’๐ฏ 3

๐ฏg

๐ฟ ๐ฏg = 2 ๐ฏg + (โˆ’ 1 )๐ฏ 3 ๐ฟ ๐ฏ 3 = 1 ๐ฏg + 0 ๐ฏ 3

Representation: state ร  vector; operation ร  matrix (quantum mechanics)

  • Construct a (generalized) vector space
  • Decompose a vector in terms of a basis: ๐‘ฅ = ๐‘ 2 ๐‘ฅ 2
  • Act T on each base vector: ๐‘‡๐‘ฅ = ๐‘‡(๐‘ 2 ๐‘ฅ 2 ) = ๐‘ 2 ๐‘‡(๐‘ฅ 2 )

Differentiation operator ๐ด = ๐‘‘/๐‘‘๐‘ก

Basis ๐‘’g = 1 , ๐‘’ 3 = ๐‘ก, ๐‘’f = ๐‘ก

3 , โ‹ฏ โ‹ฏ ๐‘’ 7 = ๐‘ก

7 Xg , ๐‘’ 7 Wg = ๐‘ก

7

Then, ๐ด๐‘’ g

3

g

f

3

7

7 X 3 = ๐‘› โˆ’ 1 ๐‘’ 7 Xg

7 Wg

7 Xg = ๐‘›๐‘’ 7

Reverse problem: Constructing the matrix of linear transformation

Differentiation operator ๐ด = ๐‘‘/๐‘‘๐‘ก on polynomial basis

๐ด 7 ร—( 7 Wg): Pn+1โ†’ Pn

Null space: ๐ด๐‘Ž{ = ๐‘‘๐‘Ž{/๐‘‘๐‘ก = 0 ,

Example : For ,

Taking derivative on an arbitrary constant = 0

k

One-sided Inverses:

  • ๐‘Ÿ = ๐‘š = ๐‘› Full rank square matrix, ๐‘€๐‘€

Xg = ๐‘€

Xg ๐‘€ = ๐ผ (two-sided)

  • ๐‘Ÿ = ๐‘› < ๐‘š Full column rank ( e.g. ๐ต = โˆซ ๐‘‘๐‘ก )

๐‘ ๐ต = {zero}, no independent columns, no free variable

k ๐ต)๐‘ฅ = 0 โŸน ๐‘(๐ต) = ๐‘(๐ต

k ๐ต) = {zero}

๐‘ป ๐‘ฉ is invertible! [ Note: ๐‘ฉ๐‘ฉ

๐‘ป is square, but NOT invertible!]

๐‘ป ๐‘ฉ

X๐Ÿ ๐‘ฉ

๐‘ป ๐‘ฉ = ๐‘ฐ ๐‘ฉ ๐ฅ๐ž๐Ÿ๐ญ

X๐Ÿ (Dimension: ๐‘ฉ ๐ฅ๐ž๐Ÿ๐ญ

X๐Ÿ

๐’ร—๐’Ž

๐‘ฉ๐’Žร—๐’ = ๐‘ฐ๐’ร—๐’ )

  • ๐‘Ÿ = ๐‘š < ๐‘› Full row rank ( e.g. ๐ด = ๐‘‘/๐‘‘๐‘ก )

k = {zero}, no independent rows, but (๐‘› โˆ’ ๐‘š) free variables

๐‘ป is invertible!

Define ๐‘จ๐‘จ

๐‘ป ๐‘จ๐‘จ

๐‘ป

X๐Ÿ = ๐‘ฐ so that ๐‘จ๐’Žร—๐’ ๐‘จ ๐ซ๐ข๐ ๐ก๐ญ

X๐Ÿ

๐’ร—๐’Ž

= ๐‘ฐ๐’Žร—๐’Ž

  • One-sided inverse is NOT unique

Example : Integration and Differentiation

รผ We can change either the free column or the free row

รผ Moore-Penrose inverse (pseudo-inverse) ๐ด

W โˆˆ ๐‘€ 7 ร— 8

๐ด๐ต = ๐ผ, but ๐ต๐ด โ‰  ๐ผ

Free column

Free row

H

H

The Moore-Penrose inverse is UNIQUE!

  • Generally defined
  • For full column (row) rank matrices, it reduces

to the particular left (right) inverse above