Three points require mentioning before beginning this whitepaper. First, I used Python and R code throughout this whitepaper and often imported objects created in Python into R for creating plots. To use Python and R coinjointly, I used the reticulate package made for R and created a conda environment to run Python code (see lines 2–12 below).

library(reticulate)

#create and use conda environment
conda_create(envname = 'blog_posts',  python_version = '3.10.11')
use_condaenv(condaenv = 'blog_posts')

#install packages in conda environment
py_packages <- c('numpy', 'pandas', 'scikit-learn', "plotnine", "statsmodels", "manim", "factor_analyzer")
conda_install(envname = 'blog_posts', packages = py_packages, pip=T)

#useful for checking what packages are loaded
py_list_packages(envname = 'blog_posts', type = 'conda')

Second, the Python packages and modules in the Python code block are needed to run all the Python code in this paper (see lines 15–22 below).

import math 
import numpy as np
import pandas as pd
import plotnine as pt

from sklearn import decomposition
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from factor_analyzer.rotator import Rotator
from factor_analyzer import FactorAnalyzer

Third, although I often include code in papers so that readers can explore concepts, I decided to not include the Python code I used to create mathematical animations given the considerable length of the script. For readers interested in how I created my animations, I used the manim library and my source code can be viewed in this script.

Introduction

In this whitepaper, I provide a deep dive into the math underlying the singular value decomposition and use this as a foundation to explain the connection with principal component analysis. To this end, I provide a detailed proof of the singular value decomposition and explain the linear algebra necessary for understanding its geometry. I then explain the meaning of the singular value decomposition by synthesizing the parts needed to run a principal component analysis. To address all these points, the following sections will follow sequentially:

1. A review of linear algebra
2. The singular value decomposition and its visualization
3. Proving the singular value decomposition
4. Understanding the singular value decomposition by synthesizing principal component analysis

A Review of Linear Algebra

Some Important Fundamentals

In this section, I provide some fundamentals of linear algebra that are necessary for understanding the geometry of the singular value decomposition. To this end, I explain the following four points:

1. Matrices define basis vectors
2. Matrix-vector multiplication transforms bases by computing weighted vector sums
3. Dot products are 1-dimensional cases of matrix-vector multiplication
4. Dot products provide orthogonal projection lengths with normalized vectors

Matrices Define Basis Vectors

Basis vectors can be understood as the coordinates used to define some n-dimension space, ℝ𝑛. Beginning with the familiar standard basis vectors—that is, vectors with only one non-zero element equivalent to one—are often used to define n-dimensional spaces. As an example, consider a two-dimensional space, ℝ2. To span ℝ2 (i.e., define any vector in ℝ2), the standard basis vectors of 𝐛𝑒𝑥=[1,0] and 𝐛𝑒𝑦=[0,1] can be used, and these vectors can be combined to form the basis matrix shown below in Equation 2.1:

𝐁𝑒=[1001].(2.1) Note that each column of 𝐁𝑒 represents a basis vector.

If 𝐛𝑒𝑥 is multiplied by one and 𝐛𝑒𝑦 is multiplied by two, the vector 𝐠𝑒=[1,2] is obtained, and this is shown below in Animation 1.

Importantly, outside from the standard basis vectors being familiar and simple, there isn’t always a strong justification for using them in applied settings. Thus, non-standard basis vectors are often used in applied settings. As an example of non-standard basis vectors, consider defining ℝ2 with basis vectors 𝐛𝑛𝑥=[0,1] and 𝐛𝑛𝑦=[1,2], which are defined as a matrix below in Equation 2.2:

𝐁𝑛=[0112].(2.2)

To define 𝐠𝑒 in the non-standard basis of Equation 2.2, the vector coordinates of 𝐠𝑛=[1.5,0.5] are used, and this is shown below in Animation 2. (In later sections, I will show how to obtain coordinates in non-standard bases.)

Matrix-Vector Multiplication Transforms Bases by Computing Weighted Vector Sums

With matrices defining basis vectors, this also allows them to apply transformations. To understand how matrices apply transformations, consider again the non-standard basis of

𝐁𝑛=[0112].(2.2 revisited)

In post-multiplying 𝐁𝑛 by 𝐠𝑛=[1.5,0.5], the first basis vector, 𝐛𝑛𝑥=[0,1], is multiplied by 1.5 and the second basis vector, 𝐛𝑛𝑦=[1,2], is multiplied by 0.5. Each weighted composite is then summed to produce 𝐠𝑒.

𝐁𝑒⁢𝐯=[0112]⁢[1.50.5]=1.5⁢[01]+0.5⁢[12]=[21].(2.3)(2.4)

Animation 3 below shows how matrix-vector multiplication of Equation 2.3 transforms bases by computing weighted vector sums. The summing of weighted basis vectors is then shown to be equivalent to applying a shearing transformation (as specified by 𝐁𝑛) to the entire matrix space.

Dot Products are 1-Dimensional Cases of Matrix-Vector Multiplication

Although it may not be obvious, dot products are simply 1-dimensional cases of matrix-vector multiplication. That is, dot products also transform bases by computing weighted vector sums. The only nuance is that the transformations applied by dot products occur in a 1-dimensional space (i.e., a line). To understand the geometry of dot products, consider two vectors of 𝐚=[1,2] and 𝐛=[3,1]. In computing the dot produict between these vectors, 𝐚⊤⁢𝐛, basis vectors still exist, but they are now 1-dimensional. More specifically, the first basis vector of 𝐛, 𝑏𝑥=3, is multiplied by 1 and the second basis vector, 𝑏𝑦=1, is multiplied by 2. I have provided Animation 4 to clearly explain the geometry of dot products.

𝐚𝐛=[12]⁢[3  1]=1⁢(3)+2⁢(1)=5(2.5)

Dot Products Provide Orthgonal Projection Lengths With Normalized Vectors

One focal process of principal component analysis is the computation of orthogonal projections: The value of one variable on another basis. In the context of principal component analysis, analysts want to know how scores on one variable translate into scores on another variable that exists in a lower dimension, As it turns out, dot products accomplish this task because the vectors being projected onto are normalized. That is, when computing orthogonal projections of a vector, 𝐱, onto a normalized vector 𝐯, 𝑃⁢𝑟⁢𝑜⁢𝑗𝐯⁡(𝐱), the equation is simply that of the dot product:

𝑃⁢𝑟⁢𝑜⁢𝑗𝐯⁡(𝐱)=𝐱𝐯(2.6)

To understand how Equation 2.6 is derived, an understanding of orthogonal projections is necessary. To this end, I provide a visualization below in Figure 1. To begin, the task shown in Figure 1 is to find the value of 𝐱 on 𝐯. For the purposes of comprehension, 𝐯 can be conceptualized as another basis that defines the span of 𝐿=𝑐⁢𝐯, and we want to know the orthogonal projection of 𝐱 on this basis. The solution to this problem becomes obvious once two facts become evident. First, the orthogonal projection will be some 𝑐 scalar multiple of 𝐯, 𝑃⁢𝑟⁢𝑜⁢𝑗𝐿⁡(𝐱)=𝑐⁢𝐯. Second, because the projection is orthogonal, then a vector must exist that has a null dot product with the projection. This is indeed the case: (𝐱−𝑃⁢𝑟⁢𝑜⁢𝑗𝐿⁡(𝐱)) is orthogonal with 𝐯. Given these two points, then Equation 2.7 below for the orthogonal projection can be obtained.

(𝐱−𝑃⁢𝑟⁢𝑜⁢𝑗𝐿⁡(𝐱))⁢𝐯=0(𝐱−𝑐⁢𝐯)⁢𝐯=0𝐱⊤⁢𝐯−𝑐⁢𝐯⊤⁢𝐯=0𝑐=𝐱⊤⁢𝐯𝐯⊤⁢𝐯(2.7) If the vector being projected onto is normalized, then it has length of 1 (i.e., 𝐯⊤⁢𝐯=1), and simply computing the dot product provides the orthogonal projection (i.e., Equation 2.6 is obtained).

Figure 1

Visualization of Vector Projection

Note. The process for deriving projections (Equation 2.7) is visualized. The depicted objective is to find the value of 𝐱 on 𝐯. For the purposes of comprehension, 𝐯 can be conceptualized as another basis that defines the span of 𝐿=𝑐⁢𝐯, and we want to know the orthogonal projection of 𝐱 on this basis. The solution to this problem becomes obvious once two facts become evident. First, the orthogonal projection will be some 𝑐 scalar multiple of 𝐯, 𝑃⁢𝑟⁢𝑜⁢𝑗𝐿⁡(𝐱)=𝑐⁢𝐱. Second, because the projection is orthogonal, then a vector must exist that has a null dot product with the projection. This is indeed the case: (𝐱−𝑃⁢𝑟⁢𝑜⁢𝑗𝐿⁡(𝐱)) is orthogonal with 𝐯. For readers interested in producing this visualization, the Python code is provided below in lines 23--125.

# Define vectors
x = np.array([3, 3])
v = np.array([1, 3])

# scalar multiplier for projection 
c = v.T.dot(x)/(x.dot(x))
v_proj = c*x

# Create DataFrame for vectors
df_vectors = pd.DataFrame({
    'x_start': [0, 0, v_proj[0], 0],  
    'y_start': [0, 0, v_proj[1], 0],
    'x_end': [v[0], x[0], v[0], 2],
    'y_end': [v[1], x[1], v[1], 2],
    'label': ['x', 'v', 'x_vproj','v_proj'], 
    'color': ["#d170c7","#d4bd04", "#33c304","#9DFFFF"]})


# Define line span of x (t * x for t in [-2, 2])
t_values = np.linspace(-5, 5, 100)
x_span = t_values * x[0]
y_span = t_values * x[1]

df_line = pd.DataFrame({'x': x_span, 'y': y_span})

# Create a dataframe for tick labels (for easier annotation)
ticks = range(-5, 6)  # From -5 to 5

df_xticks = pd.DataFrame({
    "x": [i for i in ticks if i != 0],  # Exclude 0
    "y": [0] * (len(ticks) - 1),  # Same length after filtering
    "label": [str(i) for i in ticks if i != 0]  # Exclude 0 from labels
})

df_yticks = df_xticks.copy()
df_yticks['x'] = df_xticks['y']
df_yticks['y'] = df_xticks['x']

# dataframe for annotations 
df_annotations = pd.DataFrame({
    "x": [-4, -4, -4, 0.75, 3.25, 1.5, 2, -4, 3.75, 3, 3, 3, 3], 
    "y": [5, 4, 3, 3.25, 2.75, 0.5, 3.5, 2, 4.25, -2, -3, -4, -5],  
    "label": [r"$\mathbf{x}^\top = [ 1 \ 3 ]$",  
              r"$\mathbf{v}^\top = [ 3 \ 3 ]$",
              r"$Proj_L(\mathbf{x}) = c\mathbf{v}$", 
              r"$\mathbf{x}$",
              r"$\mathbf{v}$",
              r"$Proj_L(\mathbf{x})$",
              r"$\mathbf{x} - Proj_L(\mathbf{x})$",
              r"$L = \{c\mathbf{v} | c \in \mathbb{R}\}$", 
              r"$L$", 
              r"$(\mathbf{x} - Proj_L(\mathbf{x}))\mathbf{v} = 0 $",
              r"$(\mathbf{x} - c\mathbf{v})\mathbf{v} = 0$",
              r"$\mathbf{x}^\top \mathbf{v} - c\mathbf{v}^\top \mathbf{v} = 0$",
              r"$c = \frac{\mathbf{x}^\top \mathbf{v}}{\mathbf{v}^\top \mathbf{v}}$"],
    "color": ["#d170c7", "#d4bd04", "#9DFFFF", 
              "#d170c7", "#d4bd04", "#9DFFFF", "#33c304", "white", "white", 
              "white", "white" , "white", "white"],  
    "size": [14]*13})


# Create plot
plot_proj = (pt.ggplot() +
    
    # Specify vectors 
    pt.geom_line(df_line, pt.aes(x='x', y='y'), alpha=0.5, color="white") +  
    pt.geom_segment(data=df_vectors,
                    mapping=pt.aes(x='x_start', y='y_start', xend='x_end', yend='y_end'),
                    arrow=pt.arrow(type="closed", length=0.12, angle=30), 
                    color=df_vectors['color']) +

    # Add vector labels 
    pt.geom_text(data=df_annotations,
                 mapping=pt.aes(x="x", y="y", label="label"),
                 color=df_annotations['color'], size=df_annotations['size']) +

    pt.geom_hline(yintercept=0, color="white") +  
    pt.geom_vline(xintercept=0, color="white") +  
    
    # Remove default tick labels and minor grid lines
    pt.scale_x_continuous(limits=(-5, 5), breaks=range(-5, 6), minor_breaks=[]) +    
    pt.scale_y_continuous(limits=(-5, 5), breaks=range(-5, 6), minor_breaks=[]) +   

    # Manually place x-axis tick labels ON the hline
    pt.geom_text(df_xticks, pt.aes(x='x', y='y', label='label'), 
                 color="white", size=10, va="top", nudge_y=-0.1) +  

    # Manually place y-axis tick labels ON the vline
    pt.geom_text(df_yticks, pt.aes(x='x', y='y', label='label'), 
                 color="white", size=10, ha="right", nudge_x=-0.1, nudge_y=0) +

    pt.theme_minimal(base_family='Helvetica', base_size=14) +    
    pt.theme(
        text=pt.element_text(color="#002241"),
        plot_background=pt.element_rect(fill="#002241"),  
        panel_background=pt.element_rect(fill="#002241"),  
        panel_grid_major=pt.element_line(color="#9DFFFF", size=0.3, alpha=0.5),  
        axis_title=pt.element_blank(),  # Remove axis titles
        axis_ticks=pt.element_blank(),  # Remove tick marks
        axis_text=pt.element_blank(),  # Remove default tick labels
        legend_position="none"))
        
plot_proj.save("images/proj_plot.png", dpi=500, width=8, height=6)

Visualizations of Elementary Matrices

As explained in the previous section, matrix multiplication transforms bases by computing weighted vector sums. As it turns out, there are three fundamental transformations that constitute every possible matrix transformation: 1) Rotation, 2) stretching, and 3) changing dimension. Importantly, each fundamental transformation corresponds to a particular type of matrix, and I will provide visualizations for each of these matrices. Below is a brief summary of the matrices and their transformations (note, there is technically a fourth type of transformation that un-transforms):

1. Diagonal matrices stretch basis vectors.
2. Orthonormal matrices rotate basis vector.
3. Inverse matrices un-transform basis vectors.
4. Rectangular matrices change the dimension space.

I also provide a geometric overview of eigenvectors and eigenvalues given their central importance to singular value decomposition and principal component analysis.

Diagonal Matrices Stretch Basis Vectors

Animation 5 below shows that diagonal matrices simply stretch the basis vectors that define a space. In the current example, the matrix of

𝐃=[2003](2.8) stretches the first basis vector, 𝐛𝑒𝑥=[1,0], by a factor of two such that it becomes 𝐛𝑑𝑥=[2,0] and the second basis vector, 𝐛𝑒𝑦=[0,1], by a factor of three such that it becomes 𝐛𝑑𝑦=[0,3]. As a result, the vector of 𝐠𝑒=[2,1] becomes 𝐠𝑒=[4,3].

Orthonormal Matrices Only Rotate Basis Vectors (Length and Angle Preserving)

Animation 6 below shows that orthonormal matrices simply rotate the basis vectors that define a space. In the current example, the matrix of

𝐐=[√22√22−√22√22](2.9) rotates the first basis vector, 𝐛𝑒𝑥=[1,0], clockwise to 𝐛𝑞𝑥=[√22,−√22], and the second basis vector, 𝐛𝑒𝑦=[0,1], to 𝐛𝑞𝑦=[√22,√22]. As a result, the vector of 𝐠𝑒=[2,1] becomes 𝐠𝑒=[3⁢√22,−√22].

Importantly, orthonormal matrices only rotate vector spaces. That is, there is no stretching/compressing of the vector space; vectors lengths remain unchanged and so do angles between vectors. In mathematical terms, orthonormal matrices preserve lengths and matrices and, given the ease with which each of these statements can be proven, I show each proof in turn.

Beginning with the length-preserving property of orthonormal matrices, consider the formula for computing vector lengths in Equation 2.10 below:

Length=𝑛∑𝑖=1𝑣2𝑖=<𝐯,𝐯>=‖𝐯‖22.(2.10) By applying Equation 2.10 above to the length of a vector that is transformed by some orthonormal matrix, 𝐐𝐯, it becomes clear that the vector’s length remains unchanged.

‖𝐐𝐯‖22=(𝐐𝐯)⊤⁢𝐐𝐯=𝐯⊤⁢𝐐⊤⁢𝐐⏟=𝐈⁢𝐯=𝐯⊤⁢𝐯◼(2.11)

As an example of the length-preserving property of orthonormal matrices, the Python code block below (lines 126–134) shows that length of the original vector, 𝐠𝑒=[2,1], and its orthonormal-transformed version, 𝐠𝑞=[3⁢√22,−√22], remains unchanged.

# original vector
g_e = np.array([2, 1]) 

# transformed vector
g_q = np.array([3*(np.sqrt(2))/2, -np.sqrt(2)/2]) 
                
# lengths of each vector
print(g_e.dot(g_e), "\n",
      np.round(g_q.dot(g_q), 6))

5 
 5.0

Ending with the angle-preserving feature of orthonormal matrices, consider the formula to computing the angles between two vectors, 𝐯 and 𝐰, shown below in Equation 2.12:

cos⁡(𝜃)=𝐯⊤⁢𝐰‖𝐯‖22⁢‖𝐰‖22(2.12)

By applying Equation 2.12 above to the angles between vectors following an orthonormal transformation, 𝐐𝐯 and 𝐐𝐰, it becomes clear that the angle between them remains unchanged (note how the proof shown in Equation 2.11 above is used in the below proof).

cos⁡(𝜃)=𝐐𝐯⊤⁢𝐐𝐰‖𝐐𝐯‖22⁢‖𝐐𝐰‖22=𝐯⊤⁢𝐐⊤⁢𝐐𝐰(𝐯⊤⁢𝐐⊤⁢𝐐𝐯)⁢(𝐰⊤⁢𝐐⊤⁢𝐐𝐰)Note: 𝐐⊤⁢𝐐=𝐈=𝐯⊤⁢𝐰‖𝐯‖22⁢‖𝐰‖22◼(2.13)

As an example of the angle-preserving property of orthonormal matrices, the Python code block below (lines 137–150) shows that the angle between the original standard basis vectors, 𝐛𝑒𝑥=[1,0] and 𝐛𝑒𝑦=[1,0], and their orthonormal-transformed versions, 𝐐𝐛𝑒𝑥 and 𝐐𝐛𝑒𝑦, remains unchanged.

# original standard basis 
b_ex = np.array([1, 0])
b_ey = np.array([0, 1])

# orthonormal basis 
Q = np.array([[np.sqrt(2)/2, np.sqrt(2)/2], 
              [-np.sqrt(2)/2, np.sqrt(2)/2]])
                
# angle between original basis vectors
original_angle = np.arccos(b_ex.dot(b_ey)/(np.linalg.norm(b_ex) * np.linalg.norm(b_ey)))
transformed_angle =  np.arccos(b_ex.T.dot(Q.T).dot(Q).dot(b_ey)/(np.linalg.norm(Q.dot(b_ex)) * np.linalg.norm(Q.dot(b_ey))))

print(math.degrees(original_angle), "\n",
      math.degrees(transformed_angle))

90.0 
 90.0

Inverse Matrices Un-Transform Basis Vectors

Animation 7 shows that matrix inverses transform vector spaces in the opposite direction and magnitude of their non-inverted counterparts. In this way, matrix inverses can be conceptualized as un-transforming vector spaces. Within Animation 7, the space is first transformed using the orthonormal matrix of

𝐐=[√22√22−√22√22].(2.9 revisited)

The vector space is then un-transformed and brought back to the standard basis by applying the inverse of 𝐐. By applying 𝐐−1, the basis vectors are rotated back to become the standard basis vectors, 𝐛𝑒𝑥 and 𝐛𝑒𝑦, and the transformed vector of 𝐠𝑛=[3⁢√22,−√22] returns back to its original standard basis coordinates of 𝐠𝑒=[2,1].

Rectangular Matrices Change Dimension Space

So far, I have only shown linear transformations that result from applying square matrices. When multiplying spaces by square matrices, the input and output dimension spaces are the same. When multiplying by rectangular matrices, however, the dimensions of the input and output spaces are different; that is, rectangular matrices change the dimension space. As an example, Animation 8 below shows the transformation that results from applying the rectangular matrix of

𝐁𝑟=[12].(2.14) The matrix 𝐁𝑟 takes an input space of two dimensions and returns values in an output space of one dimension. Thus, two-dimension vectors become scalars (i.e., one dimensional). In this case, the first basis vector, 𝐛𝑒𝑥=[1,0], is transformed to become a scalar value of b𝑟𝑥=1, and, likewise, the second basis vector, 𝐛𝑒𝑦=[0,1], becomes b𝑟𝑦=2. As a result, the vector of 𝐠𝑒=[2,1] becomes g𝑟=[4].

Importantly, because the input and output dimensions spaces are different, the input and output vector live in different dimension spaces, and so no input vector can lie in the span of any output vector (and vice-versa).

Eigenvectors (and Eigenvalues)

Due to the inherent connection between eigenvectors and singular value decomposition, it is necessary to provide an overview of the geometry of eigenvectors. That being said, although it is useful to understand the geometry of eigenvectors, their importance in singular value decomposition comes more from their meaning (which will be explained in later sections) and less so from their geometry. Nonetheless, I briefly present the geometry of eigenvectors and eigenvalues.

Animation 9 below shows that eigenvectors remain on their span following some linear transformation and eigenvalues represent the extent to which the eigenvectors are stretched. The geometry of eigenvectors and eigenvalues is summarized in Equation 2.15 below

𝐀𝐯=𝜆⁢𝐯.(2.15)

In words, if some eigenvector, 𝐯, is pre-multiplied by a (square) matrix, 𝐀, then the resulting vector is simply a scalar multiplication of 𝐯, where 𝜆 represents the scalar value. As an example from Animation 9 shown below, when the eigenvector 𝐯1=[√22,√22] remains on its span following a pre-multiplication

𝐀=[1102],(2.16) because the resulting vector, 𝜆⁢𝐯1 is simply twice the value of 𝐯1. In Animation 9, the light blue lines represent the eigenbases and each eigenvector remains on its basis following a linear transformation by 𝐀.

One last important point to note is that only square matrices can have eigenvectors because non-square (or rectangular matrices) change the dimension space (see section on rectangular matrices). Because rectangular matrices change the dimension space, it is impossible for any vector to remain on its span.

The Singular Value Decomposition and its Visualization

Having covered the necessary fundamentals, I will introduce the singular value decomposition and its geometry. In short, any matrix 𝐀∈ℝ𝑛×𝑝 (𝐀 can also be complex) can be decomposed such that

𝐀=𝐔⁢Σ⁢𝐕⊤,(3.1) where each of the three matrices in the decomposition has the following characteristics:

1. 𝐔∈ℝ(𝑛×𝑛): left singular vectors of 𝐀, which are the eigenvectors of 𝐀⁢𝐀⊤. Because symmetric matrices have full sets of orthonormal eigenvectors (see Appendix A3), 𝐔 is orthonormal.
2. 𝚺∈ℝ(𝑛×𝑚): rectangular matrix with singular values along its diagonal. Singular values are equivalent to the square roots of the eigenvalues of 𝐀⁢𝐀⊤ (or equivalently of 𝐀⊤⁢𝐀) divided by 𝑁−1 (see Equation 5.3).
3. 𝐕∈ℝ(𝑚×𝑚): right singular vectors of 𝐀, which are the eigenvectors of 𝐀⊤⁢𝐀. Because symmetric matrices have full sets of orthonormal eigenvectors (see Appendix A3), 𝐕 is orthonormal.

Applying what I presented previously in the fundamentals of linear algebra, the singular value decomposition implies that the linear transformation applied by any matrix can be broken down into three constituent transformations in the following order:

1. Rotation: 𝐕⊤ is an orthonormal matrix and so rotates basis vectors (see section on orthonormal matrices). The astute reader will notice that the transpose of an orthogonal matrix is equivalent to its inverse, so 𝐕⊤ is more technical an un-rotation of basis vectors.
2. Stretching with possible dimension change: because 𝚺 only has nonzero values along its diagonal, these values will stretch basis vectors (see section on diagonal matrices). The dimension space can also change following a transformation by Σ. For example, if the number of rows, 𝑛, is less than the number of columns 𝑝, then the 𝑝−𝑛 remaining columns of 𝚺 will contain zeroes that will remove the 𝑝−𝑛 remaining dimensions of 𝐕⊤.
3. Rotation: 𝐔 is an orthonormal matrix and thus rotates basis vectors (see section on orthonormal matrices).

Animation 10 below provides an geometric visualization of each transformation applied by each matrix of the singular value decomposition. First, it is shown that 𝐀 transforms the standard basis vectors such that 𝐛𝐱 becomes [2,0], 𝐛𝐲 becomes [1,2], and 𝐛𝐳 becomes [1,3]. The second animation then shows that each standard basis vector lands on its coordinates after three transformations in the order of a rotation, stretching (with a dimension reduction), and a rotation. First, the (transposed) matrix of right singular vectors is orthonormal and so applies a rotation. Second, the matrix of singular values is off-diagonal and so applies a stretching of basis vectors along with a reduction in the dimension (from ℝ3 to ℝ2. Third, and last, the matrix of left singular vectors applies a rotation of the basis vectors.