On manifold learning and interpretation of PCA
This is a must-read article about manifold learning. The article gives a great intuition about what is manifold learning and why should we know it. I found this article very exciting, inspiring and extremely brain-entertaining.
Science 22 December 2000:
Vol. 290. no. 5500, pp. 2268 – 2269
Some recent famous works
There are some famous algorithms on this topic that I surveyed — LLE, MDS, GPCA.
Multidimensional Scaling (MDS)
This is a classic data visualization and dimensionality reduction method. You can find a lot of good tutorials and books on this topic.
The cost function is to preserve the graph distance between each point in the space. This algorithm can be viewed as MDS + all-point shortest path in combination.
ISOMAP on Science magazine 2000
Locally Linear Embedding (LLE)
The objective function is to preserve the weights coming from neighborhood points. First the algorithm will find neighborhood points of each point in the input space (high-dimension space). Then estimate each point by using linear combination (weighted sum) of its neighbors. After that find a projection on the low-dimension space that preserves the weights obtained in the previous step.
article on Science magazine 2000
An Introduction to Locally Linear Embedding [pdf] (2000)
Professor Sam Roweis’s homepage on LLE [link]
Related topics and resources
There are a lot more resources on this topic
This is a good book and also available online for free “Principal Manifolds for Data Visualisation and Dimension Reduction” [link]
Interpretation of PCA
It’s also beneficial for the interested readers in this area to know variety of interpretation of PCA. That will make you understand the overview of this area and come up with a lot of new ideas on this manifold learning topic.
- This is a very understandable tutorial by Lindsay I Smith (2002). The paper covers all the basic knowledge from the beginning — what’s eigenvector, what’s covariance, how to use it. This is a great one. [pdf]
- Yet, I have another very good PCA tutorial “A Tutorial on Principal Component Analysis” (2005) by Jonathon Shlens. This tutorial paper is my favorite one. The paper give a good intuition about how to use PCA and show how to derive the principal components from the beginning. Very understandable tutorial. Also the paper did a really good job on explaining several crucial theorems on linear algebra related to PCA. The paper also discuss about the connection between PCA and SVD interestingly. At the end of the paper, the author discussed about the previous works on PCA and several types of PCA and their importance. [pdf]
- Interpret PCA as minimal approximation error as on a very recent paper “New Routes from Minimal Approximation Error to Principal Components” (2007). [pdf]
- I believe there are still more for this type of paper. Actually the key of all thing is “How can you pick the objective function!!!” The objective function will determine every thing about an algorithm.