### Archive

Archive for March, 2010

## Contrastive Divergence for Parameter Estimation in DSCRFs

Recently, I have been working on Deformable-Structure Conditional Random Fields (DSCRFs) for image classification which is about CRFs that can change the graph structure to fit the data (image) and make inference (of pixel label) simultaneously. One problem of this approach is when we estimate the parameters (I guess everyone in the field knows what I’m talking about ^_^), so I’m looking for some optimization algorithms to deal with this. Of course, first thing I’m thinking of is variational approximation. I have tried some already such as mean-field, structure variational, but there is one popular method, “Contrastive Divergence” (CD), that I have heard and wanted to try. I have read some papers on it, and here are what I really recommend to read.

1. “Note on Contrastive Divergence” by Oliver Woodford [pdf]: For me, this paper is the best; precise, intuitive and make you hungry to know more!
2. “Training Products of Experts by Minimizing Contrastive Divergence” by Geoffrey E. Hinton [pdf]: I guess this is the original paper of CD. This is the first paper I read on this topic, the paper did a good job to make me understand the math underlying CD, however, I did not have an intuitive idea of what CD really is after that first reading. Surprisingly, after I read [1], then come back to [2], I found that I can put pieces together and get a better intuition of this topic. So, I really recommend reading [1] before [2].

Video lecture related to this topic

Using Fast Weights to Improve Persistent Contrastive Divergence

Tijmen Tieleman

## Topic Models

Last week I met with a long-lost friend in ICASSP 2010 held in Dallas, TX. More precisely, the very nice friend of mine, Duangmanee (Pew), was my senior student when we were in the same high-school. In the conference, we had a good time (almost an hour) discussing about our lives, updates on our others friends (heeheehee…a nice way to say “gossips”) and our research, and I’m very lucky that Pew is an expert on Topic Models that I’m interested in. Since I’m a beginner on this topic, so I think I will have to learn some more fundamental works on this topic first prior to understanding Pew’s paper.

This post is my effort to list all good papers, notes and tutorials on topic models in the hope that it might be useful for other beginners like me. Please feel free to suggest in order to make this post of the most useful to learners.

Video lecture

Topic Models

David Blei
2 videos

Independent Factor Topic Models

Duangmanee (Pew) Putthividhya

Useful links (I’m working on the list)

LDA on Wiki

Blei, David M.; Ng, Andrew Y.; Jordan, Michael I; Lafferty, John (January 2003). “Latent Dirichlet allocation”. Journal of Machine Learning Research 3: pp. 993–1022. doi:10.1162/jmlr.2003.3.4-5.993

Blei, David M.; Lafferty, John D. (2006). “Correlated topic models”. Advances in Neural Information Processing Systems

D. Blei and M. Jordan. Modeling annotated data. In Proceedings of the 26th annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 127–134. ACM Press, 2003. [PDF]

K. Barnard, P. Duygulu, N. de Freitas, D. Forsyth, D. Blei, and M. Jordan. Matching words and pictures. Journal of Machine Learning Research, 3:1107–1135, 2003. [PDF]

Blei, David M.; Jordan, Michael I.; Griffiths, Thomas L.; Tenenbaum; Joshua B (2004). “Hierarchical Topic Models and the Nested Chinese Restaurant Process”. Advances in Neural Information Processing Systems 16: [pdf]

Hanna M. Wallach (2008), “Structured Topic Model for Language” [PhD thesis]

Tomoharu Iwata, Takeshi Yamada, Naonori Ueda, “Modeling Social Annotation Data with Content Relevance using a Topic Model,” Advances in Neural Information Processing Systems (NIPS2009), 835-843, 2009 [pdf]

## Interesting papers in ICASSP 2010, Dallas, TX (Unfinished)

Wednesday, March 17

IVMSP-P4.1: MULTISCALE SEGMENTATION FOR MRC DOCUMENT COMPRESSION USING A MARKOV RANDOM FIELD MODEL
Eri Haneda, Charles Bouman, Purdue university, United States

IVMSP-P4.3: A LOW COMPLEXITY METHOD FOR DETECTION OF TEXT AREA IN NATURAL IMAGES
Katherine L. Bouman, University of Michigan, United States; Golnaz Abdollahian, Mireille Boutin, Edward Delp, Purdue University, United States

– Application on cell phone worth thinking about…we have to use simple and efficient.

– This work is simple and seems to work well in practice.

IVMSP-P4.4: FAST SEMI-SUPERVISED IMAGE SEGMENTATION BY NOVELTY SELECTION
Antonio Paiva, Tolga Tasdizen, University of Utah, United States

-Simple, but very important idea…very practical. Just change the perspective, we gain the speed!

IVMSP-P4.5: IMPROVING IMAGE SEGMENTATION VIA SHAPE PCA RECONSTRUCTION
Hui Wang, Hong Zhang, University of Alberta, Canada

– Simple idea, but very interesting, and should be applied to more complicated contour objects

IVMSP-P4.8: COMPLEXITY-BASED BORDER DETECTION FOR TEXTURED IMAGES
Tomas Crivelli, University of Buenos Aires, Argentina; Agustin Mailing, Bruno Cernuschi-Frias, University of Buenos Aires and Consejo Nacional de Investigaciones Científicas y Técnicas, Argentina

– Use Taylor series to approx log likelihood and get Gaussian. Check out paper by Kashyap and Chellappa “Estimation choice of neighbors..”

IVMSP-P4.9: ON SAND RIPPLE DETECTION IN SYNTHETIC APERTURE SONAR IMAGERY
David Williams, Enrique Coiras, NATO Undersea Research Centre, Italy

– Smart way to formulate using 2 orthogonal (Gabor) filters in absolute value

Branch and Bound for discrete optimization

SPSA Algorithm

Factor Graph for Optimization Problem

Optimization Problem –> Dynamic Programming

My work should change to Deformable Factor Graphs

Topic Model + Image Segmentation

Categories: Uncategorized

## Eclipse IDE for 64-bit Windows

Eclipse IDE for 64-bit Windows

http://lingpipe-blog.com/2009/03/05/eclipse-ide-for-64-bit-windows-and-64-bit-java/

I’m now trying Eclipse 3.5 (“Galileo”)

This build requires a 64-bit JVM, and will not run with a 32-bit JVM.

You can, for example, use the Sun 64-bit 1.5 JVM for AMD64.  Note

that the Sun 1.4.2 JVM for AMD64 is 32-bit and therefore cannot be

used to run this build.

http://www.cs.cmu.edu/~pattis/15-1XX/common/handouts/javaeclipseinstallation.html

Categories: Reviews, Tutorials