## Contrastive Divergence for Parameter Estimation in DSCRFs

Recently, I have been working on Deformable-Structure Conditional Random Fields (DSCRFs) for image classification which is about CRFs that can change the graph structure to fit the data (image) and make inference (of pixel label) simultaneously. One problem of this approach is when we estimate the parameters (I guess everyone in the field knows what I’m talking about ^_^), so I’m looking for some optimization algorithms to deal with this. Of course, first thing I’m thinking of is variational approximation. I have tried some already such as mean-field, structure variational, but there is one popular method, “Contrastive Divergence” (CD), that I have heard and wanted to try. I have read some papers on it, and here are what I really recommend to read.

- “Note on Contrastive Divergence” by Oliver Woodford [pdf]: For me, this paper is the best; precise, intuitive and make you hungry to know more!
- “Training Products of Experts by Minimizing Contrastive Divergence” by Geoffrey E. Hinton [pdf]: I guess this is the original paper of CD. This is the first paper I read on this topic, the paper did a good job to make me understand the math underlying CD, however, I did not have an intuitive idea of what CD really is after that first reading. Surprisingly, after I read [1], then come back to [2], I found that I can put pieces together and get a better intuition of this topic. So, I really recommend reading [1] before [2].

Video lecture related to this topic

Using Fast Weights to Improve Persistent Contrastive Divergence

Tijmen Tieleman

## Topic Models

Last week I met with a long-lost friend in ICASSP 2010 held in Dallas, TX. More precisely, the very nice friend of mine, Duangmanee (Pew), was my senior student when we were in the same high-school. In the conference, we had a good time (almost an hour) discussing about our lives, updates on our others friends (heeheehee…a nice way to say “gossips”) and our research, and I’m very lucky that Pew is an expert on *Topic Models* that I’m interested in. Since I’m a beginner on this topic, so I think I will have to learn some more fundamental works on this topic first prior to understanding Pew’s paper.

This post is my effort to list all good papers, notes and tutorials on topic models in the hope that it might be useful for other beginners like me. Please feel free to suggest in order to make this post of the most useful to learners.

Video lecture

Topic Models

David Blei

*2 videos*

Independent Factor Topic Models

Duangmanee (Pew) Putthividhya

Useful links (I’m working on the list)

LDA on Wiki

Blei, David M.; Ng, Andrew Y.; Jordan, Michael I; Lafferty, John (January 2003). “Latent Dirichlet allocation”. *Journal of Machine Learning Research* **3**: *pp.* 993–1022. doi:10.1162/jmlr.2003.3.4-5.993

Blei, David M.; Lafferty, John D. (2006). “Correlated topic models”. *Advances in Neural Information Processing Systems*

D. Blei and M. Jordan. **Modeling annotated data**. In Proceedings of the 26th annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 127–134. ACM Press, 2003. [PDF]

K. Barnard, P. Duygulu, N. de Freitas, D. Forsyth, D. Blei, and M. Jordan. **Matching words and pictures**. Journal of Machine Learning Research, 3:1107–1135, 2003. [PDF]

Blei, David M.; Jordan, Michael I.; Griffiths, Thomas L.; Tenenbaum; Joshua B (2004). “Hierarchical Topic Models and the Nested Chinese Restaurant Process”. Advances in Neural Information Processing Systems 16: [pdf]

Hanna M. Wallach (2008), “Structured Topic Model for Language” [PhD thesis]

Tomoharu Iwata, Takeshi Yamada, Naonori Ueda, “Modeling Social Annotation Data with Content Relevance using a Topic Model,” Advances in Neural Information Processing Systems (NIPS2009), 835-843, 2009 [pdf]

## Interesting papers in ICASSP 2010, Dallas, TX (Unfinished)

Wednesday, March 17

**IVMSP-P4.1: MULTISCALE SEGMENTATION FOR MRC DOCUMENT COMPRESSION USING A MARKOV RANDOM FIELD MODEL**

Eri Haneda, Charles Bouman, Purdue university, United States

**IVMSP-P4.3: A LOW COMPLEXITY METHOD FOR DETECTION OF TEXT AREA IN NATURAL IMAGES**

Katherine L. Bouman, University of Michigan, United States; Golnaz Abdollahian, Mireille Boutin, Edward Delp, Purdue University, United States

– Application on cell phone worth thinking about…we have to use simple and efficient.

– This work is simple and seems to work well in practice.

**IVMSP-P4.4: FAST SEMI-SUPERVISED IMAGE SEGMENTATION BY NOVELTY SELECTION**

Antonio Paiva, Tolga Tasdizen, University of Utah, United States

-Simple, but very important idea…very practical. Just change the perspective, we gain the speed!

**IVMSP-P4.5: IMPROVING IMAGE SEGMENTATION VIA SHAPE PCA RECONSTRUCTION**

Hui Wang, Hong Zhang, University of Alberta, Canada

– Simple idea, but very interesting, and should be applied to more complicated contour objects

**IVMSP-P4.8: COMPLEXITY-BASED BORDER DETECTION FOR TEXTURED IMAGES**

Tomas Crivelli, University of Buenos Aires, Argentina; Agustin Mailing, Bruno Cernuschi-Frias, University of Buenos Aires and Consejo Nacional de Investigaciones Científicas y Técnicas, Argentina

– Use Taylor series to approx log likelihood and get Gaussian. Check out paper by Kashyap and Chellappa “Estimation choice of neighbors..”

**IVMSP-P4.9: ON SAND RIPPLE DETECTION IN SYNTHETIC APERTURE SONAR IMAGERY**

David Williams, Enrique Coiras, NATO Undersea Research Centre, Italy

– Smart way to formulate using 2 orthogonal (Gabor) filters in absolute value

Branch and Bound for discrete optimization

SPSA Algorithm

Factor Graph for Optimization Problem

Optimization Problem –> Dynamic Programming

My work should change to Deformable Factor Graphs

Topic Model + Image Segmentation

## Eclipse IDE for 64-bit Windows

Eclipse IDE for 64-bit Windows

http://lingpipe-blog.com/2009/03/05/eclipse-ide-for-64-bit-windows-and-64-bit-java/

I’m now trying Eclipse 3.5 (“Galileo”)

Important note from the download site:

This build requires a 64-bit JVM, and will not run with a 32-bit JVM.

You can, for example, use the Sun 64-bit 1.5 JVM for AMD64. Note

that the Sun 1.4.2 JVM for AMD64 is 32-bit and therefore cannot be

used to run this build.

Here is the instruction on how to download and install

http://www.cs.cmu.edu/~pattis/15-1XX/common/handouts/javaeclipseinstallation.html

## LaTeX Editor (LEd) built-in DVI viewer shows blank page

Q: I’m using Windows7 64-bit and my LaTeX editor (build 0.536501) shows blank page for built-in DVI. What should I do?

A: You may want to check in the main menu Configuration>Options. Then expand the list in Application>DVI viewer. Make sure that you put the right path to GSDLL32.DLL directory and PostScript fonts. Make sure you have Ghostscript installed in your computer. In my case, I install GPL Ghostscript 8.64 for 64-bit Windows (link), so here is my configuration

Hope this helps!