-
Notifications
You must be signed in to change notification settings - Fork 0
/
simulated-annealing-for-dirichlet-priors-in-lda.html
106 lines (94 loc) · 4.85 KB
/
simulated-annealing-for-dirichlet-priors-in-lda.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
<!doctype html>
<html lang="en">
<head>
<!-- Required meta tags -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<title> Simulated Annealing for Dirichlet Priors in LDA | akuz.me/nko
</title>
<link rel="canonical" href="https://akuz.me/simulated-annealing-for-dirichlet-priors-in-lda.html">
<link rel="stylesheet" href="https://akuz.me/theme/css/bootstrap.min.css">
<link rel="stylesheet" href="https://akuz.me/theme/css/font-awesome.min.css">
<link rel="stylesheet" href="https://akuz.me/theme/css/pygments/default.min.css">
<link rel="stylesheet" href="https://akuz.me/theme/css/theme.css">
<link rel="alternate" type="application/atom+xml" title="Full Atom Feed"
href="https://akuz.me/feeds/all.atom.xml">
<meta name="description" content="When estimating the parameters of the LDA (Latent Dirichlet Allocation) model using Gibbs sampling, if we set the Dirichlet priors to the fixed target values (usually small), then we reduce the mixing of the samples from the target distribution from the beginning, even though we haven’t found a good …">
<script>
(function(i, s, o, g, r, a, m) {
i['GoogleAnalyticsObject'] = r;
i[r] = i[r] || function() {
(i[r].q = i[r].q || []).push(arguments)
}, i[r].l = 1 * new Date();
a = s.createElement(o);
a.async = 1;
a.src = g;
m = s.getElementsByTagName(o)[0];
m.parentNode.insertBefore(a, m)
})(window, document, 'script', 'https://www.google-analytics.com/analytics.js', 'ga');
ga('create', 'UA-47495265-1', 'auto');
ga('send', 'pageview');
</script>
</head>
<body>
<header class="header">
<div class="container">
<div class="row">
<div class="col-sm-12">
<h1 class="title"><a href="https://akuz.me/">akuz.me/nko</a></h1>
<ul class="list-inline">
<li class="list-inline-item"><a href="/">Home</a></li>
<li class="list-inline-item text-muted">|</li>
<li class="list-inline-item"><a href="https://akuz.me/pages/about.html">About</a></li>
<li class="list-inline-item"><a href="https://akuz.me/pages/papers.html">Papers</a></li>
<li class="list-inline-item"><a href="https://akuz.me/pages/software.html">Software</a></li>
</ul>
</div>
</div> </div>
</header>
<div class="main">
<div class="container">
<h1> Simulated Annealing for Dirichlet Priors in LDA
</h1>
<hr>
<article class="article">
<header>
<ul class="list-inline">
<li class="list-inline-item text-muted" title="2014-01-25T00:00:00+00:00">
<i class="fa fa-clock-o"></i>
Sat 25 January 2014
</li>
<li class="list-inline-item">
<i class="fa fa-folder-open-o"></i>
<a href="https://akuz.me/category/papers.html">Papers</a>
</li>
<li class="list-inline-item">
<i class="fa fa-user-o"></i>
<a href="https://akuz.me/author/akuz.html">akuz</a> </li>
</ul>
</header>
<div class="content">
<p>When estimating the parameters of the LDA (Latent Dirichlet Allocation) model using Gibbs sampling, if we set the Dirichlet priors to the fixed target values (usually small), then we reduce the mixing of the samples from the target distribution from the beginning, even though we haven’t found a good approximation yet.</p>
<p><img alt="Simulated Annealing" class="img-fluid d-block mx-auto" src="https://akuz.me/images/SimAnn2.png"></p>
<p>An alternative would be to initialise the Dirichlet priors with relatively high parameters alpha, and then gradually decrease them during burn-in period. This will allod the sampler to locate the approximate area of interest faster at the initial stages, while still sampling at the target prior values after burn-in.</p>
<p>This article describes application of simulated annealing technique for MCMC inference of multinomial random distributions with Dirichlet priors in LDA. It is implemented in my NLP library for optimised Gibbs sampling for LDA (see software). The full article can be found <a href="https://akuz.me/pdfs/akuz_sim_ann_lda.pdf">here</a> (PDF).</p>
</div>
</article>
</div>
</div>
<footer class="footer">
<div class="container">
<div class="row">
<ul class="col-sm-6 list-inline">
<li class="list-inline-item"><a href="https://akuz.me/authors.html">Authors</a></li>
<li class="list-inline-item"><a href="https://akuz.me/archives.html">Archives</a></li>
<li class="list-inline-item"><a href="https://akuz.me/categories.html">Categories</a></li>
</ul>
<p class="col-sm-6 text-sm-right text-muted">
Generated by <a href="https://github.com/getpelican/pelican" target="_blank">Pelican</a>
/ <a href="https://github.com/nairobilug/pelican-alchemy" target="_blank">✨</a>
</p>
</div> </div>
</footer>
</body>
</html>