J.Wang upidea
upidea
/ SparkGibbsLDA.scala
Created
July 1, 2018 02:17
— forked from waleking/SparkGibbsLDA.scala
We implement gibbs sampling for LDA by Spark. This version performs much better than alpha version, and now can handle 3196204 words, 100 topics, 1000 sample iterations on server in 161.7 minutes. To solve the long time consuming in collect() process in alpha version, we utilize the cache() method as line 261 and line 262. We also solve a pile o…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package topic | |
| import spark.broadcast._ | |
| import spark.SparkContext | |
| import spark.SparkContext._ | |
| import spark.RDD | |
| import spark.storage.StorageLevel | |
| import scala.util.Random | |
| import scala.math.{ sqrt, log, pow, abs, exp, min, max } | |
| import scala.collection.mutable.HashMap |