Tech Facts
Renren website log-in mechanism
To check other people's profile, the first thing is to log-in and we need to imitate that with Java program
To learn the log-in mechanism of Renren, I used Http Analyzer to monitor http requests during log-in behavior.
From the analyzer, it is clear what name-value pairs are sent by post method and to what address.
After getting these information, we can imitate the log-in request to Renren.
Some main classes and methods of RenrenAnalyze packet
Renren Analyze package crawl data from Renren and data analyze
CrawlSearchedStatus.java contains the configuration variables and main method of the program
RenrenSpider.java make http requests, imitate log-in behavior, crawl pages
dbhelper.java connect with MySQL database, insert and update data efficiently
RenrenStats.java Process and analyze the big data acquired and generate statistic results
SwingGUI package Add Graphic User Interface to the program, make it more user friendly
Configuration before crawling
variable setups for the main method
/**txt file path of Renren account username and password to log-in*/
String accountsFilePath = "D:/Renren/account.txt";
/**the start page number of crawling target */
int offset = 9420;
/**keyword of timeline statuses to crawl in English*/
String tableName = "party";
/**keyword of timeline statuses to crawl in Chinese*/
String keyword = "聚会";
/**the start proxy host to use*/
int proxyIndex=9;
/**if need proxy host to crawl at beginning*/
boolean needProxy = false;
/**pause time (in seconds) after every crawl*/
double sleepSec = 0.5;