{"id":300,"date":"2011-12-17T00:14:26","date_gmt":"2011-12-16T16:14:26","guid":{"rendered":"http:\/\/ykyi.net\/?p=300"},"modified":"2011-12-17T00:14:26","modified_gmt":"2011-12-16T16:14:26","slug":"distributed-framework-for-iterative-computations-on-massive-datasets","status":"publish","type":"post","link":"https:\/\/ykyi.net\/?p=300","title":{"rendered":"Distributed Framework for Iterative Computations on Massive Datasets"},"content":{"rendered":"
I must confess I've only got a few words and a vague skeleton of what the Professer Lixin Gao said due to the language barrier, during the whole speech Ms Gao spoke English, and the topic is so pedantic which is full of the complicated mathematic formulation.Personally\uff0c\u6211\u672a\u6709\u6d89\u53ca\u8fc7\u6d77\u91cf\u6570\u636e\u5e76\u884c\u8ba1\u7b97\u4ee5\u81f3\u4e91\u8ba1\u7b97\u76f8\u5173\u7684\u4efb\u4f55\u6280\u672f\uff0c\u57fa\u672c\u4e0a\u5bf9\u4e8e\u6d77\u91cf\u6570\u636e\u7684\u6240\u6709\u6280\u672f\u90fd\u8ba9\u6211\u89c9\u5f97\u975e\u5e38\u5730<\/span>cutting-edge<\/span>\u3002\u5e26\u7740\u5b66\u4e60\u5f00\u62d3\u773c\u754c\u62d3\u5c55\u77e5\u8bc6\u9762\u7684\u826f\u597d\u613f\u671b\u53c2\u52a0\u4e86<\/span>Ms. Gao<\/span>\u7684\u5b66\u672f\u8bb2\u5ea7\u3002<\/span>Ms. Gao<\/span>\u8bb2<\/span>PPT<\/span>\u65f6\u5168\u662f\u82f1\u6587\u554a\u3002\u6211\u52aa\u529b\u5730\u542c\uff0c\u52aa\u529b\u5730\u542c\uff01\uff01\uff01\u9002\u5f53\u5730\u8bb0\u4e9b\u7b14\u8bb0\u3002\u5176\u5b9e\u8fd8\u662f\u5f88\u559c\u6b22\u542c\u82f1\u8bed\u7684\u3002<\/span> <\/p>\n

\u8bb2\u5ea7\u5148\u5217\u4e3e\u4e86\u4e00\u4e9b\u5927\u578b\u4e92\u8054\u7f51\u516c\u53f8\u7684\u6570\u636e\u663e\u793a\u96c6\u7fa4\u8ba1\u7b97\u7684\u89c4\u6a21\u3002\u6bd4\u5982\u8c37\u6b4c\u5728\u6570\u636e\u4e2d\u5fc3\u6709\u4e00\u767e\u4e07\u53f0\u670d\u52a1\u5668\uff01\uff01\uff01Holy gosh<\/span>\uff0c<\/span>1,000,000 servers<\/span>\uff01\uff01\uff01<\/span><\/p>\n

\u7136\u540e\u8c08\u5230\u4e86\u5f88\u591a\u5927\u89c4\u6a21\u6570\u636e\u7684\u8ba1\u7b97\u5bf9\u65f6\u95f4\u7684\u8981\u6c42\u975e\u5e38\u5730\u4e25\u683c\u3002\u4e8e\u662f\u8c08\u5230\u4e86\u76ee\u524d\u6d41\u884c\u7684\u7b97\u6cd5\uff1aMapReduce<\/span>\u3002<\/span>MapReduce\u662fgoogle<\/span>\u63d0\u5021\u7684<\/span>\u4e00\u79cd\u7f16\u7a0b\u6a21\u578b\uff0c\u7528\u4e8e\u5927\u89c4\u6a21\u6570\u636e\u96c6\uff08\u5927\u4e8e1TB<\/span>\uff09\u7684\u5e76\u884c\u8fd0\u7b97\u3002<\/span>“Map”<\/span>\u548c <\/span>“Reduce”<\/span>\uff0c\u548c\u4ed6\u4eec\u7684\u4e3b\u8981\u601d\u60f3\uff0c\u90fd\u662f\u4ece\u51fd\u6570\u5f0f\u7f16\u7a0b\u8bed\u8a00\u91cc<\/span>\u7ee7\u627f\u6765\u7684\uff0c\u8fd8\u6709\u4ece\u77e2\u91cf\u7f16\u7a0b\u8bed\u8a00\u91cc\u501f\u6765\u7684\u7279\u6027\u3002\u4ed6\u6781\u5927\u5730\u65b9\u4fbf\u4e86\u7f16\u7a0b\u4eba\u5458\u5728\u4e0d\u4f1a\u5206\u5e03\u5f0f\u5e76\u884c \u7f16\u7a0b\u7684\u60c5\u51b5\u4e0b\uff0c\u5c06\u81ea\u5df1\u7684\u7a0b\u5e8f\u8fd0\u884c\u5728\u5206\u5e03\u5f0f\u7cfb\u7edf\u4e0a\u3002\u6211\u81ea\u5df1\u7684\u7406\u89e3\u662f\uff1amap\u7d22\u5f15\u6574\u4e2a\u6d77\u91cf\u6570\u636e\uff0c\u751f\u6210<\/span>key-value pair which is used to facilitate the quick query and computing<\/span>\u3002\u800c<\/span>reduce<\/span>\u5219\u662f\u7528\u6765\u5206\u6790\u548c\u7edf\u7b79\u57fa\u4e8e\u952e\u503c\u5bf9\u7684\u4e2d\u95f4\u8ba1\u7b97\u7ed3\u679c\uff0c\u4f7f\u4e2d\u95f4\u7ed3\u679c\u8fc5\u901f\u5411\u6700\u7ec8\u7684\u6b63\u786e\u7ed3\u679c\u6536\u655b\u3002<\/span><\/p>\n

Ms. Gao \u63d0\u51fa<\/span>MapReduce<\/span>\u867d\u7136\u662f\u5de5\u4e1a\u754c\u975e\u5e38<\/span>popular<\/span>\u7684\u89e3\u51b3\u65b9\u6848\uff0c\u4f46\u662f\u4e0d\u53ef\u4ee5\u4ee5\u8fed\u4ee3\u7684\u65b9\u5f0f\u8ba1\u7b97\u3002\u800c<\/span>Ms. Gao<\/span>\u6240\u505a\u7684\u5de5\u4f5c\u5373\u662f\u4f7f<\/span>MapReduce<\/span>\u652f\u6301\u8fed\u4ee3\u7684\u529f\u80fd\u3002\uff08<\/span>A framework that support iterative computing in the cloud.<\/span>\uff09\u4ece\u4e8b\u60c5\u4e00\u822c\u89c4\u5f8b\u7684\u54f2\u5b66\u9ad8\u5ea6\u770b\u5f85\u8fd9\u4e2a\u95ee\u9898\uff0c\u5bf9\u4e8e\u89c4\u6a21\u76f8\u5f53\u5927\u6216\u8005\u6700\u7ec8\u7ed3\u679c\u662f\u76f8\u5f53\u4e0d\u660e\u786e\u7684\u95ee\u9898\uff0c\u4f3c\u4e4e\u90fd\u53ef\u4ee5\u7528\u5230<\/span>'<\/span>\u8fed\u4ee3<\/span>'<\/span>\u7684\u601d\u60f3\u3002\u4f8b\u5982\uff0c\u8fed\u4ee3\u7684\u8f6f\u4ef6\u5f00\u53d1\u6a21\u5f0f\u5219\u662f\u5f53\u4eca\u4e92\u8054\u7f51\u5e94\u7528\u7684\u4e3b\u8981\u5f00\u53d1\u6a21\u5f0f\u3002\u6bcf\u4e00\u6b21\u8fed\u4ee3\u90fd\u4f7f\u4ea7\u54c1\u5411\u7406\u60f3\u4e2d\u6700<\/span>perfect<\/span>\u7684\u72b6\u6001\u63a5\u8fd1\u3002<\/span>Ms. Gao<\/span>\u7528\u8fed\u4ee3\u7684\u601d\u60f3\u5904\u7406\u6d77\u91cf\u6570\u636e\u5e94\u8be5\u4e5f\u662f\u540c\u6837\u7684\u9053\u7406\u7f62\uff01<\/span><\/p>\n

Ms. Gao\u7684\u89e3\u91ca\u8bf4\u6539\u8fdb\u540e\u652f\u6301\u8fed\u4ee3\u7684<\/span>iMapReduce<\/span>\u53ef\u4ee5\u4f7f<\/span>map<\/span>\u540e\u7684\u6570\u636e\u652f\u6491<\/span>reduce,reduce<\/span>\u7684\u6570\u636e\u4ea6\u8f6c\u800c\u652f\u6491<\/span>map<\/span>\uff0c\u5f62\u6210\u4e00\u4e2a\u8fed\u4ee3\u53cd\u590d\u7684\u8fc7\u7a0b\uff0c\u5176\u4e2d\u5728<\/span>map<\/span>\u9636\u6bb5\u52a0\u5165<\/span>static data<\/span>\u3002\u8f83\u4e4b\u539f\u59cb\u7684<\/span>MapReduce<\/span>\uff0c\u65b0\u7684\u7b97\u6cd5\u4f7f\u5f97<\/span>static data<\/span>\u7684\u89c4\u6a21\u663e\u8457\u5730\u51cf\u5c11\u3002<\/span>Ms. Gao<\/span>\u5c55\u793a\u4e86\u51e0\u4e2a\u56fe\u8868\u8bc1\u660e\u6539\u8fdb\u540e\u589e\u52a0\u4e86\u8fed\u4ee3\u529f\u80fd\u7684<\/span>MapReduce(<\/span>\u79f0\u4e4b\u4e3a <\/span>iMapReduce )<\/span>\u7684\u901f\u5ea6\u4f18\u4e8e\u539f\u59cb\u7684<\/span>MapReduce.<\/span>\u56fe\u8868\u4e2d\u6709\u5c55\u793a\u7528<\/span>iMapReduce<\/span>\u8ba1\u7b97<\/span>PageRank<\/span>\u7684\u503c\u3002<\/span>PageRank<\/span>\uff1f\u662f\u4e0d\u662f\u5224\u65ad\u7f51\u7ad9\u6743\u91cd\u7684\u90a3\u4e2a\u4e1c\u4e1c\u554a\uff1f\u54e5\u7684\u7f51\u7ad9<\/span>http:\/\/ykyi.net<\/span><\/a>\u7684<\/span>PageRank<\/span>\u624d<\/span>0<\/span>\u554a\uff01\uff01\uff01\u624d<\/span>0<\/span>\u554a<\/span>~~<\/span>\u7528\u4ec0\u4e48\u7b97\u6cd5\u53ef\u4ee5\u7b97\u6210<\/span>9<\/span>\u554a\uff01<\/span><\/p>\n

PPT\u8f6c\u5230\u8ba1\u7b97\u56fe\u7684\u6307\u5b9a\u4e24\u70b9\u6700\u77ed\u8def\u5f84\u7684<\/span>Dijkstra<\/span>\u7b97\u6cd5\u3002\u6cea\u6d41\u6ee1\u9762\uff0c\u7ec8\u4e8e\u6709\u4e00\u4e2a\u6211\u61c2\u7684\u7b97\u6cd5\u4e86\u554a\uff01<\/span>Ms. Gao<\/span>\u8bf4\u5148\u627e\u5230\u4e24\u4e2a\u7ed3\u70b9\u95f4\u7684\u6700\u77ed\u8def\u5f84\uff0c\u4e8e\u662f\u8d4b\u4e8e\u8fd9\u4e2a\u6700\u77ed\u8def\u5f84\u7ecf\u8fc7\u7684\u7ed3\u70b9\u6700\u9ad8\u7684\u6743\u91cd\uff0c\u5176\u5b83\u7684\u8def\u5f84\u5c31\u8d4b\u4e8e\u76f8\u5bf9\u4f4e\u7684\u6743\u91cd\u3002\u63d0\u51fa\u4e86<\/span>PriorityQueue<\/span>\u7684\u6570\u636e\u7ed3\u6784\u3002\u628a\u6743\u91cd\u5f15\u5165\u5230<\/span>iMapReduce<\/span>\u7b97\u6cd5\uff0c\u79f0\u4e4b\u4e3a<\/span>pMapReduce<\/span>\u7b97\u6cd5\u3002\u5f15\u5165\u6743\u91cd\u540e\u7684<\/span>pMapReduce<\/span>\u7b97\u6cd5\u6bd4<\/span>iMapReduce<\/span>\u7b97\u6cd5\u7684\u901f\u5ea6\u66f4\u5feb\u3002\u751f\u6d3b\u4e2d\u6211\u4eec\u505a\u4e8b\u60c5\u4e5f\u8981\u4f18\u5148\u5904\u7406\u6700\u91cd\u8981\u7684\u4e8b\u60c5\uff0c\u9053\u7406\u662f\u4e00\u6837\u7684\u554a\uff01\u4e16\u95f4\u4e07\u4e8b\u4e07\u7269\uff0c\u65e0\u8bba\u591a\u4e48\u590d\u6742\uff0c\u62bd\u8c61\u51fa\u6765\u7684\u4e00\u822c\u89c4\u5f8b\u5176\u5b9e\u90fd\u662f\u76f8\u5f53\u5730\u4e00\u81f4\u3002\u6613\u7ecf\u4e2d\u8bf4\uff1a\u5f62\u800c\u4e0b\u8005\u8c13\u4e4b\u5668\uff0c\u5f62\u800c\u4e0a\u8005\u8c13\u4e4b\u9053\uff01<\/span>Oops, The ancient Tao philosophy is f<\/span>ull Of Wisdom.<\/p>\n

\u6700\u540e\u9636\u6bb5\uff0c\u518d\u6b21\u5c55\u793a\u4e86\u4e00\u4e2a\u56fe\u8868\uff0cMs. Gao<\/span>\u5b8c\u6210\u7684\u5efa\u7acb\u5728<\/span>Apache<\/span>\u7684<\/span>Hadoop<\/span>\u4e0a\u7684<\/span>pMapReduce<\/span>\u8981\u660e\u663e\u7684\u4f18\u4e8e\u539f\u59cb\u7684<\/span>Hadoop<\/span>\u3002\u5e76\u4e14\u4e5f\u4f18\u5148\u5176\u5b83\u591a\u79cd\u89e3\u51b3\u65b9\u6848\uff0c\u5176\u5b83\u7684\u540d\u5b57\u6211\u90fd\u6ca1\u542c\u8fc7\u554a\uff01\u5728\u63d0\u95ee\u9636\u6bb5\uff0c\u5c3d\u7ba1\u6e29\u6b66\u5c11\u8001\u5e08\u79ef\u6781\u9f13\u52b1\u540c\u5b66\u63d0\u95ee\u4f46\u6ca1\u6709\u540c\u5b66\u63d0\u51fa\u95ee\u9898\uff0c\u5927\u6982\u5927\u5bb6\u90fd\u4e0d\u61c2\u5427\uff01\u671d\u7ea2\u9633\u8001\u5e08\u548c\u53e6\u4e00\u4f4d\u7814\u7a76\u6d77\u91cf\u6570\u636e\u7684\u8001\u5e08\u8c08\u4e86\u4e00\u70b9\u70b9\u3002\u4ed6\u4eec\u63d0\u5230<\/span>iterative<\/span>\u7684\u7b97\u6cd5\u5728\u4ed6\u4eec\u7684\u5b9e\u9645\u5e94\u7528\u4e2d\u6548\u679c\u5e76\u4e0d\u662f\u592a\u660e\u663e\uff0c\u4e0d\u5982<\/span>memory-based<\/span>\u7684\u89e3\u51b3\u65b9\u6848\u3002<\/span>Ms. Gao<\/span>\u8c03\u51fa\u4e00\u5f20\u56fe\u8868\uff0c\u89e3\u91ca\u8bf4\u67d0\u79cd<\/span>memory-based<\/span>\u7684\u89e3\u51b3\u65b9\u6848\u6ca1\u6709\u8fed\u4ee3\u7684\u7b97\u6cd5\u5feb\u3002<\/span><\/p>\n

\u4f5c\u4e3a\u4f17\u591a\u6253\u9171\u6cb9\u7684\u540c\u5b66\u4e2d\u7684\u4e00\u5458\uff0c\u6211\u6700\u540e\u7684\u5fc3\u5f97\u4f53\u4f1a\u662f\uff1a\u5b66\u672f\u771f\u662f\u592a\u9ad8\u6df1\u4e86\u554a\uff01\u6211\u8fd8\u662f\u505a\u5de5\u7a0b\u5427\uff01\uff01\uff01\u8f66\uff0c\u6211\u6709\u5f97\u62e3\u54a9\uff1f<\/p>\n

\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/<\/p>\n

Topic <\/strong><\/p>\n

Distributed Framework for Iterative Computations on Massive Datasets<\/div>\n
 <\/div>\n
Speaker<\/strong><\/div>\n
Prof. Lixin Gao<\/div>\n
 <\/div>\n
Abstract<\/strong><\/div>\n
Iterative algorithms are pervasive in many applications such as search engine algorithms,machine learning, and recommendation systems. These applications typically involve a dataset of massive scale. Fast iterative computations of the massive datasets are essential for these applications. This is particular important for on-line query such as keyword based search query. In this talk, we present an overview of MapReduce framework, and propose two frameworks, iMapReduce and pMapReduce, that enable fast iterative computations. By providing the support of iterative computations and prioritized execution, we can ensure faster convergence of the iterative process. Both iMapReduce and pMapReduce preserve the MapReduce distributed computing framework and is particularly efficient for online queries such as top-k queries. We implement iMapReduce and pMapReduce based on Apache Hadoop and evaluate its performance. Our evaluation results show that pMapReduce can reduce the computation time by two orders of magnitude comparing to that achieved with MapReduce. At the end of the talk, I will provide an overview of on-going projects in my research group.<\/div>\n
 <\/div>\n
Bio<\/strong><\/div>\n
\"\"<\/strong><\/div>\n
\n
Lixin Gao is a professor of Electrical and Computer Engineering at the University of Massachusetts at Amherst. She received her Ph.D. degree in computer science from the University of Massachusetts at Amherst in 1996. Her research interests include social networks, and Internet routing, network virtualization and cloud computing. Between May 1999 and January 2000, she was a visiting researcher at AT&T Research Labs and DIMACS. She was an Alfred P. Sloan Fellow between 2003-2005 and received an NSF CAREER Award in 1999. She won the best paper award from IEEE INFOCOM 2010 and her paper in ACM Cloud Computing 2011 was honored with “Paper of Distinction”. She received the Chancellor’s Award for Outstanding Accomplishment in Research and Creative Activity in 2010, and is a fellow of IEEE.<\/div>\n<\/p><\/div>\n
 <\/div>\n
 <\/strong><\/p>\n

 <\/p>\n

Date and Venue<\/strong><\/div>\n
Dec.16\uff08Friday\uff0913:30pm-14:30pm<\/div>\n
Lecture Theater A101, School of Software, Sun Yat-sen University<\/div>\n<\/p><\/div>\n<\/div>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

I must confess I've only got a few words and a vague skeleton of what the Professer Lixin Gao said due to the language barrier, during the whole speech Ms Gao spoke English, and the topic is so pedantic which is full of the complicated mathematic formulation.Personally\uff0c\u6211\u672a\u6709\u6d89\u53ca\u8fc7\u6d77\u91cf\u6570\u636e\u5e76\u884c\u8ba1\u7b97\u4ee5\u81f3\u4e91\u8ba1\u7b97\u76f8\u5173\u7684\u4efb\u4f55\u6280\u672f\uff0c\u57fa\u672c\u4e0a\u5bf9\u4e8e\u6d77\u91cf\u6570\u636e\u7684\u6240\u6709\u6280\u672f\u90fd\u8ba9\u6211\u89c9\u5f97\u975e\u5e38\u5730cutting-edge\u3002\u5e26\u7740\u5b66\u4e60\u5f00\u62d3\u773c\u754c\u62d3\u5c55\u77e5\u8bc6\u9762\u7684\u826f\u597d\u613f\u671b\u53c2\u52a0\u4e86Ms. Gao\u7684\u5b66\u672f\u8bb2\u5ea7\u3002Ms. Gao\u8bb2PPT\u65f6\u5168\u662f\u82f1\u6587\u554a\u3002\u6211\u52aa\u529b\u5730\u542c\uff0c\u52aa\u529b\u5730\u542c\uff01\uff01\uff01\u9002\u5f53\u5730\u8bb0\u4e9b\u7b14\u8bb0\u3002\u5176\u5b9e\u8fd8\u662f\u5f88\u559c\u6b22\u542c\u82f1\u8bed\u7684\u3002  \u8bb2\u5ea7\u5148\u5217\u4e3e\u4e86\u4e00\u4e9b\u5927\u578b\u4e92\u8054\u7f51\u516c\u53f8\u7684\u6570\u636e\u663e\u793a\u96c6\u7fa4\u8ba1\u7b97\u7684\u89c4\u6a21\u3002\u6bd4\u5982\u8c37\u6b4c\u5728\u6570\u636e\u4e2d\u5fc3\u6709\u4e00\u767e\u4e07\u53f0\u670d\u52a1\u5668\uff01\uff01\uff01Holy gosh\uff0c1,000,000 servers\uff01\uff01\uff01 \u7136\u540e\u8c08\u5230\u4e86\u5f88\u591a\u5927\u89c4\u6a21\u6570\u636e\u7684\u8ba1\u7b97\u5bf9\u65f6\u95f4\u7684\u8981\u6c42\u975e\u5e38\u5730\u4e25\u683c\u3002\u4e8e\u662f\u8c08\u5230\u4e86\u76ee\u524d\u6d41\u884c\u7684\u7b97\u6cd5\uff1aMapReduce\u3002MapReduce\u662fgoogle\u63d0\u5021\u7684\u4e00\u79cd\u7f16\u7a0b\u6a21\u578b\uff0c\u7528\u4e8e\u5927\u89c4\u6a21\u6570\u636e\u96c6\uff08\u5927\u4e8e1TB\uff09\u7684\u5e76\u884c\u8fd0\u7b97\u3002“Map”\u548c “Reduce”\uff0c\u548c\u4ed6\u4eec\u7684\u4e3b\u8981\u601d\u60f3\uff0c\u90fd\u662f\u4ece\u51fd\u6570\u5f0f\u7f16\u7a0b\u8bed\u8a00\u91cc\u7ee7\u627f\u6765\u7684\uff0c\u8fd8\u6709\u4ece\u77e2\u91cf\u7f16\u7a0b\u8bed\u8a00\u91cc\u501f\u6765\u7684\u7279\u6027\u3002\u4ed6\u6781\u5927\u5730\u65b9\u4fbf\u4e86\u7f16\u7a0b\u4eba\u5458\u5728\u4e0d\u4f1a\u5206\u5e03\u5f0f\u5e76\u884c \u7f16\u7a0b\u7684\u60c5\u51b5\u4e0b\uff0c\u5c06\u81ea\u5df1\u7684\u7a0b\u5e8f\u8fd0\u884c\u5728\u5206\u5e03\u5f0f\u7cfb\u7edf\u4e0a\u3002\u6211\u81ea\u5df1\u7684\u7406\u89e3\u662f\uff1amap\u7d22\u5f15\u6574\u4e2a\u6d77\u91cf\u6570\u636e\uff0c\u751f\u6210key-value pair which is used to facilitate the quick query and computing\u3002\u800creduce\u5219\u662f\u7528\u6765\u5206\u6790\u548c\u7edf\u7b79\u57fa\u4e8e\u952e\u503c\u5bf9\u7684\u4e2d\u95f4\u8ba1\u7b97\u7ed3\u679c\uff0c\u4f7f\u4e2d\u95f4\u7ed3\u679c\u8fc5\u901f\u5411\u6700\u7ec8\u7684\u6b63\u786e\u7ed3\u679c\u6536\u655b\u3002 Ms. Gao \u63d0\u51faMapReduce\u867d\u7136\u662f\u5de5\u4e1a\u754c\u975e\u5e38popular\u7684\u89e3\u51b3\u65b9\u6848\uff0c\u4f46\u662f\u4e0d\u53ef\u4ee5\u4ee5\u8fed\u4ee3\u7684\u65b9\u5f0f\u8ba1\u7b97\u3002\u800cMs. Gao\u6240\u505a\u7684\u5de5\u4f5c\u5373\u662f\u4f7fMapReduce\u652f\u6301\u8fed\u4ee3\u7684\u529f\u80fd\u3002\uff08A framework that support iterative computing in the cloud.\uff09\u4ece\u4e8b\u60c5\u4e00\u822c\u89c4\u5f8b\u7684\u54f2\u5b66\u9ad8\u5ea6\u770b\u5f85\u8fd9\u4e2a\u95ee\u9898\uff0c\u5bf9\u4e8e\u89c4\u6a21\u76f8\u5f53\u5927\u6216\u8005\u6700\u7ec8\u7ed3\u679c\u662f\u76f8\u5f53\u4e0d\u660e\u786e\u7684\u95ee\u9898\uff0c\u4f3c\u4e4e\u90fd\u53ef\u4ee5\u7528\u5230'\u8fed\u4ee3'\u7684\u601d\u60f3\u3002\u4f8b\u5982\uff0c\u8fed\u4ee3\u7684\u8f6f\u4ef6\u5f00\u53d1\u6a21\u5f0f\u5219\u662f\u5f53\u4eca\u4e92\u8054\u7f51\u5e94\u7528\u7684\u4e3b\u8981\u5f00\u53d1\u6a21\u5f0f\u3002\u6bcf\u4e00\u6b21\u8fed\u4ee3\u90fd\u4f7f\u4ea7\u54c1\u5411\u7406\u60f3\u4e2d\u6700perfect\u7684\u72b6\u6001\u63a5\u8fd1\u3002Ms. Gao\u7528\u8fed\u4ee3\u7684\u601d\u60f3\u5904\u7406\u6d77\u91cf\u6570\u636e\u5e94\u8be5\u4e5f\u662f\u540c\u6837\u7684\u9053\u7406\u7f62\uff01 Ms. Gao\u7684\u89e3\u91ca\u8bf4\u6539\u8fdb\u540e\u652f\u6301\u8fed\u4ee3\u7684iMapReduce\u53ef\u4ee5\u4f7fmap\u540e\u7684\u6570\u636e\u652f\u6491reduce,reduce\u7684\u6570\u636e\u4ea6\u8f6c\u800c\u652f\u6491map\uff0c\u5f62\u6210\u4e00\u4e2a\u8fed\u4ee3\u53cd\u590d\u7684\u8fc7\u7a0b\uff0c\u5176\u4e2d\u5728map\u9636\u6bb5\u52a0\u5165static data\u3002\u8f83\u4e4b\u539f\u59cb\u7684MapReduce\uff0c\u65b0\u7684\u7b97\u6cd5\u4f7f\u5f97static data\u7684\u89c4\u6a21\u663e\u8457\u5730\u51cf\u5c11\u3002Ms. Gao\u5c55\u793a\u4e86\u51e0\u4e2a\u56fe\u8868\u8bc1\u660e\u6539\u8fdb\u540e\u589e\u52a0\u4e86\u8fed\u4ee3\u529f\u80fd\u7684MapReduce(\u79f0\u4e4b\u4e3a iMapReduce )\u7684\u901f\u5ea6\u4f18\u4e8e\u539f\u59cb\u7684MapReduce.\u56fe\u8868\u4e2d\u6709\u5c55\u793a\u7528iMapReduce\u8ba1\u7b97PageRank\u7684\u503c\u3002PageRank\uff1f\u662f\u4e0d\u662f\u5224\u65ad\u7f51\u7ad9\u6743\u91cd\u7684\u90a3\u4e2a\u4e1c\u4e1c\u554a\uff1f\u54e5\u7684\u7f51\u7ad9http:\/\/ykyi.net\u7684PageRank\u624d0\u554a\uff01\uff01\uff01\u624d0\u554a~~\u7528\u4ec0\u4e48\u7b97\u6cd5\u53ef\u4ee5\u7b97\u62109\u554a\uff01 PPT\u8f6c\u5230\u8ba1\u7b97\u56fe\u7684\u6307\u5b9a\u4e24\u70b9\u6700\u77ed\u8def\u5f84\u7684Dijkstra\u7b97\u6cd5\u3002\u6cea\u6d41\u6ee1\u9762\uff0c\u7ec8\u4e8e\u6709\u4e00\u4e2a\u6211\u61c2\u7684\u7b97\u6cd5\u4e86\u554a\uff01Ms. Gao\u8bf4\u5148\u627e\u5230\u4e24\u4e2a\u7ed3\u70b9\u95f4\u7684\u6700\u77ed\u8def\u5f84\uff0c\u4e8e\u662f\u8d4b\u4e8e\u8fd9\u4e2a\u6700\u77ed\u8def\u5f84\u7ecf\u8fc7\u7684\u7ed3\u70b9\u6700\u9ad8\u7684\u6743\u91cd\uff0c\u5176\u5b83\u7684\u8def\u5f84\u5c31\u8d4b\u4e8e\u76f8\u5bf9\u4f4e\u7684\u6743\u91cd\u3002\u63d0\u51fa\u4e86PriorityQueue\u7684\u6570\u636e\u7ed3\u6784\u3002\u628a\u6743\u91cd\u5f15\u5165\u5230iMapReduce\u7b97\u6cd5\uff0c\u79f0\u4e4b\u4e3apMapReduce\u7b97\u6cd5\u3002\u5f15\u5165\u6743\u91cd\u540e\u7684pMapReduce\u7b97\u6cd5\u6bd4iMapReduce\u7b97\u6cd5\u7684\u901f\u5ea6\u66f4\u5feb\u3002\u751f\u6d3b\u4e2d\u6211\u4eec\u505a\u4e8b\u60c5\u4e5f\u8981\u4f18\u5148\u5904\u7406\u6700\u91cd\u8981\u7684\u4e8b\u60c5\uff0c\u9053\u7406\u662f\u4e00\u6837\u7684\u554a\uff01\u4e16\u95f4\u4e07\u4e8b\u4e07\u7269\uff0c\u65e0\u8bba\u591a\u4e48\u590d\u6742\uff0c\u62bd\u8c61\u51fa\u6765\u7684\u4e00\u822c\u89c4\u5f8b\u5176\u5b9e\u90fd\u662f\u76f8\u5f53\u5730\u4e00\u81f4\u3002\u6613\u7ecf\u4e2d\u8bf4\uff1a\u5f62\u800c\u4e0b\u8005\u8c13\u4e4b\u5668\uff0c\u5f62\u800c\u4e0a\u8005\u8c13\u4e4b\u9053\uff01Oops, The ancient Tao philosophy is full Of Wisdom. \u6700\u540e\u9636\u6bb5\uff0c\u518d\u6b21\u5c55\u793a\u4e86\u4e00\u4e2a\u56fe\u8868\uff0cMs. Gao\u5b8c\u6210\u7684\u5efa\u7acb\u5728Apache\u7684Hadoop\u4e0a\u7684pMapReduce\u8981\u660e\u663e\u7684\u4f18\u4e8e\u539f\u59cb\u7684Hadoop\u3002\u5e76\u4e14\u4e5f\u4f18\u5148\u5176\u5b83\u591a\u79cd\u89e3\u51b3\u65b9\u6848\uff0c\u5176\u5b83\u7684\u540d\u5b57\u6211\u90fd\u6ca1\u542c\u8fc7\u554a\uff01\u5728\u63d0\u95ee\u9636\u6bb5\uff0c\u5c3d\u7ba1\u6e29\u6b66\u5c11\u8001\u5e08\u79ef\u6781\u9f13\u52b1\u540c\u5b66\u63d0\u95ee\u4f46\u6ca1\u6709\u540c\u5b66\u63d0\u51fa\u95ee\u9898\uff0c\u5927\u6982\u5927\u5bb6\u90fd\u4e0d\u61c2\u5427\uff01\u671d\u7ea2\u9633\u8001\u5e08\u548c\u53e6\u4e00\u4f4d\u7814\u7a76\u6d77\u91cf\u6570\u636e\u7684\u8001\u5e08\u8c08\u4e86\u4e00\u70b9\u70b9\u3002\u4ed6\u4eec\u63d0\u5230iterative\u7684\u7b97\u6cd5\u5728\u4ed6\u4eec\u7684\u5b9e\u9645\u5e94\u7528\u4e2d\u6548\u679c\u5e76\u4e0d\u662f\u592a\u660e\u663e\uff0c\u4e0d\u5982memory-based\u7684\u89e3\u51b3\u65b9\u6848\u3002Ms. Gao\u8c03\u51fa\u4e00\u5f20\u56fe\u8868\uff0c\u89e3\u91ca\u8bf4\u67d0\u79cdmemory-based\u7684\u89e3\u51b3\u65b9\u6848\u6ca1\u6709\u8fed\u4ee3\u7684\u7b97\u6cd5\u5feb\u3002 \u4f5c\u4e3a\u4f17\u591a\u6253\u9171\u6cb9\u7684\u540c\u5b66\u4e2d\u7684\u4e00\u5458\uff0c\u6211\u6700\u540e\u7684\u5fc3\u5f97\u4f53\u4f1a\u662f\uff1a\u5b66\u672f\u771f\u662f\u592a\u9ad8\u6df1\u4e86\u554a\uff01\u6211\u8fd8\u662f\u505a\u5de5\u7a0b\u5427\uff01\uff01\uff01\u8f66\uff0c\u6211\u6709\u5f97\u62e3\u54a9\uff1f \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ Topic Distributed Framework for Iterative Computations on Massive Datasets   Speaker Prof. Lixin Gao   Abstract Iterative algorithms are pervasive in many applications such as search engine algorithms,machine learning, and recommendation systems. These applications typically involve a dataset of massive scale. Fast iterative computations of the … Continue reading “Distributed Framework for Iterative Computations on Massive Datasets”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[96],"class_list":["post-300","post","type-post","status-publish","format-standard","hentry","category-life_and_others","tag-96"],"_links":{"self":[{"href":"https:\/\/ykyi.net\/index.php?rest_route=\/wp\/v2\/posts\/300","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ykyi.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ykyi.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ykyi.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ykyi.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=300"}],"version-history":[{"count":0,"href":"https:\/\/ykyi.net\/index.php?rest_route=\/wp\/v2\/posts\/300\/revisions"}],"wp:attachment":[{"href":"https:\/\/ykyi.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=300"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ykyi.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=300"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ykyi.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=300"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}