基於Apache Spark異構叢集系統任務排程優化之研究

吳秉諭

山西大学，您好！

感谢采购，人社期刊库｜人社论文库，期间：2025/01/01~2025/12/31

感谢试用，科学期刊库｜科学论文库，期间：2024/10/20~2025/01/20

浏览高级检索

关闭
检索检索高级检索出版物检索
浏览期刊主题浏览核心期刊浏览期刊拼音浏览

关闭

查询符号(半角)	查询符号意义说明
空格	表示为「AND」两个查询词之交集
双引号 ( " " )	片语以双引号标示开始及结束，而且只寻找出现顺序相同的字词，例： " image process "
?	表示一个字母切截，输入两个?表两个字母，依此类推，例：输入「Appl?」，查得结果应为appl e , appl y … ( 常用于英文字查询 )
*	表示不限字母切截，由0~n. 例：输入「appl*」，查得结果应为appl e , appl es , appl y , appl ied , appl ication … ( 常用于英文字查询 )
AND、OR、NOT	布尔逻辑组合关键字，用来扩大或缩小查询范围的技巧。 (1) AND ：缩小查询范围 (2) OR ：扩大查询范围(3) NOT：排除不相关的范围

DOI 是数位物件识别码（ D igital O bject I dentifier ）的简称，
为物件在网路上的唯一识别码，可用于永久连结并引用目标物件。

使用DOI作为永久连结

每个DOI号前面加上「 http://dx.doi.org/ 」便成为永久网址。
如以DOI号为 10.5297/ser.1201.002 的文献为例，此文献的永久连结便是： http://dx.doi.org/ 10.5297/ser.1201.002 。
日后不论出版单位如何更动此文献位置，永久连结所指向的位置皆会即时更新，不再错失重要的研究。

引用含有DOI的文献

有DOI的文献在引用时皆应同时引用DOI。若使用APA、Chicago以外未规范DOI的引用格式，可引用DOI永久连结。

DOI可强化引用精确性、增强学术圈连结，并给予使用者跨平台的良好使用经验，目前在全世界已有超过五千万个对象申请DOI。如想对DOI的使用与概念有进一步了解，请参考（）。

书目管理工具

相关连结

问题回报

全文下载

基於Apache Spark异构丛集系统任务排程优化之研究

The Study of Task Scheduling Optimization on Apache Spark Heterogeneous Cluster System

吴秉谕 , 硕士　　导师：陈弘明　　共同指导教授：陈世颖

繁体中文

Apache Spark ；云端运算；巨量资料；调度策略；任务排程调度策略；异构丛集； Apache Spark ； Cloud Computing ； Big Data ； Dispatching Strategy ； Task Scheduling and Dispatching strategy ； Heterogenic Clusters

0 个人觉得这篇文章赞

摘要 │ 参考文献 (94) │ 国际计量

摘要〈TOP〉

并列摘要〈TOP〉

As the hardware equipment is continue to be upgraded, the hardware resources of the company data center are constantly renewed and replaced. Consequently, newly added nodes cause the cluster nodes to express heterogeneity. However, due to cluster heterogeneity, the processing abilities of the each node are different. As a result, when a task is assigned to different nodes, it affects the loading of each node differently. In addition, heterogeneity causes issues in tasks themselves as well. For example: When assigning nodes, one should consider the need for different resources of CPU intensives and I/O intensives. Take the Apache Spark data analytic framework as an example, the current Spark does not take heterogeneity nor resource utilization of cluster nodes into consideration. Therefore, the nodes of each cluster demonstrate uneven loading when they are performing tasks. This causes partial system overloading and resource depletion, and limits the overall efficiency to the lesser capable nodes. As a result, overall computational performance drops. In order to counteract the problems discussed above, this study suggests a new scheduling strategy that can optimize Spark’s performance in relation to heterogeneos clusters. This study proposes a new hierarchical scheduling strategy that first divides nodes with similar calculating abilities into groups. During this process, test assignments are used to assess preliminary executing time. Then, historical data and machine learning techniques are used to further accurately estimate the execution time. Finally, with the strategy explained above, a more efficient task scheduling algorithm are proposed and implemented.

参考文献 ( 94 ) 〈TOP〉

[1] Jo, M., Maksymyuk, T., Strykhalyuk, B., & Cho, C. H. (2015). Device-to-device-based heterogeneous radio access network architecture for mobile cloud computing. IEEE Wireless Communications, 22(3), 50-58.
连结：
[2] Beloglazov, A., Buyya, R., Lee, Y. C., & Zomaya, A. (2011). A taxonomy and survey of energy-efficient data centers and cloud computing systems. Advances in computers, 82(2), 47-111.
连结：
[5] Vasile, M. A., Pop, F., Tutueanu, R. I., Cristea, V., & Kołodziej, J. (2015). Resource-aware hybrid scheduling algorithm in heterogeneous distributed computing. Future Generation Computer Systems, 51, 61-71.
连结：
[6] Di Nitto, E., Ghezzi, C., Metzger, A., Papazoglou, M., & Pohl, K. (2008). A journey to highly dynamic, self-adaptive service-based applications. Automated Software Engineering, 15(3), 313-341.
连结：
[7] Xu, X., Cao, L., & Wang, X. (2016). Adaptive task scheduling strategy based on dynamic workload adjustment for heterogeneous Hadoop clusters. IEEE Systems Journal, 10(2), 471-482.
连结：

国际计量

〈TOP〉

E-mail ：

文章公开取用时，将寄通知信至您填写的信箱地址

E-mail ：

使用DOI作为永久连结

引用含有DOI的文献

DOI可强化引用精确性、增强学术圈连结，并给予使用者跨平台的良好使用经验，目前在全世界已有超过五千万个对象申请DOI。 如想对DOI的使用与概念有进一步了解，请参考 （ ） 。

DOI可强化引用精确性、增强学术圈连结，并给予使用者跨平台的良好使用经验，目前在全世界已有超过五千万个对象申请DOI。如想对DOI的使用与概念有进一步了解，请参考（）。