A Complexity-Aware Label Based Yarn Scheduler for Cloud Video Transcoding

10.29428/9789860544169.201801.0182

Li-Ying Sung；Yo-Zen Kan；Jiann-Jone Chen；Yao-Hong Tsai

Cloud Video Transcoding ； Task Scheduling ； Hadoop Yarn ； Complexity-Aware Scheduling ； Label-Based Scheduling ； Neural Network ； Hadoop Yarn ； MapReduce job ； Label-Based Scheduling ； Cloud Video Transcoding ； Neural Network ； FFmpeg

NCS 2017 全國計算機會議

2017（2018 / 01 / 01）

973 - 978

繁體中文

近年來雲端多媒體系統應用日趨普遍，然而因用戶的網路環境與裝置不盡相同，因此必須提供轉碼服務來解決異質網路與裝置的問題。為了提升雲端轉碼效率，本論文研究如何運用Yarn來改善雲端動態任務排程。我們設計基於Yarn平台標籤式排程架構(LBCSNN)演算法，根據運算節點的記憶體狀態去分配轉碼工作到相對應的隊列上，給不同使用者獨立排程，並運用CAS演算法，優先排程複雜度較高的任務，避免複雜度較高的任務過度集中在某個工作節點上，並使用機器學習預測影片轉碼時間，以及結合指數平滑推測法，降低整體作業的完成時間。實驗結果顯示，本論文所提出的方法，能讓資源使用率有效的提升，資源使用率能維持在90%以上，並縮短約30%～50%的轉碼時間。

Cloud systems and multimedia devices have been changed our daily life. However, as user devices and network environments are heterogeneous, the cloud system has to perform transcoding in order to provide bandwidth and device compatible media formats. To improve the cloud-based video transcoding efficiency, we proposed to utilize the Yarn model to estimate the processing time for one segment such that the resource scheduling process can be carried out efficiently. We adopted the Complexity-aware Scheduler (CAS) algorithm to monitor worker node processing status and utilize the label-based scheduling to improve the accuracy and efficiency. Experiments showed that using Yarn framework and label-based scheduling method can help to maintain the resource utilization rate above 90% and shorten the overall transcoding time about 30% ~ 50%.