支援混合式巨量資料整合管理框架

A Framework for Hyper Big Data Integration and Management

張志宏(Chih-Hung Chang)；施志承(Jhih-Cheng Shih)；朱正忠(William C. Chu)

雲端計算 ； 巨量資料 ； 資訊倉儲 ； 資料整台 ； Cloud Computing ； Big Data ； Information Warehouse ； Data Integration

修平學報

32期（2016 / 03 / 01）

65 - 84

繁體中文

巨量資料應用的特性無非是4V，也就是資料巨量性(Volume)、資料多樣性(Variety)、成長速度(Velocity)與資料準確性(Veracity)。雖然雲端環境具備大量運算資源，但要在大量資料中，快速的進行巨量資料處理，備妥必要的工具、程序、方法及流程，設法增加開採速度，使得巨量資料進一步成為資料服務平台變成為一大挑戰，因此1)具備大量資料儲存與分析能力之雲端計算核心平台、2)高速平行資料處理技術、3)高吞吐量的資料處理和雲端傳輸、與4)高價值資料的分析與萃取，便成巨量資料處理與資訊倉儲服務平台不可或缺的四項要素。在本論文中，我們建立一個異質巨量資料整台處理與資訊倉儲框架，提供更精確有效具整台能力之的雲端資訊倉儲服務平台，讓SaaS服務開發者進行後續加值應用。

While dealing with the issues of Big Data, there are 4V flipped on the page, Volume, Variety, Velocity, and Veracity. Although cloud computing has already embraced capabilities with a large number of computation resources, problems with topics of friendly tools, effective processes, methodologies, working flows, efficient data mining, etc., still exist and make Big Data become the major barrier when trying to lift up cloud data service platform to the usage of high availability and efficiency to the enterprise. Hence, four characteristics should be revealed while we trying to apply for Heterogeneous Big Data processing and Information warehouse service platforms, which are (1) a core cloud computing platform for large amount data storage and analysis; (2) technologies for high speed parallel data processing; (3) mechanisms for high throughput cloud computing data transformation; and (4) mechanisms for high-value data analysis and extraction. In this paper a cloud data warehouse management framework for Heterogeneous Big Data integration will be proposed. The framework aims at a well-integrated cloud Information warehouse service platform to more accurate and efficient data service, and then leave feasibility and extension to relative SaaS application explorers expectantly.

1. Andrews, G.R.(2000).Foundations of Multithreaded, Parallel, and Distributed Programming.Addison–Wesley.
2. Chu, C.W.,Chang, C.H.,Lu, C.W.,Yang, H.,Jiau, H.C.,Chung, Y.C.,Qiao, B.(2003).Enhancing Software Maintainability by Unifying and Integrating Standards.Advances in Software Maintenance Management: Technologies and Solutions,Hershey PA:
3. Chu, C.W.,Lu, C.W.,Chen, J.N.,Chang, C.H.,Yang, C.T.,Hsiung, P.A.,Lee, H.M.(2012).Cloud Computing in Taiwan.IEEE Computer,45(5),48-56.
4. Lakshmi, N.V.,Rani, K.S.(2012).Privacy Preserving Association Rule Mining in Vertically Partitioned Databases.International Journal of Computer Applications,39(13),29-35.
5. Lith, A.,Mattson, J.(2010).Göteborg,Department of Computer Science and Engineering, Chalmers University of Technology.
6. Lu, C.W.,Chu, C.W.,Chang, C.H.,Lian, W.D.,Yang, D.L.(2003).Integrating Divers Paradigms in Evolution and Maintenance by an XML-based Unified Model.Journal of Software Maintenance and Evolution,15(3),111-144.
7. Peng, S.,Xu, Q.,Ling, X.B.,Peng, X.,Du, W.,Chen, L.(2003).Molecular classification of cancer types from microarray data using the combination of genetic algorithms and support vector machines.FEBS letters,555,358-362.
8. Sanjeev, A.,Boaz, B.(2009).Computational Complexity - A Modern Approach.Cambridge.
9. Tsang, I.V.,Kwok, J.T.,Cheung, P.M..Core Vector Machines: Fast SVM Training on Very Large Data Sets.Journal of Machine Learning Research,6,363-392.
10. White, T.(2012).Hadoop: The Definitive Guide.O'Reilly Media.
11. 董飛 （2015）。後Hadoop時代的大數據架構，[On-line].Available: http://www.yixiaoer.com/?p=2648
12. 道格. (2013, Dec.). " NoSQL 憑什麼成功?". [On-line].Available: http://news.networkmagazine.com.tw/magazine/2013/12/11/60643/