基於OpenStack雲端系統結合Ceph軟體定義分散式儲存系統效能優化之研究

A Study on the Performance Optimation for Ceph Software Defined Storage System based on OpenStack Cloud System

白凱仁

OpenStack ； Ceph ； 軟體定義儲存 ； 雲端運算 ； 分層快取 ； OpenStack ； Ceph ； Software-defined Storage ； Cache Tiering ； Cloud Computing

臺中科技大學資訊工程系碩士班學位論文

2016年

碩士

陳弘明

繁體中文

近年來資料量暴增、雲端運算技術普及與物聯網應用帶動下，資料儲存的處理需求足年倍增，導致許多組織與企業紛紛建置資料中心。由於傳統的伺服器架構限制，資源通常難以有效地利用，而導致研究的效能損失、缺乏資料安全性與可靠性。有鑑於此，基於軟體定義式基礎架構的發展被紛紛提出，其中以開源碼OpenStack雲端系統最受企業關注，其提供了基於軟體定義式的運算、網路與儲存等模型，使資源經過虛擬化方便以軟體進行彈性分配與使用。然而，OpenStack中虛擬化儲存服務預設下缺乏統一的儲存後台，造成儲存系統缺乏管理性與彈性，同時自動容錯與擴展性也受到挑戰，因此，許多企業採用了開源碼Ceph作為OpenStack儲存統一後台。本研究提出一基於OpenStack雲端系統結合Ceph軟體定義分散式儲存系統效能優化方法，針對Ceph儲存系統進行混合儲存放置規則設計與分層快取儲存優化建置，並搭配Linux核心快取技術優化儲存系統，提供一高效能儲存服務解決方案。本研究基於論文提出之三種優化方法進行組合觀察效能差異，並且透過不同混合儲存放置規則等，可能影響儲存效能之相關因素進行實驗，藉此評估與分析最佳的參數設定與優化方法組合，實驗結果顯示建立與設計一混合儲存放置規則，將對儲存效能進行最佳的分配與利用。

In recent years, due to explosive increase in data volume, popularization of cloud computing and the need of internet of things, the processing demand of data storage was in double increase year by year, which led to higher and higher complication in the data centers built by many organizations and enterprises. Due to the limitation of traditional server architecture, the resources were usually difficult to be effectively used, which led to serious effectiveness loss and the lack of data security and reliability. Therefore, to deal with such issue, development based on software-defined basic architecture was proposed one after another, among them, open source OpenStack cloud system caught the greatest attention from the enterprise. However, in OpenStack, the defaulted virtualized storage service was lack of universal storage platform, which led to the lack of managerial characteristic and flexibility in the storage system, meanwhile, indexes such as automatic fault tolerance and ease of expandability were also challenged, therefore, to cope with such issue, open source Ceph is adopted as the storage system of OpenStack. In this study, methods for optimization of the performance of OpenStack cloud system in association with Ceph software-defined distributed storage system were proposed to make storage placement rule design and layered cache storage optimization setup on Ceph storage system. This study has made further investigation and evaluation on Ceph distributed storage system, three optimization methods were proposed in this study to observe the effectiveness differences through different combinations, meanwhile, experiments were conducted on related factors that might possibly affect the storage effectiveness such as different mixed storage placement rule so as to evaluate and analyze optimized parameter setup and optimized method combination. Experimental result showed the proposed storage optimization techniques will make the system to achieve optimized storage effectively.

1. [1] Mell, Peter, and Tim Grance. "The NIST definition of cloud computing." (2011).
   連結：
2. [2] Subashini, Subashini, and Veeraruna Kavitha. "A survey on security issues in service delivery models of cloud computing." Journal of network and computer applications 34.1 (2011): 1-11.
   連結：
3. [3] MLA Zhang, Qi, Lu Cheng, and Raouf Boutaba. "Cloud computing: state-of-the-art and research challenges." Journal of internet services and applications 1.1 (2010): 7-18.
   連結：
4. [4] 鐘嘉德、高天助、楊嘉栩(2010)：雲端運算與產業發展。研考雙月刊，34(4)，20-31.
   連結：
5. [8] Yang, Chao-Tung, et al. "Implementation of a software-defined storage service with heterogeneous storage technologies." Advanced Information Networking and Applications Workshops (WAINA), 2015 IEEE 29th International Conference on. IEEE, 2015.
   連結：
6. [9] Sefraoui, Omar, Mohammed Aissaoui, and Mohsine Eleuldj. "OpenStack: toward an open-source solution for cloud computing." International Journal of Computer Applications 55.3 (2012).
   連結：
7. [10] MLA Lordan, Francesc, et al. "Servicess: An interoperable programming framework for the cloud." Journal of grid computing 12.1 (2014): 67-91.
   連結：
8. [13] 蔡權昱, and 蔡錫鈞. NCTU CStack: OpenStack 與 Ceph 的整合與應用. Diss. 2013.
   連結：
9. [15] Wang, Feiyi, et al. "Performance and scalability evaluation of the Ceph parallel file system." Proceedings of the 8th Parallel Data Storage Workshop. ACM, 2013.
   連結：
10. [16] Song, Yang, Rakesh Jain, and Ramani Routray. "Queueing-based storage performance modeling and placement in OpenStack environments." 2014 21st International Conference on High Performance Computing (HiPC). IEEE, 2014.
    連結：
11. [18] Labrinidis, Alexandros, and Hosagrahar V. Jagadish. "Challenges and opportunities with big data." Proceedings of the VLDB Endowment 5.12 (2012): 2032-2033.
    連結：
12. [20] MLA Gracia-Tinedo, Raúl, et al. "IOStack: Software-Defined Object Storage." IEEE Internet Computing 20.3 (2016): 10-18.
    連結：
13. [21] Alba, Alfredo, et al. "Efficient and agile storage management in software defined environments." IBM Journal of Research and Development 58.2/3 (2014): 5-1.
    連結：
14. [22] Jararweh, Yaser, et al. "Software defined cloud: Survey, system and evaluation." Future Generation Computer Systems 58 (2016): 56-74.
    連結：
15. [23] Dykstra, Josiah, and Alan T. Sherman. "Design and implementation of FROST: Digital forensic tools for the OpenStack cloud computing platform." Digital Investigation 10 (2013): S87-S95.
    連結：
16. [24] MLA Nakanishi, Hideya, et al. "Revised cloud storage structure for light-weight data archiving in LHD." Fusion Engineering and Design 89.5 (2014): 707-711.
    連結：
17. [28] van der Ster, Daniel C., et al. "Ceph-based storage services for Run2 and beyond." Journal of Physics: Conference Series. Vol. 664. No. 4. IOP Publishing, 2015.
    連結：
18. [31] Yamato, Yoji. "Use case study of HDD-SSD hybrid storage, distributed storage and HDD storage on OpenStack." Proceedings of the 19th International Database Engineering & Applications Symposium. ACM, 2015.
    連結：
19. [34] Emaru, Hironori, and Yoshiaki Takai. "Performance Management for the Dynamic Storage Tiering by Virtual Volume Clustering." Utility and Cloud Computing (UCC), 2011 Fourth IEEE International Conference on. IEEE, 2011.
    連結：
20. [35] Armbrust, Michael, et al. "A view of cloud computing." Communications of the ACM 53.4 (2010): 50-58. APA
    連結：
21. [36] 王耀聰, and 陳威宇. "雲端運算簡介." 國家高速網路與計算中心.(2008).
    連結：
22. [39] Howard, John H., et al. "Scale and performance in a distributed file system." ACM Transactions on Computer Systems (TOCS) 6.1 (1988): 51-81.
    連結：
23. [40] Chao, Hsiang-Ching, et al. "A seamless and reliable distributed network file system utilizing webspace." Web Site Evolution, 2008. WSE 2008. 10th International Symposium on. IEEE, 2008.
    連結：
24. [41] Chih-Tuan. C., Chao-Tung Y., "Performance Comparison of Hadoop Distributed File System and Ceph",Tunghai University,2014.
    連結：
25. [44] Konstantin Shvachko , Hairong Kuang , Sanjay Radia , Robert Chansler, "The Hadoop Distributed File System", Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), p.1-10, May 03-07, 2010.
    連結：
26. [47] Seshadri, Sangeetha, et al. "Software defined just-in-time caching in an enterprise storage system." IBM Journal of Research and Development 58.2/3 (2014): 7-1.
    連結：
27. [48] Ouyang, Jian, et al. "SDF: software-defined flash for web-scale internet storage systems." ACM SIGPLAN Notices 49.4 (2014): 471-484.
    連結：
28. [49] Breiter, Gerd, et al. "Software defined environments based on TOSCA in IBM cloud implementations." IBM Journal of Research and Development 58.2/3 (2014): 9-1.
    連結：
29. [50] Li, C-S., et al. "Software defined environments: An introduction." IBM Journal of Research and Development 58.2/3 (2014): 1-1.
    連結：
30. [53] Rosado, Tiago, and Jorge Bernardino. "An overview of openstack architecture." Proceedings of the 18th International Database Engineering & Applications Symposium. ACM, 2014.
    連結：
31. [56] Robles, Gregorio, et al. "Estimating development effort in free/open source software projects by mining software repositories: a case study of openstack." Proceedings of the 11th Working Conference on Mining Software Repositories. ACM, 2014.
    連結：
32. [57] Wen, Xiaolong, et al. "Comparison of open-source cloud management platforms: OpenStack and OpenNebula." Fuzzy Systems and Knowledge Discovery (FSKD), 2012 9th International Conference on. IEEE, 2012.
    連結：
33. [59] Bist, Meenakshi, Manoj Wariya, and Amit Agarwal. "Comparing delta, open stack and Xen Cloud Platforms: A survey on open source IaaS." Advance Computing Conference (IACC), 2013 IEEE 3rd International. IEEE, 2013.
    連結：
34. [63] Titcheu Chekam, Thierry, et al. "On the Synchronization Bottleneck of OpenStack Swift-like Cloud Storage Systems." IEEE International Conference on Computer Communications, San Francisco, CA 10-15 April 2016. IEEE Xplore®, 2016.
    連結：
35. [64] Corradi, Antonio, Mario Fanelli, and Luca Foschini. "VM consolidation: A real case based on OpenStack Cloud." Future Generation Computer Systems 32 (2014): 118-127.
    連結：
36. [70] Weil, Sage A., et al. "CRUSH: Controlled, scalable, decentralized placement of replicated data." Proceedings of the 2006 ACM/IEEE conference on Supercomputing. ACM, 2006.
    連結：
37. [73] Avilés-González, Ana, Juan Piernas, and Pilar González-Férez. "Scalable Metadata Management Through OSD+ Devices." International Journal of Parallel Programming 42.1 (2014): 4-29.
    連結：
38. [79] Guruswami, Venkatesan, and Madhu Sudan. "Improved decoding of Reed-Solomon and algebraic-geometric codes." Foundations of Computer Science, 1998. Proceedings. 39th Annual Symposium on. IEEE, 1998.
    連結：
39. [80] MacKay, David JC. "Fountain codes." IEE Proceedings-Communications 152.6 (2005): 1062-1068. APA
    連結：
40. [82] Srinivasan, Mohan, Paul Saab, and V. Tkachenko. "Flashcache." (2010).
    連結：
41. [85] Absar, Javed, and Francky Catthoor. "Analysis of scratch-pad and data-cache performance using statistical methods." Asia and South Pacific Conference on Design Automation, 2006.. IEEE, 2006.
    連結：
42. [88] Zheng, Qing, et al. "Cosbench: A benchmark tool for cloud object storage services." Cloud Computing (CLOUD), 2012 IEEE 5th International Conference on. IEEE, 2012.
    連結：
43. 參考文獻
44. [5] iThome網路原生報,"VMware 2014新戰略：從軟體定義資料中心邁向軟體定義企業"，http://www.ithome.com.tw/news/91278.
45. [6] iThome網路原生報, "突破傳統儲存架構困境，軟體定義式儲存走向開放、統合，帶動應用創新", http://www.ithome.com.tw/tech/90713.
46. [7] Carlson, Mark, et al. "Software defined storage." Storage Networking Industry Assoc. working draft, Apr (2014).
47. [11] Weil, Sage A., et al. "Ceph: A scalable, high-performance distributed file system." Proceedings of the 7th symposium on Operating systems design and implementation. USENIX Association, 2006.
48. [12] Dreyer, Malte, Jens Döbler, and Daniel Rohde. "Building Service Platforms using OpenStack and CEPH: A University Cloud at Humboldt University." EUNIS Journal of Higher Education (2015).
49. [14] van der Ster, Daniel, and Arne Wiebalck. "Building an organic block storage service at CERN with Ceph." Journal of Physics: Conference Series. Vol. 513. No. 4. IOP Publishing, 2014. APA
50. [17] Chen, Hsinchun, Roger HL Chiang, and Veda C. Storey. "Business Intelligence and Analytics: From Big Data to Big Impact." MIS quarterly 36.4 (2012): 1165-1188.
51. [19] Thereska, Eno, et al. "IOFlow: a software-defined storage architecture." Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles. ACM, 2013.
52. [25] MLA Espinal, Xavier, et al. "Disk storage at CERN: Handling LHC data and beyond." Journal of Physics: Conference Series. Vol. 513. No. 4. IOP Publishing, 2014.
53. [26] Presti, G. Lo, et al. "Streamlining CASTOR to manage the LHC data torrent." Journal of Physics: Conference Series. Vol. 513. No. 4. IOP Publishing, 2014.
54. [27] Mascetti, Luca, et al. "Disk storage at CERN." Journal of Physics: Conference Series. Vol. 664. No. 4. IOP Publishing, 2015.
55. [29] Zhang, X., S. Gaddam, and A. T. Chronopoulos. "Ceph Distributed File System Benchmarks on an Openstack Cloud." 2015 IEEE International Conference on Cloud Computing in Emerging Markets (CCEM). IEEE, 2015.
56. [30] MLA Arteaga, Dulcardo, et al. "A trace-driven analysis of solid-state caching in cloud computing systems."
57. [32] MLA Wang, Jiangtao, Wenyu Lai, and Xiaofeng Meng. "SFCM: A SSD-Friendly Cache Management Policy for Hybrid Storage Systems." International Conference on Web-Age Information Management. Springer Berlin Heidelberg, 2013.
58. [33] Meng, Fei, et al. "vCacheShare: automated server flash cache space management in a virtualization environment." 2014 USENIX Annual Technical Conference (USENIX ATC 14). 2014.
59. [37] 黃重憲, 所謂, and 所以說. "淺談雲端運算 (Cloud Computing)." 國立台灣大學計算機與資訊網路中心, Mar 20 (2009).
60. [38] 江政哲, and 張迺貞. "初探雲端運算." A Preliminary Study on Cloud Computing, 國立臺灣師範大學 (2010).
61. [42] Biardzki, Christoph, and Thomas Ludwig. "Analyzing metadata performance in distributed file systems." International Conference on Parallel Computing Technologies. Springer Berlin Heidelberg, 2009.
62. [43] Bai, Songlin, and Hao Wu. "The performance study on several distributed file systems." 2011 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery. 2011. APA
63. [45] Ghemawat, Sanjay, Howard Gobioff, and Shun-Tak Leung. "The Google file system." ACM SIGOPS operating systems review. Vol. 37. No. 5. ACM, 2003.
64. [46] Carlson, Mark, et al. "Software defined storage." Storage Networking Industry Assoc. working draft, Apr (2014).
65. [51] Huang, Ming-Jen, Chun-Fang Huang, and Wen-Shyen Eric Chen. "Architecting a Software-Defined Storage Platform for Cloud Storage Service." Services Computing (SCC), 2015 IEEE International Conference on. IEEE, 2015.
66. [52] OpenStack. "Open source software for building private and public cloud." 2014-07-19]. https://www. openstack. org (2013).
67. [54] Venzano, Daniele, and Pietro Michiardi. "A measurement study of data-intensive network traffic patterns in a private cloud." Proceedings of the 2013 IEEE/ACM 6th International Conference on Utility and Cloud Computing. IEEE Computer Society, 2013.
68. [55] MLA Pettit, Justin, and Eric Lopez. "OpenStack: OVS Deep Dive." (2013).
69. [58] Yadav, Sonali. "Comparative study on open source software for cloud computing platform: Eucalyptus, OpenStack and OpenNebula." International Journal Of Engineering And Science 3.10 (2013): 51-54.
70. [60] Kumar, Rakesh, et al. "Open source solution for cloud computing platform using OpenStack." International Journal of Computer Science and Mobile Computing 3.5 (2014): 89-98.
71. [61] OpenStack Foundation,"2016 OpenStack User Survey".
72. [62] Fifield, Tom, et al. OpenStack Operations Guide. " O'Reilly Media, Inc.", 2014.
73. [65] MLA Gao, Xiaoming, et al. "Building a distributed block storage system for cloud infrastructure." Cloud Computing Technology and Science (CloudCom), 2010 IEEE Second International Conference on. IEEE, 2010.
74. [66] MLA Arnold, Joe. OpenStack Swift: Using, Administering, and Developing for Swift Object Storage. " O'Reilly Media, Inc.", 2014.
75. [67] Weil, Sage A. Ceph: reliable, scalable, and high-performance distributed storage. Diss. UNIVERSITY OF CALIFORNIA SANTA CRUZ, 2007.
76. [68] Weil, Sage A., et al. "Dynamic metadata management for petabyte-scale file systems." Proceedings of the 2004 ACM/IEEE conference on Supercomputing. IEEE Computer Society, 2004.
77. [69] Weil, Sage A., et al. "Rados: a scalable, reliable storage service for petabyte-scale storage clusters." Proceedings of the 2nd international workshop on Petascale data storage: held in conjunction with Supercomputing'07. ACM, 2007.
78. [71] Lamport, Leslie. "Paxos made simple." ACM Sigact News 32.4 (2001): 18-25.
79. [72] Chandra, Tushar D., Robert Griesemer, and Joshua Redstone. "Paxos made live: an engineering perspective." Proceedings of the twenty-sixth annual ACM symposium on Principles of distributed computing. ACM, 2007.
80. [74] Singh, Karan. Learning Ceph. Packt Publishing Ltd, 2015.
81. [75] EMC Corporation Communities,"The case for tiered storage in the enterprise"
82. [76] Ceph Storage Community Resources，"Ceph Storage Cluster Erasure code Documentation"
83. [77] Lin, W. K., Dah Ming Chiu, and Y. B. Lee. "Erasure Code Replication Revisited." Peer-to-Peer Computing. 2004. APA
84. [78] Wicker, Stephen B., and Vijay K. Bhargava. Reed-Solomon codes and their applications. John Wiley & Sons, 1999.
85. [81] Papailiopoulos, Dimitris S., and Alexandros G. Dimakis. "Locally repairable codes." IEEE Transactions on Information Theory 60.10 (2014): 5843-5855.
86. [83] Mituzas, Domas. "Flashcache at Facebook: From 2010 to 2013 and beyond." (2014).
87. [84] Ming Zhao, "Generic block-level cache utility based on Linux device mapper framework", https://github.com/mingzhao/dm-cache.
88. [86] The Linux Kernel Archives, " Bcache Documentation", https://www.kernel.org/doc/Documentation/bcache.txt.
89. [87] Axboe, Jens. "Flexible i/o tester." Freshmeat project website (2011).
90. Ceph Community Resources，"rados – rados object storage utility Documentation", http://docs.ceph.com/docs/hammer/man/8/rados/.