以巨量資料技術Apache HBase建置高效能網路流量分析系統

葉祐欣

資訊安全 ； 網路流量 ； 巨量資料 ； NoSQL ； Apache HBase

資訊安全通訊

20卷4期（2014 / 10 / 01）

105 - 119

繁體中文

近年來巨量資料與雲端運算的議題相當熱門，而在背後推動相關技術持續演進的關鍵腳色，就是開放原始碼的巨量資料平台Apache Hadoop。本文探討在巨量資料的浪潮下，如何利用相關技術解決網路流量分析在效能方面的問題，並詳細介紹建置於Hadoop之上的NoSQL資料庫Apache HBase其具體的設計哲學與架構，透過深入了解HBase運作模式後，提出一個應用於網路流量的即時分析與搜索系統，除了能夠協助資訊安全分析師快速進行網路流量分析外，也能透過與其他資訊安全系統整合進行入侵、異常與回溯式偵測，即時排除組織的資訊安全危害。

1. 樊國楨、韓宜蓁、季祥(2014)。黑暗首爾（Dark Seoul）資訊安全事故（2013-03-20）及其防禦方法初論。資訊安全通訊，20(2)，24-36。
   連結：
2. Apache Flume, “Flume 1.5.0 User Guide,” http://flume.apache.org/FlumeUserGuide.html (2014/7/12).
3. Apache HBase, “Rowkey Design,” https://hbase.apache.org/book/rowkey.design.html (2014/7/12).
4. K. Zetter, “Google Hack Attack Was Ultra Sophisticated, New Details Show.” http://www.wired.com/2010/01/operation-aurora/ (2013/9/9).
5. J. Quittek, T. Zseby, B. Claise, and S. Zander, “Requirements for IP Flow Information Export (IPFIX),” RFC 3917 (Informational), http://www.ietf.org/rfc/rfc3917.txt (2014/7/12).
6. Apache HBase, “Secondary Indexes and Alternate Query Paths,”https://hbase.apache.org/book/secondary.indexes.html (2014/7/12).
7. Apache Flume, “Flume NG refactoring,”https://issues.apache.org/jira/browse/FLUME-728 (2014/7/12).
8. B. Claise, “Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information,” RFC 5101 (Proposed Standard), http://www.ietf.org/rfc/rfc5101.txt (2014/7/12).
9. Brewer, E. A.(2000).Towards Robust Distributed Systems.Symposium on Principles of Distributed Computing (PODC)
10. Chang, F.,Dean, J.,Ghemawat, S.,Hsieh, W. C.,Wallach, D. A.,Burrows, M.,Chandra, T.,Fikes, A.,Gruber, R. E.(2008).Bigtable: A Distributed Storage System for Structured Data.ACM Trans. Comput. Syst.,26(2),1-26.
11. Cooke, E.,Jahanian, F.,Mcpherson, D.(2005).The Zombie Roundup: Understanding, Detecting, and Disrupting Botnets.Workshop on Steps to Reducing Unwanted Traffic on the Internet (SRUTI)
12. Dean, J.,Ghemawat, S.(2008).MapReduce: Simplified Data Processing on Large Clusters.Commun. ACM,51(1),107-113.
13. Gao, M.,Zhang, K.,Lu, J.(2006).Efficient packet matching for gigabit network intrusion detection using TCAMs.Proc. of 20th International Conferece on Advanced Information Networking and Applications (AINA'06)
14. Ghemawat, S.,Gobioff, H.,Leung, S.-T.(2003).The Google File System.Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles
15. Gu, G.,Perdisci, R.,Zhang, J.,Lee, W.(2008).Botminer: Clustering analysis of network traffic for protocol-and structure-independent botnet detection.Proc. of 17th USENIX Security Symposium (USENIX Security'08)
16. Juels, A.,Yen, T.-F.(2012).Sherlock Holmes and the Case of the Advanced Persistent Threat.Proceedings of the 5th USENIX Conference on Large-Scale Exploits and Emergent Threats
17. Karasaridis, A.,Rexroad, B.,Hoeflin, D.(2007).Wide-scale botnet detection and characterization.Proc.of the first conference on First Workshop on Hot Topics in Understanding Botnets (HotBots'07)
18. Lai, H.,Cai, S.,Huang, H.,Xie, J.,Li, H.(2004).A parallel intrusion detection system for high-speed networks.Proc. of the Second International Conference Applied Cryptography and Network Security (ACNS'04)
19. Lazarevic, A.,Kumar, V.,Srivastava, J.(2005).Intrusion detection: A survey.Managing Cyber Threats
20. Lee, M.,Shon, T.,Cho, K.,Chung, M.,Seo, J.,Moon, J.(2007).An approach for classifying internet worms based on temporal behaviors and packet flows.Proc. of 3rd Int. Conf. on Intelligent Computing (ICIC 2007)
21. Morin, B.,Mé, L.(2007).Intrusion detection and virology: an analysis of differences, similarities and complementariness.Journal in Computer Virology,3,39-49.
22. Song, S.,Chen, Z.(2007).Adaptive network flow clustering.IEEE In-ternational Conference on Networking, Sensing and Control (ICNSC07)
23. Sperotto, A.,Schaffrath, G.,Sadre, R.,Morariu, C.,Pras, A.,Stiller, B.(2010).An overview of IP flow-based intrusion detection.IEEE Commun. Surv. Tutorials,12(3),343-356.
24. Strayer, W.,Lapsely, D.,Walsh, R.,Livadas, C.(2008).Botnet detection based on network behavior.Botnet Detection
25. Taleb, T.,Fadlullah, Z. M.,Hashimoto, K.,Nemoto, Y.,Kato, N.(2007).Tracing back attacks against encrypted protocols.Proc. of the 2007 international conference on Wireless communications and mobile computing (IWCMC'07)
26. Thonnard, O.,Bilge, L.,O'Gorman, G.,Kiernan, S.,Lee, M.(2012).Industrial Espionage and Targeted Attacks: Understanding the Characteristics of an Escalating Threat Research in Attacks, Intrusions, and Defenses.Lecture Notes in Comupter Science,7462,64-85.
27. Wagner, A.,D¨ubendorfer, T.,Plattner, B.,Hiestand, R.(2003).Experiences with worm propagation simulations.Proc. of 2003 ACM workshop on Rapid malcode (WORM'03)
28. Zou, C.,Gong, W.,Towsley, D.(2002).Code red worm propagation modeling and analysis.Proc. of 17th USENIX Security Symposium (USENIX Security'08)

1. 陳立邦(2017)。以平行基因演算法於Hadoop平台上建立投資組合。中原大學資訊管理學系學位論文。2017。1-52。