DOI: 10.3724/SP.J.1001.2009.00138

Journal of Software (软件学报) 2009/20:1 PP.138-151

Context Sensitive Host-Based IDS Using Hybrid Automaton

A key function of a host-based intrusion detection system is to monitor program execution. Models constructed based on static analysis have the advantage of not producing false alarms; still, they can not be put into practice due to imprecision or inefficiency of the model. The prior work has shown a trade-off between efficiency and precision. In particular, models based upon non-deterministic finite state automaton (DFA) are efficient but lack precision. More accurate models based upon pushdown automaton (PDA) are very inefficient to operate due to non-determinism in stack activity. DYCK model, VPStatic model and IMA use some subtle approaches to achieve more determinism by extracting information about stack activity of the program or inserting code to expose program state or just inline the local automaton but still can not solve the problem of indirect call/JMP. This paper presents a new training-free model (hybrid finite automaton, HFA) to gain more determinism and resolves indirect call/JMP through static-dynamic hybrid approach. The results show that in run-time, these models slowed the execution of the test programs by 5% to 10%. This paper also formally compares HFA with some typical models and proves that HFA is more accurate than others and it is more suitable for dynamic linked applications. Some technical details of the protocol type system on Linux and experimental results are also presented in the paper.

Key words:intrusion detection,hybrid automaton,training-free,call context sensitive,Linux

ReleaseDate:2014-07-21 14:29:35

Funds:Supported by the National Natural Science Foundation of China under Grant Nos.60403006, 60503046 (国家自然科学基金)

[1] Denning D. An intrusion detection model. IEEE Trans. on Software Engineering, 1987,13(2):222-232.

[2] Forrest S. A sense of self for UNIX processes. In: Proc. of the IEEE Symp. on Security and Privacy. Oakland: IEEE Press, 1996. 120-128.

[3] Hofmeyr SA, Forrest S, Somayaji A. Intrusion detection using sequences of system calls. Journal of Computer Security, 1998, 6(3):151-180.

[4] Helman P, Bhangoo J. A statistically based system for prioritizing information exploration under uncertainty. IEEE Trans. on Systems, Man and Cybernetics, Part A: Systems and Humans, 1997,27(4):449-466.

[5] Lee W, Stolfo SJ. Data mining approaches for intrusion detection. In: Proc. of the 7th USENIX Security Symp. San Antonio, 1998. 26-40.

[6] Lee W, Stolfo SJ, Chan PK. Learning patterns from UNIX process execution traces for intrusion detection. In: AAAI Workshop on AI Approaches to Fraud Detection and Risk Management. AAAI Press, 1997. 50-56.

[7] Sekar R, Bendre M, Bollineni P, Dhurjati D. A fast Automaton-Based method for detecting anomalous program behaviors. In: IEEE Symp. on Security and Privacy. Oakland: IEEE Press, 2001.144-155. pdf

[8] Feng HH, Kolesnikov OM, Fogla P, Lee W, Gong W. Anomaly detection using call stack information. In: Proc. of the 2003 IEEE Symp. on Security and Privacy. Oakland: IEEE Press, 2003.62-75.

[9] Wagner D, Dean D. Intrusion detection via static analysis. In: Proc. of the IEEE Symp. on Security and Privacy. Oakland: IEEE Press, 2001. 156-168.

[10] Giffin J, Jha S, Miller B. Detecting manipulated remote call streams. In: Proc. of the 11th USENIX Security Symp. San Francisco: 2002. 61-79.

[11] Giffin J, Jha S, Miller B. Efficient Context-Sensitive intrusion detection. In: Proc. of the 11th Network and Distributed System Security Symp. San Diego, 2004.

[12] Feng HH, Giffin JT, Huang Y, Jha S, Lee W, Miller BP. Formalizing sensitivity in static analysis for intrusion detection. In: IEEE Symp. on Security and Privacy. IEEE Press,2004.194-208. doi= rep= rep1 type=pdf

[13] Gopalakrishna R, Spafford EH, Vitek J. Efficient intrusion detection using automaton inlining. In: 2005 IEEE Symp. on Security and Privacy. IEEE Press, 2005. 18-31.

[14] Hopcroft J. An nlogn algorithm for minimizing states in a finite automaton. Theory of Machines and Computations. New York: Academic Press, 1971. 189-196.

[15] Wagner D, Soto P. Mimicry attacks on host-based intrusion detection systems. In: Proc. of the 9th ACM Conf. on Computer and Communications Security. New York: ACM Press, 2002. 255-264.

[16] Sandeep B, Abhishek C, Sekar R. Dataflow anomaly detection. In: Proc. of the 2006 IEEE Symp. on Security and Privacy. Washington: IEEE Press, 2006. 48-62.