Eagle: User Profile-based Anomaly Detection for Securing Hadoop Clusters

Santa Clara, Oct. 29 2015-Nov. 1 2015
Eagle: User Profile-based Anomaly Detection for Securing Hadoop Clusters
Chaitali Gupta, Ranjan Sinha, Yong Zhang
eBay Authors
Abstract

Existing Big data analytics platforms, such as Hadoop, lack support for user activity monitoring. Several diagnostic tools such as Ganglia, Ambari, and Cloudera Manager are available to monitor health of a cluster, however, they do not provide algorithms to detect security threats or perform user activity monitoring. Hence, there is a need to develop a scalable system that can detect malicious user activities, especially in real-time, so that appropriate actions can be taken against the user. At eBay, we developed such a system named Eagle, which collects audit logs from Hadoop clusters and applications running on them, analyzes users behavior, generates profiles per user of the system, and predicts anomalous user activities based on their prior profiles. Eagle is a highly scalable system, capable of monitoring multiple eBay clusters in real-time. It includes machine-learning algorithms that create user profiles based on the user's history of activities. As far as we know, this is the first activity monitoring system on the Hadoop-ecosystem for the detection of intrusion-related activities using behavior-based profiles of users. When a user performs any operation in the cluster, Eagle matches current user action against his prior activity pattern and raises alarm if it suspects anomalous action. We investigate two machine-learning algorithms: density estimation, and principal component analysis (PCA). In this paper, we introduce the Eagle system, discuss the algorithms in detail, and show performance results. We demonstrate that the sensitivity of the density estimation algorithm is 93%, however the sensitivity of our system increases by 4.94% (on average) to 98% (approximately) by using an ensemble of the two algorithms during anomaly detection.

Another publication from the same category: Machine Learning and Data Science

IEEE Computing Conference 2018, London, UK

Regularization of the Kernel Matrix via Covariance Matrix Shrinkage Estimation

The kernel trick concept, formulated as an inner product in a feature space, facilitates powerful extensions to many well-known algorithms. While the kernel matrix involves inner products in the feature space, the sample covariance matrix of the data requires outer products. Therefore, their spectral properties are tightly connected. This allows us to examine the kernel matrix through the sample covariance matrix in the feature space and vice versa. The use of kernels often involves a large number of features, compared to the number of observations. In this scenario, the sample covariance matrix is not well-conditioned nor is it necessarily invertible, mandating a solution to the problem of estimating high-dimensional covariance matrices under small sample size conditions. We tackle this problem through the use of a shrinkage estimator that offers a compromise between the sample covariance matrix and a well-conditioned matrix (also known as the "target") with the aim of minimizing the mean-squared error (MSE). We propose a distribution-free kernel matrix regularization approach that is tuned directly from the kernel matrix, avoiding the need to address the feature space explicitly. Numerical simulations demonstrate that the proposed regularization is effective in classification tasks.

Keywords