Apache Argus

Security on Hadoop has been catch-as-catch-can for much of the product's lifetime, but a new Apache project that entered incubation earlier this year -- Apache Argus -- addresses it in a consistent manner.

With the delivery of YARN, which powers Hadoop’s ability to run multiple workloads operating on shared data sets within a single cluster, a heightened requirement for a centralized approach to security policy definition and coordinated enforcement has surfaced.

Argus will deliver this comprehensive approach to central security policy administration across the core enterprise security requirements of authentication, authorization, accounting and data protection. It already extends baseline features for coordinated enforcement across Hadoop workloads from batch, interactive SQL and real–time IN Hadoop. And we will leverage the extensible architecture of this security platform to apply policies consistently against additional Hadoop ecosystem components (beyond HDFS, Hive, and HBase) including Storm, Solr, Spark, and more. It truly represents a major step forward for the Hadoop ecosystem by providing a comprehensive approach – all completely as open source.

Argus did not start as a community initiative; it's the open-sourced version of a commercial product, XA Secure, that Hortonworks acquired and transformed into an Apache-hosted project. The idea, as Hortonworks explained earlier this year, is to provide a centralized way to define and enforce security policy across Hadoop and all its components. This includes access controls down to the folder and file level in HDFS, and to the table and column level in Hive and HBase. But don't expect automatic Argus integration -- this project has a long road ahead for Hortonworks and everyone else contributing to the Hadoop ecosystem.

In May, Hortonworks acquired XA Secure and made a promise to contribute this technology to the Apache Software Foundation.  In June, we made it available for all to download and use from our website and today we are proud to announce this technology officially lives on as Apache Argus, an incubator project within the ASF.