IBM has just announced a new In-Hadoop analytics technologies to accelerate the conversion of data into valuable insight for the business. IBM is delivering machine learning, R, and many new features that can run over large-scale data in the new IBM BigInsights for Apache Hadoop.

Apache Hadoop is a powerful technology for collecting and economically storing a very large set of highly variable data. But as the size and complexity of Hadoop applications continues to grow, data science has become a critical function for unlocking meaningful insights. IBM BigInsights for Apache Hadoop includes a broad data science toolset to query data, visualize, explore and conduct distributed machine learning at scale.

IBM BigInsights for Apache Hadoop will introduce three new modules:

• IBM BigInsights Analyst will include IBM’s SQL engine and IBM’s spreadsheet and visualizations to find data quickly. On average, millions of SQL queries are run each year. With BigInsights Analyst, the efficiency of these queries has been shown in some cases to improve by approximately 2x to 4x on Apache Hadoop depending on the shuffle size. The ANSI compliant SQL means queries can run unchanged against Hive, HBase and relational databases.
• IBM BigInsights Data Scientist will deliver a new machine-learning engine that automatically tunes its performance over large-scale data to find interesting patterns– plus over a dozen industry-specific algorithms such as Decision Trees, PageRank and Clustering to help tackle complex problems out of the box. It will also provide native support for open source R statistical computing helping clients leverage their existing R algorithms, or gain from the more than 4,500 freely available statistics packages from the R community.
• IBM BigInsights Enterprise Management will introduce new management tools designed to help allocate resources and optimize workflows. These tools will allow deployments that can scale to large numbers of users and clusters, and will help satisfy high workload demand. The tools will provide multi-tenancy and multi-instance support in a cluster.

Also announced today is IBM Open Platform with Apache Hadoop. Based on open source software, the platform will provide the necessary data access controls and authentication for an enterprise.