Paul Sonderegger is Chief Strategist at search vendor, Endeca. Before joining Endeca, he spent six years as an analyst at Forrester Research, focusing on search technology and experience design. We asked him why we are seeing business intelligence and search merging this year.

FierceContentManagement: Do you see Endeca as a business intelligence company, a search, company or both?

Paul Sonderegger: Endeca is a search and business intelligence (BI) company. Combining the two together is what defines us. We believe this combination is the best way to answer the new grand challenge facing BI: Accommodating diversity at scale. This diversity is in both the data BI has to bring together and in the decisions BI has to support.

Traditional BI worked on carefully curated data in the data warehouse. But now BI has to bring together a huge diversity of information including data in the data warehouse, data in enterprise apps, documents in content management systems or file servers, plus blog posts, twitter feeds, and user reviews from the Internet. And now, more people rely on this growing diversity of information for more and more kinds of daily decision-making.

Combing search, with its excellent user experience and flexibility with unstructured content, and BI, with its powerful analytics and dimensional navigation, is the key to helping people both find what they need and understand what they found in highly diverse data at large scale.

FCM:When did BI and search become two sides of the same coin?

PS: The technology to make this possible is a relatively recent phenomenon, but the three macro trends driving them together have been building for years. 

The first is massive information proliferation. IDC predicts that the digital data created in a year will be 35 zettabytes by 2020. That's a stack of CDs reaching half way to Mars. But the volume isn't the big story, the variety is. All that data will be spread across the diversity of information types listed above.  

The second trend is the consumerization of IT. People who book their vacations, manage their retirement portfolios, and keep up with their friends and family online expect the same ease of use in the tools they use at work. 

But it's the third trend, commodity supercomputing, that makes available the abundant processing power needed to unify BI and search. With the advent of multi-core, multi-chip, 64-bit servers what used to take an elite multi-million dollar machine just five years ago can now be done with commodity hardware. This opens up new possibilities for innovative companies--and we like to think we're one--to take a fundamentally new approach to unifying diverse data and making it usable for people with no training in technology.

FCM: What impact do analytics have for companies trying to understand their web traffic?

PS: Web traffic data is massive. Analytics are a powerful way to aggregate large volumes of data to make trends, changes and outliers in the data easier to see. However, this is just one half of understanding what's going on with the people visiting the site. The other half is putting the context around the facts and figures to better understand why the trends are going in a certain direction, why they're changing the way they are (or not), and why the outliers exist. Without this context, web traffic data is like a daily stock ticker--it's up or it's down, but it still doesn't tell you what investment to make next.

FCM: What role can analytics and search play in understanding internal audiences?

PS: The shift to web-based apps in the enterprise opens up a new opportunity to measure and improve their performance, much as eBusiness teams do with external websites. But the parallel is not exact. There are some crucial differences, including the size of the user population and the patterns of use. Because the employee population in even large enterprises is orders of magnitudes smaller than the customer population, quantitative data plays a smaller role and qualitative information a larger one.

For example, a website with flat growth in unique visitors is probably in trouble. But an enterprise app in the same state may simply have reached all the employees it should. Similarly, patterns of use may depend not on the personal preference of the users, but company practices. For example, the new sales portal with all the latest presentations and product documentation may go unused not because it's not well designed, but because the company tracks sales activity through the salesforce automation app and, therefore, that's where salespeople spend their time.

FCM: How is search changing in the enterprise?

PS: Search in the enterprise has stratified into two distinct markets--keyword search and search applications. Keyword search puts a search box in the company portal, departmental wiki or content management system. It does what we've all been trained to expect of search--retrieve documents in response to a query. Search applications support specific decision points in a larger process, like part selection in a product design process.

But many of these decision points rely on diverse data from many sources to make a better choice in unanticipated circumstances, like figuring out why warranty claims on a top-selling product just went up or why the shipping forecast doesn't match what products were actually shipped where last week. This is where search intersects with BI and improves daily decision-making one decision at a time, millions of times over.

Related Articles:
One on One with Content Management's Movers and Shakers 
Endeca makes ecommerce suite mobile 
Endeca announces commerce suite upgrade 
Daniel Tunkelang leaving Endeca for Google 
Endeca and Informatica join forces in Informatica 9