Paul Doscher is president and chief executive officer at Lucid Imagination. He has more than 30 years of experience working in technology, many in high-level positions. We asked him about the state of enterprise search.

FierceContentManagement: In the age of big data, analytics and the increasing role for business intelligence tools, how is the role of search changing in the Enterprise?

Paul Doscher: The role of search is changing because the data has changed. It's not about searching databases or files in a document management system anymore. The typical enterprise now produces enormous amounts of both human-generated and machine-generated data, and there are valid business reasons for indexing all of it and making it searchable. On the human-generated side email was the beginning, then IM, and now twitter feeds. Social media, and the move to adopt this paradigm for business interactions, will only accelerate this data fragmentation.

On the machine-generated side, log data is being created continuously by many business processes, and more kinds of devices. Your electric meter, your car and your phone all spin off log data.

What is common to both is that the ability to mine that data for better intelligence and decision-making is potentially extremely valuable. Unlike traditional BI, combining search with big data tools gives you the ability to analyze data in an ad hoc way that doesn't require database schema updates. Using big data tools to understand what employees or customers are talking about, for example, and using that feedback to improve the information you provide and the search experience you present, creates a natural loop that improves the customer experience and the business bottom line.

FCM: What are the advantages of using an open source search tool for a company?

PD: Industry consolidation, the complexity of the data, and the unique needs of each company mean the open source community is rapidly becoming the safest vendor. Open source technology is inherently extensible; if a company needs something that the base product does not provide, then they can build it themselves and potentially contribute it back to the community. It's becoming harder for proprietary vendors to match the pace of innovation of an active open source community.

FCM: How important is it for companies to be able to build custom search applications beyond the pure nuts and bolts you'll find in most enterprise search tools?

PD: As a search platform vendor, we deal almost exclusively with teams building custom applications, so for our customer base it is absolutely essential. These are generally search-enabled applications, in which search is an important component but the search experience has to fit into the larger goals of the application. In other words, there may not be a search box in the traditional sense.

FCM: How important is search to both the end user experience inside the enterprise and for a visitor to the company website?

PD: It's critically important to both. Thanks in part to Google (NASDAQ: GOOG), all computer users expect to be able to search, and an application without at least a simple search box is now almost inconceivable.

Inside the firewall, search has always been about arming employees with the best information to help them make business decisions. As the data sources multiply, and more of the data is stored outside the database, search will become more important as the primary way that users access information.

Website visitors generally fall into two groups, browsers and searchers. As site search becomes more sophisticated, and techniques like faceting allow users to navigate through search results, the distinction is blurring. Drilling down into search results is an effective way to navigate, and it has a big advantage. A user never encounters a category with no results, since the navigational tree is assembled from the data that is available now.

FCM: What kinds of creative ways are you finding your customers have been using the ability of your product to build custom search applications? 

PD: Faceted search on e-commerce sites is an excellent example of a search experience that is presented as something other than the classic search box.  

And of course, the rise of interest in using big data technologies to solve new kinds of data-related problems is opening up many new ways to use search.

Related Articles:
Lucid Imagination brings open source search to big data
Lucid Imagination includes SharePoint connector
One on One with Eric Gries of Lucid Imagination