Hollywood has taught us that intelligence is secret: signal intelligence, like communications intercepts; imagery intelligence, like spy satellites; and human intelligence, like Mata Hari. But intelligence analysts say that many of their best sources aren’t secret, they’re open source (that’s “open source” as in publicly available, not as in free source code.) But feed it into a search and BI application, and open source intelligence can become classified.

The title of this post – Unclassified + Unclassified = Classified – comes from a law enforcement agency that is a customer of Endeca. We built them a demo that combined an unclassified data source of theirs with a publicly available data source, and they said that the combination of the two would be classified. How’s that possible?

We can’t share a classified use case, but here’s an analogous one we can share. This is a generic demo our sales engineers built on open source information, like US government contracts. Starting from raw feeds, you can pull out all the structure, and do some text mining to extract more structure. Then, with faceted search and BI, an intelligence analyst can navigate, change groups, analyze, and drill down. And with that, you expose important relationships that weren’t readily apparent before. For example, you might discover a spike in contracts where the contractor is headquartered in the same district as a member of the appropriations committee. The sources are still open, but the relationships become apparent for the first time, and that’s enough to make it classified. It’s a nice reminder that the intelligence is in the human analyst, not in the data.

When I was a graduate student in Computer Science, many years ago, I was intrigued to discover the existence of a program called ‘YACC’. My curiosity was piqued not so much by the reputation of this program (for its use was widespread throughout academia), but simply the name: the acronym stood for ‘Yet Another Compiler Compiler’. In so doing, the authors had modestly acknowledged that their output was not the first of its kind, nor was it likely to be the last. They had, in one self-effacing step, acknowledged that their work stood on the shoulders of others that had gone before – although leaving open the question of why “another” one was needed.

The extremely strong response to the launch of the Endeca UI Design Pattern Library (e.g., over 15,000 visits in the first two days) suggests that “yet another”  library is actually addressing a real need and interest in the community, despite there being a number of excellent predecessor UI design pattern collections.  In this context, we want to acknowledge those predecessors and review where our Library might overlap, complement, or build on others. In addition, we’d like to share some of our plans for where we go from here – i.e., how we intend to grow the library going forward.

Several predecessor collections have had a formative influence on the direction and content of the Endeca pattern library. One of the most popular of these is the Yahoo Pattern Library.

This site currently comprises 59 UI design patterns aimed at the web design and development community. As you’d expect from an organization that has its roots in search, it offers guidance on several topics in which we share a common interest, notably Breadcrumbs, AutoComplete and Search Pagination. In fact, we’d very much welcome a dialogue with the curators at Yahoo as we develop our own viewpoint (and patterns) on the latter topic.

The Yahoo patterns are thematically organized into a number of top-level categories; a model which we’ve adopted through our own Topic pages.  But whereas ours are focused specifically on the challenges associated with search and information discovery, Yahoo’s extends across the full spectrum of Web 2.0 issues to embrace themes such as ‘Rich Interaction’ and ‘Social’.

The Yahoo Library also offers a good example of how design patterns should be structured (which perhaps should be a design pattern in itself), with each pattern consisting of a title, a problem, a context, and a solution. In addition, the patterns are tagged with an “adherence rating” (which gives a sense of the maturity of the design solution) and many offer links to code examples & related blog articles.

Another popular UI design resource, particularly for those who specialize in search & information discovery, is Peter Morville’s collection of Search Patterns.

This collection is particularly interesting as in includes “behavior patterns”, i.e. user behaviors represented at a higher level of abstraction that may subsume several lower-level tasks or patterns. It’ll be interesting to see how these behavior patterns map onto our own notions of Usage or what we sometimes refer to as “Modes of Discovery” (such as Expanding/Exploring, Narrowing/Refining, etc.). This is something we’ll cover in greater detail in an upcoming post.

As you’d expect for a collection focused on search, there are many areas of common interest, notably AutoComplete, Faceted Navigation, and Pagination. Morville’s collection also addresses one of the most common challenges we face at Endeca: how to present a collection of heterogeneous search results (such as product records, user reviews, user guides, etc.) in a way that is comprehensible and navigable. Our current library includes some initial work in this area in the Mixed Content Results with Spotlights pattern, but there are many other solutions to this problem (e.g., segmenting with tabs or zones, etc.) and Morville’s collection alludes to a more complete range of approaches through the examples in the Structured Results and Federated Search sections. This is also an issue we plan to explore further in future work.

Search Patterns is not structured as formally as the Yahoo library, but what it lacks in narrative and textual detail it more than makes up for in visuals: each point is richly illustrated with numerous examples and screenshots. Moreover, the scope of the collection extends beyond discrete UI elements to address overall page architectures and layouts (referred to as “Contexts”), with examples grouped into common themes such as “eCommerce”, “Enterprise”, “Site Search” and so on. This level of holistic thinking is a crucial aspect of any design solution and is a theme which we plan to explore further in our own collection.

One further resource we should acknowledge is Martijn van Welie’s Interaction Design Pattern Library.

This collection goes far beyond what might be expected from the work of a single individual, offering over a hundred patterns organized into three top-level categories, many of which we reference in our own patterns. It provides extensive coverage of search and navigation topics, with entries for Breadcrumbs, Faceted Navigation, and various aspects of search (including Search Box, AutoComplete, Search Results, and so on). Like Search Patterns, van Welie’s site gives consideration to the wider design context, with specific categories for Site Types (e.g. eCommerce, Portal, Corporate, etc.) and Experiences (e.g. Information Seeking).

So where next for the Endeca pattern library?  As mentioned above, we know there are certain important areas that we’d like to explore further, such as:

• Developing a more principled view on Usages or “Modes of Discovery”
• Extending our range of solutions for presenting heterogeneous content, giving greater consideration to page and application design and contextual issues
• Addressing a growing “backlog” of UI challenges such as the use of visualized analytics, text analytics, advanced forms of faceted navigation and exploration, mobile search and discovery contexts, and so on.

In this respect, we’ll be drawing on the work of those mentioned above (along with numerous others, such Jenifer Tidwell, UIPatterns.com, Quince, PatternTap, etc.) in developing our own point of view.

In so doing, we hope to combine what we learn from the wider UX community with what we learn from our own research and design work, synthesizing useful principles for designing great search and discovery experiences and applications.  We don’t mind being “YAPL” if we help grow and accelerate community knowledge and innovation

Interviewer Brett Bonfield pulls off a tough balancing act in a new conversation with me and the founder of the web search engine DuckDuckGo, Gabe Weinberg: How do you ask the same set of questions of both a web search and search applications company? You can read the interview on the best-named library blog ever, In the Library with the Lead Pipe.

Web search and search applications have very little in common, so when we wind up in the same discussion, it’s usually a sure-fire sign it’s a remedial one. This is an exception. Here’s Brett’s premise:

As it turns out, librarians aren’t the only ones competing with Google. In fact, we’re not even the only ones offering an alternative to Google when it comes to helping people find information.

That librarians are in competition with Google is a provocative premise, but one you’ll commonly hear from them. It stems from a look into the near future where the web is “good enough” in two ways: nearly everything is digital and available via web search, making library collections unnecessary; and search engines are effective enough that reference librarians become obsolete.

That extreme future without libraries seems unlikely to me in the near term, though it will surely reduce budgets and butts in seats. But to me, the savior of libraries will be the ongoing need for humans in the HCIR equation. One small piece of evidence this is true: Noah Huffman presented on the Endeca-powered OPAC at Duke (and the entire Triangle Research Libraries Network) at the Society of American Archivists conference yesterday, reporting a 475% increase in traffic – though noting ominously that most visits come from the open web (or so it was Tweeted at #saa10). That is, better search can lead to more library traffic, not less.

At any rate, in the interview, both Gabe and I talk a lot about alternatives to web search that focus on HCIR. Although we never field Brett’s question head on about the competition between libraries and web search, the secret, like soylent green, is people.

Every UX designer working on faceted search and discovery applications faces a key challenge.  How to present facets that instantly tell users:

• What’s available within the collection or information space?
• Which actions will help users meet their goals?

To meet this challenge, faceted user interfaces need to summarize the information space in a readily comprehensible way and provide clear “information scent” that guides users to action.  Visual facet presentations could be helping much more than they are today.

Early presentations of faceted navigation back in the late 90’s often showed lists of hypertext links as the primary means of summarizing the information space, providing information scent, and inviting action.

Over the years, others have attempted to enrich and enhance faceted search and discovery by “envisioning information” and embedding visualizations within faceted interfaces. For example, Willet et all (2007) discussed the use of “scented widgets” that embed visualizations within graphical UI controls, such as range sliders with histograms (Also see our Range Slider UI Design Pattern)

However, despite the value of visualizations, many search and faceted navigation interfaces continue to rely on primarily textual presentations and appear to pay limited attention to the visual aspects of the user interface. While textual presentations are obviously effective in many contexts (and readily accessible to the vast majority of users), in others they may inadvertently contribute to information overload and unnecessary cognitive work.

For example, a textual list of color choices for shoes may be readily comprehensible to a user looking for blue running shoes, but it may also inadvertently create subtle cognitive challenges.  When a “blue” facet value is presented in black text or vice versa  — when the visual rendering conflicts with the meaning of the facet — we may be subtly creating the conceptual equivalent of the “Stroop effect” in which discrepancies between perceptual cues cause delays in reaction times.   In contrast, some sites present an enriched “color picker” which shows readily recognizable color swatches in a visual palette.

Color pickers are obvious examples of the principle of matching the presentation format to the semantic nature of a facet to optimize user recognition and action, but there are others.  For example, data values within continuous quantitative facets (such as price and date ranges) are frequently represented as a set of arbitrary ranges presented as text links.  Range sliders with histograms are often a better match for the semantic nature of this type of facet as well as the user’s conceptual model, and consequently can be a more effective UI choice. (See Range Slider UI Design Pattern for more detail).

Likewise, categorical facet values (such as car body types) may often be better represented using visual or iconic representations rather than simple text links.

The same principle of visualizing facets applies to common UX challenges in B2B e-commerce applications (e.g., industrial suppliers) with broad, deep, and varied product catalogs.  How do we help users find physical products (e.g., a bolt, a power supply, etc.) in a deep catalog when they don’t yet know what the entity might be called?  Such “I’ll  know it when I see it” situations are excellent candidates for iconic representations of facets, such as simple line drawings.

The examples I’ve given so far are all designed to help users discover single items in a collection or understand and apply specific facets. But there are also many scenarios in which people need to understand bigger subsets of a collection and/or the patterns in the collection – in other words, they need to see the forest for the trees. A sales executive evaluating and formulating sales strategy and tactics needs to understand and “see” the impact of varied facets (e.g., geography, price, sales rep) on sales outcomes;  an auto quality specialist aiming to proactively identify and quickly mitigate quality issues, needs to “see” which models, components, suppliers, and locations are most frequently associated with warranty claims.  Visualizations of facets and facet patterns can help here too.  Rich visual summaries and analyses of aggregated information — charts, graphs, etc. –  not only can provide “information scent,”  but also can convey critical info and insight in and of themselves (as Tufte taught and demonstrated in numerous ways)….but that’s another blog

(For an example of the use of visualized analytics in faceted interfaces see the Analytics Dashboard UI design pattern)

Although I’m obviously not the first person to call attention to the value of visualization in faceted search and discovery interfaces (e.g., Marti Hearst on information visualization in search interfaces), I hope that more people in the community will consider and experiment with the use of visualizations to enhance the user experience (while keeping accessibility considerations in mind).

More broadly, I hope that by working to optimize the fit between the presentation, the semantic nature of facets, and the users’ mental models, we can identify innovative ways to effectively convey meaning and facilitate discovery in faceted information solutions.

TEDx conferences, the local offshoots of TED, are more experimental in format than the classic TED talk. An innovation of TEDxBoston is the “Adventure” — an immersive trip that puts the big ideas of TED into the context of a physical location. This year, there were nearly two dozen, including a tour of Dean Kamen’s Willy Wonka-like DEKA factory, a helicopter flight over the city, a re-creation of Paul Revere’s ride on bicycles with Olympian Nicole Freedman, and a visit to the Suffolk County lock-up with its forward-thinking sheriff.

Endeca hosted an audience of 75 for one of the Adventures, The Future of Search panel, at our headquarters. And if you wanted any more evidence that search is still red hot, it turned out to be one of the most requested TEDxBoston Adventures.

Boston is a perfect city to assemble a search panel, and the mix for this one included:

• Justin Boyan, VP Data Integration, ITA Software
• Pawan Deshpande, CEO, HiveFire
• Don Dodge, Developer Advocate, Google
• Boris Evelson, VP and Principal Analyst, Forrester Research
• Adam Ferrari, CTO, Endeca

Paul Sonderegger moderated a wide ranging tour not just of search, but of searching – the human activity of seeking out information and applying it to decisions. His questions covered 3 broad areas:

• People and personal life: How is search affecting our privacy? How do disparities in access to search play out around the globe?
• Business and work life: Is search making us more productive? How are the best companies using search to gain competitive advantage?
• Government and the citizen: What is the impact of search on regulation? How is search making goverment more transparent?

The night ended with predictions from each panelist about which aspects of search will matter most in the next 3-5 years. So what is the future of search? Social graphs, location-based search, semantic web, the convergence of search and analytic technologies, and richer content aggregation.

But earlier on, Paul had another interesting question about the future of search: With better search, could we have foreseen the financial crisis? Or to generalize Paul’s question, can we use search to see the future?

The search box is imbued with nearly magical powers these days. But if you stop to think about it, even with all that power, search is rarely thought of as a crystal ball to the future. Contrast that, with, say, business intelligence tools, which are widely used for forecasting and predictive modeling. Oracle even named themselves after a seer. So why the difference?

Let’s re-ask Paul’s question a different way: With better BI tools, could we have foreseen the financial crisis? Asked this way, I think the answer is obvious. We didn’t need better tools, we needed better people using the tools.

That complementary pairing of person and computer is the heart of HCIR, or Cyborg BI. Readers of Search Facets likely already think of search as a conversation with the data, akin to BI. Yet sampling the audience at the panel, most people outside the biz still have a mental model of the search box as a tool for fact finding, or query/response. To paraphrase William Gibson, the future of search is already here, it’s just unevenly distributed.

Our UX team at Endeca gets a steady stream of questions about how to design effective search and discovery experiences. Just some of our FAQs:

• “What’s the best way to handle faceted bread crumbs?”
• “How should we present faceted navigation?”
• “How should we present results that include diverse types of content?”

In the course of solving problems like these in many contexts, we’ve been building up a library of UI design patterns specializing in search and discovery, and especially in faceted search and exploration. We previewed it for Jared Spool’s UIE virtual seminar “Leveraging Search & Discovery Patterns For Great Online Experiences,” and today marks the official launch of the library.

A pattern library serves as a knowledge base for designs that are known to work well, and provides kernels you can adapt to your specific case. This follows in the footsteps of some well-known collections like Yahoo’s Design Pattern Library and Peter Morville’s Search Patterns.

But when should you look for and apply patterns? How do you identify the most relevant and applicable ones? When you find one or more relevant patterns, what should you do with them?

We believe that UI design patterns should be used as an integral part of a human centered design process. This means that patterns should be part and parcel of the iterative process of understanding users, creating potential solutions, and evaluating and optimizing solutions.

First, it’s essential to understand the user context.

• Are the users knowledgeable about the domain or are they novices?
• What are the users’ typical goals and scenarios and modes of discovery?
• Are users attempting to locate specific known entities, attempting to learn about a domain, comparing and evaluating options and tradeoffs as part of a decision making process, etc.?
• What are the info assets that the users are working with?
• Are they simple objects with a few attributes or complex entities and concepts with a wide range of attributes?

Certain patterns may be for very specific users and scenarios, while others may work across a wide range. For example, Horizontal Faceted Navigation Multiselect is specifically designed to help knowledgeable users engage in tradeoff analysis when trying to find entities that match complex sets of criteria – for example design engineers trying to find the best components to use together to build a product. In contrast, Vertical Stack Faceted Navigation can be viewed as a multipurpose “Swiss Army Knife” that can work fairly well for both the “knowledgeable seeker” (e.g., the pro photographer looking for a replacement lens) and the “uncertain explorer” (the photo novice looking for an affordable camera to take on a family vacation).

Next, patterns provide an evaluative lens. These are guiding principles that help us spot potential gaps or sources of confusion. For example, the Breadbox pattern can help us know whether an implementation of faceted “breadcrumbs” includes clear affordances for a user to modify his or her search criteria. It also cautions that it might confuse the user if mixed with “you are here” style “breadcrumbs.”

When using patterns as an evaluative lens, it is essential to do this with the user context in mind, not as an abstract exercise. For example, patterns are extremely useful as reference points during scenario-based walkthroughs of designs. This helps ensure that that all the moving parts in your application work together to aid successful discovery by users.

Patterns are not simply evaluative, they are generative, helping us create new UIs and experiences. As with a spoken language, we can creatively “make infinite use of finite means” — a finite set of words and grammatical rules enable us to create infinite meaningful sentences. Vertical Faceted Nav may include facet values as text links, icons, range sliders, or color pickers. The presentation of “more” facet values can be presented via fly-out menus or expanded lists, with or without a scroll bar.  Breadcrumbs in a breadbox may be presented as visual “building blocks,” layed out vertically or horizontally, or as simple text presented as a readable “sentence.”

Patterns should inspire us to create novel solutions. We should extend patterns, and break outside the box as needed to improve discovery experiences, take advantages of new opportunities (e.g., afforded by technology innovation), and solve new problems. In this spirit, one limitation of the Breadbox pattern points to a possible enhancement: not only should we be able to remove filters, we should also be able to modify filters directly from the breadcrumb itself (e.g., see Greg Nudelmann’s article on “super powered breadcrumbs”.) Likewise, one limitation of standard vertical stack faceted navigation is that it can be difficult for users to “see” the tradeoffs and relationships between various attributes as they make choices. This led to new ideas such as open horizontal faceted navigation with multiselect.

So UI design patterns can and should be used as much more than simple point-in-time evaluation tools. They’re generative tools that we creatively apply throughout the process of designing, evaluating, and continuously improving human centered solutions.

We’re often asked about how Endeca’s BI offering compares to QlikView — more than usual with their “heavily oversubscribed” IPO this morning of their parent company QlikTech (QLIK).

The comparisons aren’t surprising. If you read their S-1 IPO filing, you’ll find spots where you could cut-and-paste “Endeca” for “QlikView.” For example,

“We have pioneered a powerful, easy-to-use business intelligence solution that enables our customers to make better and faster business decisions. Our software platform, QlikView, combines enterprise-class analytics and search functionality with the simplicity and ease-of-use found in office productivity software tools for a broad set of business users.”

Despite this, we rarely compete. One reason is that our products have material differences. But we don’t usually get down to the product because upstream of that, we’re each going at this expanding BI market from different directions.

Appeasing the Angry Mob

Like QLIK, Endeca is addressing the need for BI on many more desks than classic BI can reach today. As Gartner puts it in their SWOT analysis of Qlik (which also mentions Endeca as a competitor), “According to Gartner BI user surveys, fewer than one-quarter of potential users use BI today.”

All those left-out users make a stink about it. Each time they get a BI report, it answers one question, but invariably raises new ones that it can’t answer in the moment, .e.g Demand went up….Why? So the consumer requests new views of the data from IT. It’s a hydra — each time a report answers a question, it engenders new ones. And that’s one root cause of the BI backlog, as Paul Sonderegger blogged in How Big Is The BI Backlog?

Angry mobs take matters into their own hands. That’s where QLIK sniffed a market for their vigilante BI. The beauty of their product is end-users can install and use it on their own, bypassing IT entirely. In fact, their business model is predicated on vigilante BI. From their S-1,

We have a differentiated business model designed to accelerate the adoption of our product by reducing the time and cost to purchase and implement our software. Our low risk approach to product sales provides a needed alternative to costly, all-or-nothing, traditional business intelligence sales models by offering free product downloads to individuals and a 30-day money back guarantee upon purchase. We initially focus on specific business users or departments within a prospective customer’s organization and seek to solve a targeted business need. After demonstrating QlikView’s benefits to initial adopters within an organization, we work to expand sales of our product to other business units, geographies and use cases with a long-term goal of broad organizational deployment.

But that final part of their “land and expand” strategy is in question. They want to give self-service BI to individual members of the angry mob, with the aspiration that IT will eventually sanction the approach. But a product designed for end-users isn’t necessarily aligned with IT objectives, like, say, data governance. Gartner cites limitations to centralization as a weakness in their SWOT:

“QlikView is rarely enterprisewide BI because of limitations with metadata.” Expanding on this, “QlikTech has very few examples in which a single QlikView instance is used for BI metadata for all BI applications, rather than for a number of disconnected QlikView implementations. Because of the nature of siloed deployments in business units, and because there is no enterprise metadata layer or a central metadata repository across QlikView applications, it is difficult to create an enterprisewide BI view with QlikView, in which cause and effect relationships are established.”

QlikView serves individual users well, but those silos don’t aggregate up into a centralized IT deployment.

Endeca comes at this large untapped market from a different direction: we’re IT’s choice for Agile BI. Our customers tend to be IT directors in the Fortune 500 that are on the other end of the angry mob. They pick Endeca because it lets them clear out a healthy amount of their BI backlog. They can publish out many new views of enterprise data, while also meeting classic IT goals on security, reliability, and scalability.

So that’s how Endeca compares to QlikTech. We both see a big market in the angry mob. But we want to serve them in different ways, and reflecting that, our products, architectures, and go-to-market are quite different. If the QLIK IPO is a good indicator of which way the BI market is going, justice is coming for the angry mob, and it’s coming from the outlaws and the sheriffs.

My favorite part of the Endeca year just started with our sixth annual call for Navigator Award nominations, recognizing the most visionary Endeca deployments. What’s most fascinating to me about the awards is hearing our customers tell their stories in their own words.

We have our own narratives about each facet of the Endeca story. For example, the dev and product management organizations build towards user personas, like Melanie Merchandiser, our retail super user who is an expert in product promotions but who is not an expert in IT. Dozens of personas like Melanie are composites of the hundreds of people like them we hear from out on the front lines. Then from that product artifact, we need to tell more stories like ones for sales, services, education, and user experience. The point is, the personas are grounded in reality, and so in turn our stories match our customers really well. But despite all that, the Navigator entries surprise me every time.

The kernel of our Agile BI story comes from a Navigator entry a couple of years back. RS Components started as an Endeca B2B ecommerce customer, using faceted search to help buyers find components. But then RS started using the platform internally to build out an Agile BI app. Their use case: having acquired several smaller distributors in Asia, they found they couldn’t make sense of their business. That’s because each acquired company had its own ERP system — whether Oracle or SAP, each with idiosyncratic schemas – and so to get visibility across them all, they’d need to reconcile them into one data warehouse. But by the time they completed that project, maybe a year later, the business had changed. Instead, they put the data into Endeca, getting an immediate rough cut, helping them to revise their application, lather, rinse, repeat. According to our narrative, our value to RS was primarily in letting them report on both structured and unstructured content for the first time. But to them, the value was in rapid iterations, made possible by pouring in multiple schemas – another face on looking across semi-structured data. Here’s how then-CIO Richard Boynett told me the story.

eBags is one of my favorites. It’s a David and Goliath story, where they beat up giants on a modest budget. Their secret – drawing on founder Peter Cobb’s background in catalog marketing, they get big results by testing endless tiny improvements to the site – which is the retail equivalent of the Agile BI story.

One ecommerce customer reported in their Navigator entry that they saw a $270 million increase in revenue in the first year after deploying Endeca, which is a lick more than even our bold marketing story would pitch. The Auto Trader guys did a Rashomon, telling their story from multiple  perspectives, one a business story from their search product manager, one an IT story from their chief technical architect. And there are many more like this.

The race is on for 2010, and I can’t wait to hear the stories.

In his post MapReduce just semi-good for semi-structured data, Adam Ferrari answered one of his FAQs about the relationship between Endeca and MapReduce, the popular big data cruncher. Now here’s one example of them complementing each other.

The question Adam answered was, if MapReduce is so powerful for processing big data, then what role does Endeca play?

By way of background, MapReduce is “a software framework for distributed processing of large data sets on compute clusters,” which is itself a sub-project of Hadoop, “open-source software for reliable, scalable, distributed computing.” They take parallel processing that was once rarefied because it required esoteric dev skills and expensive hardware and make it accessible to people with mortal IT skills and cheap hardware.

Adam answered that the details matter. What kind of data are you crunching, and how do you want to query it? For example, if you understand the structure of the data and know how you want to query it, MapReduce is perfect. On the other hand, if you have heterogeneous, semi-structured data, then we know empirically that you likely won’t know in advance how you want to query it, so instead you’ll need to explore and refine it. Endeca fits that use case.

Another complement to Hadoop is Hive, “a data warehouse infrastructure built on top of Hadoop that provides tools to enable easy data summarization, adhoc querying and analysis of large datasets data stored in Hadoop files.”

Taken together, Hadoop, Hive, and Endeca can give you an Agile BI solution for big data.

And in fact, Vinay Mohta, a product manager at Kayak, the vertical travel site, has been blogging about this very use case. Vinay is a perfect early adopter because he’s an Endeca veteran, having served as a both a core software architect and a product manager. From his blog:

I’ve been using Hadoop and Hive for the last six months and have been pretty impressed with how well it works.  To state the obvious, if you can correctly formulate your query, nothing beats this approach.  It’s been very useful for doing cohort analysis and large scale lifetime value computations on a relatively high traffic site.  There are of course limits to what you want to keep in Hadoop / Hive; however, the convenience and the growing feature set are reducing that limit more and more.

Hive is not a good store as a backend for a BI product, since it offers no caching at all.  However, a workflow where you crunch data in Hadoop/Hive and then export to a MySQL table (or an Endeca instance) for use in a BI tool works very well.

Vinay’s not the only one. We’ve heard from quite a few customers that have Hadoop and  Endeca together in their workflow. These are fun to track because once people are in an Agile workflow, they inevitably invent new use cases.

I’d love to hear how you’re using it. I’d also like to know your motivation. Is it because it’s a quick path to Agile BI, or is there a qualitative difference between these new tools and a traditional enterprise data warehouse?

There’s still so much room for innovation on faceted search user experiences. Here’s a great improvement that’s still rarely seen in the wild: graying out dead ends instead of removing them. “Gray ends” are just for certain cases, but in those conditions, they make a big difference. Moreover, they exemplify one of the great Edward Tufte lessons.

An excellent implementation is up right now at B&H, a professional photo, video, and audio store based in New York. (Don’t miss it if you get a chance to visit their brick and mortar store. They gave me a tour, and it was clear they had designed the store experience around the well-understood needs of their pro customers — plus, it’s fun to walk down an entire row of broadcast television cameras.)

In this example, I searched for camera lenses made by Canon and narrowed the results by picking the least expensive range in facet:Price. Now if you look at the facet:Focal Length Type, voila – gray ends. They show me that when I selected the lowest price band, some focal length types, like “Super Telephoto,” are no longer available to me.

There are a few obvious benefits here, but also some nuances worth discussing. I made one more selection in facet:Focal Length Type, “Zoom Super Wide,” to highlight some of the nuances.

-In faceted spaces, when you make an explicit selection in one facet, you are also making implicit selections in the other facets. You already know this, but it’s not obvious to most users. Gray ends make it obvious, which helps them understand how facets work, and avoids confusing them when things disappear for reasons that aren’t obvious.

-To the searcher, linking explicit and implicit selections helps them understand tradeoffs. By showing the gray ends, people can understand the tradeoffs they’re making across facets without being forced to ping pong between screens. In the B&H example, I can perform sophisticated price/performance optimizations with a simple interface.

-I made that additional selection, Zoom Super Wide, to show that Focal Length Type is a multi-select facet – using the check boxes, I can now expand my selection to include, say, Wide and Super Wide lenses too. (Note it’s multi-select OR, as opposed to AND, which would have narrowed my selection.) Multi-select facets are the primary use case for gray ends. After making my first selection in the facet, I have clear visual cues from the check boxes, grays, and category counts that I can make additional choices in this facet. (B&H had an earlier implementation of multi-select without check boxes and grays, and they told me that with that earlier interface, no one noticed you could multi-select.)

-In ecommerce, gray ends can be good for merchandisers because they show abundance. For example, if there’s a facet for brands and I make a selection in a different facet that implicitly turns most of the brands into dead ends, I can still display to the shopper that I carry those brands. (Of course, as Barry Schwartz teaches us in The Paradox of Choice: Why More Is Less, additional choices can have the unexpected affect of making it so difficult for shoppers to make a decision that they’re more likely to leave empty handed.)

-You’ll notice that in facet:Price, the dead ends have been removed instead of grayed out. I can select $100-199 or $350-$499, but all the other choices in $0-$500 are gone. That’s because B&H treats price as a hierarchical facet, and from a UX standpoint, gray ends don’t work well with hierarchy. The same holds with multi-select and hierarchy, where instead of gray ends you get split ends. I’ll leave the reasoning as an exercise for the reader.

-Gray ends and multi-select can bring unwanted attention to dirty data, which everyone has in spades. In particular, things get mildly confusing for users when your records aren’t marked with at least one value from each gray end facet. For example, in the B&H example, there’s a facet for Camera Compatibility with just two choices, Full Frame or APS-C. You’ll notice in this screen shot that both are grayed out. That suggests that the remaining lenses aren’t compatible with either, but in this case, I think they’re just not tagged. The simple fix is to programmatically populate a value for “Unspecified” to gray end facets that don’t have at least one tag. The expensive fix is to clean all your data, but I’m not a utopian.

Gray ends are an exemplar of Edward Tufte’s advice to “always show comparisons adjacent in space rather than over time.” That is, if you want people to understand the difference between a “before” and “after” screen, when you redraw the screen, you’re asking them to rely on their memory to make the comparison. It’s always a risk to rely on memory, but it’s an even bigger risk here because we know people on the “before” screen were just focusing on the facet in which they made their explicit selection, rendering the others cognitively invisible. With gray ends, nothing has disappeared, so they get to compare the two states adjacent in space, no memory required. Tufte’s advice here is a classic for faceted UX work in general.

If you’re a faceted UX historian, the first website I know of with an implementation of gray ends was a mutual fund evaluator that Fidelity built in the UK around 2001. Gray ends are still pretty rare, but they shouldn’t be, so I expect we’ll start seeing more of this goodness for multi-select facets.