Search is a hard problem, it really is. Let me show this by using an example of a food chain that wants to add Drupal as their homepage of the whole chain. Of course, like many other organisations they do not only have Drupal running but also a subset of other web frameworks open and closed source systems.
Drupal Search has a great ecosystem of modules to integrate with technologies such as Solr. However, it needs more vision and direction to grow and be a great platform where other developers feel comfortable with and are able to make the right decisions. Also
We all love test and staging environments but it becomes a problem when you have Solr integrated in your project and you have a test solr, a staging solr and a production solr core. To re-index a site is not a big problem, but what if you want to go live and switch indexes immediately?
Drupal.org has wrestled with this problem. So I wanted to show you how they do it.
Recently I've been involved in drupal.org by upgrading the site to the latest version of Apache Solr Search Integration (from 6.x-1.x to 6.x-3.x, and in the near future to 7.x-1.x). This upgrade path is necessary as we still want to have a unified search capability across Drupal 6 and Drupal 7 sites, for example groups.drupal.org and drupal.org.
The module already exists since the early days of Drupal 6, where there was not even a discussion about entities. We were talking about nodes as if they were the only important thing that would ever exist in Drupal. Anything could be done with nodes, you name it! And I'll tell you... I've seen some ugly things that people have done with those poor nodes...
Have you ever worked with the Apache Solr Search Integration project? I certainly hope so! At Acquia, we have invested a lot of time to make this module stable for Drupal 7. We did not make any sacrifices in regards of speed and/or optimizations.