Search your beans with Lucene — Search

We already mapped beans to documents and indexed them. Let’s see what’s happening when you type in the box, add a filter, exclude two keys.

Start with no criteria at all and enter the MatchAll query:

// Open an index searcher using the same writer
IndexSearcher searcher = new IndexSearcher(DirectoryReader.open(writer));

// Search and retrieve the 10 first hits
TopDocs hits = searcher.search(MatchAllDocsQuery.INSTANCE, 10);

// Join stored ids back to beans:
for (var hit : hits.scoreDocs) {
    String id = searcher.storedFields().document(hit.doc).get("id");
    Track track = trackDatabase.get(id);
    if (track != null) {
        // Do something with the track, e.g., print it or add it to a list
    }
}

Type “Bob”

Analyze the query string and build the query from the tokens:

TokenStream ts = analyzer.tokenStream("title", "Bob");
// → bob

// build query for bob
Query bob = new BooleanQuery.Builder()
  .add(new BoostQuery(new TermQuery(new Term("title", "bob")), 4.0f), BooleanClause.Occur.SHOULD)
  .add(new BoostQuery(new PrefixQuery(new Term("title", "bob")), 1.0f), BooleanClause.Occur.SHOULD)
  .add(new BoostQuery(new TermQuery(new Term("artist", "bob")), 3.0f), BooleanClause.Occur.SHOULD)
  .add(new BoostQuery(new PrefixQuery(new Term("artist", "bob")), 0.75f), BooleanClause.Occur.SHOULD)
  // … genre^2 / album^1.5 / label^1 / comment^0.5 (+ prefixes) …
  .setMinimumNumberShouldMatch(1)
  .build();

// Search and retrieve the 10 first hits
TopDocs hits = searcher.search(bob, 10);

buildQuery is the fold the playground expands: each analyzed field is a SHOULD TermQuery with a boost, plus a trailing PrefixQuery at a quarter of that boost on the last typed token:

Lucene prints it as:

((title:bob)^4.0 (title:bob*)^1.0 (artist:bob)^3.0 (artist:bob*)^0.75
 (genre:bob)^2.0 (genre:bob*)^0.5 (album:bob)^1.5 (album:bob*)^0.375
 (label:bob)^1.0 (label:bob*)^0.25 (comment:bob)^0.5 (comment:bob*)^0.125)~1

Several tokens are AND-ed (MUST each token’s builder). Only the last one is a prefix; earlier words stay exact. bob sincla requires bob and a sincla… prefix. ouse does not find House — not an infix.

You remember the posting list we generated in the previous post?

artist:bob       →  1, 2, 3
artist:claude    →  4, 5
artist:francois  →  4, 5, 6
artist:marley    →  2
artist:sinclar   →  1, 3
artist:valery    →  6

You see the match with our search (artist:bob)^3.0?

Add a filter (include Club)

Wrap the free-text builder as MUST and add a FILTER — constrain, do not score. Same leaf the playground emits:

// Single filter on genre for club
Query genre = new TermQuery(new Term("genre.raw.normalized", "club"));

// Combine queries in a bool query
Query bool = new BooleanQuery.Builder()
  .add(bob, BooleanClause.Occur.MUST)
  // Filter in
  .add(genre, BooleanClause.Occur.FILTER)
  .build();

Lucene prints it as:

+(((title:bob)^4.0 (title:bob*)^1.0 … )~1) #genre.raw.normalized:club

Index a normalized keyword twin (genre.raw.normalized) next to genre.raw, so Club and club hit the same docs. Do not wildcard it.

Exclude two keys (4A and 4B)

Same MUST + FILTER, plus MUST_NOT for excluded keys. Several keys on the same dimension are OR (SHOULD, minShouldMatch = 1):

// Filter on keys 4a or 4b
Query keys = new BooleanQuery.Builder()
    .add(new TermQuery(new Term("key.code", "4a")), BooleanClause.Occur.SHOULD)
    .add(new TermQuery(new Term("key.code", "4b")), BooleanClause.Occur.SHOULD)
    .setMinimumNumberShouldMatch(1)
    .build();

Query bool = new BooleanQuery.Builder()
    .add(bob, BooleanClause.Occur.MUST)
    .add(genre, BooleanClause.Occur.FILTER)
    // Filter out
    .add(keys, BooleanClause.Occur.MUST_NOT)
    .build();

Lucene prints it as:

+(((title:bob)^4.0 … )~1) #genre.raw.normalized:club -((key.code:4a key.code:4b)~1)

This means:

Scoring plays an important role in determining the relevance of search results. Some clauses contribute to the score, while others act as filters or exclusions.

A BooleanQuery with only MUST_NOT matches nothing — add a MUST MatchAllDocsQuery if the user excludes without typing. Blank everything is MatchAllDocsQuery.

Give me the “best” results first (sort by score)

If you want to understand how the score is computed, you can use the explain method on the IndexSearcher for a specific document and query:

// Explain how the score is computed for the first hit
searcher.explain(query, hits.scoreDocs[0].doc);

This gives something like:

That tree is Lucene’s BM25 breakdown. For each matching term the score is:

N is documents that have the field, n those that contain the term, freq occurrences in this document, dl / avgdl the field length vs the average. k1 and b are the usual BM25 knobs (defaults 1.2 and 0.75).

Take title:bob on doc 3473 from the screenshot. Across the index, 7 titles contain bob (n = 7) out of 4 322 docs that have a title (N = 4 322). In this hit, bob appears once in the title (freq = 1), the title is 5 tokens long (dl = 5), and the average title length is about 4.24 (avgdl). The field boost is 4:

Same walk for album:bob: only 3 albums contain the term (n = 3) out of 3 088 docs with an album (N = 3 088), still freq = 1, dl = 5, avgdl \approx 3.77, boost 1.5:

Add the prefix leaves (title:bob* → 1, album:bob* → 0.375) and you get the hit score: \(10.77 + 1 + 4.08 + 0.375 \approx 16.23\).

The full demo lives on GitHub: lucene-search-tracks.