How to build and search the embeddings index

The [embeddings] extra adds a per-profile vector store and semantic search over a researcher’s corpus, plus cross-profile ranking via ProfileRegistry.

The index lives inside the profile at meta/embeddings.sqlite, so it travels with the profile directory.

Build the index for one profile

Importing researcher_profiles.embeddings attaches .build_index() and .search() to ResearcherProfile:

import researcher_profiles.embeddings  # attaches the methods
from researcher_profiles import ResearcherProfile

p = ResearcherProfile.from_files("path/to/jane-doe")
report = p.build_index()
print(report.added, report.updated, report.skipped, report.backend_name)

The index chunks and embeds the expertise sections, the SOUL document, and each paper summary. report is an IndexReport with added, updated, skipped, removed, backend_name, and duration_s.

Rebuilds are incremental: unchanged chunks are skipped by content hash. Pass force=True to drop and rebuild from scratch:

p.build_index(force=True)

From the CLI

researcher-profiles index path/to/jane-doe
researcher-profiles index path/to/jane-doe --force
researcher-profiles index path/to/jane-doe --backend st:all-MiniLM-L6-v2

Expected output:

backend=st:all-MiniLM-L6-v2 added=4 updated=0 skipped=0 removed=0 duration=5.17s

Search one profile

search returns a list of SearchHit ranked by cosine similarity (0–1, higher is better):

hits = p.search("region set enrichment", k=3)
for h in hits:
    print(round(h.score, 3), h.source_type, h.source_id, h.section)

Expected output:

0.595 expertise expertise Region set analysis
0.568 paper_summary doe2016example None
0.344 soul soul None

Filter by source type. Which types exist depends on the profile’s level: paper_summary, expertise, and soul on a full profile; paper_abstract on a lite one; plus grant, cv, and web on a deep profile that carries those sources.

hits = p.search("region set enrichment", k=5, filter={"source_type": "paper_summary"})

An empty or whitespace-only query returns an empty list rather than raising.

From the CLI

researcher-profiles search path/to/jane-doe "region set enrichment" -k 3
researcher-profiles search path/to/jane-doe "chromatin" --type paper_summary

Rank across many profiles

ProfileRegistry sits over a directory of profiles and ranks them against a query using a centroid prefilter, a chunk-level re-rank, and optional MMR diversification. Each profile must already have a built index:

from researcher_profiles import ProfileRegistry

reg = ProfileRegistry.from_directory("path/to/profiles")
matches = reg.rank_against("region set enrichment analysis", k=5)
for m in matches:
    print(round(m.score, 3), m.profile.slug, m.evidence.top_papers)

rank_against returns Match objects carrying the matched profile, a score, and a MatchEvidence record (top chunks, top papers, overlapping topics, centroid score). To rank against an indexed subset only, pass ProfileRegistry.from_directory(path, skip_unindexed=True).

Backends

The default backend is a sentence-transformers model (st:all-MiniLM-L6-v2). The first build downloads the model. Override the backend per build with the backend= argument / --backend flag, or globally with the RESEARCHER_PROFILES_EMBEDDING_BACKEND environment variable. An index records the backend it was built with; opening it with a different backend raises IndexBackendMismatchError unless you rebuild with force=True.

Setting RESEARCHER_PROFILES_DISABLE_INDEX=1 turns build_index into a no-op — useful for search-only consumers or tests.