Embeddings

Researcher Profiles can include pre-computed vector embeddings for semantic search (finding profiles by meaning rather than keywords). A profile with embeddings is called a “Searchable Profile.”

This document defines the embedding index format: the files in embeddings/, how they’re structured, and how consumers verify compatibility.

---

1. Why flat blobs

Vector search over HTTP means reading the entire vector set. SQLite is optimized for random row access, which is the wrong pattern here. Flat blobs are one contiguous read per model.

.cache/embeddings.sqlite is a derived local index at tier private. The servable form lives in embeddings/.

---

2. Two embedding surfaces

Surface	Location	Purpose
Per-profile chunks	<slug>/embeddings/	One vector per text chunk
Collection centroids	collection/embeddings/	One aggregate vector per profile

Both use the index.json and blob format below. They differ in these fields:

Field	Per-profile chunks	Collection centroids
normalized	false	true
row_key	chunk_index	slug
rows	integer chunk indexes	profile slugs
chunks file (section 5)	present	absent
probe	required	present when the contributing profiles supply one

---

3. embeddings/index.json

Field	Type	Description
backend_spec	string	Model identifier (e.g. st:all-MiniLM-L6-v2)
file	string	Relative path to blob (e.g. st-all-minilm-l6-v2.bin)
dtype	string	float32
byte_order	string	little
dim	integer	Vector dimensionality
count	integer	Number of rows
layout	string	row_major
normalized	boolean	Whether rows are L2-normalized
metric	string	cosine
row_key	string	chunk_index or slug
rows	array	Row manifest in blob order
sha256	string	Hex digest of blob
probe	object	{text, vector} for model verification

The blob filename is a lowercased slug of backend_spec with : and / replaced by -. Consumers MUST resolve the blob by the file field rather than rebuilding the slug.

3.1 Probe

Every per-profile index MUST include a probe. A collection centroid index includes one when the contributing profiles provide it. A consumer embeds probe.text with its own model and computes cosine similarity against probe.vector:

• Above 0.99: verified
• 0.9 to 0.99: degraded (usable but flagged)
• Below 0.9: incompatible

---

4. Blob format

Raw, headerless binary:

• Layout: row-major, contiguous
• Size: exactly count * dim * 4 bytes
• Type: IEEE 754 float32
• Byte order: little-endian

No header, no padding, no magic bytes.

---

5. Chunk metadata

Per-profile indexes carry a chunks file next to the blob: the file path with .bin replaced by .chunks.json (for example st-all-minilm-l6-v2.chunks.json). It is a JSON array with one entry per row:

Field	Type	Description
source_type	string	expertise, soul, paper_summary, paper_abstract, grant, cv, web
source_id	string	Document identifier
chunk_index	integer	Zero-based index within source
section	string/null	Section heading
char_count	integer	Chunk text length

The file does not contain the chunk text; fetch the source document to display a hit.

---

6. Privacy

Chunk tiers follow the derivation rule. A chunk’s tier is the most restrictive of the profile default, the role default for its source type, and the effective tier of the source artifact it was cut from (so a tier set on one file, a section tier such as soul, and derivedFrom inheritance all apply). A chunk whose source artifact is unknown is private. A chunk from a private source is itself private and MUST NOT appear in a public blob. cv, web, and grant chunks resolve to private by their role default and are dropped from a public blob accordingly.

The same rule governs every use of a chunk at serve time, not only the blob: a chunk the caller may not read MUST NOT be returned, and MUST NOT contribute to a score or to evidence shown to that caller (for example the chunk half of a /match score and its top_papers).

Collection centroids MAY be computed over all chunks (including private) because a single averaged vector cannot reconstruct individual inputs.

---

7. Cross-model comparison

Consumers MUST compare backend_spec strings for exact equality before computing similarity. They MUST refuse to compute when the strings differ, because cosine similarity across models is meaningless.