The profile format

This document explains the on-disk researcher profile format: what a profile directory contains, how the pieces relate, and the design thinking behind treating a profile as a portable directory of files. It is background reading. For exact field lists, see the JSON Schema reference; for the Python objects, see the Python API reference.

A profile is a directory

A researcher profile is not a database row or a single document. It is a directory of plain files, keyed by a slug — a lowercase, hyphenated identifier that is also the directory name:

<slug>/
├── config.json                         # {schema_version: 2, name, orcid, slug, level, ...}
├── profile.yaml                        # structured metadata (ProfileMetadata; carries level)
├── personality/
│   ├── expertise.md                    # per-topic sections, each citing paper_ids
│   └── SOUL.md                         # narrative voice / mission
├── sources/
│   ├── papers.yaml                     # {papers: [PaperRecord, ...]}
│   ├── citations.json                  # citation graph over the papers (optional)
│   ├── papers/<paper_id>.md            # full text (optional)
│   ├── summaries/<paper_id>.summary.md # per-paper summary
│   ├── grants.yaml                     # {grants: [GrantRecord, ...]} (deep only)
│   ├── cv.md                           # ingested CV (deep only)
│   └── web/<n>-<host>.md               # extracted web pages (deep only)
└── meta/
    └── embeddings.sqlite               # embedding index (optional; sqlite-vec)

Everything except profile.yaml is optional as far as loading goes — the SDK tolerates a missing sources/, a missing citations.json, or an absent index. This tolerance is deliberate: profiles are often incomplete while they are being assembled, and a partial profile should still load and be inspectable.

Why a directory of files

Keeping a profile as files rather than a single blob has three consequences worth naming:

• Portable. You can zip it, copy it, check it into version control, or diff it. The embedding index lives inside the directory (meta/embeddings.sqlite), so a profile carries its own search index with it.
• Inspectable. Every artifact is human-readable YAML or Markdown, except the binary sqlite index. You can read a profile without the package installed.
• Composable. A directory of profile directories is itself a meaningful unit — the thing a ProfileRegistry or the HTTP server sits over.

The files

config.json and the schema_version gate

config.json holds identity and pipeline bookkeeping. Its most important field is schema_version, which defaults to 2. Version 2 is the current, post-restructure format. The gate exists so that an old-format (version 1) profile fails fast on load rather than being silently coerced into a shape it does not actually have. If you are producing profiles, set schema_version: 2.

profile.yaml — the metadata

profile.yaml is the structured record: name, ORCID, affiliation, field and subfields, a prose summary, training and career history, expertise topic labels, research interests, collaborators, and disambiguation evidence. It is validated by the ProfileMetadata model. The model is tolerant — it allows extra keys and strips surrounding whitespace — but it does enforce a few invariants, most notably that name is non-empty and that any orcid matches the ORCID format.

personality/expertise.md and SOUL.md

These two Markdown documents carry the qualitative voice of the profile that structured metadata cannot. expertise.md is organized into topic sections, each describing what the researcher does and citing supporting papers by paper_id in square brackets ([doe2016example]). SOUL.md is a first-person-ish narrative of how the researcher frames problems, what they value, and what they distrust.

Both are loaded as raw strings. The persona methods (ask, review, innovate, riff) inject them into the LLM system prompt so generated text is grounded in the researcher’s own framing.

sources/papers.yaml — the corpus

papers.yaml is a mapping with a single papers: key whose value is a list of paper records. A bare top-level list is rejected by the strict PapersFile validator — it is a marker of old-format drift. Each entry carries a title (required) plus optional identity and metadata fields: paper_id, year, journal, first_author, authorship position, corresponding-author flag, a pipeline status, and more. See the papers schema.

sources/summaries/<paper_id>.summary.md

One Markdown summary per paper, named by the paper’s paper_id. The SDK exposes these as a lazy mapping of paper_id -> markdown body. Summaries may optionally begin with a YAML frontmatter block conforming to the SummaryFile schema (paper_id, source_kind, source_hash, written_at); when present that block records provenance. The loader reads the whole file as the summary body and does not require frontmatter.

sources/grants.yaml, sources/cv.md, sources/web/ — the non-paper sources

These three are the supplied private resources that define a deep profile (see Profile depth levels); lite and full profiles do not have them.

grants.yaml is a mapping with a grants: key holding a list of grant records — each with an id and title (required) plus optional funder, award number, role (pi/co_pi/co_i/other), status (funded/pending/completed), dates, abstract, provenance source, and URL. As with papers.yaml, a bare top-level list is rejected. See the grants schema.

cv.md is the person’s CV converted to Markdown, and web/<n>-<host>.md holds extracted main text from each of their websites, one file per page. Both carry a YAML frontmatter block recording provenance (source/url and fetched_at); the frontmatter is stripped before indexing so it never becomes searchable content.

Each of the three is independently optional — a deep profile with grants but no CV is well-formed. What matters is that at least one is supplied, since that is the whole point of the level.

meta/embeddings.sqlite

An optional per-profile vector index built by the [embeddings] extra. It stores chunk-level embeddings of the expertise sections, the SOUL document, and each paper summary, along with index metadata (the backend name, schema version, build timestamp). Because it lives inside the profile directory, the profile remains self-contained and portable. See Build and search the embeddings index.

What the index contains depends on the profile’s depth level. A full profile indexes the expertise sections, the SOUL document, and the paper summaries (expertise, soul, paper_summary chunks). A lite profile has none of those, so its index is abstract-only: it holds paper_abstract chunks built from each paper’s abstract text and nothing else. A deep profile indexes everything full does plus a grant, cv, or web chunk set for whichever non-paper sources it actually carries.

A spec plus an SDK

The package plays two roles at once, and it helps to keep them separate in your mind.

As a specification, the package defines what a valid profile is. The authoritative definition is the set of Pydantic v2 models in researcher_profiles.schema (on-disk artifacts) and researcher_profiles.api_models (HTTP wire types). JSON Schema files generated from those models are checked into schemas/, so a consumer can validate a profile directory without installing the package — using any JSON Schema validator in any language. See Export JSON Schemas.

As an SDK, the package gives you everything you can do with a conforming profile: load and inspect it (ResearcherProfile), collect many of them (ProfileRegistry), search over them (the [embeddings] extra), serve them over HTTP (the [api] extra), consume them remotely (the [client] extra), generate grounded text as the researcher (the [llm] extra), and load them into a relational database (the [sql] extra).

Profile depth levels

Not every profile carries the same amount of content. A profile has a level — one of lite, full, or deep — that records how deeply it was built. The field lives in both config.json and profile.yaml; when it is absent, the profile is treated as full. A profile is built at a level by whatever process emits it; the package only reads the level, it does not set it.

The three tiers form a single axis of increasing depth:

• lite — a public bibliometric identity: metadata plus the works list, with an abstract-only search index (paper_abstract chunks over paper abstracts). A lite profile has no per-paper summaries, no expertise.md or SOUL.md, and no persona. It is the shallowest useful profile.
• full — the complete bundle: paper summaries, personality/expertise.md, personality/SOUL.md, and a full index over all of those. A full profile is persona-ready. This is the default a profile falls back to when no level is set.
• deep — everything full has, over an exhaustive corpus, plus non-paper sources: sources/grants.yaml, sources/cv.md, and sources/web/. Its index carries grant/cv/web chunks alongside full’s.

Levels are also an input hierarchy

The more useful way to read the tiers is by what you must supply to reach each one:

Level	What you must provide	Where the content comes from
lite	name + ORCID	public APIs only (ORCID, OpenAlex)
full	name + ORCID	public APIs + public PDFs
deep	name + ORCID and grant records and/or a CV and/or website URLs	the above, plus supplied private resources

lite and full are fully automatable from a seed identity: everything they contain is publicly discoverable. deep is not, and that is the point. Grant records, a CV, and a person’s website URLs cannot be found through any public API — they must be given, per person. A deep profile’s value comes precisely from those supplied inputs, so a deep build with none of them configured is just a full build wearing a different label, and a producer should refuse it rather than emit it.

The supplied inputs are named in config.json: grants_source (grants-data / manual / none), reporter_supplement (an opt-in public NIH RePORTER supplement), cv_source (a path or URL), and websites (a list of URLs). A human fills those in; everything downstream — fetching, converting, chunking, indexing — is automated.

Level and persona availability

The consequence that flows from the level is persona availability. A profile has a synthesized persona when it carries non-empty expertise.md and SOUL.md — the two documents the persona methods build their system prompt from. Only full and deep profiles expose a persona; a lite profile never synthesizes those documents and so is never persona-ready.

This is exactly what the has_persona property reports, and it is what the persona methods (ask / review / innovate / riff) gate on: calling one on a profile without a synthesized persona raises PersonaUnavailableError rather than role-playing an empty persona. Over HTTP the same condition returns 409.

Creation is out of scope

The package is intentionally creation-method-agnostic. It does not tell you how to build a profile, and it does not ship a build pipeline. A conforming profile directory can be produced by any process that emits the format — written by hand (as in the tutorial), generated by a script, or assembled by a larger pipeline. The package’s contract is narrow and stable: define the format, validate against it, and provide the tools to consume it.

One consequence: a couple of SDK surfaces that audit a profile against richer build-time quality contracts (the validate() method and the audit CLI subcommand) depend on an external build package and are not part of the core install. Schema-level validation — the part that answers “is this a well-formed profile?” — is fully available in core. See Validate a profile for the distinction.