CenterForOpenScience · aaxelb · Jul 2, 2026 · Jul 2, 2026 · Jul 2, 2026 · Jul 10, 2026
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
@@ -9,7 +9,8 @@ In short, SHARE/trove holds metadata records that describe things and makes thos
 
 
 ## Parts
-a look at the tangles of communication between different parts of the system:
+a slightly simplified look at the tangles of communication between different parts of the system,
+as currently implemented:
 
 ```mermaid
 graph LR;
@@ -48,6 +49,50 @@ graph LR;
     subscribers-->oaipmh;
 ```
 
+### /trove/ingest
+a slightly simplified look at how metadata records are ingested, as currently implemented:
+```mermaid
+sequenceDiagram
+    participant ms as metadata source
+    box shtrove
+    participant ss as web server
+    participant sd as db (postgres)
+    participant sw as worker (celery)
+    participant sq as queues (rabbitmq)
+    participant si as indexer
+    participant se as elasticsearch
+    end
+    ms ->> ss: POST /trove/ingest
+    ss ->> sd: save ResourceIdentifier(s)
+    ss ->> sd: save Indexcard
+    ss ->> sd: save ResourceDescription(s)
+    ss ->> sq: enqueue derive task
+    ss ->> ms: 201 CREATED (success!)
+    sq -->> sw: receive derive task
+    sd <<-->> sw: load ResourceDescription(s)
+    sw ->> sd: save DerivedIndexcards
+    sw ->> sq: enqueue indexer message
+    sq -->> si: bulk receive messages
+    sd <<-->> si: bulk load metadata records
+    si ->> se: bulk index
+```
+
+### /trove/index-card-search
+a slightly simplified look at how search requests are served, as currently implemented:
+```mermaid
+sequenceDiagram
+    participant c as client
+    box shtrove
+    participant ss as web server
+    participant sd as db (postgres)
+    participant se as elasticsearch
+    end
+    c ->> ss: GET /trove/index-card-search
+    ss <<-->> se: query for result ids (and context)
+    ss <<-->> sd: load metadata records
+    ss ->> c: respond/stream search results
+```
+
 ## Code map
 
 A brief look at important areas of code as they happen to exist now.
@@ -91,9 +136,7 @@ Uses the [resource description framework](https://www.w3.org/TR/rdf11-primer/#se
 
 ### Identifiers
 
-Whenever feasible, use full URI strings to identify resources, concepts, types, and properties that may be exposed outwardly.
-
-Prefer using open, standard, well-defined namespaces wherever possible ([DCAT](https://www.w3.org/TR/vocab-dcat-3/) is a good place to start; see `trove.vocab.namespaces` for others already in use). When app-specific concepts must be defined, use the `TROVE` namespace (`https://share.osf.io/vocab/2023/trove/`).
+Whenever feasible, use full [IRI](https://www.rfc-editor.org/rfc/rfc3987.html) strings (utf-8) to identify resources, concepts, types, and properties that may be exposed outwardly (without converting to URI or using to send requests). Prefer using open, standard, well-defined namespaces wherever possible ([DCAT](https://www.w3.org/TR/vocab-dcat-3/) is a good place to start; see `trove.vocab.namespaces` for others already in use). When app-specific concepts must be defined, use the `TROVE` namespace (`https://share.osf.io/vocab/2023/trove/`).
 
 A notable exception (non-URI identifier) is the "source-unique identifier" or "suid" -- essentially a two-tuple `(source, identifier)` that uniquely and persistently identifies a metadata record in a source repository. This `identifier` may be any string value, provided by the external source.
 

diff --git a/README.md b/README.md
@@ -8,10 +8,8 @@ SHARE/trove (aka SHARtrove, shtrove) is is a service meant to store (meta)data y
 
 note: this codebase is currently (and historically) rather entangled with [osf.io](https://osf.io), which has its shtrove at https://share.osf.io -- stay tuned for more-reusable open-source libraries and tools for working with (meta)data
 
+see [how-to docs](./how-to/README.md) for help setting up and using various things
+
 see [ARCHITECTURE.md](./ARCHITECTURE.md) for help navigating this codebase
 
 see [CONTRIBUTING.md](./CONTRIBUTING.md) for info about contributing changes
-
-see [how-to/use-the-api.md](./how-to/use-the-api.md) for help using the api to add and access (meta)data
-
-see [how-to/run-locally.md](./how-to/run-locally.md) for help running a shtrove instance for local development
diff --git a/how-to/README.md b/how-to/README.md
@@ -0,0 +1,11 @@
+# shtrove how-to
+
+see [how-to/use-the-api.md](./use-the-api.md) for help using the api to add and access (meta)data
+
+see [how-to/run-locally.md](./run-locally.md) for help running a shtrove instance for local development
+
+see [how-to/use-the-admin-site.md](./use-the-admin-site.md) for help using the included admin site to manage indexing and ingested data
+
+see [how-to/wrangle-index-strategies.md](./wrangle-index-strategies.md) for help updating
+
+see [how-to/relate-to-osf.md](./relate-to-osf.md) for help understanding how this relates to OSF at the moment
diff --git a/how-to/relate-to-osf.md b/how-to/relate-to-osf.md
@@ -0,0 +1,147 @@
+# how this relates to osf.io (as of 2026-07)
+
+OSF uses shtrove to hold metadata records for all its public items, and uses the [trove search api](./use-the-api.md) to power various search/discover interfaces:
+- `osf.example/search`
+- `osf.example/preprints/<id>/discover`
+- `osf.example/registries/<id>/discover`
+- `osf.example/institutions/<id>`
+- `osf.example/institutions/<id>/dashboard`
+- `osf.example/user/<id>`
+
+## ingestion from OSF
+
+how OSF metadata (see [osf.io repo](https://github.com/CenterForOpenScience/osf.io)) gets indexed for search:
+1. gather an OSFMAP metadata record (as [linked open data](https://en.wikipedia.org/wiki/Linked_open_data))
+2. send that record to shtrove for ingestion
+3. derive multiple representations/serializations
+4. index by multiple indexing strategies in parallel
+
+### 1. gather metadata
+in `osf.metadata.osf_gathering`, each metadata property defined in OSF's metadata application
+profile ([OSFMAP](https://osf.io/8yczr)) has a "gatherer" function that gets data from OSF's
+database for the given item and yields values as portable/interoperable RDF linked data
+
+(note: this is the same way that an OSF item's metadata is gathered when a user downloads
+a metadata record or when sending an update to e.g. Datacite -- the RDF graph is an abstract
+data structure than can be serialized/rendered in various ways)
+
+after an update occurs to an item on OSF that is public and meant to be discoverable, 
+background tasks (see OSF's `api.share.utils`) gather all available metadata and send
+it to SHARE/trove
+
+### 2. send to shtrove
+metadata records are sent in [turtle](https://en.wikipedia.org/wiki/Turtle_RDF) format with
+a POST request to shtrove's `/trove/ingest` api -- for details of query params, see
+[how-to docs](https://github.com/CenterForOpenScience/SHARE/blob/develop/how-to/use-the-api.md#posting-index-cards)
+
+in OSF, all descriptive metadata is sent in a "main" record, while administrative
+metadata (specific to OSF usage, rather than describing the item itself) and,
+in the near future, custom CEDAR metadata records are gathered and sent in separate 
+"supplementary" records -- all these records are connected by the same `focus_iri`
+(identifying the item in OSF) and the knowledge graphs they contain must each be a tree
+rooted at that focus iri
+
+requests are authenticated by a pre-created access token -- for historical reasons,
+each OSF "preprint provider" and "registration provider" has a separate access token,
+and OSF has an additional static token that is used both for creating provider tokens and
+for indexing items that don't belong to a provider (e.g. projects)
+
+based on request headers and query params, shtrove ensures in the database:
+* `share.Source`/`share.SourceConfig` for the authenticated user, with...
+* `share.SourceUniqueIdentifier` for given `record_identifier`, with...
+* `trove.ResourceIdentifier` for given `focus_iri`
+
+(see `trove.digestive_tract.sniff`)
+
+then shtrove parses the request body, and may create (or update) in the database:
+
+* `trove.ResourceIdentifier` (for each described resource and its types)
+* `trove.Indexcard` (with identifiers and type-identifiers for the focus resource)
+* "resource descriptions" in turtle format:
+    * if non-supplementary, `trove.LatestResourceDescription` and `trove.ArchivedResourceDescription`
+    * if supplementary, only `trove.SupplementaryResourceDescription`
+
+(see `trove.digestive_tract.extract`)
+
+shtrove then enqueues a background `task__derive` and responds to OSF with `201 CREATED` success
+
+(note: from this point on, all background tasks are in SHARE's own queues and workers,
+unaffected by OSF's chronic celery-task logjams)
+
+### 3. derive representations
+after an update has been received for an `Indexcard`, `trove.digestive_tract.task__derive`
+combines that card's `LatestResourceDescription` and each current `SupplementaryResourceDescription`
+and creates a `DerivedIndexcard` for each deriver in `trove.derive.DEFAULT_DERIVER_SET`
+
+current derivers include:
+
+* "osfmap_json_mini": a [json-ld](https://en.wikipedia.org/wiki/JSON-LD) format using [OSFMAP](https://osf.io/8yczr) property keys, with some noisier properties (e.g. list of all files in a project) omitted -- this is the format expected by OSF's current search frontends
+* "sharev2_elastic": a json format for back-compat with 2016's [SHAREv2 search api](https://staging-share.osf.io/api/v2/search/creativeworks/_search), a raw pass-thru to elasticsearch
+* "oaidc_xml": a minimal xml format used in shtrove's [/oai-pmh/](https://staging-share.osf.io/oai-pmh/?verb=ListRecords&metadataPrefix=oai_dc) feed, following the [OAI Protocol for Metadata Harvesting](https://www.openarchives.org/OAI/openarchivesprotocol.html)
+
+once all relevant `DerivedIndexcard`s are saved, `task__derive` enqueues messages
+(using `share.search.index_messenger`) to notify shtrove's "indexer daemon" that the
+given `Indexcard` is ready for indexing
+
+### 4. index multiple ways
+shtrove's indexer daemon (see `share.search.daemon`) is meant for efficient bulk-indexing
+to multiple indexing strategies in parallel -- each index strategy has its own separate
+message queues and daemon threads for urgent and nonurgent updates, where each thread fetches
+a chunk of messages and streams updates to elasticsearch's bulk-index api (or according to logic
+in the indexing strategy)
+
+(note: on reflection, there's some logical redundancy between urgent/nonurgent queues and
+update/backfill message types that shouldn't cause problems but could likely be simplified)
+
+there are currently only two indexing strategies: "sharev2_elastic8" and "trovesearch_denorm"
+
+the "sharev2_elastic8" strategy loads the sharev2_elastic `DerivedIndexcard` and puts the json
+into an elasticsearch8 index as-is, to support the legacy sharev2 search (passthru to elasticsearch)
+
+the "trovesearch_denorm" strategy loads and traverses the combined knowledge graph
+(`LatestResourceDescription` and each `SupplementaryResourceDescription`) to populate two
+elasticsearch8 indexes: one index that supports requests to `/trove/index-card-search` and
+another index that supports `/trove/index-value-search`
+(see [search api docs](https://staging-share.osf.io/trove/docs) for details of expected behavior)
+-- allows indexing and querying on arbitrary graphs, including on metadata properties that
+were not known ahead of time (e.g. from CEDAR template instances)
+
+(note: this multi-strategy approach allows for experimenting and side-by-side comparison of
+different implementations of the trove search api, which offers some complex functionality that
+could be implemented a variety of ways -- if there were multiple trovesearch strategies
+(as there was before settling on trovesearch_denorm), can select a non-default one with
+the `indexStrategy` query param)
+
+### OSF-thru-shtrove ingestion sequence
+(slightly simplified)
+```mermaid
+sequenceDiagram
+    box OSF
+    participant oq as queues (rabbitmq)
+    participant od as db (postgres)
+    participant ow as worker (celery)
+    end
+    box shtrove
+    participant ss as web server
+    participant sd as db (postgres)
+    participant sw as worker (celery)
+    participant sq as queues (rabbitmq)
+    participant si as indexer
+    participant se as elasticsearch
+    end
+    oq -->> ow: receive update task
+    od <<-->> ow: gather metadata
+    ow ->> ss: POST /trove/ingest
+    ss ->> sd: save ResourceIdentifier(s)
+    ss ->> sd: save Indexcard
+    ss ->> sd: save ResourceDescription(s)
+    ss ->> sq: enqueue derive task
+    ss ->> ow: 201 CREATED (success!)
+    sq -->> sw: receive derive task
+    sd <<-->> sw: load ResourceDescription(s)
+    sw ->> sd: save DerivedIndexcards
+    sw ->> sq: enqueue indexer message
+    sq -->> si: bulk receive messages
+    sd <<-->> si: bulk load metadata records
+    si ->> se: bulk index
+```
diff --git a/how-to/use-the-admin-site.md b/how-to/use-the-admin-site.md
@@ -0,0 +1,94 @@
+# how to use the admin site
+
+shtrove has a [django admin site](https://docs.djangoproject.com/en/stable/ref/contrib/admin/)
+at `/admin/` that can be used for interacting with the database and managing search indexes
+-- this document describes how to use the admin site for some common tasks
+
+## prerequisite: creating an admin user
+an admin user is automatically created as part of initial setup (database migrations)
+when the `SHARE_ADMIN_PASSWORD` environment variable (or django setting) is non-empty,
+with username from `SHARE_ADMIN_USERNAME` (default "admin")
+
+in local development, when [DEBUG](https://docs.djangoproject.com/en/stable/ref/settings/#debug)
+is true and you shouldn't have any sensitive data, `SHARE_ADMIN_PASSWORD` defaults to "password"
+
+other `ShareUser` instances (perhaps created by logging in with OSF OAuth, below) can be granted
+admin-site permissions -- find them at `/admin/share/shareuser/`, set "Staff status"
+(`is_staff`) to give access to the admin site, and either select specific permissions
+or set "Superuser status" (`is_superuser`) to give all permissions.
+
+### with OSF account
+when set up with OSF, you can also click "login with osf" (or go to `/accounts/osf/login/`)
+to login with an OSF account -- this will create a `ShareUser` instance that can be granted
+admin-site permissions, as above
+
+## how to view metadata about a specific resource
+to find metadata about a specific resource, use the `Indexcard` model as a starting point -- click
+on "Indexcards" in the "TROVE" section of `/admin/` (or go to `/admin/trove/indexcard/` directly)
+and search by either indexcard uuid or source-unique identifier (e.g. OSF id)
+
+an Indexcard's detail view has links to:
+- current version of core metadata ("latest resource description")
+- past versions of core metadata ("archived description set")
+- current supplementary metadata ("supplementary description set")
+- different serializations of current metadata ("derived indexcard set")
+
+if an Indexcard has been marked deleted, its "latest" and "derived" data will be removed
+but "archived" will remain
+
+## how to view index statuses
+click on the "elasticsearch indexes" link in the admin-site header (or go to `/admin/search-indexes`)
+to see the current status of all configured index strategies, including their queues and indexes
+
+for each available index strategy, this page provides:
+- status of urgent and non-urgent queues
+    - depth (count of enqueued messages)
+    - approximate indexing rate over the past 30 seconds
+- for the current version of this strategy:
+    - backfill status (if any)
+        - for manual tracking of backfill progress
+        - click link to view/reset backfill status and error message, if any
+    - lifecycle controls (each displayed only when safe to do):
+        - setup (create indexes -- note only the current version can be set up)
+        - start keeping live (receive new metadata updates)
+        - start backfill (schedule task to enqueue indexer messages for all existing cards)
+        - mark backfill complete (to be done manually, when queue clear)
+        - make default for searching within this strategy (only when backfill marked complete)
+        - delete -- must confirm by typing "really really"
+    - status of each specific index within this strategy version:
+        - key (short name within the strategy)
+        - created date
+        - kept-live state (whether it receives new metadata updates)
+        - doc count (as of last index refresh), with direct link to view index
+        - link to elasticsearch mappings
+        - full index name
+- for each prior version of this same strategy (that still has existing indexes):
+    - unique checksum of index mappings/settings
+    - lifecycle controls (each displayed only when safe to do):
+        - start keeping live (receive new metadata updates)
+        - make default for searching within this strategy (only when backfill marked complete)
+        - delete (only when not default for searching) -- must confirm by typing "really really"
+    - status of each specific index within this strategy version:
+        - key (short name within the strategy)
+        - created date
+        - kept-live state (whether it receives new metadata updates)
+        - doc count (as of last index refresh), with direct link to view index
+        - link to elasticsearch mappings
+        - full index name
+
+see [how to wrangle index strategies](./wrangle-index-strategies.md) for more context
+
+## how to diagnose ingestion problems
+
+if something is missing from search:
+1. does it have an Indexcard (see above for finding it)? if not:
+    - requests to `/trove/ingest` are either failing or not being sent (check OSF task logs? or shtrove web-server errors?)
+2. does the Indexcard have a `deleted` date? if so:
+    - the source (OSF) apparently sent a DELETE request (check why?)
+3. does the Indexcard have `DerivedIndexcard`s? if not:
+    - `task__derive` may be failing (check for errors at `/admin/share/celerytaskresult/`)
+    - ...or not running at all (check for a backup in `digestive_tract` queues, or problems with shtrove's `worker`)
+4. is there a large queue depth at `/admin/search-indexes`? if so:
+    - check shtrove's `indexer` logs for errors
+5. if all above seems fine:
+    - compare the search query to the metadata record -- any mismatched assumptions?
diff --git a/how-to/use-the-api.md b/how-to/use-the-api.md
@@ -11,7 +11,7 @@
 (see [openapi docs](https://share.osf.io/trove/docs) for detail and available parameters)
 
 
-### Posting index-cards
+## Posting index-cards
 > NOTE: currently used only by other COS projects, not yet for public use, authorization required
 
 `POST /trove/ingest?focus_iri=...`: