Short answer

A searchable geological data index is a catalogue that lets anyone find the right dataset by where, when, what and whose — rather than emailing colleagues or digging through network drives. It is built on three layers: a spatial store (PostGIS) for the data, structured metadata (ISO 19115 lineage, domain vocabularies like GeoSciML) describing each asset, and a search API (STAC for imagery/raster collections, OGC API - Features or Records for vector and catalogue queries). The payoff is reuse: surveys, boreholes and samples already paid for become discoverable instead of re-acquired.

Why a catalogue, not a folder

Geological organisations accumulate enormous, heterogeneous archives — borehole logs, assay tables, geophysical surveys, satellite scenes, field photos, reports — usually scattered across drives with cryptic names. The cost is invisible until someone needs "all the magnetic surveys flown over this licence block before 2015" and there is no way to answer except manual hunting. A searchable index makes the corpus answerable by query, which is the difference between a data asset and a data graveyard.

The design goal is that every dataset is findable by at least four facets:

  • Spatial: does it intersect this area of interest?
  • Temporal: was it collected in this date range?
  • Type: is it a borehole, a DEM, a Sentinel-2 scene, a geochemistry table?
  • Provenance / attributes: survey method, instrument, licence, owner, quality flags.

STAC for raster and imagery collections

For imagery and gridded data, the SpatioTemporal Asset Catalog (STAC) is the practical standard. STAC describes each scene or product as a JSON Item carrying:

  • a geometry and bbox (the footprint),
  • a datetime,
  • assets (links to the actual files — COGs, sidecars), and
  • typed properties via extensions (e.g. EO, projection, raster extensions).

Items are grouped into Collections (e.g. "regional ASTER mosaics", "UAV orthophotos"), and the whole thing is either a static catalogue (plain JSON files servable from object storage) or a STAC API that adds query endpoints. A STAC API search lets a client ask, in one request, for all items within a polygon, within a date range, with cloud cover below a threshold — exactly the multi-facet query a folder cannot serve. Pair STAC with Cloud-Optimized GeoTIFFs and clients can stream the relevant window of each matched asset without downloading the whole file.

PostGIS and OGC APIs for vector and point data

STAC is built around assets and footprints; for dense feature data — boreholes, samples, mapped contacts — a spatial database plus a feature API is the better fit.

Store the features in PostGIS with a spatial index, so location queries are fast even over millions of points:

CREATE INDEX idx_boreholes_geom ON boreholes USING GIST (geom);
CREATE INDEX idx_boreholes_date ON boreholes (collared_date);

-- "boreholes inside the licence area, collared since 2020"
SELECT id, depth_m, collared_date
FROM boreholes b
JOIN licence_areas a ON a.code = 'PL-1187'
WHERE ST_Intersects(b.geom, a.geom)
  AND b.collared_date >= DATE '2020-01-01';

Expose that to consumers through an OGC API - Features endpoint (served by pygeoapi or GeoServer over the same PostGIS tables). Clients then issue spatial, temporal and CQL attribute filters over HTTP — for example bbox, datetime, and filter=depth_m>500 — with JSON output and pagination, no SQL access required.

For cataloguing the datasets themselves (rather than individual features), OGC API - Records (the successor to CSW) provides a record-search endpoint over ISO 19115 metadata.

Metadata and semantics make search meaningful

An index is only as good as the metadata it searches. Two layers matter:

  1. Structural metadata — ISO 19115/19139 for dataset descriptions, including the lineage element (how the data was produced), spatial/temporal extent, CRS, and licence. This is what catalogue search queries against.
  2. Domain semantics — geological meaning needs controlled vocabularies, or "granite" in one dataset and "granitic rock" in another will never match. GeoSciML (the IUGS/OGC geoscience data model) and the CGI vocabularies (lithology, geologic time, contact type) give shared terms so an index can group equivalent concepts and federate across sources.

Encode units, CRS, source vintage and licence directly into each item's properties so the facts travel with the asset and survive handover, rather than living in a forgotten spreadsheet.

Worked example: indexing a mixed archive

A consultancy wants one searchable index over imagery, DEMs and 40 years of boreholes:

  1. Inventory and normalise. Reproject everything to a common reference (e.g. EPSG:4326 for the catalogue footprints), validate borehole geometries with ST_MakeValid, and harmonise lithology terms to the CGI vocabulary.
  2. Catalogue rasters with STAC. Convert scenes/DEMs to COGs, write a STAC Item per asset (footprint, datetime, instrument, resolution) into Collections, and serve via a STAC API (e.g. stac-fastapi over PostGIS with pgstac).
  3. Load features into PostGIS. Boreholes, samples and mapped lines into typed tables with GiST and date indexes.
  4. Expose feature search through pygeoapi (OGC API - Features) and dataset search through OGC API - Records over the ISO metadata.
  5. Front it with a map search UI that fans a single query out to both APIs by bbox and date, returning a unified result list.
  6. Automate ingestion so new surveys are catalogued on arrival (footprint, metadata, vocabulary mapping) rather than accumulating uncatalogued.

The test of success: a geologist draws a polygon, sets a date range, picks "geophysics", and gets every matching dataset — regardless of which silo it physically lives in.

Pitfalls and why they happen

  • Indexing files without metadata. A catalogue of filenames is barely better than a folder; without footprint, date and type, the facets that make search useful do not exist.
  • Free-text geology terms. Without a controlled vocabulary, semantically identical units never match across datasets — the index fragments.
  • No spatial index. ST_Intersects without a GiST index does a full table scan; on large archives the search is unusably slow.
  • Mutable footprints / vintages. Overwriting an item when data is reprocessed breaks the ability to find what a past analysis used. Version items instead.
  • Building search before defining the questions. An index designed without knowing the real queries ends up exposing storage structure rather than answers.

Quality control

  • Confirm every catalogued item has a valid footprint, datetime, CRS and licence before it is searchable.
  • Validate that domain terms map to the controlled vocabulary; reject or flag unmapped values.
  • Test representative queries (a known polygon, a known date range) and confirm expected datasets are returned and nothing is silently missing.
  • Check spatial indexes exist and queries use them (EXPLAIN ANALYZE).
  • Re-run a sample query after each ingestion to confirm new data appears and old results are unchanged.

Bathyl perspective

We build indexes so that data already collected stays findable and reusable for years, not lost on the next drive migration. Each catalogued asset carries its footprint, vintage, CRS, licence and harmonised geological terms, so a query returns answers a reviewer can trust and trace back to source.

Related reading

Sources