Design

The purpose of these design documents is to describe the design of Propagate in enough detail to guide a sustainable implementation (i.e., maintainable in the long term). It also ensures that the team has a shared understanding of the package’s scope and architecture.

Propagate’s design builds on our overall Seedcase design principles and patterns and follows the same principles and patterns described there.

Purpose

The overall aim of Propagate is to streamline requesting and subsetting data from a data package in order to create the sample population needed to answer specific research questions.

The central concept in Propagate is a request that describes a subset of data from a data package. This request is intended to be both machine-readable and human-readable, allowing them to be shared, reviewed, and archived. They also provide a reproducible specification for regenerating the same data subset at a later time.

Overall, Propagate:

  1. Creates structured instructions for requesting a subset of data.
  2. Provides an interactive interface for creating those instructions from metadata.
  3. Checks the instructions against the metadata before applying them to the data.
  4. Generates reproducible subsets from a data package based on the instructions.

Requirements

Specifically, Propagate must:

Core functionality

  • Define a machine-readable and human-readable instruction specification for requesting data subsets.
  • Store enough information in the instructions to reproduce the subset in the future.
  • Support checking instructions against the metadata before generating a subset.
  • Assume one request per research project per data package.
  • Create a subset of the data package from a request that includes the inclusion criteria and a list of required variables (the requested resources and the requested rows and columns).

Data request functionality

  • Generate instructions interactively from the data package metadata via a web interface without downloading or installing software.
  • Support an “export” functionality: completed instructions should be downloadable as a file.
  • Support a “send” functionality: the requester’s email client should be opened with a pre-filled email to the data owner containing the instruction file as an attachment.
  • Allow users to save and resume partially completed requests.
  • Include the request metadata in the instructions (e.g., requester name and email).
  • Ensure that the request process and interface is simple and easy to use for nontechnical users.
  • For more technically experienced users, provide an interface designed for them, e.g. a terminal user interface, to create the request interactively without needing to open a web app.

Data management functionality

  • Generate a web component via a command-line interface for creating instructions interactively from data package metadata.
  • Support embedding the web component in a static website without requiring a server. In the Seedcase ecosystem, this would be the website generated by Template Data Package.
  • Generate data subsets in Parquet format for rectangular data from the instructions and the data from the data package.
  • Support previewing the planned subset operation before execution.
  • Check that the request is correctly structured, that it can be parsed to SQL, and that it has instructions that can be applied to the data package.

Out of scope

Propagate does not:

  • Manage data access approvals.
  • Transfer data subsets to the requester or to different servers.
  • Host web or server services.
  • Modify source data or metadata.
  • Act as a general-purpose query engine.
  • Store or organize the requests, the data, or the metadata.