Outputs

Propagate produces three main outputs based on the inputs described in the Inputs page. See the Design overview page for a description of the user types, Requester and Owner, referred to below.

The outputs are:

HTML and WebAssembly (WASM) files for a web application

The Owner uses Propagate to create HTML and WASM files that form a web application from the data package metadata file (for example, but not exclusively, a datapackage.json file). This application allows Requesters to browse and select the columns and rows they want to include in their request. The application can be added to a static website.

We decided on HTML and WASM for the web application for a few reasons. We wanted a fully client-side application, where all application logic runs in the browser rather than on a backend server. This allows the Owner to host the application as static files on any static website without needing to set up or pay for an application server. We also didn’t want to develop the web application in multiple languages (e.g. JavaScript for the browser interface and Python for everything else). Compiling to WASM allows us to write all components of the application in a single language. We chose Rust as the development language because it compiles directly to WASM and has more potential for future data engineering and software development work than JavaScript.

The web application includes:

  • Search and filter functionality to allow Requesters to easily find the resources and columns they are interested in, including a filter for showing only the selected columns.
  • Checkboxes to select which resources and columns to include in the request.
  • The selection of multiple columns (using Shift+click for selecting a range of columns and Ctrl+click/Cmd+click for selecting multiple non-adjacent columns).
  • Row-based conditions based on specific values with AND, OR, and NOT logic (e.g., only include rows with a value in a column above a certain threshold or with a specific value).
  • Saving of in-progress selections, allowing Requesters to resume their work later.
  • A request review showing the selected resource, columns, and rows.
  • Owner names and email addresses (extracted from the metadata file, e.g. the contributors field in the datapackage.json), allowing Requesters to choose an Owner to send the request to. Clicking a button opens an email for the selected Owner with request.yaml attached.

The web application does not send the request and it is not involved in its approval; the Requester sends the email manually, and the Owner decides whether to fulfil the request outside of Propagate. The web application also doesn’t contain any extensive custom keybindings. This is because:

  • Most browser users don’t use keybindings when navigating web pages, or only use a few basic ones.
  • Browsers have their own keybindings that could conflict with custom ones.
  • Some browser extensions can override keybindings, which could break the web application.
  • Adding keybindings is an extra implementation detail to consider and maintain, especially when considering the best practices for creating an accessible interface. Custom keybindings are largely something more technical users would use, and we don’t design the web application for these types of technical Requesters. For them we include a terminal user interface, which is described more on the terminal user interface page.

While technically the web application could be built and ran locally by an experienced Requester, it isn’t a workflow we design for or encourage. This is because it is an inherently more complex workflow (with more sources of failure) and we don’t want to maintain or develop this type of functionality.

Data request

The data request file, request.yaml, is the output of the Requester’s inputs and selections in the web application or terminal interface.

The request includes:

  • Name of the project the data is requested for.
  • Rationale for the request.
  • The date and time of request creation.
  • The Requester’s name and email address.
  • Details about which resources, columns, and rows the Requester wants for the data subset.

Data subset

The data subset is a subset of the data in the data package that is filtered based on the request to include only the selected resources, columns, and rows (if applicable). A data subset would be considered the sample/statistical population that is used to answer a set of related research questions (all part of a research project). The subset contains one file per original resource file (i.e., Propagate does not join data files into fewer files or a single file). The data subset format is Parquet. See our Why Parquet decision post for the rationale. Propagate bundles the Parquet file(s) into a single tar file.

The subset is created from a request that contains a set of inclusion criteria defining the population they need to answer their questions and a set of names for the variables/columns needed to answer the research question(s). Propagate currently only supports inclusion criteria defined as a list of AND conditions. This is because, most of the time inclusion criteria are quite simple and are usually AND conditions (e.g. people above 30 and who have diabetes). By only supporting AND, we can keep our implementation simple.

Potential OR conditions at the inclusion stage are (we anticipate) fairly rare occurrences. If more complex row filtering is needed, the Requester can ask for all rows (or at least a larger subset than they need) and do the row filtering themselves. The Requester can also reach out to the Owner of the data package and ask them to create a new column corresponding to the more advanced filter. If OR inclusion criteria turns out to be commonly required, Propagate could be extended to support these in the future.