flowchart TD
user_requester_web("Web UI Requester<br>[person]")
user_requester_tui("TUI Requester<br>[person]")
user_owner("Owner<br>[person]")
data(["Data<br>[Parquet]"])
metadata(["Metadata<br>[e.g. datapackage.json]"])
request(["Request<br>[yaml]"])
propagate("seedcase-propagate<br>[Rust app]")
template_dp("template-data-package<br>[Copier template]")
website("Static website<br>[Quarto]")
subset(["Data subset<br>[Parquet]"])
propagate --"Uses"--> metadata
propagate --"Uses"--> data
propagate --"Uses"--> request
user_owner --"Uses"--> propagate
user_owner --"Optionally uses"--> template_dp
user_requester_web --"Uses"--> website
user_requester_tui --"Uses"--> propagate
template_dp --"Uses"--> propagate
template_dp --"Builds"--> website
website --"Generates"--> request
propagate --"Generates"--> request
propagate --"Generates"--> subset
user_requester_web --"Receives"--> subset
user_requester_tui --"Receives"--> subset
%% Styling
classDef artifact fill:transparent, stroke-dasharray: 5 5
class metadata,data,request,subset artifact
Architecture
This document contains a description of Propagate’s expected user types and naming scheme and fleshes out its architecture in a series of C4 models. For design details of the Seedcase Project as a whole, see the Seedcase Design documentation.
The purpose of this page is to ensure that the team shares a common understanding of the implementation and to outline the design to anyone else interested in the internal workings of the package.
User types
Propagate supports two primary user types: a Requester and an Owner.
Requester
The term “Requester” is used throughout the documentation to refer to any user who wants to request a subset of data from a data package. We use “Requester” as a generic user type, however, there are different types of users within this “Requester” category, varying in technical experience and how they work with data.
PhD/Master’s students: This person (likely) has limited technical experience and needs some more structured, graphical guidance on requesting data. Because their studies are time-limited, they need some way of quickly requesting data and receiving it so they can continue their research and finish on time. They are also likely to not know exactly what data they might need, so they will (likely) need more guidance from the Owner on this. As they do their research, they likely will make several updates to their request over time.
Researchers: This person (likely) has more technical experience and is likely more familiar with what data they need for their research. They will likely make fewer updates to their request for a given project, request data with a narrower scope, and create requests for multiple projects over time or even at the same time. They will likely have bigger, more complex projects that may require more involvement from the Owner on reviewing and approving the request (so there may be more communication between the two). They may copy and slightly modify the request for a new project, so they will need some way of confirming that the new request is still correct. While they will appreciate efficient ways of requesting data, they will also value a structured, guided process as they may not be requesting data often enough to remember the process every time.
Analysts or data processors: This person will likely be the most technically experienced Requester. They will likely be working on multiple, very narrow projects at the same time and value efficiency highly. For example, they would likely be opt to use a terminal-based workflow if it allowed them to request data more efficiently. They will probably not communicate too often with the Owner, mostly just about the final confirmation of the request and getting access to the data subset, as they likely will be familiar with the data and the request process. They value efficiency and speed over a more guided process.
Owner
The term “Owner” is used throughout the documentation to refer to any user who is responsible for setting up the request web application, receiving requests and creating subsets of data from those requests. We use “Owner” as a generic user type, however, there are different types of users within this “Owner” category (including an actual data owner).
Data owner: This person (or people) is the one with the legal rights to the data, who is accountable for its use and (potential) misuse. This person is the one with the final say on which requests are approved and which are not. They control where the web application is hosted, where requests are stored, and who has access to the original data and to the hosting service for the web application. For any given data package, there is always at least one data owner. Depending on who is involved in managing the data package, there may only be one person, the data owner, so they will be the one using all the functionality from Propagate. If there are others involved, e.g. a data manager, then the data owner may, at a minimum, be reviewing and approving any requests received. It is their responsibility to ensure that the needs of the research project (as specified in the received
request.yaml) match the requested data subset.Data manager: This person (or people) is the one given permission to manage the data package, including setting up the request web application, handling requests and generating subsets. They may or may not be given permission to approve requests (usually not). They also may or may not actually review the requests themselves for legitimate use of the data for the given project, though they will want to check that the request is correct and can correctly create a subset.
Both data owner and data manager are likely very busy, so they need a way to consistently and predictably receive and review requests, and to generate subsets of data from the data package efficiently.
Naming
| Object | Description |
|---|---|
| package | A data package that contains a collection of related data resources and metadata. |
| resource | One or more data files that are part of a data package. |
| metadata | Information about the data, such as its description, structure, and origin. |
| data | The actual data within a data package given in Parquet format. |
| (research) project | This is the research project associated with a specific request for the whole or a subset of the data package. A project would be something that creates an academic output (e.g. paper). |
| request | The contents of the request.yaml file specifying the data requested together with metadata about the request itself. |
| Action | Description |
|---|---|
| build | Generate the WASM bundle that makes up the web component for creating requests. |
| check | Check the request against the metadata. |
| subset | Generate a subset of data based on the request provided. |
| create-request | Create a request file interactively using the terminal. |
| Action | Description |
|---|---|
| select resource | Add instructions for selecting a specific resource from the data package. |
| select field | Add instructions for selecting a specific field from the resource. |
| add filter | Add instructions for filtering the data based on certain criteria. |
| save progress | Save the current contents of the request form to local storage in the browser. |
| export | Download the contents of the request form as a request file. |
| send | Open the user’s email client with an email to the data owner pre-filled with the request file as an attachment. |
C4 Models
This section contains the C4 Models for Propagate. The C4 Model is an established visualisation approach to describe the architecture of a software system. It breaks the system down into four levels of architectural abstraction: system context, containers, components, and code.
System context
The system context diagram shows the users and external systems that interact with Propagate.
Template Data Package is a Copier template that makes creating and developing data packages easier. One of its features is building a website (via Flower) that displays package metadata. Optionally, it can also include the interactive request form generated by Propagate. The owner can use Template Data Package to build the website with the request form or they can generate the form directly from Propagate.
Container
In C4, a container diagram zooms in on the system to show the containers within it, such as web applications or databases. This diagram displays the main containers in Propagate, their responsibilities, and how they interact, including the technologies used for each.
Propagate is primarily designed for use as a CLI.
flowchart TD
subgraph "seedcase-propagate"
common("Common<br>[Rust]<br>Shared domain<br>models and logic")
request_builder("Request builder<br>[Rust]<br>Shared logic")
subset_builder("Subset builder<br>[Rust]")
web("Web<br>[Rust, Dioxus]<br>UI components")
tui("TUI<br>[Rust, ratatui]<br>UI components")
cli("CLI<br>[Rust]<br>Build, check, subset<br>and request commands")
subset_builder -- "Uses" --> common
request_builder -- "Uses" --> common
web -- "Uses" --> request_builder
tui -- "Uses" --> request_builder
cli -- "Uses" --> subset_builder
cli -- "Uses" --> web
cli -- "Uses" --> tui
end
form_assets(["Request<br>form web assets<br>[WASM bundle]"])
web --"Builds"--> form_assets
%% Styling
style seedcase-propagate fill:#FFFFFF, color:#000000
style form_assets fill:transparent, stroke-dasharray: 5 5
Component/code
In C4, a component is “a grouping of related functionality encapsulated behind a well-defined interface”, like a class or a module, while code refers to the basic building blocks, such as classes and functions.
Warning
This section is a work in progress.