diff --git a/docs/design/api-discovery.md b/docs/design/api-discovery.md index 23fe8783d8a..9b722e4e7ec 100644 --- a/docs/design/api-discovery.md +++ b/docs/design/api-discovery.md @@ -4,4 +4,95 @@ title: API Discovery for Workflow Design --- # Discovering APIs to inform your workflow automation design -How to analyze API docs and draft technical diagrams (technical WF diagram, update your solution architecture diagram) \ No newline at end of file +This article explains how to analyze API documentation and draft a technical workflow diagram. + +## What is an API? +APIs tell applications how to communicate. An API is the “messenger” that: +1. Tells you how to form a request, +2. Delivers your request to the provider that you’re requesting it from, and then +3. Delivers the response back to you + +| ![Workflow](/img/api_diagram.png) | +|:--:| +| *[Source](https://snipcart.com/blog/integrating-apis-introduction)*| + +OpenFn connects with APIs via http requests sent via the web. OpenFn can automate any tasks supported by the APIs of the applications it integrates with (e.g., if an app’s API supports sending payments, then OpenFn can automate sending payments). + +## How to analyze API documentation + +In the beginning of the design process, you should explore the target system’s API documentation to determine the options for integration. + + +### Determine integration options +Consider these questions to determine your integration options–even if an API is not available. : +1. Is there a RESTful API? + - If yes, OpenFn can connect out-of-box! REST API is the gold standard for most modern web apps, and typically supports JSON data format. +2. Is there a webhook? + - Most mobile data collection apps offer this feature. Some call it “data forwarding”, “web callback”, or “HTTP push API”. + - Webhooks automatically send messages or notifications when something happens (e.g., when a new form is submitted, notify external services like OpenFn). These event-based notifications enable real-time data integration or automated actions. +3. Otherwise, what are other options for importing/exporting data from the target applications? + - Can you connect directly to a database? + - Is there a way to import/export files? (JSON, CSV, XLS, or XML) + - Is there a legacy API (e.g., SOAP) that we can communicate with via HTTP requests? + +:::tip +OpenFn can connect any app, even if an API is not supported. See the [“Adaptors”](/adaptors) section to learn more. + +::: + + +### Authentication +API documentation will typically have a section dedicated to authentication options. Look for this to discover which authentication methods are supported, and whether configuration work will be required to set up a new user or API credential. + +Consider that authentication methods that leverage API Keys or OAuth are typically more secure than basic authentication (username/password). + +:::tip + +As early as possible, request an API credential from the system administrator of the app you’re trying to integrate with. This way, you can test authentication with a developer/test environment to verify that you’re able to connect. + +::: + +### API Endpoints +Analyze the documentation to see which resources/entities and features are supported by the API. For example, if you want to register Patient records via the API, search for reference to the “/patients” endpoint (or whatever this resource is called in your target application). + +This section of the documentation will include an overview of which HTTP request methods (i.e., POST, GET, etc.) and request parameters are supported, as well as example HTTP requests you can send to the API. + +__HTTP request methods will inform you which operations are supported by the API.__ +1. __C__reate → POST +2. __R__ead → GET +3. __U__pdate → PUT or PATCH +4. __D__elete → DELETE + +For example, if you want to query Patient records from an app, see if the API documentation supports `GET ‘/patients’`. + +### Limits +Be on the lookout for API limits. Documentation will often have a dedicated section that will describe if there are any limits or considerations for API requests and rates, concurrency, and record limits. Understanding these limits upfront can inform your integration design to ensure high-performing, scalable automation. + +## Technical Workflow Diagramming + +The output of API discovery should be a “technical” workflow diagram. This diagram is different from the functional workflow diagram produced during [“Discovery”](/documentation/next/design/discovery) in that it captures the technical specifications for how to integrate with target applications. These specifications include the specific methods/operations (e.g., GET, POST) and the database/API names of the target resources (i.e., specific API endpoints or database tables). + +![Workflow](/img/api_example.png) + +__When drafting your technical specifications, consider the following:__ +1. __Plan for failure. Your workflows will fail. Consider what happens when they do…__ + - Should individuals be notified? + - How can the workflow be re-processed safely? + - How to ensure no duplicate data is created? +2. __Where possible, use unique identifiers to build idempotent automation. Check for existing records in the target system using an available unique identifier:__ + - System record UUIDs (e.g., record_id: asjd2910-b8zy1s0a), + - Unique codes (e.g., HOUSEHOLD-10013) and + - Unique combination of attributes (e.g., familyName + phoneNumber + village + districtCode) +3. __If the target system does not have native “upsert” operation or built-in duplicate-checking before insert, implement an upsert (“update or insert”) pattern to…__ + - Check if a record exists using a unique identifier… + - If yes, update the record. + - If not, insert a new record. +4. __Don’t forget to consider data volumes. Depending on whether you need to handle 1, 10000, or 1M+ records, your workflow approach may need change.__ + - Estimate the file size of the data to be extracted + - Consider API limits (records returned per page, request rate limits) + - Consider bulk operations & batching requests + + +Check out the technical workflow diagram below for syncing forms submissions from KoboToolBox to DHIS2. The original functional diagram can be found [here](/documentation/next/design/discovery#workflow-requirements-gathering). + +![Workflow](/img/technical_example.png) \ No newline at end of file diff --git a/docs/design/design-overview.md b/docs/design/design-overview.md new file mode 100644 index 00000000000..2b827ec727d --- /dev/null +++ b/docs/design/design-overview.md @@ -0,0 +1,44 @@ +--- +sidebar_label: Design Process Overview +title: Design Process Overview +--- + +This article outlines the high-level steps to design automated workflows, inspired by the OpenFn core team’s standard implementation process. + +Typically the design process happens outside of OpenFn, in conversation and collaboration with relevant business/program and technical stakeholders. Then, once the design is finalized, the workflow configuration, testing, monitoring, and management is handled in OpenFn. + +## Key Terms + +Before you dive in, make sure you have a clear understanding of these key terms we’ll reference throughout this documentation: + +### Workflow +The set of instructions that determine how to solve a problem or accomplish a task. They are often broken down into smaller, independent tasks. + +![Workflow](/img/workflow.png) + + +### Workflow Automation +The use of software to perform these tasks independently, in accordance with predefined business rules, and without the need for human input. + +![Workflow Automation](/img/workflow_automation.png) + +### Data Integration + +The process of combining data from different sources into a centralized view. Data integration is a means of achieving workflow automation. Its tasks may be streamlined, automated, and managed by a workflow automation tool. + +![Data Integration](/img/data_integration.png) + + +## Introduction + +Workflow automation design features 5 main steps that are covered in depth in other articles: +1. [Discovery & Scoping](/documentation/next/design/discovery) +2. [Workflow Design](/documentation/next/design/design-workflow) +3. [API Discovery and Technical Design](/documentation/next/design/api-discovery) +4. [Data Element Mapping Specifications](/documentation/next/design/mapping-specs) +5. [Workflow Specifications](/documentation/next/design/workflow-specs) + +### Example Use Case +Throughout the design documentation we will reference the fictional data collection & workflow automation scenario below: + +_PatientCare is a health NGO that runs a network of community health workers who provide care to patients in remote areas in Guinea. PatientCare workers collect patient data in [KoboToolBox](https://www.kobotoolbox.org/). The Guinean government uses [DHIS2](http://dhis2.org) as its national health information system (HIS) and requires PatientCare to register all patient data in the HIS._ \ No newline at end of file diff --git a/docs/design/design-workflow.md b/docs/design/design-workflow.md index 8eddc46cb8d..2dab0726a5e 100644 --- a/docs/design/design-workflow.md +++ b/docs/design/design-workflow.md @@ -4,4 +4,37 @@ title: Design your first workflow to automate --- # Designing your first OpenFn workflow -How to diagram; intro to BPMN standard & links to resources \ No newline at end of file + +This article explains how to use the information gathered during discovery to determine the specific workflow steps, design your workflow, and draft a diagram to document the steps in the process you would like to automate. + +## Why diagram your workflow? +During requirements gathering you might outline the new workflow with a list of steps or use existing documentation on a business process/protocol. +__For example:__ +1. New patient visits clinic +2. Worker registers patient in mobile app (KoboToolBox) +3. Every day, sync new patients to national health information system (DHIS2) + +Next, consider visually outlining the structure and flow of a workflow to ensure it can be more easily understood by various stakeholders. Diagraming can help to capture: +1. The right flow/sequence of steps, +2. Dependencies, +3. Redundancies, and +4. Who is responsible for each step + + +## Main steps to workflow diagramming +1. Diagram the human/manual process steps of this workflow, +2. Identify opportunities for automation, +3. Detail the functional steps of the ideal automation process, +4. Share the diagram with all stakeholders for final sign off, and update it as necessary + +The output of this exercise is clear documentation on how a business process will be executed by automation, humans, and often a combination of both. + +## Diagram using global standards +When diagramming, consider using global standards like BPMN (business process model and notation) so that diagrams are consistent and can be understood by outside parties. + +1. There are lots of online resources to learn basic concepts like this post. +2. The OpenFn team has compiled some learning resources and BPMN diagramming tools in this document. + +Your organization may already be using a different standard notation (e.g., UML). What’s important is consistency across your documentation to ensure that your solutions can be easily understood by other stakeholders. + +![Workflow](/img/BPMN_example.png) \ No newline at end of file diff --git a/docs/design/discovery.md b/docs/design/discovery.md index 74ce37f13e2..3d73d41117f 100644 --- a/docs/design/discovery.md +++ b/docs/design/discovery.md @@ -4,4 +4,125 @@ title: Discovery & Scoping for OpenFn Projects --- # Discovery & Scoping for OpenFn Projects -Share key scoping questions and introduce solution architecture diagram \ No newline at end of file + + +This article outlines key discovery and scoping questions to confirm the business value, core workflow requirements, technical feasibility, and client capacity when starting a new implementation. This article will be referencing the example use case introduced in the [series introduction](/documentation/next/design/design-overview#example-use-case). + + +:::tip +To quickly export and/or share these questions, please see this [slide deck](https://docs.google.com/presentation/d/1WIc_uNAqapILF7redhTnZXpPRo1jFPSmAjoAplGt42w/edit?usp=sharing). + +::: + + +## Key Questions +### Business Value Assessment + +The first step in the discovery process is a business value assessment to determine the potential return on investment, efficiency gains, and other valuable outcomes to be used to monitor success. + +__Questions to ask:__ + +1. What are the workflow(s) you want to automate? +2. How are the workflows currently being managed? 21 + - Is there an existing manual or semi-automatic business process? + - If yes, how much staff time is spent managing these workflows? +3. What problem(s) will automation solve for? What efficiencies or benefits to be gained? What is the cost of inaction? + - If we do not automate these workflows, then what is the status quo? + - Is the current workflow slow, insecure, or leading to poor data quality or service delivery? + + +__Example:__ +1. I want to automate syncing case data from KoboToolBox to DHIS2 Tracker. +2. Our team currently spends 3 hours a week manually exporting the Kobo Data and inputting it to DHIS2. +3. This automation will eliminate the risk of human error in manually entering the data, save us money and time, and enable us to work with more patients. + +## Workflow Requirements Gathering +Use the questions below to determine the specific workflow steps. The output of these questions should be a draft workflow or business process diagram (consider drafting the diagram in [BPMN](https://www.bpmn.org/) to leverage a standardized notation). + +__Questions to ask:__ +1. What triggers the workflow and how often should it run? (e.g., real-time or scheduled) + - Is there a user action or system event that should trigger the workflow to run? (e.g., real-time on form submission, or when record status set to “closed”) + - Or should this workflow be scheduled at a specific day/time? (e.g., every day at 12:00) +2. Does the workflow require a one-way or two-way data flow? + - E.g., if the workflow is sending one record from System A to System B, does the data need to flow only one way? Or once the data is synced to System B, should something be sent back to System A for a bi-directional data flow? +3. What are the expected data volumes? (e.g., 100 referrals every month, 12k forms every year) + + +__Example:__ +The workflow should sync patient data from KoboToolBox to DHIS2 each time a form is submitted (i.e. real-time sync). There are a maximum of 5000 patients registered in Kobo per month. + +![Workflow](/img/functional_example.png) + + + +### Technical feasibility assessment +The answers to the questions below will help you draft a solution diagram to document specifically which instances will be connected & which integration interfaces to use. + +1. How many instances of the target systems exist? (I.e., Are you connecting to 1 or 2 DB instances?) +2. Are the target systems built? Is any configuration expected to change? + - If config is still in progress, then consider revisiting this project when the systems are stable. +3. Is there an available REST API? + - If yes, please provide the documentation. + - If not yet, then consider revisiting this project after this API has been built and tested. + - If no, then what are other available methods for data import/export? + - Is it possible to secure a direct database connection? + - Or is there a webhook or method for forwarding data to an external system? + - Or a way to export/import data using file exports? What data formats are available? +4. Where are the target systems hosted? Any known security requirements or authentication considerations? (e.g., firewalls, VPN requirements, IP whitelist requirements) +5. Is there an available test environment we can access to test integration with the application? (If not, is there a public demo of the application running on the same version that you’re currently running so that we can test the APIs?) + +Example: +Only one instance of PatientCare and DHIS2 exist for this integration and they have already been built with REST APIs. They are both hosted on PatientCare managed servers that require IP whitelisting for access. + +![Workflow](/img/technical_example.png) + + + +### Capacity assessment +The answers to the questions below will help you identify project roles for the implementation design & delivery, as well as a plan for training, rollout, ongoing administration, and support. + +1. Does each target system have a full-time system administrator? + - Are the administrators available to support integration setup & testing? + - Will the administrators be able to provide a test/developer environment for testing? + - Who will learn how to administer OpenFn? +2. What are their technical backgrounds? + - What are other resources available to provide ongoing support? + - Does anyone in the organization have experience with Javascript or JSON? +3. Is there a desire to learn how to manage the OpenFn implementation independently? +4. Who at the organization will be responsible for ongoing governance of the solution and overseeing change management? Are resources available to meet regularly to review change requests? + +__Example:__ + + +| Name | Role | +| -------- | ------- | +| Ian | OpenFn System Admin who will be responsible for ongoing mgmt & monitoring | +| Melody | PatientCare Admin, who will train their system’s users on the workflow | +| Arnis | DHIS2 Admin, who will train their system’s users on the workflow | +| Ramona | Programs focal point who will champion users, inform workflow requirements, and meet regularly with users to collect feedback, propose changes, and review change requests with the system administrators | + + + + + +### Documenting the Solution Architecture + +Once you’ve gathered the key solution requirements, consider creating a “Solution Architecture” diagram to document the following: +1. Different solution components +2. Data flows between these components (highlighting data exchange within the organization and between third party services) +3. Types of data exchanged +4. Authentication/access points + +Such diagrams promote transparency, help to identify potential data exposure risks, and provide documentation of compliance with data protection requirements. Check out the example solution architecture diagrams below. + + +__Example 1:__ + +![Workflow](/img/solution_diagram1.png) + + +__Example 2:__ + +| ![Workflow](/img/solution_diagram2.png) | +|:--:| +| *[Source](https://lucid.app/lucidchart/1e997197-2d67-4393-8394-a532d83561b2/edit?invitationId=inv_85b809a1-6fbd-4275-abdc-618fbd56e90d&page=0_0#)*| \ No newline at end of file diff --git a/docs/design/mapping-specs.md b/docs/design/mapping-specs.md index 3638c8c5ecf..a5d061ce375 100644 --- a/docs/design/mapping-specs.md +++ b/docs/design/mapping-specs.md @@ -4,4 +4,53 @@ title: Writing Data Element Mapping Specifications --- # Mapping data elements to define data integration & automation rules -Mapping template & process overview \ No newline at end of file + +This article walks through the data element mapping process used to develop entity- and field-level specifications for how data points should be exchanged, cleaned, and/or transformed in a data integration workflow. In basic terms, data mapping is the process of connecting a data field from one source to a data field in another source (e.g., System A "patient" = "person" in System B). + + +A data element mapping specification is a special type of data dictionary that serves as (1) documentation on how you are translating meaning between systems, and (2) specifications for developers building the workflow automation solution. + +For each automation step in your workflow, you will document which data elements (or metadata) will be referenced, as well as the “rules” for how these data elements should be mapped, reassigned, cleaned, transformed, and/or calculated. + +![mapping](/img/mapping_example.png) + + + +__To draft a data element mapping specification, you’ll need to…__ + +1. Export the metadata or ask for a list of data elements from the target systems, +2. Procure a sample “input” record from the source system and procure a sample output record from the destination system. At best, this is an example JSON payload or a link to example records. At worst, this is a screenshot or a CSV file with “dummy” data. +3. Start “mapping” the data elements and recording transformation rules! + +| ![mapping](/img/mapping_process.png) | +|:--:| +| *The data mapping process for data integration solutions.*| + + +## OpenFn Mapping Specification Template +You can document data elements, mappings, and rules using the OpenFn mapping specification template. This [template](https://docs.google.com/spreadsheets/d/19sPRLP4zeFgFbtOL1wKh-rc7D0KPMu3etmOOG_x5t68/edit#gid=1275153608) was created by the OpenFn team as a result of lessons learned from implementing data integrations solutions for NGOs and government partners around the world. It is used on all OpenFn projects and is maintained by the OpenFn team. + + +## Mapping Considerations + +### Maintaining Mapping Specifications +Once your OpenFn project is live, the Mapping Specs document may be the business-friendly way your users interact with your solution. If you make any changes, make sure the Mapping Spec always matches your job code. Also consider versioning your mapping specs so stakeholders have access to historical implementations of the solution. + +:::info + +While the OpenFn XLS-based mapping template is helpful for collaborating with other stakeholders on defining the mapping requirements, once these specifications are set, you might consider capturing these mapping rules in a database table or in an application like [Open Concept Lab](https://openconceptlab.org/) (which has a user-friendly web app for recording data dictionaries & mapping rules, and REST API support). This would then allow you to dynamically query these mapping rules using OpenFn, to ensure your integration is utilizing the latest and greatest specifications. + +::: + +### Functional vs. Technical Mapping +After your organization (or “the business”) determines the functional data element mapping rules for source/target systems, you'll need to consider which other technical data elements are required in order for the integration to work. These may include system-specific fields, IDs, and/or API parameters that are “under the hood" and may not be visible to the end user, but are required by the target system to share the data. + + +### Mapping to Individual or Aggregate Entities + +Consider if your integration requires a 1-to-1 exchange of individual records, or if there is a need for individual records to be summaries or aggregated. +Your workflow may require you to map individual entities (i.e., 1-to-1 mapping). For example, you can map a patient from KoboToolBox to a patient in DHIS2. You should use the [default OpenFn mapping template](https://docs.google.com/spreadsheets/d/19sPRLP4zeFgFbtOL1wKh-rc7D0KPMu3etmOOG_x5t68/edit#gid=1275153608) for such scenarios. + +However, if your workflow requires mapping individual entities to an aggregate/summarized entity (i.e., many-to-1 mapping), then you can use OpenFn’s [aggregate mapping template](https://docs.google.com/spreadsheets/d/1JVcM7FEkCeezHXONRaAaEPFks9lS8xO_q51jql_hUtc/edit) to start. + . For example, you might collect individual patient records in KoboToolBox, but want to send an aggregated count of patients to DHIS2 for key indicator results reporting (e.g. the number of patients under 18 years old). + \ No newline at end of file diff --git a/docs/design/overview.md b/docs/design/overview.md index ea1396fc902..a4fe4a5e1c7 100644 --- a/docs/design/overview.md +++ b/docs/design/overview.md @@ -197,4 +197,4 @@ implementing partners. Share this documentation with any technical implementation team, or check out the [Build](/documentation/build/jobs) documentation section to learn how to -implement these design specifications using OpenFn. +implement these design specifications using OpenFn. \ No newline at end of file diff --git a/docs/design/workflow-specs.md b/docs/design/workflow-specs.md index 70a0cbb67c4..05719c811c1 100644 --- a/docs/design/workflow-specs.md +++ b/docs/design/workflow-specs.md @@ -4,4 +4,24 @@ title: Writing Workflow Automation Specifications --- # Writing specifications for workflow automation solutions -Specifications checklist; hand off to implementers \ No newline at end of file + +__The key outputs of the the design process are:__ + +1. [Functional Workflow diagram](/documentation/next/design/discovery#workflow-requirements-gathering) +2. [Technical Workflow diagram](/documentation/next/design/discovery#workflow-requirements-gathering) +3. [Solution Architecture Diagram](/documentation/next/design/discovery#documenting-the-solution-architecture) +4. [Data element mapping specifications](/documentation/next/design/mapping-specs) + + +Given these, you’ll be ready to finalize your workflow specifications and hand-off to developers for job-writing! + +Each “task” or “step” in the OpenFn swimlane of your technical diagram can be implemented as a distinct operation in your workflow configuration. In the example diagram below, you might implement 1 job with 3 chained operations, or 3 jobs with 1 operation each. + +![workflow](/img/workflow_specs.png) + +__The workflow specifications should link to all of the design artifacts and highlight the following:__ +1. The required number of OpenFn jobs and the function of each +2. Links to sample input/output and API documentation +3. Unique identifiers +4. Expected data volumes +5. Authentication requirements \ No newline at end of file diff --git a/sidebars-main.js b/sidebars-main.js index e8064687a42..8aa61ef6d1a 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -16,8 +16,15 @@ module.exports = { type: 'category', label: 'Design Workflows', items: [ - 'design/overview', + //'design/overview', + 'design/design-overview', + 'design/discovery', + 'design/design-workflow', + 'design/api-discovery', + 'design/mapping-specs', + 'design/workflow-specs' //'design/discovery' + ], }, // { diff --git a/static/img/BPMN_example.png b/static/img/BPMN_example.png new file mode 100644 index 00000000000..0664c20e217 Binary files /dev/null and b/static/img/BPMN_example.png differ diff --git a/static/img/api_diagram.png b/static/img/api_diagram.png new file mode 100644 index 00000000000..27922413542 Binary files /dev/null and b/static/img/api_diagram.png differ diff --git a/static/img/api_example.png b/static/img/api_example.png new file mode 100644 index 00000000000..a477e3e0896 Binary files /dev/null and b/static/img/api_example.png differ diff --git a/static/img/data_integration.png b/static/img/data_integration.png new file mode 100644 index 00000000000..84b589b1ef7 Binary files /dev/null and b/static/img/data_integration.png differ diff --git a/static/img/functional_example.png b/static/img/functional_example.png new file mode 100644 index 00000000000..9480c3fdb4d Binary files /dev/null and b/static/img/functional_example.png differ diff --git a/static/img/mapping_example.png b/static/img/mapping_example.png new file mode 100644 index 00000000000..a45dab214cf Binary files /dev/null and b/static/img/mapping_example.png differ diff --git a/static/img/mapping_process.png b/static/img/mapping_process.png new file mode 100644 index 00000000000..a8b790824f8 Binary files /dev/null and b/static/img/mapping_process.png differ diff --git a/static/img/solution_diagram1.png b/static/img/solution_diagram1.png new file mode 100644 index 00000000000..7650448f0ba Binary files /dev/null and b/static/img/solution_diagram1.png differ diff --git a/static/img/solution_diagram2.png b/static/img/solution_diagram2.png new file mode 100644 index 00000000000..6b5e9ec3a83 Binary files /dev/null and b/static/img/solution_diagram2.png differ diff --git a/static/img/technical_example.png b/static/img/technical_example.png new file mode 100644 index 00000000000..fc353c3f151 Binary files /dev/null and b/static/img/technical_example.png differ diff --git a/static/img/workflow.png b/static/img/workflow.png new file mode 100644 index 00000000000..5e2488413a8 Binary files /dev/null and b/static/img/workflow.png differ diff --git a/static/img/workflow_automation.png b/static/img/workflow_automation.png new file mode 100644 index 00000000000..cc626665783 Binary files /dev/null and b/static/img/workflow_automation.png differ diff --git a/static/img/workflow_specs.png b/static/img/workflow_specs.png new file mode 100644 index 00000000000..ed263111516 Binary files /dev/null and b/static/img/workflow_specs.png differ