Merge pull request #381 from OpenFn/design-docs2

Design docs2

docs/design/api-discovery.md

@@ -4,4 +4,95 @@ title: API Discovery for Workflow Design
 ---
 
 # Discovering APIs to inform your workflow automation design
-How to analyze API docs and draft technical diagrams (technical WF diagram, update your solution architecture diagram)
+This article explains how to analyze API documentation and draft a technical workflow diagram.
+
+## What is an API?
+APIs tell applications how to communicate. An API is the “messenger” that: 
+1. Tells you how to form a request,
+2. Delivers your request to the provider that you’re requesting it from, and then 
+3. Delivers the response back to you
+
+| ![Workflow](/img/api_diagram.png) |
+|:--:|
+| *[Source](https://snipcart.com/blog/integrating-apis-introduction)*|
+
+OpenFn connects with APIs via http requests sent via the web. OpenFn can automate any tasks supported by the APIs of the applications it integrates with (e.g., if an app’s API supports sending payments, then OpenFn can automate sending payments). 
+
+## How to analyze API documentation
+
+In the beginning of the design process, you should explore the target system’s API documentation to determine the options for integration. 
+
+
+### Determine integration options
+Consider these questions to determine your integration options–even if an API is not available. :
+1. Is there a RESTful API? 
+    - If yes, OpenFn can connect out-of-box! REST API is the gold standard for most modern web apps, and typically supports JSON data format.
+2. Is there a webhook? 
+    - Most mobile data collection apps offer this feature. Some call it “data forwarding”, “web callback”, or “HTTP push API”.
+    - Webhooks automatically send messages or notifications when something happens (e.g., when a new form is submitted, notify external services like OpenFn). These event-based notifications enable real-time data integration or automated actions. 
+3. Otherwise, what are other options for importing/exporting data from the target applications? 
+    - Can you connect directly to a database? 
+    - Is there a way to import/export files? (JSON, CSV, XLS, or XML)
+    - Is there a legacy API (e.g., SOAP) that we can communicate with via HTTP requests? 
+
+:::tip 
+OpenFn can connect any app, even if an API is not supported. See the [“Adaptors”](/adaptors) section to learn more.
+
+:::
+
+
+### Authentication
+API documentation will typically have a section dedicated to authentication options. Look for this to discover which authentication methods are supported, and whether configuration work will be required to set up a new user or API credential. 
+
+Consider that authentication methods that leverage API Keys or OAuth are typically more secure than basic authentication (username/password). 
+
+:::tip 
+
+As early as possible, request an API credential from the system administrator of the app you’re trying to integrate with. This way, you can test authentication with a developer/test environment to verify that you’re able to connect. 
+
+:::
+
+### API Endpoints
+Analyze the documentation to see which resources/entities and features are supported by the API. For example, if you want to register Patient records via the API, search for reference to the “/patients” endpoint (or whatever this resource is called in your target application). 
+
+This section of the documentation will include an overview of which HTTP request methods (i.e., POST, GET, etc.) and request parameters are supported, as well as example HTTP requests you can send to the API.
+
+__HTTP request methods will inform you which operations are supported by the API.__ 
+1. __C__reate → POST
+2. __R__ead → GET
+3. __U__pdate → PUT or PATCH
+4. __D__elete → DELETE
+
+For example, if you want to query Patient records from an app, see if the API documentation supports `GET ‘/patients’`. 
+
+### Limits
+Be on the lookout for API limits. Documentation will often have a dedicated section that will describe if there are any limits or considerations for API requests and rates, concurrency, and record limits. Understanding these limits upfront can inform your integration design to ensure high-performing, scalable automation. 
+
+## Technical Workflow Diagramming
+
+The output of API discovery should be a “technical” workflow diagram. This diagram is different from the functional workflow diagram produced during [“Discovery”](/documentation/next/design/discovery) in that it captures the technical specifications for how to integrate with target applications. These specifications include the specific methods/operations (e.g., GET, POST) and the database/API names of the target resources (i.e., specific API endpoints or database tables). 
+
+![Workflow](/img/api_example.png)
+
+__When drafting your technical specifications, consider the following:__
+1. __Plan for failure. Your workflows will fail. Consider what happens when they do…__
+    - Should individuals be notified? 
+    - How can the workflow be re-processed safely? 
+    - How to ensure no duplicate data is created? 
+2. __Where possible, use unique identifiers to build idempotent automation. Check for existing records in the target system using an available unique identifier:__ 
+    - System record UUIDs (e.g., record_id: asjd2910-b8zy1s0a),
+    - Unique codes (e.g., HOUSEHOLD-10013) and 
+    - Unique combination of attributes (e.g., familyName + phoneNumber + village + districtCode)
+3. __If the target system does not have native “upsert” operation or built-in duplicate-checking before insert, implement an upsert  (“update or insert”) pattern to…__
+    - Check if a record exists using a unique identifier…
+    - If yes, update the record.
+    - If not, insert a new record.
+4. __Don’t forget to consider data volumes. Depending on whether you need to handle 1, 10000, or 1M+ records, your workflow approach may need change.__
+    - Estimate the file size of the data to be extracted 
+    - Consider API limits (records returned per page, request rate limits)
+    - Consider bulk operations & batching requests
+
+
+Check out the technical workflow diagram below for syncing forms submissions from KoboToolBox to DHIS2. The original functional diagram can be found [here](/documentation/next/design/discovery#workflow-requirements-gathering). 
+
+![Workflow](/img/technical_example.png)

docs/design/design-overview.md

@@ -0,0 +1,44 @@
+---
+sidebar_label: Design Process Overview
+title: Design Process Overview
+---
+
+This article outlines the high-level steps to design  automated workflows, inspired by the OpenFn core team’s standard implementation process. 
+
+Typically the design process happens outside of OpenFn, in conversation and collaboration with relevant business/program and technical stakeholders. Then, once the design is finalized, the workflow configuration, testing, monitoring, and management is handled in OpenFn. 
+
+## Key Terms 
+
+Before you dive in, make sure you have a clear understanding of these key terms we’ll reference throughout this documentation: 
+
+### Workflow
+The set of instructions that determine how to solve a problem or accomplish a task. They are often broken down into smaller, independent tasks. 
+
+![Workflow](/img/workflow.png)
+
+
+### Workflow Automation
+The use of software to perform these tasks independently, in accordance with predefined business rules, and without the need for human input. 
+
+![Workflow Automation](/img/workflow_automation.png)
+
+### Data Integration
+
+The process of combining data from different sources into a centralized view. Data integration is a means of achieving workflow automation. Its tasks may be streamlined, automated, and managed by a workflow automation tool.
+
+![Data Integration](/img/data_integration.png)
+
+
+## Introduction
+
+Workflow automation design features 5 main steps that are covered in depth in other articles:
+1. [Discovery & Scoping](/documentation/next/design/discovery)
+2. [Workflow Design](/documentation/next/design/design-workflow)
+3. [API Discovery and Technical Design](/documentation/next/design/api-discovery)
+4. [Data Element Mapping Specifications](/documentation/next/design/mapping-specs)
+5. [Workflow Specifications](/documentation/next/design/workflow-specs)
+
+### Example Use Case
+Throughout the design documentation we will reference the fictional data collection & workflow automation scenario below:
+
+_PatientCare is a health NGO that runs a network of community health workers who provide care to patients in remote areas in Guinea. PatientCare workers collect patient data in [KoboToolBox](https://www.kobotoolbox.org/). The Guinean government uses [DHIS2](http://dhis2.org) as its national health information system (HIS) and requires PatientCare to register all patient data in the HIS._

docs/design/design-workflow.md

@@ -4,4 +4,37 @@ title: Design your first workflow to automate
 ---
 
 # Designing your first OpenFn workflow
-How to diagram; intro to BPMN standard & links to resources
+
+This article explains how to use the information gathered during discovery to determine the specific workflow steps, design your workflow, and draft a diagram to document the steps in the process you would like to automate. 
+
+## Why diagram your workflow?
+During requirements gathering you might outline the new workflow with a list of steps or use existing documentation on a business process/protocol. 
+__For example:__
+1. New patient visits clinic
+2. Worker registers patient in mobile app (KoboToolBox)
+3. Every day, sync new patients to national health information system (DHIS2)
+
+Next, consider visually outlining the structure and flow of a workflow to ensure it can be more easily understood by various stakeholders. Diagraming can help to capture: 
+1. The right flow/sequence of steps,
+2. Dependencies,
+3. Redundancies, and
+4. Who is responsible for each step 
+
+
+## Main steps to workflow diagramming
+1. Diagram the human/manual process steps of this workflow,
+2. Identify opportunities for automation,
+3. Detail the functional steps of the ideal automation process, 
+4. Share the diagram with all stakeholders for final sign off, and update it as necessary
+
+The output of this exercise is clear documentation on how a business process will be executed by automation, humans, and often a combination of both. 
+
+## Diagram using global standards
+When diagramming, consider using global standards like BPMN (business process model and notation) so that diagrams are consistent and can be understood by outside parties. 
+
+1. There are lots of online resources to learn basic concepts like this post. 
+2. The OpenFn team has compiled some learning resources and BPMN diagramming tools in this document. 
+
+Your organization may already be using a different standard notation (e.g., UML). What’s important is consistency across your documentation to ensure that your solutions can be easily understood by other stakeholders. 
+
+![Workflow](/img/BPMN_example.png)

docs/design/discovery.md

@@ -4,4 +4,125 @@ title: Discovery & Scoping for OpenFn Projects
 ---
 
 # Discovery & Scoping for OpenFn Projects
-Share key scoping questions and introduce solution architecture diagram
+
+
+This article outlines key discovery and scoping questions to confirm the business value, core workflow requirements, technical feasibility, and client capacity when starting a new implementation. This article will be referencing the example use case introduced in the [series introduction](/documentation/next/design/design-overview#example-use-case). 
+
+
+:::tip 
+To quickly export and/or share these questions, please see this [slide deck](https://docs.google.com/presentation/d/1WIc_uNAqapILF7redhTnZXpPRo1jFPSmAjoAplGt42w/edit?usp=sharing).
+
+:::
+
+
+## Key Questions
+### Business Value Assessment
+
+The first step in the discovery process is a business value assessment to determine the potential return on investment, efficiency gains, and other valuable outcomes to be used to monitor success.
+
+__Questions to ask:__
+
+1. What are the workflow(s) you want to automate? 
+2. How are the workflows currently being managed? 21
+    - Is there an existing manual or semi-automatic business process? 
+    - If yes, how much staff time is spent managing these workflows?  
+3. What problem(s) will automation solve for? What efficiencies or benefits to be gained? What is the cost of inaction? 
+    - If we do not automate these workflows, then what is the status quo? 
+    - Is the current workflow slow, insecure, or leading to poor data quality or service delivery? 
+
+
+__Example:__
+1. I want to automate syncing case data from KoboToolBox to DHIS2 Tracker. 
+2. Our team currently spends 3 hours a week manually exporting the Kobo Data and inputting it to DHIS2.
+3. This automation will eliminate the risk of human error in manually entering the data, save us money and time, and enable us to work with more patients.
+
+## Workflow Requirements Gathering
+Use the questions below to determine the specific workflow steps. The output of these questions should be a draft workflow or business process diagram (consider drafting the diagram in [BPMN](https://www.bpmn.org/) to leverage a standardized notation). 
+
+__Questions to ask:__
+1. What triggers the workflow and how often should it run? (e.g., real-time or scheduled)
+    - Is there a user action or system event that should trigger the workflow to run? (e.g., real-time on form submission, or when record status set to “closed”)
+    - Or should this workflow be scheduled at a specific day/time? (e.g., every day at 12:00)
+2. Does the workflow require a one-way or two-way data flow? 
+    - E.g., if the workflow is sending one record from System A to System B, does the data need to flow only one way? Or once the data is synced to System B, should something be sent back to System A for a bi-directional data flow? 
+3. What are the expected data volumes? (e.g., 100 referrals every month, 12k forms every year)
+
+
+__Example:__
+The workflow should sync patient data from KoboToolBox to DHIS2 each time a form is submitted (i.e. real-time sync). There are a maximum of 5000 patients registered in Kobo per month.
+
+![Workflow](/img/functional_example.png)
+
+
+
+### Technical feasibility assessment
+The answers to the questions below will help you draft a solution diagram to document specifically which instances will be connected & which integration interfaces to use. 
+
+1. How many instances of the target systems exist? (I.e., Are you connecting to 1 or 2 DB instances?) 
+2. Are the target systems built? Is any configuration expected to change? 
+    - If config is still in progress, then consider revisiting this project when the systems are stable. 
+3. Is there an available REST API? 
+    - If yes, please provide the documentation. 
+    - If not yet, then consider revisiting this project after this API has been built and tested.
+    - If no, then what are other available methods for data import/export? 
+        - Is it possible to secure a direct database connection?
+        - Or is there a webhook or method for forwarding data to an external system? 
+        - Or a way to export/import data using file exports? What data formats are available? 
+4. Where are the target systems hosted? Any known security requirements or authentication considerations? (e.g., firewalls, VPN requirements, IP whitelist requirements)
+5. Is there an available test environment we can access to test integration with the application? (If not, is there a public demo of the application running on the same version that you’re currently running so that we can test the APIs?)
+
+Example:
+Only one instance of PatientCare and DHIS2 exist for this integration and they have already been built with REST APIs. They are both hosted on PatientCare managed servers that require IP whitelisting for access. 
+
+![Workflow](/img/technical_example.png)
+
+
+
+### Capacity assessment
+The answers to the questions below will help you identify project roles for the implementation design & delivery, as well as a plan for training, rollout, ongoing administration, and support.
+
+1. Does each target system have a full-time system administrator? 
+    - Are the administrators available to support integration setup & testing? 
+    - Will the administrators be able to provide a test/developer environment for testing? 
+    - Who will learn how to administer OpenFn? 
+2. What are their technical backgrounds?
+    - What are other resources available to provide ongoing support? 
+    - Does anyone in the organization have experience with Javascript or JSON?
+3. Is there a desire to learn how to manage the OpenFn implementation independently? 
+4. Who at the organization will be responsible for ongoing governance of the solution and overseeing change management? Are resources available to meet regularly to review change requests?
+
+__Example:__
+
+
+| Name | Role |
+| -------- | ------- |
+| Ian | OpenFn System Admin who will be responsible for ongoing mgmt & monitoring |
+| Melody | PatientCare Admin, who will train their system’s users on the workflow |
+| Arnis | DHIS2 Admin, who will train their system’s users on the workflow |
+| Ramona | Programs focal point who will champion users, inform workflow requirements, and meet regularly with users to collect feedback, propose changes, and review change requests with the system administrators |
+
+
+
+
+
+### Documenting the Solution Architecture
+
+Once you’ve gathered the key solution requirements, consider creating a “Solution Architecture” diagram to document the following: 
+1. Different solution components 
+2. Data flows between these components (highlighting data exchange within the organization and between third party services)
+3. Types of data exchanged
+4. Authentication/access points
+
+Such diagrams promote transparency, help to identify potential data exposure risks, and provide documentation of compliance with data protection requirements. Check out the example solution architecture diagrams below. 
+
+
+__Example 1:__
+
+![Workflow](/img/solution_diagram1.png)
+
+
+__Example 2:__
+
+| ![Workflow](/img/solution_diagram2.png) |
+|:--:|
+| *[Source](https://lucid.app/lucidchart/1e997197-2d67-4393-8394-a532d83561b2/edit?invitationId=inv_85b809a1-6fbd-4275-abdc-618fbd56e90d&page=0_0#)*|