forked from HHS/TANF-app
-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #2499 from raft-tech/2344-research-synthesis
#2344 Research Syntheses & Readme Update
- Loading branch information
Showing
4 changed files
with
313 additions
and
11 deletions.
There are no files selected for viewing
106 changes: 106 additions & 0 deletions
106
...h-Syntheses/2022, Fall - Understanding STT Error Message Interpretation copy.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| #2022, Fall - Understanding STT Error Message(s) Interpretations | ||
| === | ||
| ___ | ||
|
|
||
| **Table of Contents:** | ||
| --- | ||
|
|
||
| * [Who we talked to](#Who-we-talked-to) | ||
| * [What we did](#What-we-did) | ||
| * [What we tested](#What-we-tested) | ||
| * [What we learned](#What-we-learned) | ||
| * [What's next](#What's-next) | ||
| ___ | ||
|
|
||
| ### Who we talked to | ||
|
|
||
| We recruited grantees based primarily on the following criteria: | ||
| * Grantees that had previously expressed interest in participating in research sessions | ||
| * Based on the error report by STT, identified the top STTs generating errors in 2021 for Section 1 - Active Cases | ||
| * Submitted data file types (TANF or SSP-MOE) to insure appropriate coverage | ||
|
|
||
| Participants that scheduled included | ||
| * 4 States | ||
| * 1 Tribe | ||
|
|
||
| ___ | ||
|
|
||
| ### What we did | ||
|
|
||
| The goals of conducting this research were to: | ||
|
|
||
| - Utilize the findings/insights from the Research Summary - Header Errors and conduct a usability review | ||
| - Explore how Grantees interpret the newly written error code descriptions based on the top error codes | ||
| - Inform the product direction on how to help Grantees resolve the errors on their own before re-submission | ||
|
|
||
| **Supporting Documentation** | ||
| * [Planning](https://hackmd.io/hFrD2ozgRFW8Bj3YOg4yig) | ||
| * [Style Guide](https://hackmd.io/J9mD0jeUSVm3QgJucSo5_g) | ||
| * [Affinity Map](https://app.mural.co/t/raft2792/m/raft2792/1664984644170/a2de5989b0699ce40e94100eb3c4145e114c153b?sender=u3ca60168b5ac4eb17bf92804) | ||
| * [Trial Run](https://github.com/raft-tech/TANF-app/issues/2193) | ||
|
|
||
| ___ | ||
|
|
||
| ### What we tested | ||
|
|
||
| We wanted to review examples of error communication language and layout to determine which elements enabled the grantees to locate and correct an error. | ||
|
|
||
| Each grantee was presented with two errors to review and provide feedback. Below is an example of the current and the revised error message. | ||
|
|
||
|  | ||
|
|
||
| Each grantee was asked to examine and dissect the context, and identify components of the error message output that would help aide in correcting the error. | ||
| ___ | ||
|
|
||
| ### What we learned | ||
|
|
||
| Jump to: | ||
|
|
||
| * [Current sentiment of the existing error message(s) from grantees](#Current-sentiment-of-the-existing-error-message(s)) | ||
| * [Elements of the new error message to help aide in correcting errors](#Elements-of-the-error-message-to-aide-in-correcting-errors) | ||
| * [Comparing between descriptions in full versus the subset within the error code](#Comparing-between-descriptions-in-full-versu-the-subset-within-the-error-code) | ||
| * [Other challenges grantees face with regards to their current process](#Other-challenges-grantees-face-with-regards-to-their-current-process) | ||
|
|
||
| #### Current sentiment of the existing error message(s) | ||
|
|
||
|  | ||
|
|
||
|
|
||
| > "when I see just this top error (legacy one) it makes me feel anxious and I would have to flip through and look for the answers" | ||
| > "I've been doing this a year and a half now and it's still the most cumbersome [to flip back and forth and figure out what it's trying to tell me]" | ||
| > "Depending on the amount [of errors] if it were just a handful 5-6 —it would probably take me about 3-4 hours to correct [because of all the lookup/decoding time before correction is possible]" | ||
| #### Elements of the error message to aide in correcting errors | ||
|
|
||
| Table layout | ||
| > "I like Error #1 because it appears to take the guesswork out with the table it seems to spell it out for me better" Just having this table would be Awesome!" | ||
| For More Details | ||
| > "the details link is good for new folks" | ||
| Displaying the full details | ||
| > "I really like seeing all of this laid out...this bottom half (new version) would be really useful and [time saving] for me" Nothing missing that would be useful to have. Nothing extra that's not adding anything to the error." | ||
| #### Comparing between descriptions in full versus the subset within the error code | ||
|
|
||
|  | ||
|
|
||
|
|
||
| > "I'm a why person—why was it coded this or why was it left blank. If it was all in one place it would be easier for me instead of flipping back and forth—and even for a new person coming in if it were all in one place" | ||
| > "Listing all of them would probably be more helpful—in my own opinion I'd just like to be able to learn more about it at a time...sometimes knowing which ones AREN'T supposed to be there is important so that you can learn..." | ||
| > "Would prefer to see all the options in case there is additional information that needs to be considered down the line" | ||
| #### Other challenges grantees face with regards to their current process | ||
| > We use Winscape/WinSCP and they require us to have staticIP for submissions and resubmissions downline. If we can just get this requirement removed it would be much easier for us to transmit files to ACF. | ||
| > reducing the time it takes the receive back the error report timing from "2-4 weeks" | ||
| ___ | ||
|
|
||
| ### What's next | ||
|
|
||
| - **[January 2023]** Continue with v3.0 error message research | ||
| ___ |
71 changes: 71 additions & 0 deletions
71
...2022, Winter - Understanding How STTs Would Use the Transmission Report copy.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| # 2022, Winter - Understanding How STTs Would Use the Transmission Report | ||
| === | ||
|
|
||
| * [Who we talked to](#Who-we-talked-to) | ||
| * [What we did](#What-we-did) | ||
| * [What we tested](#What-we-tested) | ||
| * [What we learned](#What-we-learned) | ||
| * [What's next](#What's-next) | ||
| ___ | ||
|
|
||
| ### Who we talked to | ||
|
|
||
| We recruited Grantees based primarily on the following criteria: | ||
| - Moderate error rate from previous data submissions | ||
| - Previously expressed interest in participating in research sessions | ||
|
|
||
| The research session consisted of the following grantees: | ||
| - 2 States | ||
| - 3 Tribes | ||
| ___ | ||
|
|
||
| ### What we did | ||
|
|
||
| The goals of conducting this research was to: | ||
|
|
||
| - Utilize the findings/insights from the Research Summary - Header Errors and conduct a usability review | ||
| - Explore how Grantees interpret the newly written error code descriptions based on the top error codes | ||
| - Inform the product direction on how to help Grantees resolve the errors on their own before re-submission | ||
| - Continue exploring the various ways that states and tribal programs interact for TANF data file submissions | ||
|
|
||
| ___ | ||
|
|
||
| ### What we tested | ||
| During this research session, we wanted to simulate the future experience of the transmission report (error report) with the Grantee. The following represents the tasks that were conducted and observed. | ||
| - Request and receive approved access to TDP | ||
| - Upload and Submit Section 1 TANF data file | ||
| - Receive email confirmation of receipt of Section 1 | ||
| - Execute and send via email to the Grantee with newly formatted transmission report in a .txt extension | ||
| - Allow time for the Grantee to resolve the errors on their own | ||
| - Grantee to resubmit Section 1 TANF data file back in for review | ||
| ___ | ||
|
|
||
| ### What we learned | ||
|
|
||
| Jump to: | ||
| * [Overall initial feedback of TDP, transmission report and process](#Overall-initial-feedback-of-TDP,-transmission-report-and-process) | ||
| * [Preference towards a more timely response post data submission](#Preferenc-towards-a-more-timely-response-post-data-submission) | ||
| * [Discovered a new data administration workflow between state and certain tribal program(s)](#Discovered-a-new-data-administration-workflow-between-state-and-certain-tribal-program(s)) | ||
|
|
||
|
|
||
| #### Overall initial feedback of TDP, transmission report and process | ||
|
|
||
| > "Everything seems to be where I can understand it" - Tribal Program | ||
| > "Pretty straightforward, there's nothing too complicated about it, it's easy to navigate through" - Tribal Program | ||
| > "This is awesome. This is super sleek" - Tribal Program | ||
| #### Preference towards a more timely response post data submission | ||
| - A confirmation email upon data file submission helps to provide confidence that submissions have been received | ||
| > "Sometimes it was a shot in the dark, because I would submit and then I wouldn't hear back sometimes months later" - Tribal Program | ||
| > "I wasn't notified that he information was "taken" but it was from my executive director - it was a hit and miss situation" - Tribal Program | ||
| #### Discovered a new data administration workflow between state and certain tribal program(s) | ||
| - A result of speaking with one of the tribal programs that submit through their state we were able to better understand their current workflow. In addition, this also helps inform the way we engage and provide support on their transition to TDP. | ||
|
|
||
| > "We submit an excel spreadsheet to the state and they are submitting for us" | ||
| > "We have a Tribal TANF Codebook we use that to manage the cases - not necessarily familiar with TDP error examples, but does recognize them as what they are and sees the connection" | ||
| ___ | ||
|
|
||
| ### What's next | ||
| - **[February 2023]** Pilot expansion for the February 15 data submission deadline | ||
| ___ |
112 changes: 112 additions & 0 deletions
112
...User-Experience/Research-Syntheses/2023, Winter - TDP 3.0 Pilot Program copy.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,112 @@ | ||
| #2023, Winter - TDP 3.0 Pilot Program | ||
| === | ||
|
|
||
| Following the [January 2023 research session](https://github.com/raft-tech/TANF-app/blob/develop/docs/User-Experience/Research-Syntheses/2022%2C%20Winter%20-%20Understanding%20How%20STTs%20Use%20the%20Transmission%20Report.md) on error message handling and resolution, the team transitioned into supporting the February 2023 pilot expansion cohort. We learned the following key insights from participating states, tribes, and territories (STT) and TDP system admin: | ||
|
|
||
| **Insight 1: The shift from fiscal (FY) to calendar year (CY) time frame selections can be confusing for those who may not be familiar with this structure.** | ||
|
|
||
| Although the distinction between FY and CY was mentioned in the pilot emails, user trainings, and within the Knowledge Center, the message was missed and some STT selected the wrong year/quarter, which would cause downstream data impacts if not corrected. | ||
|
|
||
| **Recommendation**: Include validation upon data submission to check for the Fiscal Year and Quarter selection based on the data set. Until that feature is developed, the home page has a FY-CY cross-reference table to alleviate confusion. | ||
|
|
||
| **Insight 2: It’s not easy for System Admins to onboard, and manage users and logs at scale.** | ||
|
|
||
| The System Admin role is integral to the success of user management within TDP, however the current steps to identify, review, and approve users can be tedious, which may result in delayed responses and potential errors, especially as the TDP user base grows. | ||
|
|
||
| **Recommendation**: Product team to add System Admin retros to help document inefficiencies within the admin tool (Django) during the various phases of user onboarding. See below for a list of system admin specific tickets already in development. | ||
| ___ | ||
|
|
||
| ### Below are the details to the February 2023 pilot expansion: | ||
|
|
||
| * [Who we talked to](#Who-we-talked-to) | ||
| * [What we did](#What-we-did) | ||
| * [What we tested](#What-we-tested) | ||
| * [What we learned](#What-we-learned) | ||
| * [What's next](#What's-next) | ||
| ___ | ||
|
|
||
| ### Who we talked to | ||
|
|
||
| We recruited grantees based primarily on the following criteria: | ||
| * Interest in participating in the pilot | ||
| * Tribal to State ratio to insure we can continue to provide a concierge level of support | ||
| * 5 or more tribal programs within a region | ||
| * Regional staff availability | ||
|
|
||
| TDP's user base is now made up of: | ||
| * 17 States | ||
| * 17 Tribes | ||
| * Two Territories | ||
|
|
||
| As of February 2023 these STTs represent the following regions: | ||
|
|
||
| | | R1 | R2 | R3 | R4 | R5 | R6 | R7 | R8 | R9 | R10 | Total | | ||
| |-|-|-|-|-|-|-|-|-|-|-|-| | ||
| | States | 2 | | 2 | 3 | 4 | 2 | | 3 | | 1 | 17 | | ||
| | Tribes | | | | | 7 | | 1 | 2 | 5 | 2 | 17 | | ||
| | Territories | | 1 | | | | | | | 1 | | 2 | | ||
| | Onboarded | 33.33% | 25.00% | 33.33% | 33.33% | 64.71% | 20.00% | 12.50% | 38.46% | 20.69% | 11.11% | | | ||
|
|
||
| ___ | ||
|
|
||
| ### What we did | ||
|
|
||
| The pilot program v3.0 was conducted in order to: | ||
|
|
||
| * **Provide** grantees the opportunity to perform the necessary tasks to complete their FY2023 Q1 data file submissions on their own | ||
| * **Identify** and document pain points, if any with TANF Data Portal | ||
| * **Observe** and document accessibility, browser and general IT compatibility concerns/issues | ||
| * **Allocate** and conduct high level research to help inform decisions for future releases | ||
| * **Build** a rapport with grantees for future expansion and utilization of the TANF Data Portal | ||
| ___ | ||
|
|
||
| ### What we tested | ||
|
|
||
| Each participant was tasked with the following: | ||
|
|
||
| * **opening** TDP in a compatible browser (i.e. not choosing Internet Explorer) | ||
| * **viewing** the home page | ||
| * **signing-in** using Login.gov | ||
| * **requesting access** to the TANF Data Portal | ||
| * **uploading** data files for FY2023 Q1 (October - December) | ||
| * **submitting** data files for FY2023 Q1 (October - December) | ||
| * **viewing** the submission notification (or submission error) | ||
| * **accessing** data files submitted historically through TANF Data Portal | ||
|
|
||
| ___ | ||
|
|
||
| ### What we learned | ||
|
|
||
| Jump to: | ||
|
|
||
| * [We observed confusion concerning which calendar quarters map to which fiscal quarters and prioritized an enhancement to the homepage to help address it](#We-observed-confusion-concerning-which-calendar-quarters-map-to-which-fiscal-quarters-and-prioritized-an-enhancement-to-the-homepage-to-help-address-it) | ||
| * [Some programs who submitted data files were not able to correct their files based on the generic error message](#Some-programs-who-submitted-data-files-were-not-able-to-correct-their-files-based-on-the-generic-error-message) | ||
| * [System Admin observations and feedback that occurred during onboarding of users and validating user submission success rate](#System-Admin-observations-and-feedback-that-occurred-during-onboarding-of-users-and-validating-user-submission-success-rate) | ||
|
|
||
| #### We observed confusion concerning which calendar quarters map to which fiscal quarters and prioritized an enhancement to the homepage to help address it | ||
|
|
||
| - The problem stems from the data files themselves organizing data in a calendar quarter schema rather than one that deals with fiscal quarters (beyond the header of each file). When users are only used to operating in that calendar quarter context it can result in incorrect selections when submitting in TDP which in turn causes downstream data misalignment. | ||
|
|
||
|  | ||
| - To aide users in selecting the correct Fiscal Year and Quarter we have designed a cross-reference table to be added to the TDP homepage as an interim solution to address their confusion until we have parsing (which will be able to identify the issue as part of the error report). | ||
|
|
||
|  | ||
|
|
||
| - The enhancement details can be found here [#1965](https://github.com/raft-tech/TANF-app/issues/1965) | ||
|
|
||
| #### Some programs who submitted data files were not able to correct their files based on the generic error message | ||
| - Specific copy and proper error messaging is recommended. | ||
|  | ||
| The details can be found here [#430](https://github.com/raft-tech/TANF-app/issues/430) | ||
|
|
||
|
|
||
| #### System Admin observations and feedback that occurred during onboarding of users and validating user submission success rate | ||
| In order to properly support additional users, the following should be taken into consideration for development. | ||
| - [Defect] VersionID missing for S3 submission history [#1007](https://github.com/raft-tech/TANF-app/issues/1007) | ||
| - [Feature] Being able to filter by STT and User Name so that system admin can view who is requesting access [#2415](https://github.com/raft-tech/TANF-app/issues/2415) | ||
| - [Feature] Reduce the number of unnecessary steps to approve a new user [#1641](https://github.com/raft-tech/TANF-app/issues/1641) | ||
|
|
||
| ### What's next | ||
| - [**March 2023**] Continue recruiting for onboarding STTs into TDP with regional team alignment so that we are able to stay on schedule for full onboarding | ||
| - [**April 2023**] Planning transmission report .csv file research in order to explore how Grantees may interact with the content for error resolution | ||
| - [**May 2023**] Support onboarding and data file submission efforts for the next data submission deadline |
Oops, something went wrong.