Identifying areas for effective and appropriate automation

[kariljordan]

Goal 5 of The Carpentries Strategic Plan reads: Strengthen organisational structure and capacity to be strategic and responsive.

One objectives under this goal is:

1. Establish workflow and documentation for our essential ongoing processes (e.g. budget creation and review, communication channel use), and identify areas for effective and appropriate automation or other improvements needed to scale support.

This issue tracks the work we are doing to automate internal workflows for:

• Membership program
• Instructor Training
• Instructor selection of workshops

[maneesha]

Leaving this as a placeholder for now - I'll come back later to update details.
EDIT: Details for each project are in a separate comment below (except for the workshop survey results)

Automation projects, April 2020 to present

• Automated emails for workshops (2020 Q3)
• Automated emails for memberships (2022 Q2)
• Instructor selection (2021-2022)
• Instructor training planning for automations (2022 Q2)
• Instructor training communications overhaul (2021 Q3)
• Instructor certificates in Helpscout (2021 Q1)
• Open training invitations (April 2021)
• Workshop survey results app (December 2019, September 2021)
• Instructor training - Trainer stat reports ()
• Instructor training - lesson contributions directly in AMY (2022 Q3)
• Scheduling teaching demos and community discussions
• Workshop survey results
• Lesson Website Automation: GitHub Actions

EDIT 2023-03-13 to change bullet list to checkbox list and add in items created by other Core Team members.

[maneesha]

Automated emails for workshops

Context:
Workshop administrators have standard template emails that they send out to workshop Hosts, Organisers, Instructors, and Helpers. These emails include information such as introductions of people serving various roles, reminders about the workshop website and survey, and post-workshop follow ups. The emails all contained the same information, customized to each workshop's details such as date, personnel, and location. Because the emails content was generally the same, it was a tedious task to draft the same emails for each workshop.

New system:
Our AMY developer created a system where emails would automatically go out to workshop personnel on set dates relative to the workshop. This ensured that workshop adminstrators did not have to manually generate the text for each email. It also ensured emails would go out at exact dates/times relative to the workshop date because they were automated, rather than depending on the workshop administrator's availability. Read more about the automated emails. Planning work on this system began in 2019, and was implemented in mid-2020.

Next steps:
As our operations change, these emails will also have to change. The Infrastructure Team (Maneesha) is working closely with the Workshop Admin team (Sher!) to identify changes needed to these emails based on eliminating the role of Supporting Instructor, running both in-person and online workshops, and implementing the new Instructor Selection project (see below). Automated emails that support other teams (such as Instructor Training) may also be considered in this context. Much of this work will depend on the availability of an AMY developer.

[maneesha]

Automated emails for memberships

Context:
When a new membership is signed, the Membership Team has a standard template they use to send to site contacts including information on how to request workshops, instructor training, as well as general information about The Carpentries. Because the email content was generally the same, it was a tedious task to draft the same emails for each membership.

Additionally, member sites would ask about their benefit usage. Providing benefit usage reports on demand becomes another time consuming task. We wanted a system where member sites would proactively receive regular updates about their membership usage to remind them to use their benefits before the membership term ended.

New system:
AMY developer time is limited. Knowing this and that unlike workshops, the membership emails can be sent at approximate times relative to the membership date, the Infrastructure Team decided to have these emails sent outside of AMY. Maneesha wrote queries and scripts to be run weekly. This would identify all memberships that were entered in AMY in the past week and identify all memberships hitting their three month, six month, or renewal checkpoint in the past week. The script runs on a cron job. It creates a draft email in Helpscout to these members. This draft is reviewed by someone on the Membership Team (Angelique), who sends it if it looks good, or reports to Maneesha if there are any concerns. Planning work on this began in late 2021. System was implemented and moved to a monitoring phase in June 2022.

Next steps:
This project is in a monitoring/maintenance phase now, and if all emails look good, we will move to having them all sent directly, rather than going through the draft review. These emails currently only cover standard memberships (Bronze, Silver, Gold, Platnimum). Future steps may also consider how we handle non-standard agreements.

[maneesha]

Instructor selection

Context:
Currently when a new workshop request comes in, if it needs us to recruit Instructors, the Workshop Administrator manually enters the necessary information into a spreadsheet and sends a call out to our Instructors mailing list letting them know new opportunities have been posted. As Instructors sign up, the WA reviews signups, manually cross-checks them in the AMY database for things like teaching experience, areas of expertise, location, etc. Based on this the WA confirms instructors to teach. Through this project, we are moving signups to be inside AMY, making it easier for instructors and core team members to manage the process.

New System
All instructor recruitment activity will happen within AMY. When a new workshop is entered in AMY, the workshop administrator will be able to open the instructor recruitment process for that workshop. A new view in AMY, available only to active certified Instructors, will allow them to see details of upcoming workshops and register their interest in teaching them. Still within AMY, the workshop administrator will be able to see who is interested in each workshop along with each instructor's profile, allowing them to easily confirm or decline the instructor to the workshop. This reduces administrative load on the Workshop Administrator and ensures none of the Instructor's information is publicly accessible. We are currently reviewing this system on the test AMY server with Core Team members and volunteer Instructors. Planning work on this project began in April 2021, with plans for implmentation in 2022 Q3.

Next steps
Concurrent with release of this system will be release of a unified Carpentries Instructor badge. Currently, as a historical relic, we have three distinct instructor badges for the three lesson programs. There is no real distinction between the three badges though. This creates technological, administrative, and branding challenges. All current Instructors will be re-issued a Carpentries badge, and all new Instructors will get a Carpentries badge.

After the new system is implemented, we will also consider changes to automated emails (see above) and alerts when new teaching opportunities are posted.

[maneesha]

Open Training Invitations

Context:
When new applicants were accepted through our open application program, a team member would periodically send them reminders to register for instructor training. This was done by manually querying accepted applicants in AMY, pasting emails in Helpscout, and sending mass email to all participants linking to our current calendar. Because Helpscout limits emails to 25 recipients, this often had to be done in multiple batches.

New system:
Maneesha set up a cron job to query all individuals who have been accepted to instructor training but have not joined an event. It then sends each individual an email (rather than a mass bcc email) reminding them to register, and sends a log to the Instructor Training mailbox listing who has received this communication. This was set up in April 2021.

Next steps:
No next steps are planned specifically for this automation, but other related steps may be considered in the context of general instructor training automations.

EDIT 2023-03-13 to add link to cron job.

[maneesha]

Instructor training planning for automations

Context
Instructor Training administrative workflows are fragmented and time-consuming. Many of the most time-consuming steps could be automated. With Kelly stepping into another role in 2022 and many pressing needs for staff time elsewhere, improving efficiency with automation to the greatest extent possible is necessary and urgent. Several next steps have previously been identified, including import of attendance data to AMY. This project will additionally include a comprehensive review to ensure that requests for AMY development in particular are maximally efficient and effective, and that all impacts of change to existing workflows have been anticipated.

New system
This was a meta-project. The Instructor Training team and Infrastructure Team reviewed all current workflows, and identified the most tedious, repetitive, or time consuming steps within each workflow. This allowed the Instructor Training team to note where changes to streamline processes could be made. We also identified what resources we would need to make these changes (Core Team time, external AMY developer time, or other resources). These discussions happened over 2022 Q2.

Next steps
No next steps directly related to Instructor Training automations are being started, because this process highlighted that we need better planning systems around these type of work.

Out of these discussions, we developed a suggested plan to add requests like this (from all teams within the Core Team) to the Infrastructure Team road map. This will include:

1. Conversations Maneesha has with each team about their needs
2. Discussions within the Infrastructure Team about these needs
3. Discussions at the Team Leads meeting to review these priorities and get them on the Infrastructure roadmap.

[maneesha]

Instructor training communications overhaul

Context
The communications experience of a Trainee from first expressing interest in Instructor Training through certification and the associated Instructor Training Team workflows have developed organically over time. The process includes many touch points between trainees, Trainers, and The Carpentries Core Team. Frequent questions from Trainees suggested the process may be inundating Trainees with too much information in too many different places.

New System
The team mapped out the course of communications that go out to trainees, and made changes to our email templates, website, handbook, and other workflows. By being methodical and intentional about all the information we were sharing, when it was being shared, we were able to move towards consistency and clarity in our communications. This sets us up well to scale our systems when we have opportunities to automate these workflows.

Next steps
The Instructor Training team continues to monitor these new communication flows and identify areas for improvement. Many of these areas will be addressed in the next steps for Instructor training planning for automations.

[maneesha]

Instructor certificates in Helpscout

Context
Instructor Certificates are generated and saved in a private GitHub repo. Approximately once a week, a Core Team member would send out certificates to all newly badged Instructor via a local script on their computer. There was no log of this, so we had no way to respond if a newly badged Instructor wanted to know if their certificate was sent out.

New System
The script was updated so that it would not only send the email to each newly badged Instructor, but would also include additional inforamtion about joining The Carpentries community. It also sent a copy of the certificate to the Instructor Training mailbox, making it easier for Core Team to track when certificates were going out. This was implemented in January 2021.

Next steps
While the system is improved, it is still manually run on a local Core Team member's computer. Ideally, this should be integrated into AMY so that an Instructor will get their certificate immediately after a badge is issued, and can access their certificate at any time by logging into AMY. Given AMY developer time is limited, this is likely low priority.

[maneesha]

Instructor training - checkout contribution directly in AMY

Context
One of the three steps towards instructor training certification is a contribution to a Carpentries repo. This includes doing the following:

1. Review the email
2. Look up the profile in AMY
3. Copy and paste the url to the contribution from email to AMY
4. Mark it as passed
5. Reply to the email.

New System
AMY has always had the capability of letting trainees submit their contribution directly in AMY, but we have not publicized this. We are moving towards having them submit their contribution directly in AMY to reduce the administrative steps on our end (all we have to do is click "accept"). By having trainees log in to AMY directly before they get certified, this will also ensure they are ready to use our new system to sign up for workshops. This will reduce administrative burden on the Core Team from new Instructors who can't log in to AMY to sign up for upcoming workshops. A plan for this was developed in 2022 Q2, with implementation (next steps) in 2022 Q3.

Next steps
A general plan for this has been developed. There are a few changes in AMY that will need to be made, and can be made concurrently with the work in the instructor selection project. At that time, we will also update our documentation and curriculum to reflect this change.

[maneesha]

Scheduling teaching demos and community discussions

Context
Long ago, teaching demos and community discussions were scheduled at fixed times each week. This meant they did not account for the availability of Trainers and Discussion Hosts to run these sessions, and did not give community members enough variety of time slots to choose from.

New System
In 2019, we began using Calendly so Trainers could choose their own slots to host teaching demos. In 2020, we included community discussion hosts to follow the same process. Each quarter, Trainers and Discussion Hosts are invited to use Calendly to sign up to run sessions at times that work best for them. Trainers'and Hosts' availability should be representative of our larger community, so the distribution of time slots should accommodate our global community.

In addition to using Calendly for Trainer/Host signups, we also use a script to generate the text for the Etherpad signup and a script to use Google Calendar's API to list these events on our community calendar. Both of these tasks had historiallly been done manually. We began this system in 2019 for Demo sessions and expanded to include Community Discussions for the 2022 Q3 sessions.

Next steps
The Etherpad is not an ideal way for people to sign up. We have looked into other scheduling options but have not found ones that meet all our needs - including features like an API with GET and POST endpoints and ability for Trainers/Hosts to view registration and contact participants.

[fmichonneau]

Workshop survey results

Context
To give access to the pre- and post-workshop survey results, we used to rely on SurveyMonkey featured. It required manual work as a link needed to be generated for each workshop based on the slug of the event.

New System
Starting in late 2019, we switched to a new system where the surveys for the workshops are captured in Typeform. The data is then sent to Airtable. We developed a Shiny app that allows us to share data from the surveys with the hosts and instructors for a workshop. This system was extended in September 2021 to also include the results from the Instructor Training surveys.

Next steps
Ideally, we would like to move away from Airtable and develop our own self-hosted database to capture the results from these surveys.

[zkamvar]

Lesson Website Automation: GitHub Actions

Context

From 2015 to the present, Carpentries lessons were built on top of a Jekyll template called carpentries/styles. The term "template" is a misnomer as it also contained the tools needed to build the lesson and these tools and stylings needed to be updated manually via pull request.

New System

The Carpentries Workbench is a new lesson infrastructure that separates content from tooling with R packages to build (sandpaper), validate (pegboard), and style (varnish) the lessons. We used GitHub Actions for deployment, with each repository containing a folder called .github/workflows/ that hold a series of workflows (which can be found at https://github.com/carpentries/sandpaper/tree/main/inst/workflows) that use GitHub Actions in https://github.com/carpentries/actions/ do deploy, validate, and maintain the workflows on a weekly basis.

These workflows are updated by creating a pull request on a weekly check when the workflows have updated. This process uses a token generated from the @carpentries-bot account (protected with 2fa) that is manually rotated on a schedule.

Because the maintenance workflows use a token, the github action check valid pr is used as a measure against malicious attacks via manipulation of the PR action.

Next steps

Lessons are to be deployed to an AWS S3 bucket folder using a unique key per lesson.
Workflows components should be validated and updated with a defined schema so that authors outside of the carpentries can define their own steps should the need them and still benefit from the workflow updates using their own tokens. A secure system for token storage and regeneration may need to be considered.

[maneesha]

@kariljordan I saw you just assigned this to me. Is there anything else you need beyond what's shared in the comments above?

[kariljordan]

HI @maneesha, not at this time. I'm reassigning François' issues. Thank you so much.

[ErinBecker]

Adding @froggleston to this issue. Will discuss during his onboarding to the Core Team.

[froggleston]

Rob and @maneesha met today and we can provide this update:

• Automated emails infrastructure

  • Condition-based scalable infrastructure needs to be developed
  • Currently on the new IT Roadmap (https://app.asana.com/0/1203796613288612/1202971413325672)

• Instructor selection

  • In October, we released the system in AMY to log into their profiles and click interest
  • All done, except teaching opportunity alerts

• Open training invitations

  • done

• Instructor training planning for automations

  • Recent discussions with Karen
  • verification of member codes and changes to checkout process

• Instructor training communications overhaul

  • Really should now be in the previous task

• Instructor certificates in Helpscout

  • Maneesha manually runs this script weekly
  • roadmapped (https://app.asana.com/0/1201014068954724/1201044632853599)
  • can do this whenever the automated email system is set up
  • only thing we can't do now is send out the automated email in AMY (see above)

• Instructor training - checkout contribution directly in AMY

  • Done, but we are now changing the checkout process
  • Still relevant but not in the same way
  • This process should stay the same once the checkout process changes have been made, but there will be a slight difference to the checking off process that trainees will do in AMY
  • Curriculum and documentation will need to reflect these changes

• Scheduling teaching demos and community discussions

  • This is from the trainee POV, not the host
  • Currently use etherpad for signups, which isn't optimal
  • Having a calendar of events and allowing trainees to select them would be best, but no complete solution is available
  • Calendly is close, but it doesn't have a POST endpoint in its API so you need to manually create the events
  • Also doesn't let trainers or hosts see who has signed up, only the Core Team
  • could eventbrite work? yes but trainers could still not see who is signed up, and we'd need to know who the host is
  • on demand access to the information would be needed too (AMY querying the eventbrite API?)
  • this could be in AMY with a calendaring app (https://djangopackages.org/grids/g/calendar/)

• Workshop survey results

  • Done, and still using the described workflow
  • Not 100% sure why moving away from Airtable was proposed
  • Working for now so low priority

• Lesson Website Automation: GitHub Actions

  • Will check in with Zhian

Next steps for Q2/Q3

• Automated emails implementations
  • Workshop logistics
  • Teaching opportunities
  • Memberships

[froggleston]

Automated emails infrastructure

• The new email automation infrastructure is progressing well, and we're hoping that we will see the first templates and automations happening Q1 2024
• This will support many of the current manual email processing tasks that are based on events that happen and information that is held in AMY, e.g. sending out information about Instructor Training approaching

Instructor training planning for automations

• including Instructor training communications overhaul
• Changes to checkout were released in AMY 4.2, and subsequent changes required to support differentiation between in-person and online workshops is ongoing
• Verification of member codes is ongoing with a release expect at the end of Q4 2023

Instructor certificates in Helpscout

• This process is still manual, but we have deployed the scripts within our AWS infrastructure to share the management of running these every week
• This will hopefully be addressed in future by the Event Scheduling work planned (see below)

Instructor training - checkout contribution directly in AMY

• See above

Scheduling teaching demos and community discussions

• This is the first new major piece of work that the Technology Team is undertaking that was net previously roadmapped
• It will require a complex mix of various tools and APIs that will need thorough development and testing time
• We are exploring potential solutions and are working through the technical requirements and test code in 2023 with potential solutions arriving in mid to late 2024

Lesson Website Automation: GitHub Actions

• Development of lesson websites is progressing, with new features such as overview and child pages now supported
• Automation around these development is supported through a variety of github actions

[elletjies]

Automation of Asana Template for Self-organised and Centrally-organised workshop administration