Web interface for GH query notifications

[karelz]

Great UI: https://octobox.io/ ... could we piggyback on it?
Maybe inject another source of notifications (based on queries and settings like label/milestone changes per query)?

It would also allow us to enable notifications for all issues in your queries, without explicitly subscribing to the issues (which is pain).

Additional ideas:

• Team per-repo queries (for people with write permissions)
• Personal queries
• Option to monitor any label/milestone changes in a query

---

GitHub issue tracking in insufficient for daily developer job on large repo like CoreFX, CoreCLR, Roslyn, VSCode, etc.

GitHub provides:

• Notifications about comments (incl. Closing) for issues you subscribed to.
• Notifications about comments where you are mentioned explicitly.
• Unusable for large repos: Notifications for all issues in the repo = madness, no one uses that

Missing features:

1. Notifications about new issues/PRs in your area -- supplemented by email alerts from command line tool
   • Scenario: As area owner, you want to know which new issues were 'sent' to your area, which new PRs are in your area, so that you can participate in triage/code review.
   • Still not ideal: No customization which notifications to get. No per-user/alert cadence customization. No catch up after vacation.
   • Note: Even external folks today want this, they work around it today by sending notifications to Twitter, e.g. @corefxissues and @dotnetissues -- everyone tools around these things
2. Notifications about comments in your area (when you didn't manually subscribe to the issue)
   • Scenario: As area owner, you want to be auto-subscribed to all issues which are currently in your area(s), so that you don't have to do it manually. Bonus: Works also on 'Closed' issues when customers start commenting and asking on them (it happens quite often).
3. Notifications about label/milestone changes for issues in your area.
   • Scenario: As key area owner, you want to make sure the bugs don't get triaged out/marked in harmful way silently, without you knowing.
4. Notifications about label/milestone changes for issues you are subscribed to.
   • Scenario: As interested person, you want to make sure the bugs don't get triaged out/marked in harmful way silently, without you knowing.
5. Notifications about PRs which touch files/directories you monitor
   • Scenario: As owner of source code, you want to know when anyone makes changes to them.
6. Option to choose which notifications are via web UI and/or email alerts (like on GitHub)

Most of the missing features [2], [3] [4] and [5] are not smoothly doable in current form of the alerts tool. The tool reached its limits.
We need to invest in new tooling (2.0) ideally with web-based UI, similar to OWA email and https://octobox.io. More details on architecture and requirements are here: #9

Similar requirements are for any large repo - VSCode, CoreCLR, Roslyn, Google repos, etc. Everyone creates their own one-off half-baked tooling. It would be nice to habe just one which works for all (open source even).

[karelz]

It will likely require data delta pulling - #142

[karelz]

Architecture:

1. Database to mirror all GH issues changes
   • New data is pulled via GitHub APIs every ~10 mins -- see Data - pull only delta against last snapshot/pull #142 for details
   • Database backend captures each issue timeline: Azure tables (the schema is pretty trivial with 1-2 indexes)
   • Feature: Import/onboard new repo (i.e. download all info about the repo)
2. Database to store user preferences (queries settings, etc.)
   • Optional: If you change your query or add a new one, you should get the option to generate its notifications for a time period.
3. Database exposes generalized notifications (e.g REST API) - push notifications
   • Optional/Future: Expose the notifications as a service for existing solutions like https://octobox.io (similar to GitHub APIs)
4. Create web UI which has simple email / http://octobox.io look & feel.
   • Functions required:
     • Delete notification
     • Mark notification/issue as follow up
     • Unsubscribe from query-notifications (in cases you didn't subscribe on GitHub explicitly) - similar to 'ignore thread' in email client
   • Enable simple per-repo customization:
     • List of queries to watch/get notifications for (default query is entire repo)
     • Per-query options:
       • Notification for every new issue in the query -- note: Today even external folks want this, they work around it today by channeling notifications to Twitter, e.g. @corefxissues and @dotnetissues
       • Notification for every gone issue in the query
       • Notification for every comment on any issue in the query
       • Notification for milestone change for any issue in the query
       • Notification for label change for any issue in the query
     • Per-notification choice to be via email or only in web UI (or both) -- ideally should be configurable per query & notification kind combo
   • TODO: Provide mockup

---

Proof-of-concept -- key components:

1. Verify capability of GitHub APIs necessary to pull issue database updates (see Data - pull only delta against last snapshot/pull #142 for details from GitHub folks)
2. Design database service API for front-end consumption (follow GitHub REST? APIs model for easier consumption from other front-ends)
3. Define Azure tables schema (assuming Azure SQL is overkill + costly) (@ChadNedzlek's recommendation)
4. Validate UI JS framework - AngularJS 1 (@ChadNedzlek's recommendation - keep just 1!) (or @terrajobst's recommendation as he did bunch of web sites with JS)
5. Design authentication (reuse existing components/technology from mc.net)

Tips & tricks:

• Dogfood ASP.NET Core.
• Database can be mocked by existing data on disk (\clrmain\public\GitHubIssuesTracker\Data)

Execution plan:

1. Proof-of-concept - 1-2w
2. Minimal viable product - cost: TBD
3. Make it OSS project, let other large-repo teams to chip in and help share the cost (e.g. VSCode, non-MS), or interns / MVPs

[karelz]

Additional features for web UI:

1. Query reports - to provide copy-pastable list of bugs into email/OneNote - supported today via command line tool
2. Query diff report - new/gone/same issues between 2 data points - supported today via command line tool
3. Query history report - number of bugs / incoming/gone - supported today via command line tool
4. multi-repo queries - burndown graphs, etc. - supported as .csv output today via command line tool
5. Fancy graphs for query burndowns?

[karelz]

List of teams who requested the tooling:

1. Roslyn team (@jinujoseph) needs multi-repo burndowns - https://github.com/dotnet/core-eng/issues/706#issuecomment-297231268 (missing feature in current tooling - catch up on history)
2. ASP.NET team (@DamianEdwards) needs multi-repo burndowns - https://github.com/dotnet/core-eng/issues/706#issuecomment-297231268 (missing feature in current tooling - catch up on history)
3. Community member (@cwe1ss) would like to monitor specific area (Tracing), but doesn't scale to monitor entire repo
   • I suspect it is true for many more community members who just didn't speak up (the top MVPs just monitor Twitter for new issues - they can't monitor new PRs in area)
   • Temporary solution: I added @cwe1ss to my email alerts, but it doesn't scale to too many people. Also the lack of configuration is bummer.
   • Feedback from @cwe1ss: this tool is awesome and very helpful! Thanks Karel!

[karelz]

Let's move the architecture to wiki: https://github.com/karelz/GitHubIssues/wiki/Architecture

[karelz]

gitter room created for easier collaboration: