Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Telecommand-based, System-wide Mission Mode Management #11

Merged
merged 25 commits into from
Sep 5, 2023

Conversation

chgio
Copy link
Member

@chgio chgio commented Sep 4, 2023

As per #4, this MR introduces the implemented management of mission modes (idle, imaging, processing) across all* subsystems from a single, small telecommand transmission.

This feature has been thoroughly, though relatively coarsely, tested across a multitude of architectures and implementations, motivating its final shape.

* the Deployer and ADCS subsystems are included in architecture, but not in logic. As these two subsystems are not impacted by mission mode changes (see Capella [LAB]/[PAB] views of this feature), I opted against writing purposefully uncovered logic.

1. Distributed, Asynchronous

Originally (b44bbf3), the feature was implemented as architected in Capella (model version of PR #41), with a central System Mode Manager only enforcing the order of subsystem mode changes, and one Subsystem Mode Tracker per subsystem, each enforcing the validity of the requested subsystem mode transition. For simplicity, all interfaces were made sporadic (asynchronous).

This approach was adopted so that the OBC could track other subsystem modes as SDL states, mainly for the purpose of verifying SDL code by comparing the resulting State Chart View with the subsystem FSM specified in Capella.

It worked (kinda; source), but this architecture relied too heavily on sporadic interfaces, implementing synchronous behaviour through waiting states. This exposed its two main flaws:

  1. unreliable behaviour, resulting in Rare mismatch between mode and last operation in subsystem mode status reports #9
  2. excessive memory footprint, preventing it from fitting on the MSP430 -- even using the full 64KB afforded by the large memory model powered by the C code generation backend

2. Hybrid, Synchronous

To fix both flaws pointed out above, the feature was refactored to use protected/unprotected (synchronous) interfaces for all links except for the forced-async G/S -- OBC serial. Due to TASTE/SDL limitations, this also required centralising the subsystem mode transition validation logic from the Subsystem Mode Trackers to the System Mode Manager (8c48c42).

This was successful in achieving its targets:

  1. Rare mismatch between mode and last operation in subsystem mode status reports #9 was fixed to very high confidence ($\geq 99.9 \, \%$, source), as tested on x86 deployment
  2. memory footprint was reduced enough to fit all subsystems on the MSP430, demonstrating full round-trip communication and full logic execution (68a0f30)

However, this architecture also had some shortcomings:

  1. not scalable to distributed deployment: as TASTE requires that any interface allocated to an inter-device link is strictly sporadic, supporting communication with actual subsystems deployed on separate hardware nodes would have re-introduced nearly all the sporadic interfaces removed with the refactoring, and with them also their memory footprint
  2. not very elegant, and nullifying the original point of the distributed architecture: due to TASTE/SDL limitations when dealing with synchronous interfaces, a separate interface was required for each target subsystem mode; the centralisation mentioned above left the Subsystem Mode Trackers basically without logic, so meaningful State Chart Views of their implementations could no longer be obtained

3. Centralised, Asynchronous

To address both shortcomings pointed out above, the feature was once again reverted to using asynchronous interfaces, but now refactored to adopt a much more centralised architecture: the System Mode Manager includes all the logic of enforcing the order of the subsystem mode changes and validating the subsystem mode transitions (d66bba28); Subsystem Mode Trackers are to be considered Subsystem Mode Managers themselves (and eventually deployed with the rest of the subsystem on a separate hardware node), and duplicate the latter logic as a redundancy measure (92c454a). This duplicate logic is never nominally exercised, so a new System Mode Manager implementation was written only including the subsystem mode change order enforcement logic, so that the subsystem mode change validation logic in the Subsystem Mode Managers can be tested. Successful tests and coverage reports give credibility to this back-up logic (source).

Again, this has been successful in achieving its targets:

  1. sporadic interfaces enable the deployment of separate subsystems on separate hardware nodes
  2. meaningful State Chart Views can once again be generated from the Subsystem Mode Managers, enabling comparison with Capella -- and on the subsystem itself, where it actually matters
    ... and did not introduce any regression, as Rare mismatch between mode and last operation in subsystem mode status reports #9 is still fixed to the original confidence (source)

Critically, however, this introduces extremely significant paradigm shifts in our approach to both software and hardware:

  1. just like the "OBC Model" repository is evolving to encompass the entire CubeSat system, this "OBC Firmware" repository is also evolving to encompass the firmware of all (or at least most) subsystems of the CubeSat
  2. as there really does not seem to be any way to avoid high-memory-footprint sporadic interfaces for convenient distributed deployment, the MSP430 is to be phased out in favour of more modern chips with larger memory banks (most likely STM32). However, the MSP430 can still be used for simpler projects, such as CloudView launches with significant subsystem cuts -- such as the one coming up soon! (source)

Giorgio Ciacchella added 24 commits June 13, 2023 15:53
ignore:
- more build files
- generated Makefiles
- more code skeletons
unignore:
- main TASTE project files

consequently remove tracked files for enforcement.

squash me into extend and enforce gitignore
add and implement subsystem mode trackers:
- eps
- comms
- payload
- pipeline

also add and implement mission mode change telecommand handler
refactor interfaces:
- "report mode change":
    - rename to "report mode status"
    - add last_op field to report
- "confirm mode change":
    - rename to "return mode status"
    - exchange success boolean

consequently refactor functions:
- mission mode change tc handler:
    - directly map system mode status report to mission mode change
      report, instead of field by field
- system mode manager:
    - remove decisions before invoking subsystem mode trackers:
      now always invoked, even for non-changes
    - add decisions based on return boolean before updating subsystem
      mode variables
- subsystem mode trackers:
    - add assignment of report.last_op fields and success booleans in
      return signals
- rename mission-mode field of Mission-Mode-Change-TC-T data type into target-mission-mode
- add target-mission-mode field to Mission-Mode-Change-Report-T data type too
- edit mission_mode_change_tc_handler and system_mode_manager functions to handle target-mission-mode field

add MSC tests exercising mission mode changes:
{idle|imaging|processing} -> {idle|imaging|processing}
together with run_tests.sh to automate their execution
(fully automatic execution requires further investigation as it relies on an instance of the GS GUI running separately)

also remove reset_uplink cyclical interface from system_mode_manager function for reproducibility of MSC tests:
mission mode changes are accepted in downlink mode for simplicity; consequently, set comms_mode_tracker to initialise to uplink instead of idle
remove sporadic interface change_comms_mode, and create unprotected
interfaces:
- change_comms_mode_idle
- change_comms_mode_uplink
- change_comms_mode_downlink
... with output parameter T-Boolean success.

create the respective exported procedures in comms_mode_tracker, and
move transition policy enforcement logic to system_mode_manager.

also remove mission -> system mode mapping procedures from
system_mode_manager, instead simply initialising a system mode matching
the initialised subsystem modes -- comms still initialises to uplink as
no mechanism to enact its idle -> uplink transition is present yet.

ref #9
remove sporadic interface change_eps_mode, and create unprotected
interfaces:
- change_eps_mode_idle
- change_eps_mode_high_power
... with output parameter T-Boolean success.

create the respective exported procedures in eps_mode_tracker, and
move transition policy enforcement logic to system_mode_manager.

ref #9
remove sporadic interface change_pipeline_mode, and create unprotected
interfaces:
- change_pipeline_mode_idle
- change_pipeline_mode_processing_fp
- change_pipeline_mode_processing_lp
... with output parameter T-Boolean success.

create the respective exported procedures in pipeline_mode_tracker, and
move transition policy enforcement logic to system_mode_manager.

ref #9
remove sporadic interface change_payload_mode, and create unprotected
interfaces:
- change_payload_mode_idle
- change_payload_mode_imaging
... with output parameter T-Boolean success.

create the respective exported procedures in payload_mode_tracker, and
move transition policy enforcement logic to system_mode_manager.

ref #9
set report_*_mode_status interfaces to unprotected for:
- comms
- eps
- pipeline
- payload

...along with complete_system_mode_change, and create respective
exported procedures in mode_change_report_collector to "exfiltrate" the
content of the report.

120 runs of the 9 tests included in work/tests/run_tests.sh recorded 0
errors in the telecommanded mission mode change process, making
transitions of eps, comms, payload, and pipeline bug-free more than
99.99% of the time, so:
close #9
remove sporadic interface change_deployer_mode, and create unprotected
interfaces:
- change_deployer_mode_idle
- change_deployer_mode_deployment
... with output parameter T-Boolean success.

create the respective exported procedures in deployer_mode_tracker, and
remove transition policy enforcement logic -- no transfer to
system_mode_manager: deployer_mode_tracker is never invoked as part of this
feature, so it could not be tested.

also set report_deployer_mode_status interface to unprotected, and create
respective exported procedure in mode_status_report_collector
"exfiltrating" the content of the report.

ref #9
rename for accuracy:
- function mode_change_report_collector -> mode_status_report_collector
- interface relay_system_mode_change_report -> relay_system_mode_status_report
- parameter system_mode_change_report -> system_mode_status_report
...and relevant SDL signals and variables accordingly.

also, set relay_system_mode_status_report to unprotected instead of
sporadic, create a matching exported procedure in
mission_mode_change_tc_handler, and move the variable-setting behaviour
into it.

ref #9
remove sporadic interface change_adcs_mode, and create unprotected
interfaces:
- change_adcs_mode_idle
- change_adcs_mode_detumbling
... with output parameter T-Boolean success.

create the respective exported procedures in adcs_mode_tracker, and
remove transition policy enforcement logic -- no transfer to
system_mode_manager: adcs_mode_tracker is never invoked as part of this
feature, so it could not be tested.

also set report_adcs_mode_status interface to unprotected, and create
respective exported procedure in mode_status_report_collector
"exfiltrating" the content of the report.

ref #9
set change_system_mode as a protected interface, rename its parameter to
target_mode_in to facilitate exfiltration, and create matching exported
procedure doing just that.

however, in order to prevent deadlocks in this async-first architecture,
roll back relay_system_mode_status_report to sporadic.

also nest the system mode management group into a wrapper function
(interface grouping within nested functions is currently not supported)

ref #9
in interface view:
- set parameters of G/S -- S/C interfaces to "UPER" encoding
- remove already-unused but parasitic print_service function

in system_mode_manager implementation:
- turn intermediate states and continuous signals (not supported in
  msp430 runtime) into goto labels

set generic deployment view to be specific to the x86 POHIC-based
runtime, and change serial port configurations to:
- devnames: "/dev/ttyACM1" (x86) ; "/dev/tty0" (msp430)
- speed: b9600
- parity: even
- use-paritybit: yes

also update Dockerfile and Docker-run.sh to build on the debian:bullseye
development branch of TASTE for cutting-edge fixes
change mock HAL implementation for x86 from C to SDL to comply with
requirements for interactive simulation (make simu), and add generated
files to .gitignore along with work/Makefile
update Dockerfile and Docker-run.sh to build with new cutting-edge forks
enabling the x86 Linux C++ runtime to use Broker for serial links.

accordingly add deploymentview_cpp to use these developments, and
interfaceview to use the C code generation backend for SDL and optimise
some stack and queue sizes for sporadic/cyclic interfaces.

thanks Michał and Maxime!
change docker names and overwrite local function configuration fields in
interfaceview.xml to enforce:
- SDL mock implementations
- Ada code generation backend
in accordance with new Capella modelling, subsystem mode trackers now
represent the subsystem mode management itself, and the central mode
manager now also acts as subsystem mode status report collector and
buffer.
therefore, roll back interface types to asynchronous for deployment on
separate hardware nodes, and apply all consequent changes:
- roll back mode-specific subsystem mode change interfaces to
  mode-generic, passing the target mode as parameter
- remove the mode status report collector function entirely
- remove all subsystem mode change confirmation interfaces
- leave subsystem mode change validation checks in place in system mode
  manager, but make them all call the same (parametrised) interface
- roll back "wait for S/S" mid-states to implement sync behaviour using
  async interfaces
- implement confirmation of subsystem mode change through parsing (and
  subsequent buffering) of their mode status reports
- remove subsystem mode change validation logic based on specific states
  and interfaces from subsystem functions: these are now as thin as they
  can possibly be, and entirely trust system mode manager with target
  mode validation

also, rename some variables across functions for clarity:
- "system_mode" to "current_system_mode"
- "target_mode" to "target_system_mode"
- "target_mode in" to "target_system_mode_in"

...and redraw the interface view to split the system mode management group
into obc and other subsystems
as a safety measure, duplicate subsystem mode change validation logic
into the subsystem mode trackers themselves.

in order to test this, add new "test_subsystem_mode_change_validation"
SDL implementation to "system_mode_manager", only enforcing the order of
subsystem mode changes and exercising subsystem mode change validations
in the subsystem mode trackers -- as with the original architecture
(b44bbf3).

also change Dockerfile to self-merge new fixes to kazoo (for SDL-Ada
coverage collection) into Michał's fork for MSP430 serial communication.

check the GitHub comments for code coverage reports as evidence of this
working :)
(reverts it back to 68a0f30)
@chgio chgio added the feature Implementation of a new feature from the Capella model label Sep 4, 2023
@chgio chgio self-assigned this Sep 4, 2023
Copy link
Member

@xszymonzur xszymonzur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job, the PR comment is very thorough, the division into 3 separate design ideas and the references to the commit comments are especially good, it’s way easier to find them now. One can really see how and why the feature evolved over time. When it comes to the content of the PR itself, it seems to be fine and working, also it’s definitely been well tested and reviewed multiple times by now. Thanks for including the test_subsystem_mode_change_validation, it might just turn out to be useful in the future.

About the last paragraph, yes ofc the MSP430 is not going to last long term as discussed, but it still might be debatable whether OBC Firmware repo covering other subsystems of the CubeSat in the future is a good thing or not.

Solid work:fire:, good to go.

@chgio chgio force-pushed the feature/gio-telecommanded-mission-mode-change branch from 492ac25 to 69441ce Compare September 5, 2023 17:25
Michał's fork of the C++ runtime for x86 Linux is the only one we got to
work so far. maybe this is just because of print_service, maybe not, but
it's not broken, so why fix it? instead, let's promote it to our
default:
- rename deploymentview_cpp.dv.xml to deploymentview.dv.xml
- rename deploymentview_cpp.ui.xml to deploymentview.ui.xml
- delete deploymentview_pohic.dv.xml and deploymentview_pohic.ui.xml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Implementation of a new feature from the Capella model
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants