-
Notifications
You must be signed in to change notification settings - Fork 47
Services
Internal to Greengrass Nucleus each Greengrass component is implemented by the GreengrassService class, so internally they are known as "services" rather than components. I'll try and be consistent and clear when talking about components versus services.
GreengrassService class is the base class for all services including external components, internal services (ex: DeploymentService), and lambdas.
GreengrassService does not implement the state machine for actually moving the service between lifecycle states, it instead is the interface to request state changes and implements the state logic like install or shutdown. The state machine is separated out in the Lifecycle class.
When created, a GreengrassService begins by identifying its dependencies and configuring a callback for when the list dependencies changes. The service then identifies any change to the list of dependencies. For any removed dependency, the dependency state listener is removed. All remaining existing or new dependencies are then configured with a dependency state listener. A dependency state listener will receive state change events for all services, identify if it is the dependency service we're interested in, and then restart this service if the dependency service is not in a good state and this service is either currently running or starting up. If all dependencies are in a healthy state, then the state listener will also send a notification to the dependencyReadyLock which will unblock the lifecycle thread which may be waiting for dependencies to be ready prior to starting this service.
After loading dependencies in the constructor, the dependency injector will call postInject where we initialize the lifecycle. This starts the lifecycle thread running in preparation to get the service running.
Each service has a set of lifecycle commands: bootstrap(), install(), startup(),
handleError(), shutdown(), and close().
bootstrap() will be executed when the Nucleus performs a deployment when isBootstrapRequired(Map<String, Object>) is true for this service and the Nucleus enters the BOOTSTRAP phase where no other services are running. Bootstrap is used to make significant changes to the OS and system level packages. What is special about bootstrap is 1. no other services are running at the same time, 2. the bootstrap step can request either a Nucleus restart or a system reboot after executing.
install() is used to install any requirements such as Python packages before executing the main service lifecycle. Greengrass does not persistently track what state services are in, therefore, a service's install() will execute every time Greengrass starts up even if everything was previously installed and running. install() runs in parallel for all services, disregarding any dependencies. Dependencies are only used to control when a service is allowed to enter the STARTING state. There is a default 120s timeout for install() to complete and transition to STARTING. If the timeout expires then Greengrass will move the service into the ERRORED state and retry the install.
startup() is used to get the service into the RUNNING state. A service should only report that it is in the RUNNING state when it is truly ready and dependencies could successfully use it at that point in time. There is a default 120s timeout to move to the RUNNING state. If the timeout expires then Greengrass will move the service into the ERRORED state and retry the startup.
handleError() is called when the service enters the ERRORED state. This method may be used to attempt some sort of error recovery procedure to prevent the error from reoccurring.
shutdown() is used to stop the service when the service is in the STOPPING state. There is a default 15s timeout to move to the FINISHED state from STOPPING. If the timeout expires then Greengrass will move the service into the ERRORED state and then move to the desired state or FINISHED if nothing else was desired.
Each service also has some query methods which services may choose to override: isBootstrapRequired(Map<String, Object>), shouldAutoStart(), and isBuiltin().
Lifecycle implements the state machine for services. There is one instance per service and each instance runs its own thread. This lifecycle thread executes the state machine by identifying what state the service is currently in, and what state it wants to get to, if it isn't already in the desired state.
To move between states, lifecycle tracks a list of desired states so that it can perform more complicated actions like restarting which require going through multiple states. In the restart case, the desired states would be FINISHED, RUNNING so that the service first needs to get to the finished state where it is no longer running, then get into the running state again.
The lifecycle thread will block on the stateEventQueue waiting for events such as a notification that we'd like to move the service to a different state.
Services can have the following states: STATELESS (this state is never used), NEW, INSTALLED, STARTING, RUNNING, STOPPING, ERRORED, BROKEN, FINISHED. A normal service would start as NEW, then INSTALLED, STARTING, RUNNING, FINISHED.
The BROKEN state means that Greengrass is giving up on restarting the service due to the service erroring 3 times within 1 hour. There is no way to opt out of this behavior. A service will get out of BROKEN if Greengrass restarts or the service moves itself to NEW (reinstallation). An external service will reinstall itself when the version, install script, runwith, or resource limits change or when any other lifecycle changes and the service is in BROKEN state.