Development #1
Development #1
Conversation
The observation class defines the basic structure to implement the observation tree. It's still a WIP. Therefore, the entire class is commented out.
|
|
|
Hi @ID-Akash and @rsjeffers I tried to address the TODOs in the TreeObservation class: package ch.sbb.matsim.contrib.railsim.observation;
import ch.sbb.matsim.contrib.railsim.qsimengine.RailsimCalc;
import ch.sbb.matsim.contrib.railsim.qsimengine.TrainPosition;
import ch.sbb.matsim.contrib.railsim.qsimengine.resources.RailLink;
import ch.sbb.matsim.contrib.railsim.qsimengine.resources.RailResource;
import ch.sbb.matsim.contrib.railsim.qsimengine.resources.RailResourceManager;
import ch.sbb.matsim.contrib.railsim.qsimengine.resources.ResourceState;
import org.matsim.api.core.v01.network.Link;
import org.matsim.api.core.v01.network.Network;
import org.matsim.api.core.v01.network.Node;
// TODO: What is the time resolution of the environment? 1 iteration is equal to home many seconds in rlsim?
// The time resolution of matsim is seconds, same for railsim. But there are seconds where nothing happens, no events in update queue.
import java.util.List;
import java.util.ArrayList;
class OtherAgent {
// Distance of agent to other agent's tail position if the train is moving in the same direction
// Otherwise, distance to BufferTip position if moving in opposite direction
double distance;
List<Double> bufferTipPosition;
List<Double> tailPosition;
double speed;
public OtherAgent(double distance, List<Double> bufferTipPosition, List<Double> tailPosition, double speed, boolean sameDirection) {
this.distance = distance;
this.bufferTipPosition = bufferTipPosition;
this.tailPosition = tailPosition;
this.speed = speed;
this.sameDirection = sameDirection;
}
boolean sameDirection;
}
class ObservationNode {
Node node;
double distNodeAgent;
double distNodeStop;
boolean isSwitchable;
OtherAgent sameDirAgent;
// OtherAgent oppDirAgent;
}
public class TreeObservation {
private final TrainPosition position;
private final RailResourceManager resources;
private final Network network;
private List<ObservationNode> observation;
public TreeObservation(TrainPosition position, RailResourceManager resources, Network network) {
this.resources = resources;
this.position = position;
this.network = network;
createTreeObs();
}
public RailLink getBufferTip() {
double reserveDist = RailsimCalc.calcReservationDistance(position, resources.getLink(position.getHeadLink()));
RailLink currentLink = resources.getLink(position.getHeadLink());
List<RailLink> reservedSegment = RailsimCalc.calcLinksToBlock(position, currentLink, reserveDist);
// TODO Verify if the links are added in the list in the sequence of occurence.
RailLink bufferTip = reservedSegment.get(reservedSegment.size() - 1);
return bufferTip;
}
private void createTreeObs() {
int depth = 3;
createTreeObs(depth);
}
// TODO: Handle cases when the agent should also see the next 2 halts. - But this may not be needed as the agent ca be penalised
// if the agent is unable to reach the next station.
private void createTreeObs(int depth) {
// Get the link of the tip of the buffer
RailLink bufferTipLink = getBufferTip();
// Get the toNode of the bufferTipLink
Node toNode = getToNode(bufferTipLink);
List<Node> exploreQueue = new ArrayList<Node>();
exploreQueue.addLast(toNode);
for (int i = 0; i < depth; i++) {
// Depth traversal: Making an observation Tree of fixed depth
int lenExploreQueue = exploreQueue.size();
while (lenExploreQueue > 0) {
// Level traversal
Node curNode = exploreQueue.getFirst();
exploreQueue.remove(0);
// Look for switches/intersections/stops on the branches stemming out of the current switch
List<Node> nextNodes = getNextNodes(curNode);
for (Node nextNode : nextNodes) {
exploreQueue.addLast(nextNode);
TrainPosition trainF = getClosestTrainOnPathF(curNode, nextNode);
TrainPosition trainR = getClosestTrainOnPathR(curNode, nextNode);
observation.add(createObservatioNode(nextNode, trainF, trainR));
}
lenExploreQueue -= 1;
}
}
}
private TrainPosition getClosestTrainOnPathR(Node curNode, Node nextNode) {
// complete route to current position of the train from schedule, if needed?
List<RailLink> previousRoute = position.getRoute(0, position.getRouteIndex());
// TODO: If all opposite trains are needed, follow the inLinks of curNode until the nextNode is reached. Store the of the path links.
List<Link> path = null;
// check for each link if there is capacity
for (Link link : path) {
RailLink railLink = resources.getLink(link.getId());
RailResource resource = railLink.getResource();
ResourceState state = resource.getState(railLink);
// TODO: Ask Christian how we get the train position of the nearest train.
}
return null;
}
private TrainPosition getClosestTrainOnPathF(Node curNode, Node nextNode) {
// complete route from current position of the train from schedule, if needed?
List<RailLink> upcomingRoute = position.getRoute(position.getRouteIndex(), position.getRouteSize());
// TODO: If all opposite trains are needed, follow the outLinks of curNode until the nextNode is reached. Store the of the path links.
List<Link> path = null;
// same procedure as above...
return null;
}
private List<Node> getNextNodes(Node curNode) {
// next switches
List<Node> switchNodes = new ArrayList<>();
// check all outgoing links from the current node
for (Link outLink : curNode.getOutLinks().values()) {
Node nextNode = outLink.getToNode();
// follow nodes with only one outgoing link until a switch is reached
while (nextNode.getOutLinks().size() == 1) {
// get the single outgoing link and follow it
// TODO: Handle end of network, will throw NoSuchElementException at the moment
nextNode = nextNode.getOutLinks().values().iterator().next().getToNode();
}
// if the next node has more than one outgoing link, it's a switch
if (nextNode.getOutLinks().size() > 1) {
switchNodes.add(nextNode);
}
}
return switchNodes;
}
private Node getToNode(RailLink bufferTipLink) {
return network.getLinks().get(bufferTipLink.getLinkId()).getToNode();
}
public List<ObservationNode> getObservation() {
return observation;
}
}The TreeObservation needs to have access to the Network, therefore i added it to the constructor. To get access to the TrainPositions of the opponent trains, we should consult Christian (@rakow), as a simple method to extract this object was not obvious to me. Best, M. |
|
The disposition is given the TrainPosition on each request, as well as for departing trains. It should be able to keep track of the state and update it accordingly on every change. The information always goes from the railsim engine to other components, and can not be directly requested. Another option would be to let the RailResourceManager keep track of some state that is commonly used in multiple dispoisition strategies. |
|
@rakow thank you for your message, I didn't fully understand could you elaborate a little on what you're proposing? For each train we need to be able to observe the trains near to it in order to make an informed decision about which direction to take |
|
Maybe I got it completely wrong, but I interpreted @rakow answer as follows: package ch.sbb.matsim.contrib.railsim.rl;
import ch.sbb.matsim.contrib.railsim.config.RailsimConfigGroup;
import ch.sbb.matsim.contrib.railsim.qsimengine.RailsimEngine;
import ch.sbb.matsim.contrib.railsim.qsimengine.deadlocks.NoDeadlockAvoidance;
import ch.sbb.matsim.contrib.railsim.qsimengine.resources.RailResourceManager;
import org.matsim.api.core.v01.Id;
import org.matsim.api.core.v01.Scenario;
import org.matsim.api.core.v01.population.Person;
import org.matsim.core.api.experimental.events.EventsManager;
import org.matsim.core.events.EventsUtils;
public class RLEnvironment {
private final Scenario scenario;
private final RailsimEngine engine;
private final EventsManager eventsManager;
private final RLDisposition disposition;
private final RailResourceManager resourceManager;
private double time = 0;
public RLEnvironment(Scenario scenario) {
RailsimConfigGroup config = new RailsimConfigGroup();
this.scenario = scenario;
this.eventsManager = EventsUtils.createEventsManager();
// TODO: Add a BasicEventHandler implementation that handles or tracks events as needed
// eventsManager.addHandler(BasicEventHandler);
// eventsManager.initProcessing();
this.resourceManager = new RailResourceManager(eventsManager, config, scenario.getNetwork(), new NoDeadlockAvoidance());
this.disposition = new RLDisposition(scenario.getNetwork(), resourceManager); // pass in disposition whatever needed
engine = new RailsimEngine(eventsManager, config, resourceManager, disposition);
}
public TreeObservation step(Id<Person> agent) {
// wait until agent with agentId calls rl disposition
// while (true) {
// engine.doSimStep(time++);
// }
// TODO: Generate tree observation internally in the disposition and let the agent decide based on it, see RLDisposition::requestNextSegment
// Maybe some synchronization needed... semaphore or something in the disposition?
// TODO: wait until action has manifested for agent, how dis this defined? Fixed time threshold, next link enter event, arrival at next station, ...
// while (true) {
// engine.doSimStep(time++);
// }
// gather new tree observation for agent
return disposition.getTreeObservation(agent);
}
public static void main(String[] args) {
// TODO: load the scenario
Scenario scenario;
RLEnvironment env = new RLEnvironment(scenario);
// step through the simulation
env.step(Id.createPersonId("a1"));
}
}And track all the train positions in the disposition: package ch.sbb.matsim.contrib.railsim.rl;
import ch.sbb.matsim.contrib.railsim.qsimengine.TrainPosition;
import ch.sbb.matsim.contrib.railsim.qsimengine.disposition.DispositionResponse;
import ch.sbb.matsim.contrib.railsim.qsimengine.disposition.TrainDisposition;
import ch.sbb.matsim.contrib.railsim.qsimengine.resources.RailLink;
import ch.sbb.matsim.contrib.railsim.qsimengine.resources.RailResourceManager;
import org.matsim.api.core.v01.Id;
import org.matsim.api.core.v01.network.Link;
import org.matsim.api.core.v01.network.Network;
import org.matsim.api.core.v01.population.Person;
import org.matsim.core.mobsim.framework.MobsimDriverAgent;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
public class RLDisposition implements TrainDisposition {
private final Network network;
private final RailResourceManager resourceManager;
// maybe more is needed here...
private final Map<Id<Person>, TrainPosition> trainPositions = new HashMap<>();
private final Map<Id<Link>, TrainPosition> trainPositionsToLink = new HashMap<>();
// note: ignores the case of multiple trains on the same link...
public RLDisposition(Network network, RailResourceManager resourceManager) {
this.network = network;
this.resourceManager = resourceManager;
}
@Override
public void onDeparture(double time, MobsimDriverAgent driver, List<RailLink> route) {
// TODO: Mark trains as active, store initial route if needed
}
@Override
public DispositionResponse requestNextSegment(double time, TrainPosition position, double dist) {
// track all trains in the network
trainPositions.put(position.getDriver().getId(), position);
trainPositionsToLink.put(position.getHeadLink(), position);
// create tree observation for current agent
TreeObservation observation = getTreeObservation(position.getDriver().getId());
// TODO: gRPC call to python with tree observation
// Then return the disposition response from python
return null;
}
@Override
public void unblockRailLink(double time, MobsimDriverAgent driver, RailLink link) {
// TODO: Release resource in resource manager
}
public TreeObservation getTreeObservation(Id<Person> agent) {
TrainPosition currentAgentPosition = trainPositions.get(agent);
// TODO: Generate the tree observations here:
// Follow the network from the current node
// lookup in the trainPositionsToLink map if there are any trains on the link
// create TreeObservation object and return it
return null;
}
}The approach is for sure not complete, since I am a bit confused now if we use gRPC or directly Python to control Java... |
|
@rsjeffers Can you give us write permissions on the fork, then we don't have to copy the code into the comments? Thanks 😄 |
There are some still some issues we must solve I guess on the railsim side. The tracked information in the disposition is always the trainState at the moment the disposition was called and not at the time step when the ObservationTree is calculated... I think for the start we can work with this and look for a solution to get the states from the engine from all trains at a specific time step when we have a running RL setup? A possible solution could be to catch the RailsimTrainsStateEvents in the event handler (interface to be implemented: BasicEventHandler) which is added to the event manager and set the updateInterval in the railsim config to one second:
Note: These are events, which are written to the MATSim output events file and not internal events of the railsim engine in the update queue!
|
|
Hi @munterfi you should already have write access, is it working now? |
|
@munterfi I think your proposal looks like what I suggested. The disposition is called each time a train needs to make a request, which is fairly often. So it should have a very up-to-date state. Rarely there might be position updates where the disposition may not be involved. You also have to keep in mind the design of the simulation. A train state is updated when necessary and not at a fixed time step. So there might be state update for t=0s and t=10s. If another train requests the disposition t=5s, the exact position of the first train will simply be not available because it has never been calculated. Enforcing state updates of all other trains may be possible, but I'm not sure if that is a good idea. In reality, you would also have to rely on the last reported position of a train, and we have a fairly low update interval (< 10s). To summarize, I would suggest using the last known position of the other trains instead of enforcing updates. You may also try to play around with reducing the update interval. Another note: If you need functionality to keep track of train states, I would rather directly implement in the rail sim engine and resource manager instead of creating an event listener. Event listener maybe possible, but it is more difficult to use and implement. The resource manager is already available to all disposition and may be used to query the state. Alternatively, you may create a dedicated class for keeping track of the states. |
|
@rakow, thanks for your feedback! |
fixed bug in observation tree in getNextNodes method. Added the code to integrate RLDisposition with Railsim
merge guice to development
PR to ask questions