Notebooks

Please read Python Module document first!

Notebooks

Example notebooks are located at python/notebook.

• history_analysis.ipynb

We consider the original notebooks templates. So, we highly recommend making copies of them for your own tests and future analysis. Do not commit changes to the notebook. If you find bugs, please let Chang-Goo ([email protected]) know. If you create a notebook useful for general purposes, please also let us know.

When you run notebooks from another directory, you may need to set up the path to pyathena appropriately. There are three (maybe more) ways to set things up:

• Adding the path to PYTHONPATH in your .bashrc file:

  export PYTHONPATH="/your-path-to-Athena-TIGRESS/python"
  

• Adding the path using sys module (in the beginning of notebooks):

  import sys
  sys.path.insert(0,"/your-path-to-Athena-TIGRESS/python")
  

• Using a startup file located at ~/.ipython/profile_default/startup/. An example startup can be found in my repo.

Installing jupyter

You may look at the official documentation, or simply type

> conda install jupyter

SSH tunneling (port fowarding)

Often you will want to analyze data that is on a remote machine (server) where the simulation was run, without moving data to a local machine. To do this, you can run jupyter on the remote machine while piping the notebook output to your local machine. The pyathena modules as well as the notebook you want to run must be on the server.

• First, connect to the server and run jupyter notebook (you may want to run this using screen)

  > ssh username@servername
  > jupyter notebook --no-browser --port=hostport
  

  For example:

  > ssh [email protected]
  > jupyter notebook --no-browser --port=8100
  

• Second, set up your browser on your local machine to listen to output from the remote machine. In the browser search window (e.g., Firefox, Chrome, Safari), type:

  localhost:localport
  

  For example

  localhost:8110
  

• Third, execute the following ssh command on your local machine to forward from hostport to your localport. This tells the remote server that information output to hostport by processes (here, jupyter) should be piped by ssh to localport on your local machine.

  > ssh -N -f -L localport:localhost:hostport username@servername
  

  For example:

  > ssh -N -f -L 8110:localhost:8100 [email protected]
  

NOTE: You may need to install your own python package (using conda; see Python Module) in order to access jupyter notebook running on the PiCSiE machines with ssh tunneling.

To understand what all the options mean, use man ssh, which gives:

     -f      Requests ssh to go to background just before command execution.  This is use-
             ful if ssh is going to ask for passwords or passphrases, but the user wants
             it in the background.  This implies -n.  The recommended way to start X11
             programs at a remote site is with something like ssh -f host xterm.

             If the ExitOnForwardFailure configuration option is set to ``yes'', then a
             client started with -f will wait for all remote port forwards to be success-
             fully established before placing itself in the background.

     -L [bind_address:]port:host:hostport
     -L [bind_address:]port:remote_socket
     -L local_socket:host:hostport
     -L local_socket:remote_socket
             Specifies that connections to the given TCP port or Unix socket on the local
             (client) host are to be forwarded to the given host and port, or Unix socket,
             on the remote side.  This works by allocating a socket to listen to either a
             TCP port on the local side, optionally bound to the specified bind_address,
             or to a Unix socket.  Whenever a connection is made to the local port or
             socket, the connection is forwarded over the secure channel, and a connection
             is made to either host port hostport, or the Unix socket remote_socket, from
             the remote machine.

             Port forwardings can also be specified in the configuration file.  Only the
             superuser can forward privileged ports.  IPv6 addresses can be specified by
             enclosing the address in square brackets.

             By default, the local port is bound in accordance with the GatewayPorts set-
             ting.  However, an explicit bind_address may be used to bind the connection
             to a specific address.  The bind_address of ``localhost'' indicates that the
             listening port be bound for local use only, while an empty address or `*'
             indicates that the port should be available from all interfaces.

     -N      Do not execute a remote command.  This is useful for just forwarding ports.

Configure jupyter for remote access

There are several tweaks you may need to use jupyter notebook. You may want to look at the jupyter document.

Especially, you may wish to consult the section Running a public notebook server.