Merge branch 'develop' into base_remaining_notes

CONTRIBUTING.md

@@ -1,13 +1,13 @@
 # How to contribute
 
-Third-party contributions are highly encouraged for PEcAn and will grow the code as well as the understanding of PEcAn and its applications.  The core development team can not add all models that exist to PEcAn or all possible scenarios and analysis that people want to conduct.  Our goal is to keep it as easy as possible for you contribute changes that get things working in your environment. 
+Third-party contributions are highly encouraged for PEcAn and will grow the code as well as the understanding of PEcAn and its applications.  The core development team can not add all models that exist to PEcAn or all possible scenarios and analysis that people want to conduct.  Our goal is to keep it as easy as possible for you contribute changes that get things working in your environment.
 There are a few guidelines that we need contributors to follow so that we can have a chance of keeping on top of things.
 
 ## PEcAn CORE vs Models vs Modules
 
 New functionality is typically directed toward modules to provide a slimmer PEcAn Core, reducing the requirements to get PEcAn running on different platforms, especially HPC machines, and to allow greater freedom for modules and models.
 
-Generally, new model should be added to the models folder and new modules should be added to the modules folder. 
+Generally, new model should be added to the models folder and new modules should be added to the modules folder.
 Exceptions include code that is reused in many models or modules and wrapper functions that call specific implementations in models; these can be placed in the core packages.
 
 If you are unsure of whether your contribution should be implemented as a model, module or part of PEcAn Core, you may visit [Chat Room](https://join.slack.com/t/pecanproject/shared_invite/enQtMzkyODUyMjQyNTgzLWEzOTM1ZjhmYWUxNzYwYzkxMWVlODAyZWQwYjliYzA0MDA0MjE4YmMyOTFhMjYyMjYzN2FjODE4N2Y4YWFhZmQ) or ask on the pecan-develop mailing list for advice.
@@ -16,9 +16,9 @@ If you are unsure of whether your contribution should be implemented as a model,
 
 - Make sure you have a GitHub account.
 - Search GitHub and Google to see if your issue has already been reported
-        - Create an issue in GitHub, assuming one does not already exist.
-	- Clearly describe the issue including steps to reproduce when it is a bug.
-	- Make sure you fill in the earliest version that you know has the issue.
+  - Create an issue in GitHub, assuming one does not already exist.
+  - Clearly describe the issue including steps to reproduce when it is a bug.
+  - Make sure you fill in the earliest version that you know has the issue.
 - Ask @dlebauer, @mdietze or @robkooper to add you to the PEcAn project if you plan on fixing the issue.
 
 ## Getting Started
@@ -33,17 +33,20 @@ At this point you will have a copy of PEcAn and you are almost ready to work on
 At this point you will have a copy of the pecan repo in your personal space. Next steps are to setup your local copy to work with the forked version.
 
 Introduce your self to GIT (if you have not done this yet), make sure you use an email associated with your GitHub account.
+
 ```bash
 git config --global user.name "John Doe"
 git config --global user.email [email protected]
 ```
 
 Switch pecan to your fork
+
 ```bash
 git remote set-url origin https://github.com/<your username>/pecan.git
 ```
 
 Setup pecan to be able to fetch from the master/develop
+
 ```bash
 git remote add upstream https://github.com/PecanProject/pecan.git
 ```
@@ -66,30 +69,30 @@ Here is a simplified workflow on how add a new feature:
 
 Update your develop (both locally and on GitHub)
 
-```
+```bash
 git fetch upstream
 git checkout develop
 git merge upstream/develop
 git push
 ```
 
-### Create a branch to do your work.
+### Create a branch to do your work
 
-A good practice is to call the branch in the form of GH-<issue-number> followed by the title of the issue. This makes it easier to find out the issue you are trying to solve and helps us to understand what is done in the branch. Calling a branch my-work is confusing. Names of branch can not have a space, and should be replaced with a hyphen.
+A good practice is to call the branch in the form of `GH-<issue-number>` followed by the title of the issue. This makes it easier to find out the issue you are trying to solve and helps us to understand what is done in the branch. Calling a branch my-work is confusing. Names of branch can not have a space, and should be replaced with a hyphen.
 
-```
+```bash
 git checkout -b GH-issuenumber-title-of-issue
 ```
 
 ### Work and commit
 
-Do you work, and commit as you see fit.Make your commit messages helpful. 
+Do you work, and commit as you see fit.Make your commit messages helpful.
 
-### Push your changes up to GitHub.
+### Push your changes up to GitHub
 
 If this is the first time pushing to GitHub you will need to extended command, other wise you can simply do a `git push`.
 
-```
+```bash
 git push -u origin GH-issuenumber-title-of-issue
 ```
 

DEBUGING.md

@@ -1,3 +1,5 @@
+# DEBUGGING.MD
+
 Adding the following to the workflow.R or to your .Rprofile will enable
 printing of a stacktrace in case something goes wrong. This will help
 with the development of PEcAn.

DEV-INTRO.md

@@ -34,10 +34,12 @@ The use of Docker in PEcAn is described in detail in the [PEcAn documentation](h
 ### Installing Docker
 
 To install Docker and docker-compose, see the docker documentation:
+
 - Docker Desktop in [MacOS](https://docs.docker.com/docker-for-mac/install/) or [Windows](https://docs.docker.com/docker-for-windows/install/)
 - Docker (e.g. [Ubuntu](https://docs.docker.com/compose/install/)) and [docker-compose](https://docs.docker.com/compose/install/) on your linux operating system.
 
 _Note for Linux users:_ add your user to the docker group. This will prevent you from having to use `sudo` to start the docker containers, and makes sure that any file that is written to a mounted volume is owned by you. This can be done using
+
 ```sh
 # for linux users
 sudo adduser ${USER} docker.
@@ -51,13 +53,13 @@ By default docker-compose will use the files `docker-compose.yml` and `docker-co
 
 For Linux/MacOSX
 
-```
+```sh
 cp docker-compose.dev.yml docker-compose.override.yml
 ```
 
 For Windows
 
-```
+```sh
 copy docker-compose.dev.yml docker-compose.override.yml
 ```
 
@@ -67,12 +69,12 @@ You can now use the command `docker-compose` to work with the containers setup f
 
 The steps in this section only need to be done the first time you start working with the stack in docker. After this is done you can skip these steps. You can find more detail about the docker commands in the [pecan documentation](https://pecanproject.github.io/pecan-documentation/master/docker-index.html).
 
-* setup .env file
-* create folders to hold the data
-* load the postgresql database
-* load some test data
-* copy all R packages (optional but recommended)
-* setup for web folder development (optional)
+- setup .env file
+- create folders to hold the data
+- load the postgresql database
+- load some test data
+- copy all R packages (optional but recommended)
+- setup for web folder development (optional)
 
 #### .env file
 
@@ -86,18 +88,18 @@ cp docker/env.example .env
 
 For Windows
 
-```
+```sh
 copy docker/env.example .env
 ```
 
-* `COMPOSE_PROJECT_NAME` set this to pecan, the prefix for all containers
-* `PECAN_VERSION` set this to develop, the docker image we start with
+- `COMPOSE_PROJECT_NAME` set this to pecan, the prefix for all containers
+- `PECAN_VERSION` set this to develop, the docker image we start with
 
 Both of these variables should also be uncommented by removing the # preceding them.  At the end you should see the following if you run the following command  `egrep -v '^(#|$)' .env`. If you have a windows system, you will need to set the variable PWD as well, and for linux you will need to set UID and GID (for rstudio).
 
 For Linux
 
-```
+```sh
 echo "COMPOSE_PROJECT_NAME=pecan" >> .env
 echo "PECAN_VERSION=develop" >> .env
 echo "UID=$(id -u)" >> .env
@@ -106,22 +108,22 @@ echo "GID=$(id -g)" >> .env
 
 For MacOSX
 
-```
+```sh
 echo "COMPOSE_PROJECT_NAME=pecan" >> .env
 echo "PECAN_VERSION=develop" >> .env
 ```
 
 For Windows:
 
-```
+```sh
 echo "COMPOSE_PROJECT_NAME=pecan" >> .env
 echo "PECAN_VERSION=develop" >> .env
 echo "PWD=%CD%" >> .env
 ```
 
 Once you have setup `docker-compose.override.yml` and the `.env` files, it is time to pull all docker images that will be used. Doing this will make sure you have the latest version of those images on your local system.
 
-```
+```sh
 docker-compose pull
 ```
 
@@ -131,11 +133,10 @@ The goal of the development is to share the development folder with your contain
 
 If you have uncommented the volumes in `docker-compose.override.yml` you will need to create the folders. Assuming you have not modified the values, you can do this with:
 
-```
+```sh
 mkdir -p $HOME/volumes/pecan/{lib,pecan,portainer,postgres,rabbitmq,traefik}
 ```
 
-
 The following volumes are specified:
 
 - **pecan_home** : is the checked out folder of PEcAn. This is shared with the executor and rstudio container allowing you to share and compile PEcAn. (defaults to current folder)
@@ -153,23 +154,22 @@ These folders will hold all the persistent data for each of the respective conta
 
 First we bring up postgresql (we will start RabbitMQ as well since it takes some time to start):
 
-```
+```sh
 docker-compose up -d postgres rabbitmq
 ```
 
 This will start postgresql and rabbitmq. We need to wait for a few minutes (you can look at the logs using `docker-compose logs postgres`) to see if it is ready.
 
 Once the database has finished starting up we will initialize the database. Now you can load the database using the following commands. The first command will make sure we have the latest version of the image, the second command will actually load the information into the database.
 
-```
+```sh
 docker pull pecan/db
 docker run --rm --network pecan_pecan pecan/db
 ```
 
-
 Once that is done we create two users for BETY,  first user is the guest user that you can use to login in the BETY interface. The second user is a user with admin rights.
 
-```
+```sh
 docker-compose run --rm bety user guestuser guestuser "Guest User" [email protected] 4 4
 docker-compose run --rm bety user carya illinois "Carya Demo User" [email protected] 1 1
 ```
@@ -178,7 +178,7 @@ docker-compose run --rm bety user carya illinois "Carya Demo User" carya@example
 
 Once the database is loaded we can add some example data, some of the example runs and runs for the ED model, assume some of this data is available. This can take some time, but all the data needed will be copied to the `/data` folder in the pecan containers. As with the database we first pull the latest version of the image, and then execute the image to copy all the data:
 
-```
+```sh
 docker pull pecan/data:develop
 docker run -ti --rm --network pecan_pecan --volume pecan_pecan:/data --env FQDN=docker pecan/data:develop
 ```
@@ -206,26 +206,26 @@ docker run -ti --rm -v pecan_lib:/rlib pecan/base:develop cp -a /usr/local/lib/R
 
 If you want to use the web interface, you will need to:
 
-1. Uncomment the web section from the `docker-compose.override.yml` file. This section includes three lines at the top of the file, just under the `services` section. Uncomment the lines that start `web:`, `  volumes:`, and `- pecan_web:`.
+1. Uncomment the web section from the `docker-compose.override.yml` file. This section includes three lines at the top of the file, just under the `services` section. Uncomment the lines that start `web:`, `volumes:`, and `- pecan_web:`.
 2. Then copy the config.php from the docker/web folder. You can do this using
 
 For Linux/MacOSX
 
-```
+```sh
 cp docker/web/config.docker.php web/config.php
 ```
 
 For Windows
 
-```
+```sh
 copy docker\web\config.docker.php web\config.php
 ```
 
-### PEcAn Development
+## PEcAn Development
 
 To begin development we first have to bring up the full PEcAn stack. This assumes you have done once the steps above. You don\'t need to stop any running containers, you can use the following command to start all containers. At this point you have PEcAn running in docker.
 
-```
+```sh
 docker-compose up -d
 ```
 
@@ -241,7 +241,7 @@ To compile the PEcAn code you can use the make command in either the rstudio con
 
 You can submit your workflow either in the executor container or in rstudio container. For example to run the `docker.sipnet.xml` workflow located in the tests folder you can use:
 
-```
+```sh
 docker-compose exec executor bash
 # inside the container
 cd /pecan/tests
@@ -252,9 +252,9 @@ A better way of doing this is developed as part of GSOC, in which case you can l
 
 # PEcAn URLs
 
-You can check the RabbitMQ server used by pecan using https://rabbitmq.pecan.localhost on the same server that the docker stack is running on. You can use rstudio either with http://server/rstudio or at http://rstudio.pecan.localhost. To check the traefik dashboard you can use http://traefik.pecan.localhost.
+You can check the RabbitMQ server used by pecan using <https://rabbitmq.pecan.localhost> on the same server that the docker stack is running on. You can use rstudio either with <http://server/rstudio> or at <http://rstudio.pecan.localhost>. To check the traefik dashboard you can use <http://traefik.pecan.localhost>.
 
-If the stack is running on a remote machine, you can use ssh and port forwarding to connect to the server. For example `ssh -L 8000:localhost:80` will allow you to use http://rabbitmq.pecan.localhost:8000/ in your browser to connect to the remote PEcAn server RabbitMQ.
+If the stack is running on a remote machine, you can use ssh and port forwarding to connect to the server. For example `ssh -L 8000:localhost:80` will allow you to use <http://rabbitmq.pecan.localhost:8000/> in your browser to connect to the remote PEcAn server RabbitMQ.
 
 # Directory Structure
 
@@ -298,7 +298,7 @@ Small scripts that are used as part of the development and installation of PEcAn
 
 If you want to start from scratch and remove all old data, but keep your pecan checked out folder, you can remove the folders where you have written the data (see `folders` below). You will also need to remove any of the docker managed volumes. To see all volumes you can do `docker volume ls -q -f name=pecan`. If you are sure, you can either remove them one by one, or remove them all at once using the command below. **THIS DESTROYS ALL DATA IN DOCKER MANAGED VOLUMES.**.
 
-```
+```sh
 docker volume rm $(docker volume ls -q -f name=pecan)
 ```
 
@@ -308,7 +308,7 @@ If you changed the docker-compose.override.yml file to point to a location on di
 
 If you want to reset the pecan lib folder that is mounted across all machines, for example when there is a new version of PEcAn or a a new version of R, you will need to delete the volume pecan_lib, and repopulate it. To delete the volume use the following command, and then look at "copy R packages" to copy the data again.
 
-```
+```sh
 docker-compose down
 docker volume rm pecan_lib
 ```
@@ -323,27 +323,27 @@ This will leverage of NFS to mount the file system in your local docker image, c
 
 First install nfs server:
 
-```
+```sh
 apt-get install nfs-kernel-server
 ```
 
 Next export your home directory:
 
-```
+```sh
 echo -e "$PWD\t127.0.0.1(rw,no_subtree_check,all_squash,anonuid=$(id -u),anongid=$(id -g))" | sudo tee -a /etc/exports
 ```
 
 And export the filesystem.
 
-```
+```sh
 sudo exportfs -va
 ```
 
 At this point you have exported your home directory, only to your local machine. All files written to that exported filesystem will be owned by you (`id -u`) and your primary group (`id -g`).
 
 Finally we can modify the `docker-compose.override.yml` file to allow for writing files to your PEcAn folder as you:
 
-```
+```sh
 volumes:
   pecan_home:
     driver_opts: