Merge pull request #10 from SpaceNetChallenge/spacenetV2

Spacenet V2
SpaceNetChallenge · Mar 21, 2017 · 21dbe6e · 21dbe6e
2 parents 57b0127 + 78d1661
commit 21dbe6e
Show file tree

Hide file tree

Showing 23 changed files with 3,783 additions and 533 deletions.
diff --git a/.idea/dictionaries/dlindenbaum.xml b/.idea/dictionaries/dlindenbaum.xml
diff --git a/.idea/vcs.xml b/.idea/vcs.xml
diff --git a/README.md b/README.md
@@ -1,7 +1,7 @@
 # SpaceNet Utilities
 
-This repository has two python packages, geoTools and evalTools. The geoTools packages is intended to assist in the preprocessing of [SpaceNet](https://aws.amazon.com/public-data-sets/spacenet/) satellite imagery data corpus to a format that is consumable by machine learning algorithms. The evalTools package is used to evaluate the effectiveness of object detection algorithms using ground truth.
-
+This repository has two python packages, geoTools and evalTools. The geoTools packages is intended to assist in the preprocessing of [SpaceNet](https://spacenetchallenge.github.io/) satellite imagery data corpus hosted on [SpaceNet on AWS](https://aws.amazon.com/public-datasets/spacenet/) to a format that is consumable by machine learning algorithms. The evalTools package is used to evaluate the effectiveness of object detection algorithms using ground truth.
+This is version 2.0 and has been updated with more capabilities to 
 ## Download Instructions
 
 Further download instructions for the [SpaceNet Dataset](https://github.com/SpaceNetChallenge/utilities/tree/master/content/download_instructions) can be found [here](https://github.com/SpaceNetChallenge/utilities/tree/master/content/download_instructions)
@@ -32,6 +32,74 @@ Hints:
 * The images provided could contain anywhere from zero to multiple buildings.
 * All proposed polygons should be legitimate (they should have an area, they should have points that at least make a triangle instead of a point or a line, etc).
 * Use the [metric implementation code](https://github.com/SpaceNetChallenge/utilities/blob/master/python/evaluateScene.py) to self evaluate.
+To run the metric you can use the following command.
+```
+python python/evaluateScene.py /path/to/SpaceNetTruthFile.csv \
+                               /path/to/SpaceNetProposalFile.csv \
+                               --resultsOutputFile /path/to/SpaceNetResults.csv
+```
+
+## Data Transformation Code
+
+To make the Spacenet dataset easier to use we have created a tool createDataSpaceNet.py
+This tool currently supports the creation of datasets with annotation to support 3 Formats
+1. [PASCAL VOC2012](http://host.robots.ox.ac.uk/pascal/VOC/)
+2. [Darknet](https://pjreddie.com/darknet/yolo/)
+3. [Segmenation Boundaries Dataset (SBD)](http://home.bharathh.info/pubs/codes/SBD/download.html)
+
+It will create the appropriate annotation files and a summary trainval.txt and test.txt in the outputDirectory
+
+### Create an PASCAL VOC2012 Compatiable Dataset
+The final product will have image dimensions of 420 pixels
+```
+python python/createDataSpaceNet.py /path/to/spacenet_sample/AOI_2_Vegas_Train/ \
+           RGB-PanSharpen \
+           --outputDirectory /path/to/spacenet_sample/annotations/ \
+           --annotationType PASCALVOC2012 \
+           --imgSizePix 400
+
+```
+### Changing the raster format
+Some GIS Images have 16-bit pixel values which openCV has trouble with.  createDataSpaceNet.py can convert the 16bit GeoTiff to an 8bit GeoTiff or 8bit JPEG 
+
+To create the 8bit GeoTiff
+```
+python python/createDataSpaceNet.py /path/to/spacenet_sample/AOI_2_Vegas_Train/ \
+           RGB-PanSharpen \
+           --outputDirectory /path/to/spacenet_sample/annotations/ \
+           --annotationType PASCALVOC2012 \
+           --convertTo8Bit \
+           --outputFileType GTiff \
+           --imgSizePix 400
+    
+```
+
+To create the 8bit JPEG
+```
+python python/createDataSpaceNet.py /path/to/spacenet_sample/AOI_2_Vegas_Train/ \
+           RGB-PanSharpen \
+           --outputDirectory /path/to/spacenet_sample/annotations/ \
+           --annotationType PASCALVOC2012 \
+           --convertTo8Bit \
+           --outputFileType JPEG \
+           --imgSizePix 400
+
+```
+
+For more Features
+```
+python python/createDataSpaceNet.py -h
+
+```
+
+
+
+## Use our Docker Container
+We have created two Docker files at /docker/standalone/cpu and /docker/standalone/gpu
+These Dockerfiles will build a docker container with all packages neccessary to run the package
+
+More documenation to follow
+
 
 ## Dependencies
 All dependencies can be found in [requirements.txt](./python/requirements.txt)

diff --git a/content/download_instructions/README.md b/content/download_instructions/README.md
@@ -1,67 +1,118 @@
-# SpaceNet Utilities
+## Hosting
+[SpaceNet](https://aws.amazon.com/public-datasets/spacenet/) is a corpus of commercial satellite imagery and labeled training data to use for machine learning research. The dataset is currently hosted as an [Amazon Web Services (AWS) Public Dataset](https://aws.amazon.com/public-datasets/).
 
-Instruction for the download of [SpaceNet](https://aws.amazon.com/public-data-sets/spacenet/) satellite imagery data corpus to a format that is consumable by machine learning algorithms.
-
-
+## Catalog
+1. Area of Interest 1 (AOI 1) - Location: Rio de Janeiro. 50cm imagery collected from DigitalGlobe’s [WorldView-2 satellite](http://satimagingcorp.s3.amazonaws.com/site/pdf/WorldView-2_datasheet.pdf). The dataset includes building footprints and 8-band multispectral data.
+2. Area of Interest 2 (AOI 2) - Location: Vegas. 30cm imagery collected from DigitalGlobe’s [WorldView-3 satellite](https://www.spaceimagingme.com/downloads/sensors/datasheets/DG_WorldView3_DS_2014.pdf). The dataset includes building footprints and 8-band multispectral data.
+3. Area of Interest 3 (AOI 3) - Location: Paris. 30cm imagery collected from DigitalGlobe’s [WorldView-3 satellite](https://www.spaceimagingme.com/downloads/sensors/datasheets/DG_WorldView3_DS_2014.pdf). The dataset includes building footprints and 8-band multispectral data.
+4. Area of Interest 4 (AOI 4) - Location: Shanghai. 30cm imagery collected from DigitalGlobe’s [WorldView-3 satellite](https://www.spaceimagingme.com/downloads/sensors/datasheets/DG_WorldView3_DS_2014.pdf). The dataset includes building footprints and 8-band multispectral data.
+5. Area of Interest 5 (AOI 5) - Location: Khartoum. 30cm imagery collected from DigitalGlobe’s [WorldView-3 satellite](https://www.spaceimagingme.com/downloads/sensors/datasheets/DG_WorldView3_DS_2014.pdf). The dataset includes building footprints and 8-band multispectral data.
+6. Point of Interest (POI) Dataset- Location: Rio de Janeiro. The dataset includes POIs.
 
 ## Dependencies
-Required that AWS CLI is installed and that an active AWS account with credit card is associated with the aws cli
+The [AWS Command Line Interface (CLI)](https://aws.amazon.com/cli/) must be installed with an active AWS account. Configure the AWS CLI using 'aws configure'
 
-Configure the AWS CLI using aws configure
 
-SpaceNet AWS Structure
+## SpaceNet Simple Storage Service (S3) Directory Structure (AOI 1)
 ```
 s3://spacenet-dataset/
 -- AOI_1_Rio
     |-- processedData
-    |   -- processedBuildingLabels.tar.gz  # Compressed 3band and 8band 200m x 200m tiles with associated building foot print labels
-                                           # This dataset is the Training Dataset for the first [Top Coder Competition](https://community.topcoder.com/longcontest/?module=ViewProblemStatement&rd=16835&pm=14439)
+    |   -- processedBuildingLabels.tar.gz  # Compressed 3band and 8band 200m x 200m tiles with associated building foot print labels                                 # This dataset is the Training Dataset for the first Top Coder Competition
     `-- srcData
         |-- rasterData
-        |   |-- 3-Band.tar.gz # 3band (RGB) Raster Mosaic for Rio De Jenairo area (2784 sq KM) collected by [WorldView-2](http://satimagingcorp.s3.amazonaws.com/site/pdf/WorldView-2_datasheet.pdf)
-        |    -- 8-Band.tar.gz # 8band Raster Mosaic for Rio De Jenairo area (2784 sq KM) collected by [WorldView-2](http://satimagingcorp.s3.amazonaws.com/site/pdf/WorldView-2_datasheet.pdf)
+        |   |-- 3-Band.tar.gz # 3band (RGB) Raster Mosaic for Rio De Jenairo area (2784 sq KM) collected by WorldView-2
+        |    -- 8-Band.tar.gz # 8band Raster Mosaic for Rio De Jenairo area (2784 sq KM) collected by WorldView-2
+         -- vectorData
+            |-- Rio_BuildingLabels.tar.gz # Source Dataset that contains Building the building foot prints traced from the Mosaic
+            |-- Rio_HGIS_Metro.gdb.tar.gz  # Source Point of Interest Dataset in GeoDatabase Format.  Best if Used with ESRI
+             -- Rio_HGIS_Metro_extract.tar # Source Point of Interest Dataset in GeoJSON with associated .jpg.  Easy to Use without ESRI toolset
+-- AOI_1_Rio
+    |-- processedData
+    |   -- processedBuildingLabels.tar.gz  # Compressed 3band and 8band 200m x 200m tiles with associated building foot print labels                                 # This dataset is the Training Dataset for the first Top Coder Competition
+    `-- srcData
+        |-- rasterData
+        |   |-- 3-Band.tar.gz # 3band (RGB) Raster Mosaic for Rio De Jenairo area (2784 sq KM) collected by WorldView-2
+        |    -- 8-Band.tar.gz # 8band Raster Mosaic for Rio De Jenairo area (2784 sq KM) collected by WorldView-2
+         -- vectorData
+            |-- Rio_BuildingLabels.tar.gz # Source Dataset that contains Building the building foot prints traced from the Mosaic
+            |-- Rio_HGIS_Metro.gdb.tar.gz # Source Point of Interest Dataset in GeoDatabase Format.  Best if Used with ESRI
+             -- Rio_HGIS_Metro_extract.tar # Source Point of Interest Dataset in GeoJSON with associated .jpg.  Easy to Use without ESRI toolset
+-- AOI_1_Rio
+    |-- processedData
+    |   -- processedBuildingLabels.tar.gz  # Compressed 3band and 8band 200m x 200m tiles with associated building foot print labels                                 # This dataset is the Training Dataset for the first Top Coder Competition
+    `-- srcData
+        |-- rasterData
+        |   |-- 3-Band.tar.gz # 3band (RGB) Raster Mosaic for Rio De Jenairo area (2784 sq KM) collected by WorldView-2
+        |    -- 8-Band.tar.gz # 8band Raster Mosaic for Rio De Jenairo area (2784 sq KM) collected by WorldView-2
          -- vectorData
             |-- Rio_BuildingLabels.tar.gz # Source Dataset that contains Building the building foot prints traced from the Mosaic
             |-- Rio_HGIS_Metro.gdb.tar.gz # Source Point of Interest Dataset in GeoDatabase Format.  Best if Used with ESRI
              -- Rio_HGIS_Metro_extract.tar # Source Point of Interest Dataset in GeoJSON with associated .jpg.  Easy to Use without ESRI toolset
 ```
 
-#To download the Imagery
-
-##To download processed 200mx200m Tiles with associated building foot prints for building foot print extraction tests do the following
+## SpaceNet Simple Storage Service (S3) Directory Structure (AOI 2-5)
 ```
-## Warning this file is 3.4 GB
-aws s3api get-object --bucket spacenet-dataset --key AOI_1_Rio/processedData/processedBuildingLabels.tar.gz --request-payer requester processedBuildingLabels.tar.gz
+├── AOI_[Num]_[City]_Train
+│   ├── geojson
+│   │   └── buildings  # Contains GeoJson labels of buildings for each tile
+│   ├── MUL            # Contains Tiles of 8-Band Multi-Spectral raster data from WorldView-3
+│   ├── MUL-PanSharpen # Contains Tiles of 8-Band Multi-Spectral raster data pansharpened to 0.3m
+│   ├── PAN            # Contains Tiles of Panchromatic raster data from Worldview-3
+│   ├── RGB-PanSharpen # Contains Tiles of RGB raster data from Worldview-3
+│   └── summaryData    # Contains CSV with pixel based labels for each building in the Tile Set.
 ```
 
+## Download instructions
 
-##To download the Source Imagery Mosaic
+### AOI 1 - Rio de Janeiro
+To download processed 200mx200m tiles of AOI 1 (3.4 GB) with associated building footprints do the following:
+```
+aws s3api get-object --bucket spacenet-dataset --key AOI_1_Rio/processedData/processedBuildingLabels.tar.gz --request-payer requester processedBuildingLabels.tar.gz
+```
+To download the Source Imagery Mosaic (3-band = 2.3 GB and 8-band = 6.5 GB):
 ```
-## Warning this file is 2.3 GB
 aws s3api get-object --bucket spacenet-dataset --key AOI_1_Rio/srcData/rasterData/3-Band.tar.gz --request-payer requester 3-Band.tar.gz
-## Warning this file is 6.5 GB
 aws s3api get-object --bucket spacenet-dataset --key AOI_1_Rio/srcData/rasterData/8-Band.tar.gz --request-payer requester 8-Band.tar.gz
 ```
-
-##To download the Source Vector Data for the Building Extraction Challenge
+To download the Source Vector Data (0.18 GB):
 ```
-## Warning this file is 0.18 GB
 aws s3api get-object --bucket spacenet-dataset --key AOI_1_Rio/srcData/vectorData/Rio_BuildingLabels.tar.gz --request-payer requester Rio_BuildingLabels.tar.gz
+```
 
+### AOI 2 - Vegas
+To download processed 200mx200m tiles of AOI 2 (23 GB) with associated building footprints do the following:
+```
+aws s3api get-object --bucket spacenet-dataset --key AOI_2_Vegas/AOI_2_Vegas_Train.tar.gz --request-payer requester AOI_2_Vegas_Train.tar.gz
 ```
 
-##To download the Rio Point of Interest Dataset in ESRI GeoDatabase Form
+### AOI 3 - Paris
+To download processed 200mx200m tiles of AOI 3 (5 GB) with associated building footprints do the following:
+```
+## Warning this file is 5 GB
+aws s3api get-object --bucket spacenet-dataset --key AOI_3_Paris/AOI_3_Paris_Train.tar.gz --request-payer requester AOI_3_Paris_Train.tar.gz
 ```
-## Warning this file is 31 GB
-aws s3api get-object --bucket spacenet-dataset --key AOI_1_Rio/srcData/vectorData/Rio_HGIS_Metro.gdb.tar.gz --request-payer requester Rio_HGIS_Metro.gdb.tar.gz
 
+### AOI 4 - Shanghai
+To download processed 200mx200m tiles of AOI 4 (23 GB) with associated building footprints do the following:
+```
+aws s3api get-object --bucket spacenet-dataset --key AOI_4_Shanghai/AOI_4_Shanghai_Train.tar.gz --request-payer requester AOI_4_Shanghai_Train.tar.gz
 ```
 
-##To download the Rio Point of Interest Dataset Extracted into GeoJSONs with associated .jpg
+### AOI 5 - Khartoum
+To download processed 200mx200m tiles of AOI 5 (4 GB) with associated building footprints do the following:
+```
+aws s3api get-object --bucket spacenet-dataset --key AOI_5_Khartoum/AOI_5_Khartoum_Train.tar.gz --request-payer requester AOI_5_Khartoum_Train.tar.gz
+```
+
+### Point of Interest Dataset in ESRI GeoDatabase Form (31 GB)
+```
+aws s3api get-object --bucket spacenet-dataset --key AOI_1_Rio/srcData/vectorData/Rio_HGIS_Metro.gdb.tar.gz --request-payer requester Rio_HGIS_Metro.gdb.tar.gz
 ```
-## Warning this file is 29 GB
-aws s3api get-object --bucket spacenet-dataset --key AOI_1_Rio/srcData/vectorData/Rio_HGIS_Metro_extract.tar --request-payer requester Rio_HGIS_Metro_extract.tar
 
+### Point of Interest Dataset Extracted into GeoJSONs with associated .jpg (29 GB)
+```
+aws s3api get-object --bucket spacenet-dataset --key AOI_1_Rio/srcData/vectorData/Rio_HGIS_Metro_extract.tar --request-payer requester Rio_HGIS_Metro_extract.tar
 ```
 
 

diff --git a/docker/Dockerfile b/docker/Dockerfile
@@ -0,0 +1,93 @@
+LABEL maintainer dlindenbaum
+
+
+## List of Python packages needed
+###dataTools
+#import numpy as np
+#from osgeo import ogr, gdal, osr
+#import cv2
+###evalTools
+###geoTools
+#from osgeo import gdal, osr, ogr
+#    import rtree
+#from osgeo import gdal, osr, ogr, gdalnumeric
+#from PIL import Image
+
+## Install External Libraries for Pillow
+#apt-get install
+#libjpeg
+#zlib
+#libtiff
+#libfreetype
+#littlecms
+#libwebp
+#openjpeg
+#
+### install Python openCV
+#libopencv-dev \
+#python-opencv \
+#python-numpy \
+#python-pip \
+#python-setuptools \
+#gdal-bin \
+#python-gdal
+
+## Install General Requirements
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        apt-utils \
+        build-essential \
+        cmake \
+        git \
+        wget \
+        vim \
+        python-dev \
+        python-pip \
+        python-setuptools
+RUN pip install --upgrade pip
+
+## Install Basics for Python
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        python-numpy \
+        python-scipy
+
+## Install Essentials for Pillow
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        libjpeg-dev \
+        zlib1g \
+        libtiff5-dev \
+        libfreetype6-dev \
+        libwebp-dev \
+        libopenjpeg-dev
+
+RUN pip install Pillow
+
+## Install GDAL Requirments
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        gdal-bin \
+        python-gdal
+
+## Instal OpenCV Requirements
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        libopencv-dev \
+        python-opencv
+
+## Install RTRee
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        libspatialindex-dev
+
+RUN pip install rtree
+
+ENV GIT_BASE=/opt/
+WORKDIR $GIT_BASE
+
+# Download spaceNetUtilities
+# FIXME: use ARG instead of ENV once DockerHub supports this
+ENV CLONE_TAG=spacenetV2
+RUN git clone -b ${CLONE_TAG} --depth 1 https://github.com/SpaceNetChallenge/utilities.git
+
+ENV PYUTILS_ROOT $GIT_BASE/utilities/python
+ENV PYTHONPATH $PYUTILS_ROOT:$PYTHONPATH
+
+WORKDIR /workspace
+
+