Sentinel AWS Fetcher

Warning: Super rough outline below!

To do the following, this will require working knowledge of command line tools, makefiles, and no fear of AWS.

To start, you'll need to set your machine up to download files from AWS. This guide here is pretty fantastic for that.

First, we need to suss out which region we want imagery for. The cloudless sentinel imagery is stored in chunks according to the following grid:

For example, for Ireland we only need two:

AOI=29U 30U

Next, we iterate over that folder and make aws requests...

fetch:
	rm -r imagery
	mkdir imagery
	for a in $(AOI); do \
		mkdir -p imagery/$$a; \
	done
	for a in $(AOI); do \
		aws \
			s3 cp s3://sentinel-s2-l2a-mosaic-120/2019/6/10/$$a \
			/Users/moriartyd/graphics/0-selfish/album-maps/talking-heads/imagery/$$a \
			--recursive; \
	done

Data comes in raw bands. This will scale the raw values so we can actually merge em' together

scaleRawBands:
	for f in $(AOI); do \
		gdal_translate \
			-of GTiff \
			-ot Byte \
			-scale 0 1024 0 80 \
			imagery/$$f/B02.tif imagery/$$f/bit-B02.tif; \
		gdal_translate \
			-of GTiff \
			-ot Byte \
			-scale 0 1024 0 80 \
			imagery/$$f/B03.tif imagery/$$f/bit-B03.tif; \
		gdal_translate \
			-of GTiff \
			-ot Byte \
			-scale 0 1024 0 80 \
			imagery/$$f/B04.tif imagery/$$f/bit-B04.tif; \
	done

Below is the messiest part of this. It converts those individual bands into RGB GeoTiffs we can work with, which is straightforward, and then also add in an alpha channel, which is not. It works, but definitely needs to be cleaned up.

bandsToRGB:
	mkdir -p imagery/rgb
	mkdir -p imagery/proj
	mkdir -p imagery/projfile
	mkdir -p imagery/alpha
	mkdir -p imagery/alphaF
	for a in $(AOI); do \
		gdal_merge.py \
			-co "PHOTOMETRIC=RGB" \
			-seperate imagery/$$a/bit-B04.tif imagery/$$a/bit-B03.tif imagery/$$a/bit-B02.tif \
			-o imagery/rgb/$$a.tif; \
		gdalwarp imagery/rgb/$$a.tif imagery/proj/$$a-proj.tif -t_srs "+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs"; \
		gdalwarp -co "TFW=YES" imagery/proj/$$a-proj.tif imagery/projfile/$$a; \
		rm imagery/projfile/$$a; \
		gdal_calc.py -A imagery/proj/$$a-proj.tif \
			--A_band=1 \
			--outfile="imagery/alphaF/$$a-proj.tif" \
			--calc="255" \
			--type=Byte; \
		gdal_edit.py -unsetnodata imagery/alphaF/$$a-proj.tif; \
		gdal_merge.py imagery/proj/$$a-proj.tif imagery/alphaF/$$a-proj.tif -o imagery/alpha/$$a-proj.tif -separate; \
	done

ImageMagick to remove all pure black and pure white values from the images. From here, you can re-add the TFW projection file and open them in QGIS for merging.

magic:
	mkdir -p imagery/magickw
	mkdir -p imagery/magickb
	mkdir -p imagery/fixed
	for a in $(AOI); do \
		convert imagery/proj/$$a-proj.tif -transparent black imagery/magickb/$$a.tif; \
	done