Glob patterns for merging a lot of files

[kaeff]

When using the CLI, the number of files is currently limited by the maximum argument list length of the shell. Merging a large number of files currently fails:

$ geojson-merge *.json
zsh: argument list too long: geojson-merge

geojson-merge could support a large number of input files using glob. Would this feature be of interest?

[tmcw]

The syntax would have to be geojson-merge "*.json" for this, I expect? This seems like a workaround of limited usefulness because people will default to using *.json. Maybe we could support geojson-merge ./directory and note that if the input is a directory name, to readdirSync on it and merge all the found geojson files?

[morandd]

I encountered the same problem trying to merge lots of files. Here's a new front-end script which uses readDir():

You can save this file as "geojson-merge-dir.js" and run it per the Example Usage:

Update - this fails on my case (merging polys for all 35,000 municipalities in France) due to javascript's max string length. It seems I will need to learn the dark magic of Stream Processing. [https://github.com/nodejs/node/issues/35973]

#!/usr/bin/env node

// Example usage:
//   node geojson-merge-dir.js path/to/my/geojson/files/  > combined.geojson

var geojsonMerge = require('geojson-merge'),
    fs = require('fs');

  let dir = process.argv[process.argv.length-1];
  if (!(fs.existsSync(dir) && fs.lstatSync(dir).isDirectory())){
    console.error('Call this with one argument, which must be a directory');
    process.exit();
  }

  if (!dir.endsWith("/")) dir += "/";

  let files = fs.readdirSync(dir).filter(fn => fn.toLowerCase().endsWith(".geojson"));

  process.stdout.write(JSON.stringify(geojsonMerge(files.map(function(fn) {
      return JSON.parse(fs.readFileSync(dir + fn));
  })), null, 2));