Error merging two large geojson files

[amirmes]

Seems someone else has this issue as well, but nobody has answered it. I am using -s (stream) and it still has a JavaScript heap out of memory error.

$ geojson-merge -s Roads_1.geojson Roads_2.geojson> combined.geojson

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [/usr/local/bin/node]
2: node::FatalTryCatch::~FatalTryCatch() [/usr/local/bin/node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [/usr/local/bin/node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
5: v8::internal::Factory::NewTransitionArray(int) [/usr/local/bin/node]
6: v8::internal::TransitionsAccessor::Insert(v8::internal::Handlev8::internal::Name, v8::internal::Handlev8::internal::Map, v8::internal::SimpleTransitionFlag) [/usr/local/bin/node]
7: v8::internal::Map::ConnectTransition(v8::internal::Handlev8::internal::Map, v8::internal::Handlev8::internal::Map, v8::internal::Handlev8::internal::Name, v8::internal::SimpleTransitionFlag) [/usr/local/bin/node]
8: v8::internal::Map::CopyReplaceDescriptors(v8::internal::Handlev8::internal::Map, v8::internal::Handlev8::internal::DescriptorArray, v8::internal::Handlev8::internal::LayoutDescriptor, v8::internal::TransitionFlag, v8::internal::MaybeHandlev8::internal::Name, char const*, v8::internal::SimpleTransitionFlag) [/usr/local/bin/node]
9: v8::internal::Map::CopyAddDescriptor(v8::internal::Handlev8::internal::Map, v8::internal::Descriptor*, v8::internal::TransitionFlag) [/usr/local/bin/node]
10: v8::internal::Map::CopyWithField(v8::internal::Handlev8::internal::Map, v8::internal::Handlev8::internal::Name, v8::internal::Handlev8::internal::FieldType, v8::internal::PropertyAttributes, v8::internal::PropertyConstness, v8::internal::Representation, v8::internal::TransitionFlag) [/usr/local/bin/node]
11: v8::internal::Map::TransitionToDataProperty(v8::internal::Handlev8::internal::Map, v8::internal::Handlev8::internal::Name, v8::internal::Handlev8::internal::Object, v8::internal::PropertyAttributes, v8::internal::PropertyConstness, v8::internal::Object::StoreFromKeyed, bool*) [/usr/local/bin/node]
12: v8::internal::LookupIterator::PrepareTransitionToDataProperty(v8::internal::Handlev8::internal::JSReceiver, v8::internal::Handlev8::internal::Object, v8::internal::PropertyAttributes, v8::internal::Object::StoreFromKeyed) [/usr/local/bin/node]
13: v8::internal::StoreIC::LookupForWrite(v8::internal::LookupIterator*, v8::internal::Handlev8::internal::Object, v8::internal::Object::StoreFromKeyed) [/usr/local/bin/node]
14: v8::internal::StoreIC::UpdateCaches(v8::internal::LookupIterator*, v8::internal::Handlev8::internal::Object, v8::internal::Object::StoreFromKeyed, v8::internal::MaybeHandlev8::internal::Object) [/usr/local/bin/node]
15: v8::internal::StoreIC::Store(v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Name, v8::internal::Handlev8::internal::Object, v8::internal::Object::StoreFromKeyed) [/usr/local/bin/node]
16: v8::internal::KeyedStoreIC::Store(v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
17: v8::internal::Runtime_KeyedStoreIC_Miss(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
18: 0x3a4a79a8427d
Abort trap: 6

[carlos-mg89]

In the first line of your error log you get the clue to solve the merge.

"FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory" this means that you need to increase the heap memory that JavaScript is using. Here the solution:

https://futurestud.io/tutorials/node-js-increase-the-memory-limit-for-your-process

[anphanngocvn]

@carlos-mg89 this doesn't help me. I am having this error:

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

[carlos-mg89]

@AnnPhan have you increased the JavaScript heap limit as explained in the link I shared? You are having the same issue, "heap limit Allocation failed".

[anphanngocvn]

@carlos-mg89 It works for me now. Thank you!

[carlos-mg89]

@AnnPhan great! If you could like / upvote my first comment, it would definitely help other people with this matter. As it seems it's the way to deal with the JavaScript heap limit errors.

Perhaps @tmcw you want to close this issue as well.

[Koesters]

Could someone point me to what file, or how I need to use "node --max-old-space-size=" on?

node --max-old-space-size=8192 "/usr/local/bin/geojson-merge"
usage: geojson-merge [-s] FILE FILE2 FILE3

  --stream (or -s): use streaming mode for large files

node --max-old-space-size=8192 /usr/local/bin/geojson-merge
usage: geojson-merge [-s] FILE FILE2 FILE3

  --stream (or -s): use streaming mode for large files

cd /usr/local/bin/
/usr/local/bin$ node --max-old-space-size=8192 geojson-merge
usage: geojson-merge [-s] FILE FILE2 FILE3

  --stream (or -s): use streaming mode for large files

sudo node --max-old-space-size=8192 ./geojson-merge
[sudo] password for user: 
usage: geojson-merge [-s] FILE FILE2 FILE3

  --stream (or -s): use streaming mode for large files