Fix incorrect handling of brk syscall when shrinking the heap

Fix incorrect handling of brk syscall when shrinking the heap #13556