io.popen results in yielded Lua thread

[carsakiller]

OS: Ubuntu container
BeamMP-Server Version: v3.4.1

Describe the bug
Previously, I have used io.popen for running some other tools. For example, I have used it as a hack to call date to get the current time in UNIX ms. It used to work just fine, reporting the current time pretty quickly. However, I have noticed that it now seems to result in the Lua thread yielding, and it no longer responds to events. Even weirder though is if I then enter the interactive Lua mode in the server console, all I need to do is press Enter and the server resumes my Lua again — until hitting io.popen again. Even weirder is that if I just run io.popen in the interactive Lua console, it executes perfectly fine, as expected.

I noticed this a few weeks ago, perhaps e2d7721 is responsible, but I haven't had the time to build the server myself and see — just want to get this reported before it gets lost. I want to say that v3.2.2 was probably the last version I can remember io.popen working in.

To Reproduce
Steps to reproduce the behavior:

1. Create a Lua plugin that calls io.popen("date"). (any program should work)
2. Notice that the server will now no longer respond to events.
3. Enter interactive Lua mode in the server console.
4. Press Enter
5. Notice that the server will resume your plugin, executing all queued events.

Expected behavior
I would expect io.popen to not yield my Lua plugin until something is written to the interactive Lua console. I presume this is an issue with stdin.

Logs
I think I did run the server in debug mode when I first noticed this and there was nothing of use, but I'll try getting a fresh container and plugin going and getting a new log.

[carsakiller]

Ok, so I ran a little test with this code:

function onSomething()
    print("START onVehicleEdited!!")
	local file, err = io.popen("date", "r")
	if err then
		print(err)
	end

	if file then
		print(file:read("a"))
	end
    print("END onVehicleEdited!!")
end

MP.RegisterEvent("onVehicleEdited", "onSomething")

And I was able to reproduce the problem by doing the following (and without even using the interactive Lua mode):

1. Press Sync in vehicle edit menu to force a onVehicleEdited event (red underline)
2. Press Enter in console 1st time (blue underline) (I presume to unblock io.popen)
3. Press Enter 2nd time (green underline) (I presume to unblock file:read)
4. My onSomething handler finally finishes, printing the date (yellow highlight)
   

[carsakiller]

I just tried again and was able to unblock by just pressing Enter once.

[lionkor]

#if defined(LUA_USE_POSIX)	/* { */

#define l_popen(L,c,m)		(fflush(NULL), popen(c,m))
#define l_pclose(L,file)	(pclose(file))

in Lua's liolib.c lines 57..60 on 5.3 do this from what I can tell.
Specifically, fflush(NULL) according to the man page, calling it like that is 1) blocking and 2) flushes all output streams. We mess with the stdout (which is an output stream), so I'm guessing(!) that this is the problem.

The solution is to make our own equivalent to io.popen.

[carsakiller]

So, if I understand correctly, you are saying io.popen is attempting to flush stdout and since commandline alters stdout by creating its own line-buffered event loop(?), we end up blocking while waiting for commandline to write to and release it?

Wouldn't this be a problem with stdin though, as execution is only resumed when Enter is pressed on the command line? Could it be that commandline is blocking stdin while trying to read in keyboard inputs and only when Enter is pressed, it is freed, allowing Lua to continue? I guess it could also be waiting for stdout to free while commandline writes to it, though…

I'm not that familiar with what is going on here, but would setting stdin to not block with O_NONBLOCK have negative consequences? Alternatively, maybe putting things in separate threads would be necessary to prevent them from blocking each other, although I thought the whole point of commandline is to be non-blocking?

Regardless, I hope there is an easy fix. Being able to use io.popen allows me to do many things not yet supported by BeamMP like making HTTP requests.