Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Headless mode awaits for requested user feedback without showing any text for what that feedback should be #5015

Open
1 task done
ryx2 opened this issue Nov 15, 2024 · 8 comments · May be fixed by #5246
Open
1 task done

[Bug]: Headless mode awaits for requested user feedback without showing any text for what that feedback should be #5015

ryx2 opened this issue Nov 15, 2024 · 8 comments · May be fixed by #5246
Labels
bug Something isn't working fix-me-experimental

Comments

@ryx2
Copy link
Contributor

ryx2 commented Nov 15, 2024 •
edited
Loading

Is there an existing issue for the same bug?

  • I have checked the existing issues.

Describe the bug and reproduction steps

Command to run:

docker run -it \
    --pull=always \
    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.13-nikolaik \
    -e SANDBOX_USER_ID=$(id -u) \
    -e WORKSPACE_MOUNT_PATH=$(pwd) \
    -e LLM_API_KEY=*** \
    -e LLM_MODEL="anthropic/claude-3-5-sonnet-20241022" \
    -v $WORKSPACE_BASE:/opt/workspace_base \
    -v /var/run/docker.sock:/var/run/docker.sock \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app-$(date +%Y%m%d%H%M%S) \
    docker.all-hands.dev/all-hands-ai/openhands:0.13 \
    python -m openhands.core.main -t "the realtor_tour_scenario.test.ts is erroring, could you investigate? I think the erroring one is due to the message coming from the wrong POV" --debug true

output (happens twice):

0.13: Pulling from all-hands-ai/openhands
Digest: sha256:28307e6ef3ca477df56e0689c0fcaa8b6f073d018cf90c945317b5b76c8566cb
Status: Image is up to date for docker.all-hands.dev/all-hands-ai/openhands:0.13
Starting OpenHands...
Setting up enduser with id 501
Docker socket group id: 0
root:x:0:
Group with id 0 already exists
Running as enduser
03:16:36 - openhands:INFO: eventstream_runtime.py:220 - [runtime default-2a338e0f97031b24] Starting runtime with image: docker.all-hands.dev/all-hands-ai/runtime:0.13-nikolaik
03:16:36 - openhands:INFO: eventstream_runtime.py:224 - [runtime default-2a338e0f97031b24] Container started: openhands-runtime-default-2a338e0f97031b24
03:16:36 - openhands:INFO: eventstream_runtime.py:227 - [runtime default-2a338e0f97031b24] Waiting for client to become ready at http://host.docker.internal:38456...
03:16:54 - openhands:INFO: eventstream_runtime.py:233 - [runtime default-2a338e0f97031b24] Runtime is ready.
03:16:54 - openhands:WARNING: state.py:119 - Could not restore state from session: sessions/default-2a338e0f97031b24/agent_state.pkl
03:16:54 - openhands:INFO: agent_controller.py:193 - [Agent Controller default] Starting step loop...
03:16:54 - openhands:INFO: agent_controller.py:316 - [Agent Controller default] Setting agent(CodeActAgent) state from AgentState.LOADING to AgentState.RUNNING


==============
[Agent Controller default] LEVEL 0 LOCAL STEP 0 GLOBAL STEP 0



==============
[Agent Controller default] LEVEL 0 LOCAL STEP 1 GLOBAL STEP 1



==============
[Agent Controller default] LEVEL 0 LOCAL STEP 2 GLOBAL STEP 2

03:17:08 - openhands:INFO: agent_controller.py:316 - [Agent Controller default] Setting agent(CodeActAgent) state from AgentState.RUNNING to AgentState.AWAITING_USER_INPUT
Request user input >> the realtor_tour_scenario.test.ts is erroring, could you investigate? I think the erroring one is due to the message coming from the wrong POV, meaning the test is either using the wrong assistant, perhaps because it's not getting the right result for isRealtor                                                      
04:01:21 - openhands:INFO: agent_controller.py:316 - [Agent Controller default] Setting agent(CodeActAgent) state from AgentState.AWAITING_USER_INPUT to AgentState.RUNNING


==============
[Agent Controller default] LEVEL 0 LOCAL STEP 3 GLOBAL STEP 3

04:01:25 - openhands:INFO: agent_controller.py:316 - [Agent Controller default] Setting agent(CodeActAgent) state from AgentState.RUNNING to AgentState.AWAITING_USER_INPUT
Request user input >> ^C
Session terminated, killing shell...^C ...killed.
^C%

Unsure exactly what it's awaiting request user input? it doesn't say what type of user input it wants

OpenHands Installation

Docker command in README

OpenHands Version

0.13

Operating System

None

Logs, Errors, Screenshots, and Additional Context

Executed this in the root of my typescript repo
@ryx2 ryx2 added the bug Something isn't working label Nov 15, 2024
@ryx2 ryx2 mentioned this issue Nov 15, 2024
Open
1 task
@enyst
Copy link
Collaborator

enyst commented Nov 15, 2024

Please use the LOG_ALL_EVENTS env var added by PR 4729 in the docker command (and dev setup too), I think it is now in the README too. Recently added, to solve this lack of information on local run.

@mamoodi
Copy link
Collaborator

mamoodi commented Nov 15, 2024

@rbren part of this issue was brought up before where the headless run now acts like running CLI mode.

@ZimaBlue307
Copy link

Hi, I encountered similar situation where Openhands asking for user input in headless mode. I'm also curious why the headless run now acts like running CLI mode. Thanks for your time.

@mamoodi
Copy link
Collaborator

mamoodi commented Nov 18, 2024

This is a bug that should be fixed.

@rbren
Copy link
Collaborator

rbren commented Nov 18, 2024

@xingyaoww probably knows how we do this in evaluation, to make sure the agent never requests user input

@xingyaoww
Copy link
Collaborator

In evaluation, we actually pass in a user_response_fn to supply a prompt to tell the agent not to ask user.

OpenHands/evaluation/swe_bench/run_infer.py

Lines 410 to 412 in a87b859

fake_user_response_fn=AGENT_CLS_TO_FAKE_USER_RESPONSE_FN[
metadata.agent_class
],

Basically it will tell the agent this:

OpenHands/evaluation/bird/run_infer.py

Lines 40 to 45 in a87b859

def codeact_user_response(state: State) -> str:
msg = (
'Please continue working on the task on whatever approach you think is suitable.\n'
'If you think you have completed the SQL, please finish the interaction using the "finish" tool.\n'
'IMPORTANT: YOU SHOULD NEVER ASK FOR HUMAN HELP OR USE THE INTERNET TO SOLVE THIS TASK.\n'
)

@github-actions GitHub Actions
Copy link
Contributor

OpenHands started fixing the issue! You can monitor the progress here.

@github-actions GitHub Actions
Copy link
Contributor

An attempt was made to automatically fix this issue, but it was unsuccessful. A branch named 'openhands-fix-issue-5015' has been created with the attempted changes. You can view the branch here. Manual intervention may be required.

This was referenced Nov 25, 2024
Draft
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working fix-me-experimental
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix issue #5015: [Bug]: Headless mode awaits for requested user feedb… All-Hands-AI/OpenHands
6 participants
@enyst @rbren @ryx2 @xingyaoww @ZimaBlue307 @mamoodi