Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat - custom code evaluation in UI #589

Closed
wants to merge 90 commits into from

Conversation

aybruhm
Copy link
Member

@aybruhm aybruhm commented Sep 12, 2023

Description

This PR addresses #569 and introduces the following:

Workflow Summary:

  • Users can now easily create and manage custom evaluations.
  • The front end provides a user-friendly interface for inputting code and managing evaluations.
  • The backend efficiently handles the storage and execution of evaluations.
  • The system ensures that users can execute evaluations and view results seamlessly.

Additional Notes:

Making API call(s):

If you plan to make an API call from the code to your local machine, please use Ngrok to tunnel it. Making an API call directly to your local machine from the custom evaluation code will not work due to the security restrictions in place.

Related Issue

Closes #569

@aybruhm aybruhm requested a review from mmabrouk September 12, 2023 22:16
Copy link
Collaborator

@aakrem aakrem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Getting this error.
    https://github.com/Agenta-AI/agenta/assets/6608260/b1220883-9137-4d42-a3cb-3f46991abea0

  2. @mmabrouk The button to create a custom evaluation is distracting the flow of the other evaluations.
    My suggestions are:

  3. Add it as a first item of the dropdown like:
    [List Custom evaluations]
    |_ Create a new custom evaluation (This is a button to redirect to the new evaluation form)
    |_ The items of list..

  4. Have it as "Text Button" under the last dropdown.

@aybruhm
Copy link
Member Author

aybruhm commented Sep 13, 2023

  1. Getting this error.
    https://github.com/Agenta-AI/agenta/assets/6608260/b1220883-9137-4d42-a3cb-3f46991abea0
  2. @mmabrouk The button to create a custom evaluation is distracting the flow of the other evaluations.
    My suggestions are:
  3. Add it as a first item of the dropdown like:
    [List Custom evaluations]
    |_ Create a new custom evaluation (This is a button to redirect to the new evaluation form)
    |_ The items of list..
  4. Have it as "Text Button" under the last dropdown.

The error has been fixed.

Copy link
Collaborator

@aakrem aakrem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aybruhm There are bugs.

  1. Scores are not saved.
  2. When we run it again we get errors
    For theses two bugs see this video
Screen.Recording.2023-09-13.at.17.55.23.mov

@MohammedMaaz

  1. In this new evaluation we can't see the code we added.
    Please propose a solution the see the code. (editing the code after running the evaluation is not needed)
  2. The Button to create a new custom evaluation is distracting the flow to run evaluations.
    Try to put it within the dropdown or as an underlined text button under the dropdown
  3. In the evaluations list we see NaN
  4. Minor but it looks like the the dropdown to list the evaluations is a different from the dropdown group. when we select another dropdown and then the custom evaluation, both got selected. Check this screenshot:
Screenshot 2023-09-13 at 17 50 49
Screenshot 2023-09-13 at 17 53 10

@MohammedMaaz
Copy link
Contributor

MohammedMaaz commented Sep 15, 2023

Closed in favor of #610

@MohammedMaaz
Copy link
Contributor

@aybruhm There are bugs.

  1. Scores are not saved.
  2. When we run it again we get errors
    For theses two bugs see this video

Screen.Recording.2023-09-13.at.17.55.23.mov

@MohammedMaaz

  1. In this new evaluation we can't see the code we added.
    Please propose a solution the see the code. (editing the code after running the evaluation is not needed)
  2. The Button to create a new custom evaluation is distracting the flow to run evaluations.
    Try to put it within the dropdown or as an underlined text button under the dropdown
  3. In the evaluations list we see NaN
  4. Minor but it looks like the the dropdown to list the evaluations is a different from the dropdown group. when we select another dropdown and then the custom evaluation, both got selected. Check this screenshot:
Screenshot 2023-09-13 at 17 50 49
Screenshot 2023-09-13 at 17 53 10

All Fixed in #610

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Custom Code Evaluation via UI
4 participants