Musical ideation through Generative AI

[walterbender]

Ticket Contents

Description

We would like to deploy generative AI in support of musical composition. The basic idea is to use AI to generate possible modifications of a phrase (or phrases) generated by the user. In other words, the user would start the composition and the AI would then present different possibilities for enriching the composition to which the user would react and further enhance.

Specifically, we would be working toward accomplishing the following:

Tune open source LLM to create the ideal output for beginners, and for learning
Create an API that translates LLM musical data to Music Blocks data, so that the learner has something to work from
Create a in-app framework for evaluating the AI output, so that the user has several plausible choices and, if they so choose, grade the choices to be used for further improvement

Goals & Mid-Point Milestone

Goals

• [model development]
• [backend deployment]
• [UX design]
• [Frontend implementation]
• [Goals Achieved By Mid-point Milestone]
• [model development]
• [backend deployment]

Setup/Installation

No response

Expected Outcome

A "button" added to Music Blocks that invokes an LLM to provide:

• phrasing suggestions
• orchestration suggestions

The suggestions would be in the form of written text and/or code blocks.

Acceptance Criteria

The mechanics of the interaction are the primary requirement. The quality of the suggestions can be improved over time.

Implementation Details

A dockerized server would run on one of the Sugar servers. That server would interface to one or more LLMs.

The messaging between Music Blocks and the server would be in the form of code snippets (individual phrases) and multiple phrases.

The interface would prompt the user to either submit a phrase or multiple phrases.

Mockups/Wireframes

No response

Product Name

Music Blocks

Organisation Name

Sugar Labs

Domain

⁠Education

Tech Skills Needed

Docker, JavaScript, Machine Learning, Python

Mentor(s)

@pikurasa @Wat

Category

Backend, Frontend

[rishabhknowss]

Is this an additional feature to an existing project, or is it a separate project that will be integrated with Music Blocks? Regarding the LLM component, will it be implemented as an API from a model provider like Hugging Face, or something else ?

[akshayw1]

Intrested to work on this, Have an idea to implement, the Hugging face,or third party API, that could facilate the same @pikurasa Is these issue open?

[arth-1]

Hello, I have started to work on adding this feature, but this is going to be a mid sized fix so i would require some help, at first i would like to know which llm will the organisation plan to use as i am not sure about the legalities of wether any can be used comertially for such a software and second regarding that we would need to tune the open source llms and train them according to the requirements of music blocks. and what is the framework in the backend which this would rely on? and we would need to Build APIs to enable communication between the LLM, Music Blocks client, and server. also Should the orchestration suggestions include instrument choices, dynamics, or tempo changes? there are many such questions i would like to discuss upon this feature in detail for the same.