Brainharmonic - Generating music from brain signal data using deep learning

Title

Brainharmonic - Generating music from brain signal data using deep learning

Leaders

Mariano Cabezas

Collaborators

Aria Nguyen Brendan Harris

Brainhack Global 2022 Event

Brainhack Australasia

Project Description

This project aims to develop a tool to generate music from brain signals using deep learning models. There has been a lot of work on generating new music from a large collection of music using deep AI models, as well as some work on generating music from brain EEG/fMRI signal through algorithmic or rule based approaches. However there has not been work done on using deep learning models to generate music from EEG/fMRI signal.

In this project, we will use deep generative models to allow EEG/fMRI data to be used as an input to generate music. More details of the design can be found on our Github page.

Link to project repository/sources

https://github.com/marianocabezas/brainharmonic

Goals for Brainhack Global

Develop a tool to generate songs from EGG/fMRI brain signals using deep learning

Good first issues

Currently we have a pipeline to generate musical motif from a time series and a deep learning model to output a song from the generated motif. This was tested with a list of EEG signals from multiple subjects. We can build upon the existing work in three aspects

1. Issue one: Develop a pipeline to process time series signals (fMRI, EEG, etc) and feed the output to the music generation part
2. Issue two Refine current DL to resolve some existing issues and produce better music
3. Issue three Process signals from different brain regions as different instruments to create multitrack songs.

Communication channels

https://mattermost.brainhack.org/brainhack/channels/brainharmonic

Skills

Python: intermediate Pytorch: beginner EEG/fMRI data processing: beginner

Onboarding documentation

https://github.com/marianocabezas/brainharmonic

What will participants learn?

EEG/fRMI data preprocessing Deep learning generative models Existing music-to-music deep AI frameworks

Data to use

No response

Number of collaborators

3

Credit to collaborators

Project contributors are credited on Readme file on project Github page

Image

Leave this text if you don’t have an image yet.

Type

coding_methods, method_development, pipeline_development

Development status

1_basic structure

Topic

deep_learning

Tools

fMRIPrep, Freesurfer, Jupyter

Programming language

Julia, Python

Modalities

EEG, fMRI

Git skills

1_commit_push

Anything else?

No response

Things to do after the project is submitted and ready to review.

• Add a comment below the main post of your issue saying: Hi @brainhackorg/project-monitors my project is ready!
• Twitter-sized summary of your project pitch.