Title
Phoneme Boundaries
Leaders
Noah R. Fram (@fram-noah)
Collaborators
Miriam Lense
Warren Jones
Camila Alviar
Project description
Many linguistic stimuli and samples are analyzed based on phonemes, or the smallest units of speech that contain meaning. One high-level form of this process involves labeling boundaries of vowels, sounds generated by filtering sound from the larynx without blocking the airflow, and distinguishing them from consonants, which do entail blocking airflow at some point. Human coders use a combination of auditory features, like the organization of vowel formants, and visual features, like mouth shape, to determine vowel boundaries. Characteristics of vowels including their timing are particularly important when analyzing child-directed speech and child-directed song. However, many naturalistic recordings of child-directed communication have occluded or degraded data in either the visual or auditory domain, making it challenging to code vowel boundaries.
This project aims to use predictive coding networks, a neuromorphic form of deep machine learning based on probabilistic inference, to resolve this issue. We will design a multimodal predictive coding network to predict vowel boundaries based on both auditory features and visual characteristics, train it on synchronized audiovisual recordings, and test how well it handles degradation or removal of auditory or visual signals. This will enable behavioral scientists to code a much wider range of ecologically valid child-directed communication signals, and will allow for more precise and reproducible characterization of audiovisual stimuli representing such signals.
Link to project repository/sources
https://github.com/fram-noah/vowel-xmodal-pcn
Concerete goals with specific tasks for Brainhack Vanderbilt 2026
- Learn about predictive coding network models, particularly in the
omni-pcn implementation.
- Build an initial model architecture for handling time-varying audiovisual information.
- Investigate windowing procedures for parsing this kind of data.
Good first issues
- Issue one: Install and start experimenting with the
pcn package.
- Issue three: Research potential audio processing approaches to prepare the dataset for use in
pcn.
Skills
- Python: intermediate to advanced, including familiarity with handling audiovisual data. We will be using crossvalidation techniques to train predictive coding network models.
- Signal processing: beginner to intermediate. We will likely be using windowing and mel spectrograms for the audio portion.
Onboarding documentation
See project GitHub repo.
What will participants learn?
Participants will learn:
- Predictive coding network models
- Differences between child-directed and adult-directed verbal communication
- Basic audio signal processing
Public data to use
For BrainHack, we will not be using any data, since we'll be focusing on learning the methodology. The model will be trained with a set of audiovisual recordings including human annotations of vowel boundaries, which cannot be posted publicly due to licensing considerations.
Number of collaborators
3
Credit to collaborators
Project collaborators will be included as authors in the publication of this coding tool.
Image
Project Summary
Use multimodal predictive coding networks to enable detection of vowel boundaries, even when data from one sensory modality is degrade or occluded
Type
method_development
Development status
0_concept_no_content
Topic
machine_learning
Tools
other
Programming language
Python
Modalities
behavioral
Git skills
2_branches_PRs
Anything else?
We will be using the Omni-PCN Python package developed by Nicole Dumont: https://github.com/nsdumont/omni-pcn.
Things to do after the project is submitted and ready to review.
Title
Phoneme Boundaries
Leaders
Noah R. Fram (@fram-noah)
Collaborators
Miriam Lense
Warren Jones
Camila Alviar
Project description
Many linguistic stimuli and samples are analyzed based on phonemes, or the smallest units of speech that contain meaning. One high-level form of this process involves labeling boundaries of vowels, sounds generated by filtering sound from the larynx without blocking the airflow, and distinguishing them from consonants, which do entail blocking airflow at some point. Human coders use a combination of auditory features, like the organization of vowel formants, and visual features, like mouth shape, to determine vowel boundaries. Characteristics of vowels including their timing are particularly important when analyzing child-directed speech and child-directed song. However, many naturalistic recordings of child-directed communication have occluded or degraded data in either the visual or auditory domain, making it challenging to code vowel boundaries.
This project aims to use predictive coding networks, a neuromorphic form of deep machine learning based on probabilistic inference, to resolve this issue. We will design a multimodal predictive coding network to predict vowel boundaries based on both auditory features and visual characteristics, train it on synchronized audiovisual recordings, and test how well it handles degradation or removal of auditory or visual signals. This will enable behavioral scientists to code a much wider range of ecologically valid child-directed communication signals, and will allow for more precise and reproducible characterization of audiovisual stimuli representing such signals.
Link to project repository/sources
https://github.com/fram-noah/vowel-xmodal-pcn
Concerete goals with specific tasks for Brainhack Vanderbilt 2026
omni-pcnimplementation.Good first issues
pcnpackage.pcn.Skills
Onboarding documentation
See project GitHub repo.
What will participants learn?
Participants will learn:
Public data to use
For BrainHack, we will not be using any data, since we'll be focusing on learning the methodology. The model will be trained with a set of audiovisual recordings including human annotations of vowel boundaries, which cannot be posted publicly due to licensing considerations.
Number of collaborators
3
Credit to collaborators
Project collaborators will be included as authors in the publication of this coding tool.
Image
Project Summary
Use multimodal predictive coding networks to enable detection of vowel boundaries, even when data from one sensory modality is degrade or occluded
Type
method_development
Development status
0_concept_no_content
Topic
machine_learning
Tools
other
Programming language
Python
Modalities
behavioral
Git skills
2_branches_PRs
Anything else?
We will be using the Omni-PCN Python package developed by Nicole Dumont: https://github.com/nsdumont/omni-pcn.
Things to do after the project is submitted and ready to review.
Hi @brainhack-vandy/project-monitors my project is ready!