• Note:

ML-for-Creative-Coding/03-segmentation at main · shiffman/ML-for-Creative-Coding

Body Segmentation and Visualization with ml5.js and BodyPix

I'm using ml5.js and its encapsulation of BodyPix to implement body segmentation in video streams and to visualize specific parts (e.g. chest).

录屏2025-02-21 10.25.39.mov

bodySegmentation by Taigen2457 -p5.js Web Editor

1.Introducing ml5.js and setting the BodyPix parameter

let bodySegmentation;
let video;
let segmentation;

let options = {
  maskType: "parts", // 使用 BodyPix 进行部分分割
};

Here, we declare the bodySegmentation variable and specify maskType as "parts," indicating that we want to segment the human body into multiple parts.

2. Preloading models

function preload() {
  bodySegmentation = ml5.bodySegmentation("BodyPix", options);
}

The preload() function loads the BodyPix model before initializing p5.js.

3. Initialize the video stream and start detection

function setup() {
  createCanvas(640, 480);
  video = createCapture(VIDEO);
  video.size(640, 480);
  video.hide();

  bodySegmentation.detectStart(video, gotResults);
}

In the setup() function, we create a 640x480 canvas and start the video stream using createCapture(VIDEO).

detectStart(video, gotResults); initiates real-time detection, with gotResults() handling the results.

4. Processing Detection Results and Visualization

function draw() {
  background(255);
  image(video, 0, 0);

  if (segmentation) {
    // Draw segmentation mask
    image(segmentation.mask, 0, 0, width, height);
    
    // Retrieve segmentation data
    let parts = bodySegmentation.getPartsId();
    let gridSize = 10; // Grid size
    
    // Iterate over video pixels and draw grid
    for (let x = 0; x < video.width; x += gridSize) {
      for (let y = 0; y < video.height; y += gridSize) {
        // Check if the grid belongs to the torso (TORSO_FRONT)
        if (segmentation.data[y * video.width + x] == parts.TORSO_FRONT) {
          fill(255, 0, 0); // Red color
          noStroke();
          // Draw red circles
          circle(x, y, gridSize);
        }
      }
    }
  }
}