Abstract

3D reconstruction could improve colonoscopy by estimating mucosal coverage and alerting clinicians to missed regions during screening. However, algorithm development is limited as no current datasets provide both a realistic in vivo appearance and dense, time-resolved 3D ground truth, especially under non-rigid deformation. We present C3VD-DEFCOL, a framework and dataset for evaluating deformable colonoscopy reconstruction with paired geometry and realistic texture. Starting from C3VD/C3VDv2 colon meshes and camera trajectories, we generate controlled deformations of the colon surface, including peristaltic waves and centerline motion, and render per-frame depth, surface normals, optical flow, camera poses, and time-stamped 3D meshes. We then use the rendered geometry, primarily depth, to condition an LTX-2.3-based sim-to-real translation model that produces RGB clips with in vivo-like mucosal color, texture, vasculature, and specular appearance while preserving the underlying 3D scene structure. The resulting dataset contains 110 videos from 11 unique colon mesh geometries, with varying camera trajectories, appearances, and parameterized deformation regimes, including three peristaltic severity levels that serve as controlled evaluation axes. We evaluate the generated videos using appearance realism, geometric consistency, and temporal consistency metrics, and use the paired ground truth to benchmark the downstream task of pose estimation in deformable 3D reconstruction. Our experiments show how pose estimation error increases with increasing deformation severity, providing a controlled stress test that is not possible with existing in vivo datasets. Overall, C3VD-DEFCOL is designed as a reproducible, quantitative evaluation platform for testing deformable 3D reconstruction algorithms, with the goal of reducing the domain gap between synthetic datasets and in vivo colonoscopy.

What's New?

• First colonoscopy dataset pairing realistic in vivo-style RGB appearance with dense, time-resolved 3D ground truth under controlled non-rigid deformation — 110 videos across 11 unique colon geometries (cecum through sigmoid), totaling 9,240 frames.
• Realistic sim-to-real RGB video generated via a geometry-conditioned diffusion model, producing in vivo-style mucosal color, texture, vasculature, and specular highlights while maintaining 3D geometry.
• Dense per-frame ground truth: RGB, depth, surface normals, optical flow, occlusion masks, camera poses, and time-stamped deforming 3D meshes for every frame.
• Open, extensible generation framework allowing researchers to produce additional videos with custom colon geometry, camera trajectory, deformation type, and severity
• Controlled deformation regimes: Rigid, low/medium/high-magnitude peristalsis, and centerline-shift to test reconstruction robustness as non-rigid motion increases.

C3VD-DEFCOL

C3VD-DEFCOL builds on C3VDv2, which consists of two distinct colon shapes, each segmented into seven to eight anatomical regions. C3VD-DEFCOL contains 110 videos with a total of 9240 frames. Code to generate additional videos is available in our GitHub repository.

The dataset is coming soon will be publicly hosted on Johns Hopkins Research Data Repository.

For each registered video frame, the dataset includes:

• RGB frame: rgb/NNNN.png represents the sim2real generated 512x512 image. NNNN denotes the 4-digit frame number within the video.
  
• Depth frame: depth/NNNN_depth.png represents the depth along the camera frame's Z-axis, clamped between 0 and 100 mm, and linearly scaled and encoded as a 16-bit grayscale image. For example, a pixel value of 16,384 corresponds to a depth of 25 mm.
  
• Surface normal frame: normals/NNNN_normals.png stores the X, Y, and Z components of the surface normal vector for each surfel in the R, G, and B color channels, respectively. Components are linearly scaled from ±1 to 0-255 and encoded as a 8-bit color image. Normal vector directions are defined with respect to the camera coordinate system: +x points right, +y points down, and +z points along the viewing direction (i.e., away from the camera).
  
• Optical flow frame: optical_flow/NNNN_flow.png depicts the optical flow from the current to the previous frame. X-direction motion (left to right, clamped between -20 to 20 pixels) is stored in the red channel, and Y-direction motion (up to down, clamped between -20 to 20 pixels) is stored in the green channel. Flow values are linearly scaled and encoded as a 8-bit color image.
  

• Occlusion frame: occlusions/NNNN_occlusion.png indicates pixels that occlude other mesh faces within 100 mm of the camera origin, assigning a value of 255 to these pixels and 0 to all others. This binary data is encoded as an 8-bit grayscale image.
  
• Diffuse Frame: diffuse/NNNN_diffuse.png encodes Lambertian reflectance, computed using the dot product of the surface normal and the direction of the incident light. Reflectance values range from 0.1 to 1.0 and are linearly scaled and encoded as an 8-bit grayscale image.
  
• Camera pose:  pose.txt contains each frame's flattened homogeneous camera-to-world transformation matrix (column major order).
• World Positions and Normals  world_vertex_positions.bin / world_vertex_normals.bin contain the 3D positions and normals of the world-space vertices.
• 3D aligned model aligned_model.obj stores the ground truth triangulated mesh.

C3VDv2

Screening Videos

3D Model Files

Registered Videos (C3VD v1)

Screening Videos (C3VD v1)

3D Model Files (C3VD v1)

C3VDv2 is licensed under a

Creative Commons Attribution 4.0 International License

The original C3VD is licensed under
CC BY-NC-SA 4.0