O4 - Identification of Fiducial Points on 3D Torso Images Using Contrastive Learning

Introduction: : In plastic surgery, surgeons refer to measurements of breast aesthetics, including breast symmetry, for surgical outcome monitoring throughout the progression of the reconstruction. Current quantitative measurements of breast morphology include direct anthropometry on the patient’s body using a measuring tape or using 2D photogrammetry and 3D stereophotogrammetry as less invasive options. Nevertheless, the increasing use of 3D imaging for breast morphometry in reconstructive surgery relies on manual identification of anatomical landmarks (fiducials), which is time-consuming and prone to operator bias. Therefore, we propose a machine learning (ML) framework for automated identification of four fiducial points in 3D torso photographs: Midline Point, Nipple, Sternal Notch, and Umbilicus.

Materials and

Methods: : The proposed ML framework uses a convolutional neural network-based feature extractor and a multi-layer perceptron to generate feature maps for each input image. During training, these maps are sampled to update a feature bank and improve the distinction between fiducial and background (clutter) points through supervised contrastive learning. Inference is performed by computing a score map for each fiducial point, where the highest score indicates the predicted point location. A localized search is performed to refine the identification of nipples using Intrinsic Shape Signatures (ISS). Due to the distinctive characteristics of the nipple compared to the surrounding points, ISS reliably identifies it as a salient point in the search region. We used 317 3D torso photographs taken before and after breast reconstruction from 129 patients, with 251 images used for k-fold training/validation, and 66 images for testing. Two former graduate students, trained by plastic surgeons, manually identified and recorded the fiducial locations on separate sets of images using custom software. We define as ground truth (GT) these locations used as reference during training, validation, and testing. Accuracy was evaluated using the percentage of correct keypoints (PCK), which considers a fiducial to be correct if its distance from the GT is within a radius determined from previously reported anthropometric data. We report the average distance between predictions and GT for evaluation.

Results, Conclusions, and Discussions:: The framework detects fiducial points within a specified acceptable radius from the ground truth in 75.13% of cases (Table 1). Table 2 provides a comparison of the performance of fiducial identification approaches in torso images against the proposed approach. Kenig et al. [1] proposed a deep learning method achieving 87% accuracy for Sternal Notch identification in breast surgery patients’ 2D frontal photographs, leveraging properties like 2D grid structure not available in our 3D point clouds dataset. Kawale et al. [2] utilized surface curvature measurements on 3D meshes and color texture. The proposed framework and Kawale et al. algorithms are tested on surface scans of women who had both native breasts. Kawale et al report equal or lower accuracy for all the points, with higher average predicted to GT distance compared to ours, indicating the proposed framework provides better predictions. Unlike their method, our approach doesn’t require manual registration of torso meshes to the upright frontal position. Additionally, the proposed framework operates on uncolored point cloud data, which enhances flexibility in 3D surface analysis and avoids any bias due to the color texture (e.g., skin color, lighting).

Figure 1a presents examples of successful midline point identification within a 15 mm radius of the manual annotation. The framework effectively predicts nipple locations within a 6.5 mm radius for unoperated breasts and when the nipple shape is occluded or inverted (Figure 1b). It successfully identifies the sternal notch within a 20 mm radius from the manual annotation (Figure 1c). The umbilicus is easier to identify than other fiducial points due to its distinct shape (Figure 1d).
In summary, this framework was successful in detecting fiducial points in 3D surface scans of breast surgery patients. The proposed framework can be extended to identify additional fiducial points on the torso. Moreover, framework accuracy can further be improved for clinical applicability i.e., the evaluation of breast symmetry in plastic surgery.

Acknowledgements and/or References (Optional):: [1] N. Kenig, J. Monton Echeverria, and L. De la Ossa, “Identification of Key Breast Features Using a Neural Network: Applications of Machine Learning in the Clinical Setting of Plastic Surgery,” Plast Reconstr Surg, vol. 153, no. 2, pp. 273E-280E, Feb. 2024, doi: 10.1097/PRS.0000000000010603.
[2] M. M. Kawale et al., “Automated Identification of Fiducial Points on 3D Torso Images,” Biomed Eng Comput Biol, vol. 5, p. BECB.S11800, Jan. 2013, doi: 10.4137/BECB.S11800.