N29 - Conversational LLM-Based Decision Support for Defect Classification in AFM Images

Introduction: : Atomic Force Microscopy (AFM) is widely employed in biophysics to investigate the mechanical and topographical characteristics [1] of individual living cells and their constituent parts because of its great force sensitivity and spatial precision. Accurately measuring height profiles, roughness, or nanoscale mechanical properties (e.g., Young's modulus, deformation, dissipation, adhesion) depends on crisp AFM height and mechanical property maps. However, image artifacts or defects obscure true sample features, leading to inaccurate nanomechanical property maps, especially on soft samples like live cells or polymers. Image clarity is critical for calculating surface roughness (Ra, RMS) or measuring particle size or fiber orientation. AFM imaging quality can be impacted by different sources [2]. Among all these sources, “tip contamination” and “not tracking” are the most frequent issues irrespective of organic or inorganic samples. Image artifacts arising from defects such as tip contamination or tracking failures can lead to inaccurate quantification and misinterpretation. Recent advances in artificial intelligence (AI) and deep learning (DL) have been encouraging towards automating image analysis tasks, but the majority of existing solutions are limited to post-processing or single types of samples. To address these limitations, we presented an end-to-end AI-based system (shown in Fig.1) that combines a vision-based DL model for defect classification with an (Large Language Model) LLM-based conversational assistant, aiming to reduce expert intervention and enable real-time corrective advice for AFM experimentalists.

Materials and

Methods: : The biological subset was composed of live cells (multiple mammalian cell lines including T98G, U87 MG, AHPC, HeLa, and NIH 3T3). The AFM was operated in Peak Force Quantitative Nanomechanical Mapping (PF-QNM) mode. To train defect detection model, we developed AFM defect dataset spanning organic (biological) and inorganic samples. All AFM images in our dataset were manually classified into four groups based on visual inspection and defect type domain knowledge: Class (1): Good Image (high-quality, defect-free AFM scans), Class (2): Imaging Artifacts (images with multiple overlapping or unclassifiable defects that do not fit into the predefined categories perfectly), Class (3): Not Tracking, Class (4): Tip Contamination. To ensure label quality, annotations were reviewed and validated by an expert in AFM imaging and analysis. AFM image processing steps included image cropping, augmentation and z-score normalization. We started with a baseline VGG16 model (transfer learning method) [3] for AFM defect classification, employed a layer-wise, systematic fine-tuning approach to obtain the optimal performance for our dataset (shown in Fig. 2). To achieve convergence and prevent oscillations, we employed the ReduceLROnPlateau scheduler. Once a defect was identified in the AFM image, we utilized an LLM to assist the AFM experimentalist in determining appropriate corrective actions to mitigate the issue. we evaluated four different models: (i) GPT-4o (from OpenAI), (ii) o3-mini (from OpenAI), (iii) Claude 3.5 Sonnet (from Anthropic), and (iv) Claude 3.7 Sonnet (from Anthropic) and we curated a benchmark dataset comprising 50 detailed questions focused on AFM image defects for a robust comparison.

Results, Conclusions, and Discussions:: Result and

Discussion: To prevent overfitting, we introduced dropout layers after the first two fully connected layers of the VGG16 classifier. However, increments (> 0.6) didn’t reflect a proportionate enhancement in test accuracy, which reflects that excessive regularization can discourage learning and overall performance. We got lower test accuracy of around 80-85% for dropout rate of 70-80%. Specifically, unfreezing the last two convolutional blocks and fully connected layers resulted in a training accuracy of 99%, validation accuracy of 89%, and test accuracy of 95% (shown in Table 1 and 2). All models were evaluated on the same set of 50 benchmark questions related to AFM defect identification and mitigation, ensuring a fair and consistent basis for comparison. We observed that the mean score for OpenAI’s reasoning model (o3-mini) is marginally higher than that of the non-reasoning model (GPT-4o) (shown in Table 3 and 4). We developed the user interface (UI) using the Streamlit library, creating a unified platform that integrates two key modules: (1) an image upload module for automatic defect classification, and (2) an LLM-powered chatbot. Overall work is available at: https:// github.com/ idealab-isu/ AFM-LLM-Defect-Guidance). This deep learning-based framework (proposed here) will provide ‘much-needed’ decision-support for experimentalists, improving biomolecular and biomedical imaging.

Conclusion: In this work, we proposed a unified framework for automated AFM defect classification and LLM-based decision assistance to enhance the efficiency of AFM experimentation. By systematically investigating data preprocessing techniques— including normalization and augmentation—alongside a transfer learning-based approach with incremental, layer-by-layer fine-tuning of the VGG16 network, we achieved robust classification performance on challenging AFM datasets. Additionally, we extended our framework by incorporating LLMs to further assist AFM experimentalists. By systematically evaluating multiple LLMs, including reasoning and non-reasoning models from OpenAI and Anthropic, we demonstrated that LLMs can provide context-aware explanations and actionable guidance for addressing imaging defects. Our findings revealed that reasoning models generally offer advantages for complex queries, particularly when built on strong baseline architectures, while non-reasoning models remain highly effective for standard, well-defined tasks.

Acknowledgements and/or References (Optional)::

References:

[1] Sarkar, A., Sohail, A., Dong, J., Prunotto, M., Shinki, K., Fridman, R., & Hoffmann, P. M. (2019). Live cell measurements of interaction forces and binding kinetics between discoidin domain receptor 1 (ddr1) and collagen i with atomic force microscopy. Biochimica et Biophysica Acta (BBA)-General Subjects, 1863(11), 129402.

[2] Bruker Corporation, AFM Image Quality, https://nanoscaleworld.bruker-axs.com/nanoscaleworld/, 2025, sPM Training Guide, Bruker Corporation, Santa Barbara, CA, USA. [Online].

[3] Li, M. A., & Xu, D. Q. (2021, May). A transfer learning method based on vgg-16 convolutional neural network for mi classification. In 2021 33rd Chinese Control and Decision Conference (CCDC) (pp. 5430-5435). IEEE.