A12 - Blood–Brain Barrier Permeability Prediction by Integrating Transformer-Derived Embeddings with Chemical Descriptors

Introduction: : The blood-brain barrier (BBB) is a highly selective barrier that separates the body from the central nervous system (CNS). It prevents most medications from entering the CNS, posing a large challenge in drug development. Accurately predicting BBB permeability early in drug discovery can optimize the drug discovery process by identifying unpromising candidates early, reducing experimental costs. Classical in-silico models use fixed, hand-crafted chemical descriptors (e.g., total polar surface area, hydrogen bond donors) that are interpretable but can miss higher-order structural information. Newer models such as transformers treat simplified molecular input line entry system strings (SMILES) as text, learning rich latent representations that capture nuances in chemical structure, but tend to lag behind chemical descriptor-based models in terms of performance. However, these models can benefit from pretraining and can be applied to many different tasks at once, making them good targets for future exploration in computational drug development. In this work we show that fusing transformer-encoded embeddings with chemical descriptors significantly boosts predictive accuracy across multiple datasets.

Materials and

Methods: : Datasets:
Three datasets were utilized in this study. Models were trained on the public benchmark dataset B3DB [1]. Model validation was conducted using two external datasets: BBBP, a widely used benchmark available through MoleculeNet [2], and NeuTox, derived from NeuTox2.0 paper [3]. To prevent bias and information leakage, datasets were filtered using Tanimoto Similarity Scores (TSS); compounds exhibiting a TSS greater than 0.9 with other datasets were removed (e.g., NeuTox compounds with a TSS ≥ 0.9 to BBBP or B3DB were dropped).
Feature Extraction:
A pretrained transformer-based model Chemformer [4] was used to extract features by using the outputs of the model’s encoder. The Chemformer encoder was fine-tuned for BBB permeability prediction by fitting it to the B3DB dataset. The fine-tuned encoder was then used to obtain a total of 512 encoded features. 208 chemical descriptors were calculated using the RDKit Featurizer available through DeepChem [5], a widely used cheminformatic software package.
Feature Fusion and Prediction
For each SMILES string, the chemical descriptors were concatenated the encoded features, resulting in a final total of 720 features. Features were then normalized about their means and standard deviations and passed to a Multi-Layer Perceptron for classification.

Results, Conclusions, and Discussions:: Using combined features, the model achieved an AUROC of 0.985 on the B3DB test set. On external validation sets, the model demonstrated strong generalization, achieving an AUROC of 0.800 on BBBP and 0.934 on NeuTox. Ablations reveal that late-stage feature concatenation provided minimal additional information, as using only encoded features resulted in an AUROC 0.985 and using only chemical descriptors resulted in an AUROC of 0.960 in the B3DB test set. This indicates that although feature concatenation did not improve performance, it did not reduce performance either.
Interestingly, chemical descriptors demonstrated a stronger ability to generalize across different datasets compared to encoded features, making them an important feature to include in models as they can boost both interpretability and generalization to unseen datasets. For example, when trained using the B3DB dataset, the encoded features result in an AUROC of 0.776 on the BBBP test set whereas the chemical descriptors result in an AUROC of 0.876. By incorporating chemical descriptors, the model can better generalize to external datasets, improving performance (combined features result in an AUROC of 0.800 on BBBP).
Overall, these results demonstrate that simple late-stage feature integration methods such as concatenation can help to improve model performance without a substantial increase in complexity. Future work may examine attentive feature integration methods or ensemble learning to boost classification performance and extract more meaningful complementary information from both feature types.

Acknowledgements and/or References (Optional):: [1] F. Meng, Y. Xi, J. Huang, and P. W. Ayers, “A curated diverse molecular database of blood-brain barrier permeability with chemical descriptors,” Sci Data, vol. 8, no. 1, p. 289, Oct. 2021, doi: 10.1038/s41597-021-01069-5.
[2] Z. Wu et al., “MoleculeNet: a benchmark for molecular machine learning,” Chem. Sci., vol. 9, no. 2, pp. 513–530, Jan. 2018, doi: 10.1039/C7SC02664A.
[3] X. Pang et al., “NeuTox 2.0: A hybrid deep learning architecture for screening potential neurotoxicity of chemicals based on multimodal feature fusion,” Environment International, vol. 195, p. 109244, Jan. 2025, doi: 10.1016/j.envint.2024.109244.
[4] R. Irwin, S. Dimitriadis, J. He, and E. J. Bjerrum, “Chemformer: a pre-trained transformer for computational chemistry,” Mach. Learn.: Sci. Technol., vol. 3, no. 1, p. 015022, Jan. 2022, doi: 10.1088/2632-2153/ac3ffb.
[5] “Licensing and Commercial Uses — deepchem 2.8.1.dev documentation.” Accessed: Mar. 24, 2025. [Online]. Available: https://deepchem.readthedocs.io/en/latest/development_guide/licence.html