Published in IEEE Access, 2025 | DOI: 10.1109/ACCESS.2025.3597601
Source code documentation plays a significant role in the software development lifecycle, improving the comprehensibility and maintainability of software projects. Despite its importance, documentation is often dismissed or fails to meet expected standards. This research empirically evaluates two prominent open-source Large Language Models, T5 (Google AI) and BART (Meta AI), for automating single-intent code comment generation on Python and Java code snippets.
We collected code-comment datasets from well-documented, high-popularity open-source GitHub repositories, fine-tuned both models, and rigorously evaluated their output using four key metrics: BLEU, Smoothed BLEU, ROUGE (1/2/L), and METEOR. The comprehensive analysis clearly highlights BART as the superior model over T5 for single-intent code comment generation, while T5 offers a substantial training-efficiency advantage.
| RQ | Question |
|---|---|
| RQ1 | How can code comment generation be automated using LLMs such as T5 and BART? |
| RQ2 | How effective are LLMs, including T5 and BART, in automating code comments generation? |
Datasets were constructed from curated, high-popularity open-source GitHub repositories, selected for active maintenance, domain diversity, and documentation quality.
| Repository | Domain | Stars | Forks | Size (MB) | URL |
|---|---|---|---|---|---|
| Django | Web framework | 70k+ | 30k+ | 308 | github.com/django/django |
| Flask | Lightweight web framework | 65k+ | 15k+ | 339 | github.com/pallets/flask |
| Keras | Neural network API | 60k+ | 20k+ | 51.7 | github.com/keras-team/keras |
| Matplotlib | Data visualization library | 18k+ | 7k+ | 518 | github.com/matplotlib/matplotlib |
| Pandas | Data analysis library | 40k+ | 18k+ | 412 | github.com/pandas-dev/pandas |
| PyTorch | Deep learning framework | 70k+ | 19k+ | 1160 | github.com/pytorch/pytorch |
| Requests | HTTP library | 50k+ | 10k+ | 18.6 | github.com/psf/requests |
| Scikit-learn | Machine learning library | 60k+ | 25k+ | 186 | github.com/scikit-learn/scikit-learn |
| TensorFlow | Deep learning framework | 180k+ | 88k+ | 1350 | github.com/tensorflow/tensorflow |
| Repository | Domain | Stars | Forks | Size (MB) | URL |
|---|---|---|---|---|---|
| Android Architecture | Android sample applications | 30k+ | 10k+ | 27.5 | github.com/android/architecture-components-samples |
| HikariCP | JDBC connection pool | 18k+ | 3k+ | 6.13 | github.com/brettwooldridge/HikariCP |
| Java Design Patterns | Design pattern implementations | 85k+ | 28k+ | 65.8 | github.com/iluwatar/java-design-patterns |
| JUnit4 | Unit testing framework (v4) | 9k+ | 2k+ | 28.0 | github.com/junit-team/junit4 |
| JUnit5 | Unit testing framework (v5) | 10k+ | 2k+ | 691 | github.com/junit-team/junit5 |
| Apache Maven | Build automation tool | 5k+ | 2k+ | 82.0 | github.com/apache/maven |
| Apache Spark | Big data processing framework | 35k+ | 26k+ | 646 | github.com/apache/spark |
| Sunflower | Android sample project | 17k+ | 3k+ | 16.6 | github.com/android/sunflower |
| Scrapy (Java wrapper) | Web scraping framework (mixed) | 45k+ | 10k+ | 32.4 | github.com/scrapy/scrapy |
Selection criteria: popularity (high stars/forks), active maintenance, domain diversity, and high documentation quality, ensuring a representative and high-quality dataset for code-comment generation.
Split: ~70% train / 15% validation / 15% test per language.
Note: Raw cloned repositories (
data_collection/raw_cloned_repos/) are excluded from version control via.gitignore. Run the data collection scripts below to generate the processed datasets locally.
Models evaluated:
| Model | Parameters | Architecture | Notes |
|---|---|---|---|
| T5 | ~60M | Encoder-decoder, text-to-text transformer | Faster training, lower compute footprint |
| BART | ~110M | Bidirectional encoder + autoregressive decoder | Higher comment quality, better generalization |
Evaluation metrics:
| Metric | Description |
|---|---|
| BLEU | n-gram precision against reference comments |
| Smoothed BLEU | BLEU with additive smoothing for short sequences |
| ROUGE-1 / ROUGE-2 / ROUGE-L | Unigram / bigram overlap and longest common subsequence |
| METEOR | Precision/recall harmonic mean with synonym and stemming matching |
EncoderDecoder_CodeComment/
├── README.md
├── .gitignore
├── .gitattributes
├── LICENSE
├── CITATION.cff
├── environment.yaml # Conda environment definition
│
├── data_collection/ # Dataset builder
│ ├── extract_python.py # AST-based docstring extraction
│ ├── extract_java.py # Regex-based comment extraction
│ ├── merge_and_clean.py # Merge Python + Java, clean entries
│ ├── split_dataset.py # 70/15/15 train/val/test split
│ ├── compare_test_data.py
│ ├── raw_cloned_repos/ # Ignored: cloned GitHub repos
│ ├── combined/ # Python + Java combined dataset
│ ├── python_only/
│ ├── java_only/
│ └── json_full_dataset.json
│
├── T5_codecomment_Model/
│ ├── T5_codecomment_python_java/
│ │ ├── code/ # Tokenize, train, infer, evaluate scripts
│ │ ├── models/ # Fine-tuned checkpoints (gitignored)
│ │ ├── Tokenized_data/
│ │ └── evaluation_results/
│ ├── T5_codecomment_python/
│ └── T5_codecomment_java/
│
├── BART_codecomment_Model/
│ ├── BART_codecomment_python_java/
│ │ ├── code/
│ │ ├── models/
│ │ ├── Tokenized_data/
│ │ └── evaluation_results/
│ ├── BART_codecomment_python/
│ └── BART_codecomment_java/
│
└── results/
├── fig_results_comparison.png # T5 vs BART metric comparison chart
└── fig_pipeline_overview.png # Colorized pipeline diagram
conda env create -f environment.yaml
conda activate codet5_bartRequired: Python 3.9+, PyTorch, Transformers, Datasets, SentenceTransformers, Evaluate, Scikit-learn.
cd data_collection
python extract_python.py
python extract_java.py
python merge_and_clean.py
python split_dataset.pyDatasets generated in data_collection/combined/, data_collection/python_only/, data_collection/java_only/.
cd T5_codecomment_Model/T5_codecomment_python_java/code
python tokenize_t5_dataset_python_java.py(Equivalent script under BART_codecomment_Model/ for BART.)
python modeltrain_t5_python_java.py \
--train_dataset_path "../Tokenized_data/train" \
--valid_dataset_path "../Tokenized_data/valid" \
--output_base_dir ".."python inference_t5_python_java.py \
--model_dir "../models/T5_Finetuned_model" \
--test_dataset_path "../../../data_collection/combined/test_data.json" \
--output_base_dir "../results"python evaluate_t5_python_java.py \
--predictions_path "../results/t5_python_java_generated_comments.json" \
--test_path "../../../data_collection/combined/test_data.json" \
--output_dir "../results/evaluation"Evaluated on the combined Python + Java test set (3-epoch training):
| Metric | T5 | BART | BART improvement |
|---|---|---|---|
| BLEU | 0.07 | 0.31 | +343% |
| ROUGE-1 | 0.76 | 0.90 | +18% |
| ROUGE-2 | 0.78 | 0.88 | — |
| ROUGE-L | 0.62 | 0.82 | — |
| METEOR | 0.68 | 0.85 | +25% |
| Smoothed BLEU | 0.07 | 0.85 | +1,114% |
Training efficiency:
| Metric | T5 | BART |
|---|---|---|
| Final training loss | 0.0361 | 0.1475 |
| Final evaluation loss | 0.0085 | 0.00378 |
| Total training time | 128,225s | 178,752s |
| Samples/second | 1.079 | 0.774 |
Key takeaway: BART consistently outperforms T5 across every evaluation metric, in both languages and combined, showing stronger fluency, content overlap, and semantic fidelity. T5 trains ~39% faster and reaches a lower training loss, making it preferable for resource-constrained or rapid-iteration settings, while BART is the better choice when comment quality and generalization matter more than training cost.
- BART showed stronger syntactic fluency and semantic fidelity across all metrics and both languages.
- T5 trains significantly faster and suits resource-constrained, real-time use cases (e.g. IDE plugins).
- Both models capture high-level function intent well but omit deeper implementation details — better suited for real-time IDE assistance than comprehensive API documentation.
- No-repeat n-gram and beam size tuning significantly impacted BLEU and ROUGE scores.
- Dataset selection — custom-built from selected GitHub repositories rather than a standard benchmark (e.g. CodeSearchNet), which may limit direct comparability.
- Performance metrics — BLEU/ROUGE/METEOR/Smoothed BLEU capture lexical overlap but not deep semantic fidelity; semantic-aware metrics (e.g. SIDE) were excluded due to resource constraints.
- Model configurations — smaller T5/BART variants were used for computational feasibility; results may not generalize to larger configurations (e.g. T5-large, BART-large).
- Generalizability — findings are specific to Python and Java and may not transfer to other programming paradigms.
- Integrate semantic-aware evaluation metrics (e.g. SIDE) alongside lexical metrics.
- Expand to additional programming languages beyond Python and Java.
- Explore larger model variants (T5-large, BART-large, CodeT5+) for multi-intent comment generation.
- Evaluate with human annotators.
- Extended results with longer training schedules (6 epochs) are explored in the accompanying thesis — see citation note below.
If you use or build on this project, please cite the paper:
D. P. Ghale and M. Dabbagh, "Automated Code Comments Generation Using Large Language Models: Empirical Evaluation of T5 and BART,"
IEEE Access, vol. 13, pp. 141420–141433, 2025, doi: 10.1109/ACCESS.2025.3597601.
Paper link: https://ieeexplore.ieee.org/document/11122447
This repository also includes a CITATION.cff file — GitHub displays a "Cite this repository" button automatically.
Dhan Prasad Ghale (Student Member, IEEE) — Master of Research, Melbourne Institute of Technology Mohammad Dabbagh (Senior Member, IEEE) — Senior Lecturer, Melbourne Institute of Technology; Honorary Senior Lecturer, Macquarie University
This repository is licensed under the MIT License — see LICENSE for details. The associated paper is published under a Creative Commons Attribution 4.0 License (CC BY 4.0).
- Google AI – T5
- Meta AI – BART
- Hugging Face Transformers
- GitHub Open-Source Community
- Melbourne Institute of Technology
If you use or build on this project, feel free to open an issue or star the repo.