Addressing Neural Network Training Challenges through Optimizers and Activation Functions

Author(s)	Ms. Madhuri Nitin Ghadge, Ms. Rohini Kapse
Country	India
Abstract	Training deep neural networks is often hindered by problems such as vanishing gradients (Sigmoid), neuronal instability (ReLU), slow convergence (SGDM), and poor generalization (Adam). This research systematically investigates how the choice of optimizers and activation functions mitigates these challenges. Using a Convolutional Neural Network (CNN) trained on the MNIST dataset, five activation functions (Sigmoid, ReLU, LeakyReLU, GeLU, SiLU) and four optimizers (SGDM, Adam, AdamW, RMSProp) were compared in 20 unique configurations. Performance was evaluated on the basis of validation accuracy, convergence speed, and gradient stability.
Keywords	Deep Learning, Convolutional Neural Networks (CNN), Activation Functions, Optimizers, Gradient Vanishing Problem, Convergence Speed, MNIST Dataset
Field	Computer > Artificial Intelligence / Simulation / Virtual Reality
Published In	Volume 7, Issue 6, November-December 2025
Published On	2025-11-20
DOI	https://doi.org/10.36948/ijfmr.2025.v07i06.61158
Short DOI	https://doi.org/hbbz53

About IJFMR Fees & Payment Current Issue Publication Archive	Submit Research Paper Track Submission Status Publication Guidelines Publication Ethics Peer Review & Plagiarism	Join as a Reviewer Editors & Reviewers Reviewer Referral Program Get Reviewer Membership Certi.	Website/Journal Policies Usage Policy Content Policies Privacy Policy

Contact Us		+91-9687-828-838	editor@ijfmr.com

International Journal For Multidisciplinary Research