AutoML Pipeline Orchestration and Explainable AI Integration in Databricks Environments

Author(s)	Praveen Kumar Reddy Gujjala
Country	United States
Abstract	This study explores the integration of automated machine learning (AutoML) capabilities with explainable AI frameworks within Databricks ecosystems for enterprise-scale deployment. The research presents a comprehensive methodology for automated model selection, hyperparameter optimization, and interpretability analysis that addresses regulatory compliance requirements while maintaining production-grade performance. Novel contributions include adaptive algorithm selection based on data characteristics, automated bias detection mechanisms, and real-time explainability dashboards for production models. The proposed framework demonstrates a 65% reduction in model development time while ensuring regulatory compliance through integrated fairness metrics and interpretability standards. Performance evaluation across multiple industry datasets shows consistent accuracy improvements of 12-18% compared to traditional manual ML approaches, with automated bias detection achieving 94% accuracy in identifying potential fairness violations before model deployment.
Keywords	AutoML, Explainable AI, Databricks, Model Interpretability, Regulatory Compliance, Bias Detection.
Field	Engineering
Published In	Volume 6, Issue 3, May-June 2024
Published On	2024-06-07
DOI	https://doi.org/10.36948/ijfmr.2024.v06i03.55444

About IJFMR Fees & Payment Current Issue Publication Archive	Submit Research Paper Track Submission Status Publication Guidelines Publication Ethics Peer Review & Plagiarism	Join as a Reviewer Editors & Reviewers Reviewer Referral Program Get Reviewer Membership Certi.	Website/Journal Policies Usage Policy Content Policies Privacy Policy

Contact Us		+91-9687-828-838	editor@ijfmr.com

International Journal For Multidisciplinary Research