International Journal For Multidisciplinary Research

E-ISSN: 2582-2160   •   Impact Factor: 9.24

A Widely Indexed Open Access Peer Reviewed Multidisciplinary Bi-monthly Scholarly International Journal

Call for Paper Volume 8, Issue 2 (March-April 2026) Submit your research before last 3 days of April to publish your research paper in the issue of March-April.
Submit Research Paper Join as a Reviewer

Plagiarism is checked by the leading plagiarism checker TurnItIn

Call for Paper

Volume 8 Issue 2
March-April 2026

Submit your research paper

Indexing Partners
Academia Advanced Sciences Index Bielefeld Academic Search Engine CiteSeer DRJI Google Scholar Independent Search Engine & Directory Network ISI (International Scientific Indexing) Issuu Mendeley Research Networks
RefSeek ResearcherId - Thomson Reuters ResearchGate Scirus Scribd Semantic Scholar UTeM - Universiti Teknikal Malaysia Melaka Wiki for Call for Papers WorldCat Zenodo

AI-Driven Orchestration for Cloud-Native Data Engineering Pipelines

Author(s) Narasimha Chaitanya Samineni
Country United States
Abstract Cloud-native data engineering pipelines form the backbone of modern data ecosystems, powering analytics, machine learning, operational intelligence, and real-time decision systems. As organizations adopt distributed cloud platforms, microservices, serverless compute, and containerized workloads, the volume and velocity of data pipelines increase dramatically. Manual orchestration, rule-based scheduling, limited observability, and fragmented operational workflows create significant challenges in reliability, scalability, and deployment automation. Artificial intelligence offers an opportunity to automate, optimize, and self-heal end-to-end pipeline operations through intelligent orchestration.
This research article proposes an AI-driven orchestration framework that improves pipeline scheduling, anomaly detection, resource optimization, metadata augmentation, data quality validation, and autonomous remediation. The study introduces a taxonomy of orchestration challenges, expands on cloud-native architectural concepts, presents two large analytical tables, and describes novel AI-driven orchestration layers. The findings highlight how AI transforms pipeline execution from static scheduling to dynamic, adaptive, self-governing systems aligned with cloud elasticity and modern DevOps practices. [1][3][5][7][9]
Keywords Cloud Native Pipelines, AI Orchestration, Data Engineering, Metadata Automation, Pipeline Observability, Autonomous Jobs, Data Quality, Serverless Compute.
Field Engineering
Published In Volume 6, Issue 4, July-August 2024
Published On 2024-08-14
DOI https://doi.org/10.36948/ijfmr.2024.v06i04.66461
View / Download PDF File

Share this



E-ISSN 2582-2160
CrossRef

CrossRef DOI is assigned to each research paper published in our journal.

IJFMR DOI prefix is
10.36948/ijfmr

Downloads
Research Paper Format Copyright Permission Form and Undertaking Form Cover Page Vol 8 Isu 2Cover Page Vol 8 Isu 1Cover Page Vol 7 Isu 6

All research papers published on this website are licensed under Creative Commons Attribution-ShareAlike 4.0 International License, and all rights belong to their respective authors/researchers.

CC-BY-SA Open Access

About IJFMR
Fees & Payment
Current Issue
Publication Archive 		Submit Research Paper
Track Submission Status
Publication Guidelines
Publication Ethics
Peer Review & Plagiarism 		Join as a Reviewer
Editors & Reviewers
Reviewer Referral Program
Get Reviewer Membership Certi. 		Website/Journal Policies
Usage Policy
Content Policies
Privacy Policy
Contact Us +91-9687-828-838 editor@ijfmr.com
Powered by Sky Research Publication and Journals