Real-Time Regulatory Reporting in Banking: From 24-Hour Batch to Sub-Two-Minute Streaming with Auditable Lineage

Jeevan Krishna Paruchuri

doi:10.36948/ijfmr.2024.v06i05.75350

Real-Time Regulatory Reporting in Banking: From 24-Hour Batch to Sub-Two-Minute Streaming with Auditable Lineage

Author(s)	Jeevan Krishna Paruchuri
Country	United States
Abstract	Regulatory reporting in retail and commercial banking has historically been built on overnight batch ETL: extract from the core banking system after end-of-day, transform overnight, submit before the morning regulatory deadline. This pattern is operationally simple and audit-friendly, and it imposes a hard floor of approximately 24 hours between when a transaction occurs and when a regulator could in principle see it. As regulatory regimes Basel IV's granularity expectations, DORA's operational resilience requirements, MiFID II's trade reporting timelines push toward shorter and more frequent reporting windows, the 24-hour floor becomes a problem. This paper presents a case study of migrating a banking regulatory reporting pipeline from a legacy overnight batch architecture to a streaming architecture with end-to-end latency under 2 minutes, while preserving and in fact strengthening the audit and lineage properties that make the legacy system defensible to regulators. The architecture is built on Attunity Replicate for mainframe CDC-based change data capture from the Oracle core banking system into per-table Kafka topics governed by a schema registry, Spark Structured Streaming with exactly-once semantics writing into Delta Lake with append-only audit semantics, and dual-region replication that drove disaster recovery time from 6 hours 20 minutes to 3 hours 45 minutes. Inter-region Kafka replication lag is under 1 minute in steady state. The pipeline handles a baseline of approximately 2 million transactions per day (~23 TPS baseline, ~230 TPS peak) with capacity for 10x spikes at month-end and during crisis scenarios. We document the regulatory requirements that drove the architecture, the technical patterns that emerged, the audit capabilities the new system provides point-in-time queries, full change history, immutable amendment records and the things that took multiple iterations to get right, most notably exactly-once semantics under Kafka offset management and schema evolution coordination across producer and consumer teams. We are honest about the operational complexity the streaming architecture introduces and about the months of work required to build compliance-team trust through hands-on demonstrations of the audit trail. The contribution is a practitioner-grounded blueprint for teams considering whether the cost of streaming regulatory reporting is justified by the regulatory and operational benefits, with explicit attention to the audit and lineage properties that compliance teams will demand.
Field	Computer Applications
Published In	Volume 6, Issue 5, September-October 2024
Published On	2024-10-10
DOI	https://doi.org/10.36948/ijfmr.2024.v06i05.75350

View / Download PDF File

E-ISSN 2582-2160

doi

CrossRef DOI is assigned to each research paper published in our journal.

IJFMR DOI prefix is
10.36948/ijfmr

Downloads

Research Paper Format Copyright Permission Form and Undertaking Form Cover Page Vol 8 Isu 2 Cover Page Vol 8 Isu 1 Cover Page Vol 7 Isu 6

All research papers published on this website are licensed under Creative Commons Attribution-ShareAlike 4.0 International License, and all rights belong to their respective authors/researchers.

CC-BY-SA

About IJFMR Fees & Payment Current Issue Publication Archive	Submit Research Paper Track Submission Status Publication Guidelines Publication Ethics Peer Review & Plagiarism	Join as a Reviewer Editors & Reviewers Reviewer Referral Program Get Reviewer Membership Certi.	Website/Journal Policies Usage Policy Content Policies Privacy Policy

Contact Us		+91-9687-828-838	editor@ijfmr.com

International Journal For Multidisciplinary Research

A Widely Indexed Open Access Peer Reviewed Multidisciplinary Bi-monthly Scholarly International Journal

Real-Time Regulatory Reporting in Banking: From 24-Hour Batch to Sub-Two-Minute Streaming with Auditable Lineage

Share this