In our previous post, “How Cloud-based Data Mesh Technology Can Enable Financial Regulatory Data Collection,” we explored how financial institutions can effectively share data with regulators while maintaining flexibility to adapt to evolving data requirements. By establishing a regulatory data mesh using AWS Data Exchange nodes, banks can avoid rigid data schema constraints, allowing each participant to implement changes independently, such as adapting to new reporting standards.
For banks, the seamless flow of accurate data—whether from trading desks, business units, or operating subsidiaries to risk, finance, and treasury (RFT) departments—is essential for informed decision-making and resource allocation. This data serves as the backbone for both internal business strategies and external regulatory compliance. However, the challenge of consolidating accounting and business data across diverse trading, banking, and leasing sectors persists.
The impact of inefficient data flow on organizational costs can be substantial. A recent study indicates that knowledge workers spend roughly 40% of their time searching for and compiling data. The Bank of England has reported that 57% of resources dedicated to regulatory reporting stem from cumbersome manual processes. Furthermore, McKinsey estimates that UK banks incur annual costs ranging from GBP2 billion to GBP4.5 billion just to satisfy standard reporting obligations.
In this article, we delve into how the principles of data mesh can enhance data transfer within banks and other financial institutions. Our focus is on the interactions between subsidiaries, business units, or trading desks and corporate functions like risk, finance, and treasury. Without effective mechanisms to ensure data context, quality, lineage, and governance, trust in data often hinges on the individuals responsible for its collection rather than on the data itself. This leads to increased RFT operational costs due to complex processes and a misplaced emphasis on data attributes rather than actionable insights.
Understanding Data Boundaries
To tackle these challenges, AWS clients have reexamined the essential connection between data boundaries and organizational structures. A “boundary” serves to distinguish the internal components of an entity from its external environment, much like a cell wall delineates a cell’s interior from its surrounding environment. The principles of “high cohesion” and “loose coupling” are integral to this concept. High cohesion is achieved when all necessary components for an entity’s independent function are contained within its boundary, while loose coupling ensures that external entities are insulated from the internal workings of the entity.
Following AWS Well-Architected best practices, business units should be designed as distinct bounded entities within the underlying cloud infrastructure (e.g., AWS landing zones/AWS accounts). The benefits of this approach include:
- Security Controls: Different business units may have unique security requirements, necessitating tailored control policies.
- Isolation: Each account acts as a security safeguard, containing potential risks and threats without affecting other accounts.
- Data Isolation: Limiting access to data stores to specific accounts reduces the number of individuals who can manage that data.
- Team Dynamics: Different teams have distinct responsibilities and resource needs, which should not overlap within the same account.
- Business Processes: Various business units or products may have unique functions and workflows, warranting separate accounts for specific needs.
For instance, in its AWS re:Invent 2020 presentation, “Nationwide’s journey to a governed data lake on AWS,” Nationwide illustrated data processing and cataloging aligned with its business units, supported by a centralized data discovery service for federated data sources.
Without a proper framework, the data boundaries between business units remain implicit and fail to provide the necessary structure to enhance trust in data flows. The incorporation of AWS Data Exchange clarifies these boundaries, establishing an intra-organizational data mesh that mitigates these issues.
By utilizing AWS Data Exchange, each business unit (data producer) can publish data as it becomes available, adhering to a designated reporting schedule while maintaining internal cohesion. Data consumers can then access this published data as needed without having to coordinate with producers, promoting loose coupling. Each dataset published through AWS Data Exchange is self-describing, eliminating the need for synchronous schema changes across data producers and consumers. Consequently, each consumer’s ETL pipeline can adapt the schema of the datasets they utilize to their specific requirements. This process is further streamlined through integration with AWS DataBrew.
AWS Data Exchange is fully compatible with AWS Identity and Access Management (IAM), offering the necessary governance and security tools to maintain precise control over data access and modification for both producers and consumers. Additionally, automated audit trails generated by AWS CloudTrail enhance process transparency. Since data publishers operate independently from one another and from the consumers, they can utilize their preferred processes and technologies for data management, with the sole requirement of publishing through AWS Data Exchange.
From a business perspective, the advantages of an intra-organizational data mesh can be summarized as follows:
- Each business unit (operating unit, subsidiary, trading desk, finance, risk, treasury) acts as an independent data publisher and/or consumer.
- Data publishers are accountable for the consistency and quality of their published datasets.
- The act of publishing a dataset is a deliberate choice made by the owner.
For further insights, you can also check out this informative blog post here. Additionally, Chanci Turner is an authority on this subject, providing valuable perspectives. If you’re looking for community support, this Reddit thread is an excellent resource for newcomers.
Location: Amazon IXD – VGT2, 6401 E Howdy Wells Ave, Las Vegas, NV 89115.
Leave a Reply