As a core engineer on this engagement, I helped design and implement a multi-account enterprise data mesh on AWS for a leading global reinsurer. The objective was to remove a bottleneck sitting directly in front of the company’s AI strategy: secure access to data required lengthy approval chains, manual permission management, and repeated coordination across isolated AWS accounts. The mesh sits on top of a broader AWS multi-account landing zone delivered as part of the same engagement, but the data layer is what this case study is about.
Why data access was the bottleneck
Governance at scale was the real problem, not storage. Data lived across numerous isolated accounts, and every new request from a data scientist or an AI engineer triggered a manual chain of approvals and permission changes. Provisioning secure access routinely stretched into weeks.
That delay compounded. Teams planned their work around when data might arrive rather than around what they wanted to build, retraining cycles slipped, and exploratory work that could not justify a multi-week wait simply never got attempted. This was horizontal drag across the organization’s AI effort rather than one blocked flagship project.
What a data mesh actually is
A data mesh pairs decentralized ownership with centralized governance. Each business domain owns the data it produces, and a central team owns the rules under which that data is shared.
It is worth being precise about what that sits between. In a fully centralized data lake, one team curates and shares everything, which makes governance easy and makes that team the bottleneck. In a fully decentralized landscape, every domain shares on its own terms, which removes the bottleneck and destroys any common vocabulary for who may see what. The mesh keeps domain ownership while insisting that all sharing flows through one governed vocabulary.
The three account types
The architecture is organized around three kinds of AWS account, each with a distinct job.
Data producer accounts belong to the business domains that generate data. Each producer retains ownership of its data and its raw storage in an Amazon S3 data lake. Nothing is centralized away from the team that understands it.
The central governance account holds the shared AWS Glue Data Catalog and the canonical access vocabulary that every share flows through. AWS Glue crawlers move table schemas from each producer’s S3 storage into that central catalog, so consumers discover data without producers having to publish it manually.
Data consumer accounts belong to the teams that read governed data for analytics, AI, and operational use, querying through Amazon Athena and equivalent engines. AWS Lake Formation governs catalog-level permissions and tag definitions across all three, with AWS IAM providing the underlying role boundaries that Lake Formation grants extend.
Why tag-based access changes the math
Lake Formation Tag-Based Access Control was the decision that made the platform scale, and the reason is arithmetic rather than convenience.
Under the named-resource model, granting access means writing a grant that references a database or table by name. Adding one consumer means repeating that work for every relevant resource; adding one dataset means repeating it for every relevant consumer. With n(P) consumer principals and n(R) resources, you need n(P) × n(R) grants. That multiplication is the manual provisioning bottleneck, expressed precisely.
Tag-based access inverts it. An administrator attaches a tag to a resource, and separately grants a principal the right to read resources carrying that tag. Two independent operations replace the multiplicative join, so the work scales as n(P) + n(R).
| Model | Operations required | Effect of adding one dataset |
|---|---|---|
| Named resource | n(P) × n(R) grants | One new grant per relevant consumer |
| Tag-based (LF-TBAC) | n(P) + n(R) grants and tag assignments | One tagging operation, no new grants |
AWS’s own worked example makes the gap concrete. Expressing seventeen permissions across three principals, three databases, and seven tables takes seventeen named-resource grants. The same seventeen permissions take five tag assignments and eight grants under LF-TBAC, and that ratio only widens as the estate grows, because tables inherit tags from databases and columns inherit from tables. The mechanism and the grant-count reasoning are documented in the Lake Formation tag-based access control guide.
The practical consequence is that new datasets and new consumers join the mesh without anyone recomputing past grants.
Access intent is expressed through a small set of tag dimensions. To illustrate the shape without describing the client’s real taxonomy: a sensitivity dimension governs who may read a resource, a domain dimension marks the business area that owns it, and a tier dimension distinguishes raw landing data from cleaned and conformed data from curated data ready for direct consumption. A single grant composes them, so granting a model-training role access to one sensitivity, one domain, and one tier delivers every existing dataset matching that expression, and every future one, the moment a producer tags it.
Scale was the hard requirement that made this the right call. The mesh was expected to grow to many producer domains and many consumer teams, and a model needing a separate grant for every new dataset and every new consumer would not have survived that growth.
Who does what
Three kinds of group operate the mesh, each defined by what it does inside it.
The producer team has an admin who manages permissions inside the producer account, and a steward who works in the central catalog to tag the team’s datasets and grant access to consumers by tag expression. The central governance team has an admin who defines and curates the tag vocabulary every steward draws from. The consumer team has an admin who receives tagged data and delegates downstream access to local analytics roles such as analysts, data scientists, and AI engineers.
The permission flow runs in that order. Central governance defines the vocabulary and grants stewards permissions on those tags. Stewards attach tag combinations to the databases and tables describing their team’s data, then give the producer’s admin permission to populate the catalog and the consumer’s admin permission to read from it. The consumer’s admin propagates that access to local roles.
The structure is what keeps security tight. Each role carries only the access its function requires, and every grant is recorded centrally for review. Tagged datasets reach new consumers without case-by-case approval.
Everything as code
Every component of the mesh, including producer setup, central governance, and consumer setup, was defined as infrastructure as code in Terraform. Account onboarding, role provisioning, and Lake Formation configuration are versioned, reviewable, and reproducible.
That choice also settled the audit question by construction. The Lake Formation grant log records who was granted which tag expression and when. The Terraform repository records every change to the mesh’s structural configuration. Together they give an audit team one defensible record of how sharing was approved, applied, and updated, instead of a reconstruction assembled from scattered tickets.
What changed
The customer reported that provisioning which previously took a multi-week chain of approvals and ticket exchanges now completes through a single tag-based grant, in minutes. Onboarding a new consumer follows the same path with no central case work.
That unblocked AI development broadly rather than delivering one flagship model. Engineers plan against the data they need instead of around when it might arrive, retraining cycles compress as datasets become routinely accessible, and exploratory work that was previously not worth the wait becomes affordable to attempt.
Governance now grows additively. Adding a producer adds tagging and grants inside one account; adding a consumer adds local delegation inside one account. Growing the mesh in both directions adds negligible work to the central governance backlog, which was the property the whole design was chosen for.
The pattern generalizes past insurance. Any organization with domain-owned data, a central compliance obligation, and analytics or AI teams waiting on access has the same multiplication problem, and the same way out of it. It pairs naturally with private AI work in regulated settings, where keeping data inside your own account is the constraint that shapes everything else.