Why Database Generated IDs Can Complicate Your System Design

When you’re creating something in your system, where does the ID come from? For most, it’s probably the database. Database generated IDs are pretty straightforward, and you’re probably used to them if you’ve done a lot of CRUD. But why is your database responsible for generating the ID? This seemingly simple decision can have a pretty big impact on your overall system design when you start thinking about asynchronous processing, messaging, and different boundaries.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

You have a client make a call to an API that ultimately inserts something into a database. In my example, I’m using a shipment in a logistics application. Ultimately, the database generates some type of ID. This could be an auto incrementing value, a UUID, or something else. That ID is then returned back to the client.

That’s pretty straightforward, and you’re probably used to this if you’ve done a lot of CRUD.

But the question is: why is your database responsible for generating the ID?

This seemingly simple decision can have a pretty big impact on your overall system design when you start thinking about asynchronous processing, messaging, workflows, and different boundaries.

Creating Something Isn’t Always a CRUD Operation

Creating a shipment isn’t just a CRUD operation.

There’s a lot going on when our client makes that API call. We might need to validate the carrier, check compliance, calculate ETAs, determine the route, and apply various business rules.

That’s not necessarily a short process.

Because of that, what we might actually want to do is process the request asynchronously. We can return back to the client saying, “Yes, we are going to create this shipment,” without requiring the shipment to be completely created before responding.

Instead of doing all the work inside the HTTP request, we place a message on a queue and return back to the client.

From there, some asynchronous process or worker picks up that message, performs all the work, and eventually saves the shipment into the database.

But if the database is generating the shipment ID, how does the client ever know what that ID is?

We could create some initial database record or generate some separate operation ID that we return to the client immediately.

But really, all we’re doing is designing around the fact that our database is generating the shipment ID.

Why not generate the ID before we even send the request?

Give the Shipment an Identity First

Instead, the client can generate the ID.

This could be a UUID, a GUID, or really any type of identifier the client already knows.

When it sends the HTTP API call, the shipment ID is included in the request. When we create the message that will be processed asynchronously, the shipment ID is part of that message as well.

We’ve already defined what the shipment ID is.

At this point, we can process everything asynchronously just as we were before. The difference is that the client already knows the identity of the shipment.

That means the client can make some type of status request using that shipment ID even though the shipment hasn’t been fully processed yet.

All the same validation and business rules are still happening while processing the message.

The distinction is that the shipment has an identity before it’s persisted to the database.

Identity Across a Workflow

While asynchronous processing is an easy way to illustrate this, it goes much deeper.

Consider a typical, simplified workflow around a shipment in a logistics application.

An order becomes available to dispatch. We assign a carrier to it. We dispatch the order, which creates the shipment. From there, we have all the different workflow events that happen over time.

The shipment arrives at the shipper. The freight gets loaded. It arrives at the consignee. It gets emptied. The freight gets delivered.

There’s an entire lifecycle.

Depending on how you’ve designed your system, you likely have different consumers in different services processing these messages. Those services might also be publishing events related to the exact same shipment.

So how are you correlating all of them?

With the shipment ID that was defined at the very beginning.

There Is No Shipment Service

Now look at this from a slightly different perspective.

There is no shipment service.

There isn’t necessarily one boundary that owns a shipment.

Instead, we have different boundaries that are all part of a workflow. Each of them understands a shipment and has certain capabilities related to it.

Dispatch is concerned with the execution of the shipment. Arrive, load, empty, and the actual milestone events.

Tracking is concerned with understanding where the truck or vehicle is driving, where it currently is, and what the ETAs are.

Billing cares about paying the carrier and charging or invoicing the customer.

Carrier management cares about compliance, insurance, and other carrier related information.

There isn’t one single concept of a shipment that every part of the system shares.

Each boundary understands its specific view of a shipment.

But Users Still See One Shipment

I need to mention a talk and video by Udi Dahan at Particular Software called Design More Decoupled Services with One Weird Trick.

It’s worth watching because it goes much deeper into the issues around sharing data between services.

That’s usually where the conversation immediately goes.

If there isn’t one shipment and the data is spread across multiple boundaries, how do we deal with all of that?

Because from a user’s perspective, there is just one shipment.

They don’t care about what service owns what data. They want to see all the information related to a shipment so they can get their job done.

If we’re building a UI that displays shipment information, we might need the general shipment information along with tracking information, the current vehicle location, ETAs, driver information, carrier information, customer information, compliance information, and more.

There’s a lot of data involved.

So how do we get all of that data?

Does it all need to live in one service just so we can create this view?

Stop Publishing Everything

One direction people often go is publishing large events to distribute all of that data around.

An OrderDispatched event occurs, and we include everything about the shipment. All of the shipment details are included in the event so other services have access to them.

Another option people use is change data capture to publish changes.

But it’s effectively the same idea.

We’re publishing what is essentially the entire schema of our shipment through events so we can distribute that data to other services.

But why do we need to do this?

Go back to generating the shipment ID at the very beginning.

What if OrderDispatched looked more like this?

That’s it.

No shipment details.

Just the ID of the shipment that was dispatched.

Compose the View From Different Boundaries

Jump back to the shipment screen.

From a user’s perspective, it is still one shipment. But we’ve defined boundaries for where different pieces of shipment data live.

Tracking has tracking information. Carrier management has carrier information. CRM has customer information. Compliance has compliance information. Dispatch has the general information related to executing the shipment.

The UI is a unified view.

That doesn’t mean all the data has to live in one place.

On the command side, when dispatch publishes an event saying that the order was dispatched, that can kick off other parts of the workflow.

We have different consumers such as tracking, compliance, CRM, billing, and others.

They already understand the shipment ID.

They were part of the process earlier and already have data related to that shipment.

For example, when we selected our carrier, that carrier might already be associated with tracking. Tracking can now follow the vehicle and understand its position on the road.

When we need to compose data for a user, tracking already has the information it needs.

It didn’t need everything copied from some central shipment service.

Identity Connects the Workflow

When you’re trying to display the shipment UI, there isn’t one central shipment service or one particular boundary that has all the aggregated information.

You’re composing it.

You get the general information from dispatch, tracking information from tracking, compliance information from compliance, customer information from CRM, and whatever else the user needs.

You’re composing that information from the boundaries that already had the data to begin with.

And all of this started because they shared the shipment identity from the beginning.

This matters when looking at the workflow as a whole because not all events are being published from the same boundary.

An order becoming available, assigning a carrier, dispatching the order, arriving at the shipper, loading freight, and delivering it can involve different parts of the system.

They’re participating in the same workflow, but they don’t need to share the same model or database.

They need to understand the identity they’re working with.

A Small Decision With a Big Impact

Generating an ID before you send the initial request seems like a pretty simple implementation detail.

But it can have a pretty profound effect on the design of your system.

It’s not just useful when you’re doing asynchronous processing. It matters when you have different boundaries and start asking questions like:

Who owns what data?

How are those boundaries communicating?

How is data being composed?

How are different parts of a workflow correlated?

And ultimately, how coupled are those parts of the system?

To be clear, generating an ID ahead of time doesn’t automatically make your system decoupled.

But generating it at the right time as part of a workflow can remove constraints you might not even realize you had.

It can also give you more visibility into what your system design should actually look like.

Join CodeOpinon! Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• 5 Event Driven Architecture Problems (And How to Avoid Them)
• CRUD vs Task Driven: Are You Losing User Intent?
• Design More Decoupled Services with One Weird Trick.

The post Why Database Generated IDs Can Complicate Your System Design appeared first on CodeOpinion.