The batch data source is a job that produces new versions of the dataset. The real-time data source, on the other hand, appends data to a real-time buffer, and this data eventually gets written to all versions of the dataset.

When the batch data source runs and produces a new version of the dataset, the Hybrid Store design pattern then does the following:

• Create a new version N+1 of the dataset (called a future version) and load the batch data into it.

• Replay recent data from the real-time buffer onto the future version N+1.

• After the buffer replay is caught up, switch the current version pointer from N to N+1. Version N can be kept as a backup. There is no future version anymore (until the next batch job).

Hybrid Stores are a way to implement Lambda Architectures, but by merging at write-time, rather than read-time. They can also support Kappa Architectures.

Venice is an open source database implementing the Hybrid Store design pattern, and other proprietary systems do so as well.

The case in favor of A leveraging B was that it would decrease the maintenance cost of system A, since more functionality would be “outsourced” to system B (owned by another team). This maintenance gain could be especially significant if system B offers a correct solution to a difficult problem. That being said, some nuances are missing.

Deeply integrating multiple systems with a single shared dependency has pros and cons. It generates more leverage of course, but what kind of leverage?

If the surface area of the leveraged functionality is large (i.e., lots of APIs), then the more likely it is that any change on either side causes a regression. System A could break the system B, or vice versa, potentially leading to outages. This risk can be mitigated with robust integration testing and other techniques, but in the end it still translates into a productivity cost. These various drags on operations and development can be called coupling cost.

If the maintenance gain - coupling cost > 0, then the leverage is positive on execution, otherwise the leverage is negative on execution.

Reassess this periodically. Yesterday’s great dependency could be today’s liability.

One of the many things they discussed in the interview is the contrast between the state of flow, and deliberate practice. Cal says that some people have been trying to reconcile these two approaches, but he argues they are actually quite different.

Flow is a state of mind where one performs some task intuitively. Time flies by. The end result may be high quality, perhaps the best one can do (at a given time).

Deliberate practice, on the other hand, is a state of peak exertion. It is trying to learn something you don’t know yet, or improving some weaker aspect of it. Time does not fly by, in fact every second may be excruciating. The end result is getting better at the thing being practiced.

Flow is about performing at peak capacity, while deliberate practice is about lifting the peak higher.

Neither states can be achieved if distracted, but that does not make them one and the same.

Cal Newport is the author of Deep Work*, a book I’ve read and heartily recommend.

*As an Amazon Affiliate I earn from qualifying purchases. You can support my blog by buying through this link.

In order to properly benefit from time off, it is better to disconnect fully, so it is commonly accepted wisdom that one shouldn't read emails on the beach. But in addition to that, I recommend not reading emails that came in during that time at all.

Why does this work? Because the people you work closely with should already know that you have upcoming time off, and have already gotten any time-sensitive context from you before that. And those you don't work closely with should be able to get what they're looking for from your teammates.

If your bus_factor == 1, fix that. Let others lead too, make yourself replaceable. Be a good example for your team by NOT being online while you're on vacation. Come back fully recharged, and encourage your teammates to do the same.

Here's the template I've used for my auto-responder for many years:

Hi,

I’m on vacation until <date>.

This email will not be read, so please slack me after this date if it is still important then.

For support, please email <team_support_email> or consult the #<team_support_channel> slack channel.

For anything else, please email my manager at <manager_email>.

-F

Feel free to steal it, tweak it, use it!

Organizations that pursue leverage may want standardization, reuse, specialization and separation of concern. Organizations that avoid it may instead reward ownership, tight integration, scrappiness and waiting for no one. Both have pros and cons. Regardless, it would be desirable for the industry to have a vocabulary to discuss leverage.

We should evaluate how leveraging something affects execution and craft:

• Execution: faster/easier versus slower/harder

• Craft: higher versus lower quality

These two dimensions are fundamentally independent, and both can be either positive or negative. We can thus map four quadrants, with craft as the horizontal axis, and execution as the vertical one:

Some organizational cultures assume that leverage is miraculous (top right quadrant) while others think it is catastrophic (bottom left). However, in reality, the most common scenarios are the top left and bottom right ones!

We can make better leverage decisions by considering these tradeoffs. A given project may benefit from quick and dirty execution leverage, and be negatively impacted by the detour of craft leverage. In another project, the reverse may be true. Of course, one should always try to avoid catastrophic leverage.

Get in the habit of asking: what kind of leverage is it?

During early iterations, we may come up with a design which is strictly better than another, but we rapidly exhaust these low hanging fruits. We end up with many local optimums, without a clear global one.

If we conclude that one these alternatives is the global optimum, it is often a contextual judgment. In other words, we are faced with tradeoffs, and decide that some dimension is more valuable than another.

Once we reach the stage where we compare refined alternatives, then we usually don’t have any opportunity left to reduce complexity, but rather, we merely move it around. We may think we reduced it but that could be an illusion due to complexity having moved outside our scope of ownership (e.g. we dumped the complexity on the users, instead of solving it).

Software is like a thermodynamic system. You cannot just increase temperature while keeping everything else the same. Increasing temperature will also increase pressure. If you increase temperature while keeping pressure fixed, then you’ll need a larger volume, and so on. Something's gotta give.

How do you navigate these design comparisons when something seems too good to be true and some side effect may be hidden?

Then scale was all the craze, both technologically and organizationally. It was not good enough to replicate your monolith; the cool kids also split theirs into microservices. We’ve been in this era for more than a decade, but the pendulum is starting to swing back the other way…

In large web companies, a request hitting a front-end can then touch hundreds or more mid-tier, back-end, cache and database servers. This is the epitome of the microservices architecture, but also where the edifice starts crumbling under its own weight. By compartmentalizing, we built a Conway-esque symmetry of the sprawling org chart. Unfortunately, it is impossible to make this many servers work flawlessly, quickly, all the time. Tail latency degrades the more moving parts are involved.

Microservices are becoming bulkier again, because their owners don’t want to pay the latency and reliability costs of RPCs. Services which heavily query databases can be refactored to eagerly cache the entire dataset.

Can a stateful service really be called "micro"? Or is it more of a macroservice? Or perhaps a microlith? Minilith. Multiliths?

It’s a brave new world out there.

In other words, the use case ceases to exist if the cache is down. This can be due to excessive throughput hitting the database, thus taking it down as well. Or it could be due to the clients being incapable of operating efficiently enough when too many requests are served slowly (e.g. due to threads being tied up too long and thus causing backpressure).

This risk can be mitigated to some extent by always reading from both the database and the cache (rather than only on cache misses), thus ensuring that the database must be provisioned for full throughput, but this doesn’t guarantee that clients can gracefully sustain degraded latency.

If a cache is existential, then the uptime of the end-to-end architecture equals:

db_uptime * client_uptime * cache_uptime

Where cache_uptime means the proportion of time where the cache hit ratio remains above the level required for the use case to survive.

A non-existential cache, on the other hand, provides the more favorable end-to-end uptime of:

db_uptime * client_uptime

Do you leverage existential caches in your architectures?

If an object’s capabilities are deemed too powerful to pass around nilly willy, we may want to constrain the scope of what gets passed to the bare minimum needed by the recipient.

One way to achieve this pattern is for the powerful class to implement an interface containing only the functions to be exposed. Then the recipient can specify this interface as the type of parameter it receives.

Another way, if the scope of interest is just a single function, is to pass a handle of that function.

These techniques can be used to reduce coupling between classes. Instead of a class having the potential to call any functions of another class, it gets to call only a subset. Increasing the scope of the interaction requires widening the keyhole, which is an opportunity to reflect on whether the class structure has grown too tangled.

This last one, decoupling, says that two components interacting together should know as little as possible about each other. This is typically achieved via an abstraction. A recent post on LinkedIn reminded me that some folks really love abstractions.

Many years ago, I heard a good rule of thumb, which stuck with me:

A well-designed abstraction needs at least three implementations.

The issue with an abstraction that is backed only by a single implementation is that it will almost inevitably be over-fitted. The abstraction will closely espouse the shape of its one implementation and will therefore not be very generic, thus making it awkward to plug-in alternative implementations under the hood. Leaky abstractions are an example of this phenomenon.

This does not necessarily mean we should never write abstractions with just one or two implementations, just that it might be a red flag. Do we really need it, or is it a premature abstraction? Do we need to think extra carefully to avoid over-fitting? What future implementations which don't exist yet might we want to plug-in, and do those inform the design of the abstraction?

What if, instead, the goal was to describe how the system should work?

Surely, it can’t be right to treat documentation like a fantasy novel. It does need to be factual.

And yet, if we write how the system should work, then analyze the gap between reality and this fantasy, we can go and plug that gap.

Or put another way, if one writes factual documentation, and struggles to make it easy to understand, it may be that the issue lies not in the doc, but in the thing being documented.

A similar idea is that your FAQ is your bug list. Why does this question need to be asked in the first place? Can’t the system be sufficiently intuitive to side step the question completely?

In the end, it may be possible to document how the system should work and how it does work, though it may require some iteration(s) of tweaking the system itself to get there.