#27 - Always Be Aware of Changes, Always Be Architecting
Always Be Architecting is one of the fundamental principles of Data Engineering, advocated by Google and amplified by Joe Reis in his book Fundamentals of Data Engineering.
The Google Cloud team believes that “As a cloud-native architect, you should always seek to refine, simplify and improve the architecture of the system”. This is the essence of the Always Be Architecting attitude. Emphasising the drivers of this attitude, Joe Reis stated in his book that data architects “constantly design new and exciting things in response to changes in business and technology”.
Indeed, we should always be architecting as changes are unavoidable. However, changes are not operational failures or future scale. These issues should already be addressed by the current architecture. In fact, Plan for Failure and Architect for Scalability are two other fundamental principles of good data architecture.
So how is Always Be Architecting different? This principle places emphasis on an important responsibility of the architect: be aware of changes.
What kind of change should an architect look out for?
Technology advancement: These are improvements of cloud providers capabilities and additions of components into the data stack. The latter is undoubtedly a prominent theme, proven by the Cambrian explosion in the data stack in the last few years. Data architects should stay on top of these technological advancements, and look out for opportunities to reduce the current workload of engineering teams, or solve more technical problems. Check out this piece from Andreesen & Horowitz, which shed lights on the key changes in the data technology landscape in the last few years.
Shift of use cases: The number of use cases for data will inevitably increase when an organisation grows, changes, or moves forward in its digital transformation journey. From ad-hoc data discovery, automated reporting, to embedded AI applications, the shift in use cases demands the enterprise’s data architects to adapt, either by adding new components into the data stack, or utilising existing components for different purposes.
Organisational structural changes: Finally, certain significant events will require architectural changes. This type of change may be less common, but not less important. Merger is a typical example. When two previously separate entities merge into one, it is important to harmonise and consolidate data representing the two entities into an united whole. The architect will need to work out how to reflect this identity change in their data model, and prepare for a smooth transition between the two chapters of their organisations.
The identification of changes helps the architect envision a suitable target architecture. Noting that this is not a static target architecture, but rather a moving one. Responding to a constantly changing environment, the architect will need to be agile and regularly update the priorities of changes. However it is better said than done, as these types of change often require investment, both in time and money. There is also a trade-off: moving fast may lead to duplication and silos, while consolidating effort on a centralised data platform will eliminate duplication but is more time-consuming.
It will take longer to go into these problems, but I’ll leave you with a final thought:
“It’s never been easier to get a new data application adopted, and it’s never been more important to properly maintain the platform.”
That’s it for this week! If you enjoy or get puzzled by the content, please leave a comment so we can continue the discussion. Throw in a like as or share as well if you know someone who may enjoy this newsletter. Thanks!