While working on an enterprise platform, one of its customers in the banking industry wanted to migrate from on-premises to running it on whatever cloud was popular at the time.
The migration was long, tedious, and overly complicated in its own right (i.e. one proposed solution for the "how do we migrate all data safely across the continent?" question involved armored trucks) but just as we reached the T-1 day mark, I realised we had forgot something.
The customer was regulated by various entities, and so it had to deliver periodic audit logs in a particular format. The raw logs (stored in a cloud hosted bucket) would not be sufficient and had to be parsed; in order to process the logs into the desired format, the customer wrote thousands of lines of code in the platform that I was in the process of migrating. This code could only run on the platform, due to some other esoteric privacy regulation.
So there I was on a Sunday, with :
- a few hours to deliver up-to-date, formatted audit logs to regulatory entities or risk legal action
- raw logs in a cloud bucket that required ingestion and processing
- a new cloud platform that could process the logs but was unable to ingest data from the other provider's cloud bucket (due to some temporary allowlisting ingress/egress issue and this being one of the first migrations onto the new cloud)
- an onprem platform being decommissioned and no longer allowed to process the logs BUT capable of ingesting them
The solution I came up with was to have the data flowing:
log bucket in cloud provider A -> decomissioned platform running onprem -> connector I wrote that evening and had no time to test -> platform running on cloud provider B
The ship was afloat the next morning and everything was in order despite cutting it close; I am now a big fan of exhaustive planning, months in advance.
The migration was long, tedious, and overly complicated in its own right (i.e. one proposed solution for the "how do we migrate all data safely across the continent?" question involved armored trucks) but just as we reached the T-1 day mark, I realised we had forgot something.
The customer was regulated by various entities, and so it had to deliver periodic audit logs in a particular format. The raw logs (stored in a cloud hosted bucket) would not be sufficient and had to be parsed; in order to process the logs into the desired format, the customer wrote thousands of lines of code in the platform that I was in the process of migrating. This code could only run on the platform, due to some other esoteric privacy regulation.
So there I was on a Sunday, with :
- a few hours to deliver up-to-date, formatted audit logs to regulatory entities or risk legal action
- raw logs in a cloud bucket that required ingestion and processing
- a new cloud platform that could process the logs but was unable to ingest data from the other provider's cloud bucket (due to some temporary allowlisting ingress/egress issue and this being one of the first migrations onto the new cloud)
- an onprem platform being decommissioned and no longer allowed to process the logs BUT capable of ingesting them
The solution I came up with was to have the data flowing:
log bucket in cloud provider A -> decomissioned platform running onprem -> connector I wrote that evening and had no time to test -> platform running on cloud provider B
The ship was afloat the next morning and everything was in order despite cutting it close; I am now a big fan of exhaustive planning, months in advance.