Digital Transformation: When to Talk and When to Listen
As we approach the one year anniversary on my current digitization project at work, I look back to see what can be learned from the experience. Project SWIFT was aimed at providing information about our global manufacturing plants to the product developers earlier in the development process. The output would be a data collection tool and a business intelligence report in Tableau. The objective was quite simple, but we encountered constant obstacles at the micro and macro level.
At the macro level, there were huge shifts occurring in the organization. Our department was undergoing a strategic shift to the cloud and becoming more data driven. This included significant organizational restructures. One of which was a new data analytics team that was positioned closer to the business. This new team absorbed my position. For new business intelligence projects, such as SWIFT, my team took the lead on the technical side. Previously, the business side would drive all requirements while IT would handle the technical side or any communication with a 3rd party for development. There was a new dynamic at play with uncertainties abound as to who takes the lead on ambiguous requirements and design decisions on project SWIFT. I believe this macro level change was the foundation for the many challenges on this project.
At the micro level, I was at the center of all discussions centered around gathering the business requirements. Most of the data required for the final report did not exist, so the early conversations were about what data needed to be captured and the relationship between each item. The initial request translated into 35 different data fields. Immediately, I was pushing back to reduce this large undertaking for a number of reasons. Besides the huge time commitment this would take, we were employing an agile methodology to get this project live as swiftly as possible. As soon as we agreed on a subset of data fields and made progress on their relationships, we hit a major obstacle.
The project added new subject matter experts who could provide more specialized knowledge from around the globe. The result of this expansion was a reality in which new data fields were requested and the previously settled data relationships were now questioned. Every ensuing meeting had ten or more people attending. You had to be selective about which questions to ask and how to present the request clearly. I remember constantly tweaking my approach to uncover which pairs of data fields had a many to many relationship. Sometimes my requests were interpreted incorrectly, but it was never entirely clear when I should interrupt a conversation. The best I could do was listen and wait for the right opening to correct the course.
As of today, we are still working on the project, but I am happy to say that we are live. While there are still a number of ongoing challenges to improve the data collection tool and the report, we have made significant progress from the start. The scope of the project was eventually reduced to 9 fields and the data model was designed in third normal form to provide flexibility for any future expansion of data collection. Without the constant pushback from the data analytics team to reduce the complexity of the project, I don’t think we would be anywhere near “go-live”.
This project provided many valuable lessons. Most of all is to understand the ambiguous nature for when it is our time to talk versus our time to listen. Steve Jobs said it best with the following, “Our job is to figure out what they’re going to want before they do”. As a data engineer, sometimes you need to think like Steve Jobs in order to guide the various stakeholders down the right path of technical simplicity, while still addressing their requirements.
~ The Data Generalist™
Data Science Career Advisor
Note: This article was published by O’Reilly Media in 97 Things Every Data Engineer Should Know.