Mastering AI Department Reorganizations: Lessons from the Trenches | by Elad Cohen | Jun, 2024

Do’s and Dont’s after five years of Data Science department reorgs

During the past five years, I’ve served as the VP of Data Science, AI, and Research at two publicly traded companies. In both roles, AI was central to the company’s core product. This provided significant resources and the opportunity to lead substantial departments — comprising 40–50 data scientists, including 2–3 group leaders and 6–8 team leads. One of the greatest challenges in this role has been structuring the department to enhance effectiveness, streamline value, and clarify roles and responsibilities. Today, I’ll share some best practices I’ve gathered through six departmental reorganizations.

Your guiding principle throughout the process should be the business need. What value is your organization creating? Are you developing models for a core product, or supporting the business with internal ML solutions? Identify the specific value your department provides and the key performance indicators (KPIs) used to measure success. Ideally, you should have explicit KPIs, but if these are unavailable, develop implicit measures to guide your efforts (i.e. how are you measuring yourself).

Once you’ve clarified the business value, map out the entire value stream for which you are responsible. This includes all processes from the initial inputs to the final product your organization delivers. For instance, if your unit creates ML models, your inputs might be data from a company database managed by another department. Your tasks could include cleaning the data, training the models, evaluating the differences, and productionalizing a final model object to handoff. Perhaps you also deploy to production and monitor the results, or you hand it off to an Engineering / ML production unit for deployment. The process involved in your value stream could require several teams, with multiple handoffs until complete. You’re going to try and optimize the throughput of your value stream — how can you make this process faster and more efficient. That will usually mean reducing the number of handoffs between teams to the minimum number possible. Handoffs between teams are almost always a source of inefficiency, but unless your value stream is very simple, you won’t be able to avoid them altogether.

Next, break down the value stream and supporting functions into smaller sub-units (groups or teams). Start by creating an idealized structure, outlining the roles and responsibilities of the various units. At this stage, I highly recommend to avoid considering personnel assignments. Avoid other constraints like aligning with other departmental structures at this stage. For example, I’ve been in companies where some departments were structured by product lines, while others by function (e.g. one team in charge of working with data vendors, regardless of product). This can require more communication overhead, but you don’t want to throw out alternatives just yet.

To see how the structure holds up, generate a list of use cases — a few projects that fall within the main value stream, and some exception cases you’ve come across recently. Walk through the process — how are they handled in the current structure? How would they be handled in the new structure? Have you reduced handoffs? Reduced decision-making and/or pushed it down in the hierarchy? Can teams handle it end to end? What new difficulties have been created?

One of the greatest challenges I’ve come across at this stage is that the main value stream is too great to be handled by one team (i.e. requires 8–20 data scientists). Breaking up the value further into orthogonal “mini value streams” isn’t a straightforward process, but one key direction that has served me well with classic ML models has been splitting the gains to Data, Features and Algorithms. One team can optimize the algorithm being chosen (including target encoding, feature selection, hyperparameter tuning and more), while another can focus on improving the quality of the data — cleaning the labels, supervising the label creation process (e.g. deciding which observations to be manually labeled), choosing the most representative data sets, weighting the observations, etc. Additional teams can focus on feature engineering, further breaking them out by the types of features involved (here again, it’s important to make sure you maintain orthogonality in the work done). This method enables multiple teams to contribute independently to the same model, leveraging many more people to squeeze out greater performance in your models. I’ve used this approach to leverage as many as 6 teams collaborating and independently improving the same xgboost model.

At this stage, seek feedback from other managers, both within and outside your department. If the changes are significant, avoid discussing specific personnel placements to ensure objective input. You want alignment on what’s the right structure for the organization, not building it around specific people. The more you can involve your people in this process, the greater buy in you can get. By understanding what you’re optimizing (the value stream), and the constraints you may have, they’ll be able to buy-in to the change you are making. This is especially true if they end up getting disappointed by some aspects of their new role. In one case, this approach enabled one of my Directors to accept and even champion a new structure, even though it reduced some of his roles and responsibilities. On the other hand, avoid consulting too many people. The sheer thought of an upcoming reorg can dramatically increase anxiety among your people.

After you feel that your new structure improves most of the current (and expected) use cases, and others agree with your analysis, begin working on placements. This stage varies wildly depending on circumstances — whether you’re growing, downsizing, or merging with another unit. As such, I don’t have a step-by-step play for this part, but do offer some guidelines:

• Keep key personnel on board: you want the right people on the bus (and consider giving them bigger roles), and the others off the bus. You need managers who can align with your change and adapt to their new role.
• Foster a growth mindset: more often than not, we think of an employee based on the current role they’re doing, not giving enough consideration to their potential to grow. Some of the greatest satisfaction in my career has been promoting someone to a new team lead or group lead role and witnessing them stepping up in a profound way, well above my expectations. I’m immensely proud whenever I get a LinkedIn notification that one of my former employees has been further promoted or is moving on to a bigger role elsewhere, and I was able to help them with that early break.
  If you can, take a chance on your people and emphasize potential and attitude, at the expense of experience or company know-how. The latter will be gained over time.
• Be human and empathic: some changes may require a manager to step aside from their current role, taking on a smaller scope. Some managers may switch back to an individual contributor role (which they excelled at, and can excel at again). This is not a demotion, and shouldn’t be seen as such. This is a lateral transition from the managerial ladder to the individual contributor ladder. Any transition can be difficult on people, so treat them with the utmost respect, explain the rationale, and if relevant do your best to help them maintain dignity.

Once you have decided on a draft of the new placements, identify the new challenges. At this stage, you may need to tweak the original roles and responsibilities. Perhaps you have a Group lead with a team in charge of a very technical, legacy system. In one case, I kept such a team under the same GL as a default, even though the roles and responsibilities didn’t match perfectly. Instead of adapting the group’s definition and changing their responsibilities to make the transition appear as if there were some deeper rationale, I was open and explained that this decision was based on that GL’s personal experience (you can make compromises, but it’s best to be transparent so others understand this is the exception to the rule).
Make necessary compromises, but make sure you don’t erode to much of the expected improvement. During the first major reorg I tried to carry out, I took this step way too far. I shifted responsibilities between managers to appease them and gain more buy-in to the process. After several iterations, I had come to the conclusion that the change no longer improved the value stream and there was no coherent narrative to the change offered. At that point, I called the whole thing off for a quarter and restarted it with the resolution not to make the same mistake again.

Create a communication plan detailing who needs to be informed, by whom, what will be communicated, and when. If someone is moving to a new manager, their current manager should inform them first, followed by a meeting with the new manager within a few hours. Information spreads quickly, so your communication plan should be swift and well-organized. You never want someone to hear about their new role by rumor. Review the plan through the eyes of the various people involved. For example — if an IC is told they’ll be moving to another team, when do you talk to the rest of the team (even if there no changes planned for them)? Everyone should be communicated eventually, even if it’s to explain the change and assure them that their role is unaffected.

With your written communication plan in place, you can be sure you won’t forget anyone and no communication takes place out of order.

In addition to one-on-one meetings, communicate the overall narrative to the entire organization — why the change is happening and what exactly will occur. Allow ample time for Q&A. If you don’t have an open-door policy, offer one in the aftermath of any large change. Proactively reach out to everyone affected to gauge how they’re doing.

• If you operate with a quarterly plan, it’s usually best to carry out the communications before planning for the next quarter, effective from the beginning of the next quarter. For example, you might communicate the changes March 1st, starting April 1st. During the remainder of the quarter, the existing teams can finish the quarter’s tasks. In parallel, employees will plan their next quarter’s tasks with their new teams.
• Have a plan B prepared for personnel assignments. Some people won’t be onboard with your changes. When creating your communication plan, you should note their responses and the order of the communication to de-risk your plan. The higher the risk the person will reject the change, and the more critical their role, the sooner you want to discuss with them. Avoid sharing too many details as this can be an iterative process, and you want to maintain a measure of flexibility.
• Pick up the book Team Topologies: Organizing Business and Technology Teams for Fast Flow by Manuel Pais and Matthew Skelton for more ideas on different team structures and value streams.

By following these guidelines, you can structure your department to enhance effectiveness, streamline value, and clarify roles and responsibilities, ensuring a smoother transition and greater overall performance.

Are you a data science leader undergoing or contemplating a reorg? Feel free to connect over Linkedin. Always happy to share my 2 cents.