Firefighting in the Dependency Hell and Notes about Managing Dependencies
Recently, LinkedIn asked me to share my insights to be eligible for the “Top Voice” badge. I realized that I have something to share with my followers in a more extended way than typical (not ChatGPT-generated) answers to those top-voice questions.
Definition and classification of dependencies
As usual, let’s start with the definition, which is rather made on my own. When you need someone to do something so you can do your work, that is a dependency. Postponing the dependency means a delay in delivering your work and the expected outcome, which causes financial losses.
Your team/service can have one or several dependencies and be a dependency for someone else. Sometimes it can be both. And dependencies might be temporary when you need something to be done once or permanent when you continuously need something to operate.
There can be external dependencies, meaning that you are waiting for something to be done outside your organization. Or that will be done inside your organization, which is an internal dependency.
We can distinguish cross-team, cross-stream, and cross-department dependencies, each with its own peculiarities.
In my world, dependency means an API or a software library. In your world, that may be anything else, like a vehicle part or a plumber to fix a pipeline leak.
Dependency is a risk
Obviously, it is the most common and unpredictably dangerous risk. Dependency also means you are limited in control and authority to influence all aspects.
You can’t eliminate all the dependencies, as a system can’t act in isolation. It uses different systems on its level and is also embedded into higher-level systems. There is an entire discipline about that, and I have an essay on a book about it.
But if you can proceed without a dependency, you should try. If having an external dependency is economically justified but related to a business-critical part of the system, then it is better to proceed on your own. That is the reason why enterprises do a lot of their stuff in-house, even though it would be reasonable to use open-source or paid vendors.
Inside an organization, we should avoid duplicating functionality even though it is inevitable on a big scale. With an internal dependency, you should ask yourself twice whether you need it to achieve your goals and whether you can rely on those who commit to delivering it for you.
External dependencies
There is a simple rule: if an external dependency has no contractual obligations, then it is a void promise. Don’t believe a salesperson who promises you a feature to be delivered in Q4. However, having a contractual agreement does not guarantee that a dependency will be resolved in time. Legal obligations are additional motivation, and it is better to have them than nothing.
You don’t have much authority and control over the outcome from your position as a dependent. You provide clear input and your expectations. Then, you have a right to accept or deny the outcome if that does not satisfy your needs. Losing time and money in case of negative scenarios cannot be fully compensated.
Internal dependencies
Let’s clarify that a contractor’s team is an external dependency as it is technically outside your organization. If there is a team that consists of contractors inside your organization, then it is an internal dependency.
Compared to external dependency, you have more control and the ability to drive things in your expected way. However, if communication between teams is broken, it will still be a challenge.
The main advantage is that you can use administrative leverage: basically, call big guys to press on other guys to make them do what you need in the desired terms. But it may backfire in different ways: put yourself in a vulnerable position for higher management or deteriorate your relationship with folks on “the other side.”
Key thoughts on managing dependencies
Transparency
- Establish all dependencies in a work management tool you are using (e.g., Jira) with ETA (Expected Time of Arrival).
- If an external dependency owner has no access to your “Jira,” then create a placeholder and update it there.
- Build a dashboard or an Excel spreadsheet to have a comprehensive view from top to bottom.
- Be able to respond at any time by explaining the current state of affairs for any stakeholders.
Communication
- Keep weekly catch-ups for internal dependencies and bi-weekly for external ones.
- Define clear acceptance criteria; don’t rely on someone else to do it instead.
- Take time to communicate things that seem to be obvious to you.
- Take time defining the naming convention to be aligned with terms and their definition; making a glossary is a good option.
- Remember, you don’t have the knowledge and resources your dependency owners do. Otherwise, you would do it by yourself.
Timeline and commitments
- Consider risks of delay due to dependencies as they are almost inevitable.
- Consider working in parallel in time-sensitive cases where possible.
- Define and strictly follow API contracts established between the parties.
- If you did some work ahead of delivering dependency, be ready to change.
- If you are a dependency and have a dependency for yourself, don’t commit until you confirm the commitments of your dependencies
Escalation
- Know an escalation path both on your and dependency’s side when things go wrong
- Be quick to escalate.
- Avoid taking responsibility for what is beyond your control.
Firefighting in the Dependency Hell: a Case Study
Disclaimer
The following case study focuses on dependency management, omitting other details that are irrelevant to the topic or might be sensitive to share publicly.
I do not necessarily speak on my behalf. Let’s assume some Product Owner (The PO) appeared in such a situation and acted in a certain way to resolve the dependencies.
Background context
There are a few details to be mentioned to provide context:
- PO joined an ongoing project at its critical stage. Thus, there was no room for cross-organizational changes.
- PO had a flexible number of responsibilities, which might vary considerably from that of the vanilla Scrum product owner.
- PO had a great team; it would be impossible without them.
Dependencies structure
This is what the team topology looks like, approximately. And that impacted how dependencies are distributed.
Three organizations are involved, each with their own development teams and management. It is obvious that each side pursues its own goals in accordance with contractual obligations.
There are five architecture layers, with a number of teams on each layer. It is not described on the diagram (to keep it simple), but a team within the layer might belong to a different specialization, responsible for one or several microservices. Each specialization has its respective PO, Architect, Delivery Manager, QAs, BAs, etc. They have their own scope, backlog, and priorities.
They are, from bottom to top:
- The Platform layer provides the core capabilities of a system. It exists outside the project.
- The Feature layer uses, customizes, and extends the core capabilities.
- The API layer that exposes RESTful API to be consumed by upstream layers.
- The Integration layers consume those APIs and provide data enrichment from 3rd party services.
- The Customer layer utilizes the final output for its own purpose.
Our heroic PO is trapped in the middle, being on the frontline of their organization and managing the API backlog for two teams. We will look at dependency management from that perspective.
Horizontal and vertical dependencies
There are 3-week Scrum sprints, so a feature path through each layer takes much time. From the horizontal perspective, there are some scenarios:
1) Core functionality without deviation: starts from the API layer.
2) Core functionality with customization: starts from the Feature layer.
3) New Core functionality: starts from the Platform layer.
The Platform has its own per-quarter release cycle with an option to provide intermediate releases on-demand within a quarter. However, there is an additional buffer time (aka upgrade) to deliver a Platform release and ensure it works with the existing customizations.
So, case 3 is more unpredictable and depends on multiple criteria. Case 2 is the most common, so our story focuses on it.
Starting from the Feature layers, it takes four sprints (12 weeks) to deliver a feature to the top. Plus, we consider end-to-end testing and other production readiness procedures. So it might take even more time.
We also consider the Horizontal dependencies within the team in the same layer. They could work in parallel, but sometimes, they must wait for their peers. So delivery might be postponed even further.
Shrinking the gap
The first step will be prioritizing work based on the above-mentioned scenarios. First of all, the PO decides to identify and prioritize what is already available without any downstream dependencies. Next is what is available from the core but requires customization of the feature layer. And last, what is not present in the core must be delivered by the Platform.
Secondly, we need to parallel the work to shrink the delivery timeline as much as possible.
In this case, it is possible for the API and Integration to work in parallel: the API teams provide an API contract for each future-coming API endpoint or expected changes for the existing one. Teams can implement based on the contracts but not merge into the main branch.
An alternative strategy would be exposing a mock API (i.e., API with mock data). That was a meaningful decision by engineering leads not to merge any mock code. We later discuss what downsides it would cause.
At some point, due to some postponements and waiting for new Platform core pieces, three layers are forced to work in parallel.
The API and Feature layers align on the API contract between them. And that is not the same API contract that exists. So, the PO has to manage two types of API contracts across the parties.
An advantage here is that all four upstream layers act as a Platform’s customers, so they request a design that does not require customization and skip the feature layer. However, the Platform, having many customers, has the right to provide a generic solution, not a custom feature that suits one customer but another one.
Even though they managed to pack an ideal delivery timeline across four technology and organizational layers, the cost is high:
- Excessive communication and efforts to manage parallel work.
- Inevitable reworks and last-minute changes in the API contracts.
- Communicating and enforcing all parties to keep the API contracts.
- “Communal” testing that leads to multiple bugs.
- Late coming or missed requirements.
- Fighting “who owns what” as each organization pushes its own agenda.
That parallel work looks pretty good on paper, but that is a fragile construct that is far from smooth operation.
Art of extreme facilitation
The API layer has to identify requirements from the upstream layers and pass them to the downstream for further gap analysis to see whether a query fits one of three scenarios. The API team lacks the specialization knowledge that the Feature team does, and they both lack the Platform knowledge.
Being responsible for requirements, the API layer’s PO has to communicate vertically between up and downstream to come up with a design that satisfies everyone and the server business needs of the customer. That means bringing everyone to one table and facilitating all parties to be productive and reach a particular conclusion.
Along with that, horizontal communication across teams in the same layers must be managed and facilitated as well. Different specializations often cannot align on something without a 3rd party facilitator, and the API folks fit that role.
After concluding the design, the next step is to make sure it is prioritized accordingly, both vertically and horizontally, and closely monitor any deviation from the plan. Those deviations are inevitable.
What if…
The PO thinks the main problem is the team structure, which complicates communication. Having five layers, where one layer is a bottleneck, is too far from being effective.
The Customer layer should merge with the Integration, and API with Features should make the cross-functional dedicated team serve API and core customization within the required specialization. That would require additional training and learning from each other, but it would benefit everyone.
The PO thinks that a dedicated API team is bullshit. And I can’t agree more after reading the Team Topologies book (I wrote about it here).
The Platform remains a constant as it exists beyond the project, and there is not much to do with it from that perspective.
The PO would also proceed with mock API instead of API contracts even though it considers some additional development and testing efforts (make a mock, test it, replace it with the actual implementation, and test it again) and risks of forgetting to remove mocks. In the end, maintaining and communicating API contracts takes the same effort, if not more.
Conclusion
The current case study explores how having multiple layers complicates effective delivery and causes multiple dependency downsides and demanding commitments to the upstream. Any changes to improve the process with good intentions mainly intensify the agony and firefighting to communicate with all parties to resolve dependency in time.
The management should carefully structure teams before the project starts and consider some flexibility for further changes down the line. Major restructuring in the late stages would put the project at risk. Decide wisely.
Originally published at https://ilyazakharau.com on April 7, 2024.
