As a developer, you must have heard about the problem that plagues even the most organised companies. It's called - Technical debt.
There is enough literature available on the internet to create awareness about technical debt and how it can affect the progress of a product and, consequently, the entire business. But, every article stresses a different solution. You will typically find the following list of solutions -
Regular Code Reviews: Conduct regular code reviews to ensure code quality, identify potential issues, and share knowledge among team members - While this is a must, it is highly dependent on the experience and knowledge level of the reviewer. And that cannot be consistent across an entire team.
Automated Testing: Implement comprehensive automated testing to catch bugs early and ensure the reliability of the codebase. - Automated testing, especially unit testing techniques like TDD, is a luxury for startups and mid-level organisations. Most of the time, such organisations don't have the time or resources to implement, and changes are too frequent.
Continuous Integration and Deployment (CI/CD): Use CI/CD practices to automate testing and deployment processes, making catching and fixing issues early easier. - In my opinion, this should be implemented irrespective of your organisation's size, especially since CI/CD solutions have become much cheaper and easier with time. This is a one-time effort that pays off in the long run. However, the automated testing part of CI/CD often remains unimplemented for the reasons I mentioned above.
Refactoring: Regularly refactor code to improve its structure and maintainability. Addressing technical debt through systematic refactoring helps prevent its accumulation. - Again, this highly depends on whether you have enough resources or structured systems to perform refactoring occasionally. It often becomes impractical if your systems are changing too frequently.
Educate and Train Teams: Ensure team members understand the importance of code quality and the long-term consequences of accumulating technical debt. - This can be done with simple motivation and structure, and it's achievable irrespective of the resources available in the organisation. Most knowledge is available for free, and if the team members systematically up-skill themselves, the same will be reflected in the quality of their output and overall system.
Agile Practices: Embrace agile development practices emphasising iterative development, collaboration, and adaptability. - I will not even bother expanding this. This is a no-brainer!
Documentation: This is the solution we are going to talk about in this blog post. This is the easiest and the most efficient way to avoid and reduce your technical debt and ensure a high-quality output.
This blog post will give you a template for defining your engineering requirements that has worked exceptionally well for me. It will help you structure your solution better. I have used it to describe and build several products that solve complex problem statements, so it's battle-tested.
As a startup developer, I never had access to big engineering teams. I learned the hard way that the best way to avoid problems, as mentioned earlier, in the long run, is to structure the product problem statements, engineering requirements and the What, Why and How of the solution well before writing a single line of code. This template will help you if you are an engineering manager or developer.
In one of my previous posts, I gave a PRD template for structuring your product requirements. Please check it out if you haven't done already. Think of it as a precursor to planning your engineering product. If your product problem statement isn't defined well, this ERD template won't be as effective in solving your problems as you would ideally want it to be.
How to use this template?
Before I share the template, let me share a few pointers on how to use it. The idea is to avoid ambiguity while using this template.
Define a single Engineering requirement document (ERD) for each product. If you are extending or changing the product, make changes in the same document. Don't create a separate doc for new features or modifications to the existing system. It will cause issues in the long run.
Most tools where you write the ERDs have a versioning system that can fetch the document's historical version if needed. Please ensure your tool has this functionality, too. If not, do a manual backup, or if you are a developer, you can always use a Git repository to do versioning of the engineering documents.
If your product depends on an existing product or system in your organization, mention the links, constraints and considerations of the older product in the new ERD.
Write down the considerations you took into account for a part of your solution, especially if your solution is experimental- a template of such considerations can be - “I took this decision to do <X> like <Y> because of <z> reason. It will help you iterate on the solution better.
Be as detailed as possible. The devil is always in the details. The template below describes what you are supposed to write at a high level. But it doesn't dictate what level of detail you can go up to. That's your choice. And trust me when I say this- the depth of your thought would count. The depth of your solution will contribute to the long-term success of your product.
If you follow the points mentioned above religiously, you will end up with an ERD document that efficiently solves 99% of the communication, scoping and product quality problems and will preemptively eliminate all the technical debt you could have incurred in the long term.
Engineering requirements template
*************** template starts here ********************
1. Product requirements
Write a high-level summary of the product requirements derived from PRD (product requirements document). If it's already present in the PRD, copy and paste it. Mention the link to the PRD.
Mention the link of the UI/UX file (Figma link, etc.), if any.
2. Functional requirements of the system
2.1 - Clarify ambiguity
Read the product docs and understand each feature well.
Think about all the edge cases from the product perspective and check if they have been properly explored in the product doc or not.
If there are any dependencies on third-party tools and services, please ensure they have been explored properly in PRD. Classify these dependencies into one of the sections below (functional requirements/ considerations/ constraints/ engineering problems to solve) and detail them in that section.
If there are any dependencies on existing systems (any existing internal services), please ensure they have been explored properly in PRD. Classify these dependencies into one of the sections below (functional requirements/ considerations/ constraints/ engineering problems to solve) and detail them in that section.
2.2 - Write each functional requirement clearly from the engineering perspective.
Engineering functional requirements essentially translate the product requirements into their tech counterpart. For example, if the product requirement is the “Login flow of LinkedIn”, in that case, the functional requirements will be the high-level description of all the APIs associated with that user journey. Please write down the possible engineering edge and corner cases and how to handle them as a requirement.
2.3- Define what is out of scope from the product and engineering perspective
Check the PRD for things that are out of scope from the product perspective. Please ask the Product/Engineering manager to update all the out-of-scope items if they need to be written. If they are already written, copy the link to that section of PRD here and then write down all the things that are out of scope from an engineering perspective.
These should be everything you factored out while designing the solution in this document.
2.4 - Write down the system considerations and constraints from an engineering perspective.
This section should create a list of -
All the constraints of the engineering systems - Write down all the things this system won’t be able to do. Also, mention why it won’t be able to do that.
Write down all the considerations you took into account from a non-functional and functional perspective for designing the solution later - The template of considerations is - I took this decision to do <X> like <Y> because of <Z> reason
2.5- Define any specific engineering problems to solve
This section should describe all the things not directly related to the product problems but are needed on the engineering side to solve the functional requirements mentioned. It can be as simple as ” We need a URL shorter system for this” or as complex as “Our existing pub-sub system won’t work in this case because of XYZ reason, so we need to come up with a different/better solution”.
3. Non-functional requirements of the system
Write down all the non-functional requirements - examples are availability, capacity, reliability, scalability, security (high level), Maintainability + Manageability, etc.
Discuss the load expected on the system with the product manager (check if it's already mentioned in the PRD or not) and mention that here
Discuss any data-related non-functional requirements here - for example - data access speed, encryption, data consistency (strong/eventual)
4. Back of the envelop estimate
4.1- Estimate load and explore it in detail -
Requests per second/operations per second/ actions per second - write down the estimated figures.
Database storage and cache storage estimates - Estimate the amount of the data stored (per minute/hour) and establish a read-write ratio.
Media storage - read/write pattern and size of data.
4.2- Synchronous vs Asynchronous communication
Figure out which parts of the functional requirements need sync communication
As a rule, for inter-service (inter-system) communications, Try to lean on async communication as much as possible.
5. Define the interfaces - High-level definition
Choose the proper interface according to the requirements and back-of-the-envelope estimate-
REST API
gRPC
GraphQL
Websockets
SSE
If you are choosing anything apart from REST API - mention the reason for that choice.
Explain each of the interfaces clearly -
Define the input parameter and type of the input parameter (query, body, URL, etc.)
Define the output very clearly (what are the fields in the response, the response structure, etc.).
This section should only describe the interfaces clearly. The detailed logic of it should be explained in a detailed section later.
Factor in the end-user behaviour in these interfaces. We design the interfaces before writing data models to ensure we have thought about end-user access and write patterns. Our data models should reflect the same thinking (detailed in the next step)
6. Data models and low-level design of the system
6.1.- Schemas
Define the different SQL and NoSQL schemas required for the functional requirements.
Define the relationship between the schemas.
Mention the data type and the corresponding validation checks like maximum length, what data is not allowed (for example - special characters), what level of precision is required (for example, time is stored in UTC nanoseconds), etc
6.2 - Explore the end user read/write patterns
Write down the read/write patterns of end users for each of these schemas - Estimate this based on the expected user behaviour in product requirements.
Define the partition keys, secondary indexes, etc., for each schema based on these patterns and functional requirements.
Define if there are any secondary models(models that contain the same data stored in a manner which is optimised for end-user experience) required for carrying out the functionality.
Check if a single document/record of this schema has the potential to become very large. If yes, evaluate whether breaking the schema will be more practical.
7. Architecture diagram with details of the tools and technologies used
7.1 - Case 1 - If it's an addition or extension to an existing functionality/system
Mention the links to all documentation related to that existing functionality/system.
Draw a diagram depicting how it will talk to the various components in the current system. The diagram should clearly show the flow of data across the entire system.
Note any major changes/decisions made in this new system that can directly or indirectly affect the existing system.
If there was a particular engineering problem to solve, note what solution you used to solve that problem.
7.2 - Case 2 - If it's a new functionality/system
Draw a diagram depicting all the components and how they communicate. The diagram should clearly show the flow of data across the entire system.
If there was a specific engineering problem to solve, note what solution you used to solve that problem.
Mark the components shared across systems - databases, file systems, media storage, etc.
Mention the tools and technologies used clearly in the diagram -
Language/framework
Databases
Queue
File systems
Media storage
Third-party tools
Utility libraries
If any of the above tools and technologies differ from the existing systems, mention the reason for choosing them.
8. Business logic of the interfaces
Detail the business logic of each interface, starting from validation checks to when the output will be generated. This should be written in a line-by-line pseudo-code format. The idea is to have a good clarity of logic before we start translating that into code. Send this logic to your peers/managers for review to cross-check if you missed anything. You may create sub-documents of this document and mention the link if there are more than five interfaces. This will ensure that this document is cleaner and more readable.
9. Client-side logic and interactions
9.1 - Compare the interfaces with the design file
Do a final check to ensure that the interfaces designed earlier cater to all the user journeys created in the design file. This is a redundant but essential check to ensure that you have not missed anything.
9.2 - Check whether all the possible error cases have a corresponding UI/UX in the file.
Your API documentation should detail all the possible error responses, and the design file should have a UI/UX element for all these cases. Check the design file for the same.
9.3 - Check whether the analytics events (and other tracking tools) are available in the product requirements.
Analytics and other tracking requirements are often missed while writing PRDs or ERDs. These events can be triggered on both the client side and the server side. Ensure that they are clearly described with
9.4 - Detail the most critical client-side validations and security checks
In many cases, client-side validations and security checks play a crucial role in avoiding catastrophic security failures. If your use case has any of these cases, they should be mentioned in this required. Example - My application requires a client certificate security for a socket connection (handshake).
10. Application and infra-level logging and alerts
In this section, answer the following questions -
What things need to be logged at the application and infra level?
What things need to be alerted at the application and infra level?
Your final application code should factor in the application layer logging and alerts.
11. Single points of failure
Write a note about the single point of failure in the system. Discuss with the leads how critical these points of failure are and what steps we should take to ensure the disaster response and recovery process is fast.
11. Security consideration
Write down the possible security failures in the solution described above
Propose a solution for addressing them.
Figure out places that will need specialised security.
12. API documentation
Mention the links to all the relevant API and other interface documentation.
*************** template ends here ********************
Ending note
I hope this template helps you as much as it has helped me. If you have any questions, doubts or suggestions, please feel free to contact me on Twitter, Linkedin or Instagram. Do share this article with your friends and colleagues.
The Startup Developer series
If you are reading this, thanks for completing this post. This blog post is a part of my ongoing series called "The Startup Developer". This entire blog series is dedicated to all the "startup developers" out there. The blog series aims to share the learning, hacks, tips and tricks that have worked for me. If I can save you even a few hours per week, I will count it as a win.
Following are the posts in this series in order -