Every time a bug, crash or performance issue appears in any tech product, its attributed to a list of usual suspects like -

1. Poor code quality
2. Wrong selection of tools and technologies.
3. Wrong selection of languages/frameworks/databases etc.
4. Selecting or Not selecting a "Microservices" architecture.
5. Wrong selection of cloud architecture or deployment

But what if I told you that there is a simple pre-requisite of building a good technology product that is easiest to execute and its often overlooked.

Its writing an Engineering requirement document !

Taking a documentation first approach is the most efficient way to avoid bugs, performance issues, crashes, technical debt and ensure a high-quality output.

This blog post will give you a template for defining your engineering requirements that has worked exceptionally well for me. It will help you structure your solution better. I have used it to describe and build several products that solve complex problem statements, so it's battle-tested.

I never had access to big engineering teams. I learned the hard way that the best way to avoid problems, as mentioned earlier, in the long run, is to structure the product problem statements, engineering requirements and the What, Why and How of the solution well before writing a single line of code. This template will help you if you are an engineering manager or developer.

In one of my previous posts, I gave a PRD template for structuring your product requirements. Please check it out if you haven't done already. Think of it as a precursor to planning your engineering product. (click here) If your product problem statement isn't defined well, this ERD template won't be as effective in solving your problems as you would ideally want it to be.

How to use this template?

Before I share the template, let me share a few pointers on how to use it. The idea is to avoid ambiguity while using this template.

1. Define a single Engineering requirement document (ERD) for each product. If you are extending or changing the product, make changes in the same document. Don't create a separate doc for new features or modifications to the existing system. It will cause issues in the long run.

2. Most tools where you write the ERDs have a versioning system that can fetch the document's historical version if needed. Please ensure your tool has this functionality, too. If not, do a manual backup, or if you are a developer, you can always use a Git repository to do versioning of the engineering documents.

3. If your product depends on an existing product or system in your organization, mention the links, constraints and considerations of the older product in the new ERD.

4. Write down the considerations you took into account for a part of your solution, especially if your solution is experimental- a template of such considerations can be - “I took this decision to do <X> like <Y> because of <z> reason. It will help you iterate on the solution better.

5. Be as detailed as possible. The devil is always in the details. The template below describes what you are supposed to write at a high level. But it doesn't dictate what level of detail you can go up to. That's your choice. And trust me when I say this- the depth of your thought would count. The depth of your solution will contribute to the long-term success of your product.

If you follow the points mentioned above religiously, you will end up with an ERD document that efficiently solves 99% of the communication, scoping and product quality problems and will preemptively eliminate all the technical debt you could have incurred in the long term.

Engineering requirements template

*************** template starts here ********************

1. Product requirements

Write a high-level summary of the product requirements derived from PRD (product requirements document). If it's already present in the PRD, copy and paste it. Mention the link to the PRD.

Mention the link of the UI/UX file (Figma link, etc.), if any.

2. Functional requirements of the system

2.1 - Clarify ambiguity

1. Read the product docs and understand each feature well.

2. Think about all the edge cases from the product perspective and check if they have been properly explored in the product doc or not.

3. If there are any dependencies on third-party tools and services, please ensure they have been explored properly in PRD. Classify these dependencies into one of the sections below (functional requirements/ considerations/ constraints/ engineering problems to solve) and detail them in that section.

4. If there are any dependencies on existing systems (any existing internal services), please ensure they have been explored properly in PRD. Classify these dependencies into one of the sections below (functional requirements/ considerations/ constraints/ engineering problems to solve) and detail them in that section.

2.2 - Write each functional requirement clearly from the engineering perspective.

Engineering functional requirements essentially translate the product requirements into their tech counterpart. For example, if the product requirement is the “Login flow of LinkedIn”, in that case, the functional requirements will be the high-level description of all the APIs associated with that user journey. Please write down the possible engineering edge and corner cases and how to handle them as a requirement.

2.3- Define what is out of scope from the product and engineering perspective

Check the PRD for things that are out of scope from the product perspective. Please ask the Product/Engineering manager to update all the out-of-scope items if they need to be written. If they are already written, copy the link to that section of PRD here and then write down all the things that are out of scope from an engineering perspective.

These should be everything you factored out while designing the solution in this document.

2.4 - Write down the system considerations and constraints from an engineering perspective.

This section should create a list of -

1. All the constraints of the engineering systems - Write down all the things this system won’t be able to do. Also, mention why it won’t be able to do that.

2. Write down all the considerations you took into account from a non-functional and functional perspective for designing the solution later - The template of considerations is - I took this decision to do <X> like <Y> because of <Z> reason

2.5- Define any specific engineering problems to solve

This section should describe all the things not directly related to the product problems but are needed on the engineering side to solve the functional requirements mentioned. It can be as simple as ” We need a URL shorter system for this” or as complex as “Our existing pub-sub system won’t work in this case because of XYZ reason, so we need to come up with a different/better solution”.

3. Non-functional requirements of the system

1. Write down all the non-functional requirements - examples are availability, capacity, reliability, scalability, security (high level), Maintainability + Manageability, etc.

2. Discuss the load expected on the system with the product manager (check if it's already mentioned in the PRD or not) and mention that here

3. Discuss any data-related non-functional requirements here - for example - data access speed, encryption, data consistency (strong/eventual)

4. Back of the envelop estimate

4.1- Estimate load and explore it in detail -

1. Requests per second/operations per second/ actions per second - write down the estimated figures.

2. Database storage and cache storage estimates - Estimate the amount of the data stored (per minute/hour) and establish a read-write ratio.

3. Media storage - read/write pattern and size of data.

4.2- Synchronous vs Asynchronous communication

1. Figure out which parts of the functional requirements need sync communication

2. As a rule, for inter-service (inter-system) communications, Try to lean on async communication as much as possible.

5. Define the interfaces - High-level definition

1. Choose the proper interface according to the requirements and back-of-the-envelope estimate-

   1. REST API

   2. gRPC

   3. GraphQL

   4. Websockets

   5. SSE

2. If you are choosing anything apart from REST API - mention the reason for that choice.

3. Explain each of the interfaces clearly -

   1. Define the input parameter and type of the input parameter (query, body, URL, etc.)

   2. Define the output very clearly (what are the fields in the response, the response structure, etc.).

   3. This section should only describe the interfaces clearly. The detailed logic of it should be explained in a detailed section later.

4. Factor in the end-user behaviour in these interfaces. We design the interfaces before writing data models to ensure we have thought about end-user access and write patterns. Our data models should reflect the same thinking (detailed in the next step)

6. Data models and low-level design of the system

6.1.- Schemas

1. Define the different SQL and NoSQL schemas required for the functional requirements.

2. Define the relationship between the schemas.

3. Mention the data type and the corresponding validation checks like maximum length, what data is not allowed (for example - special characters), what level of precision is required (for example, time is stored in UTC nanoseconds), etc

6.2 - Explore the end user read/write patterns

1. Write down the read/write patterns of end users for each of these schemas - Estimate this based on the expected user behaviour in product requirements.

2. Define the partition keys, secondary indexes, etc., for each schema based on these patterns and functional requirements.

3. Define if there are any secondary models(models that contain the same data stored in a manner which is optimised for end-user experience) required for carrying out the functionality.

4. Check if a single document/record of this schema has the potential to become very large. If yes, evaluate whether breaking the schema will be more practical.

7. Architecture diagram with details of the tools and technologies used

7.1 - Case 1 - If it's an addition or extension to an existing functionality/system

1. Mention the links to all documentation related to that existing functionality/system.

2. Draw a diagram depicting how it will talk to the various components in the current system. The diagram should clearly show the flow of data across the entire system.

3. Note any major changes/decisions made in this new system that can directly or indirectly affect the existing system.

4. If there was a particular engineering problem to solve, note what solution you used to solve that problem.

7.2 - Case 2 - If it's a new functionality/system

1. Draw a diagram depicting all the components and how they communicate. The diagram should clearly show the flow of data across the entire system.

2. If there was a specific engineering problem to solve, note what solution you used to solve that problem.

3. Mark the components shared across systems - databases, file systems, media storage, etc.

4. Mention the tools and technologies used clearly in the diagram -

   1. Language/framework

   2. Databases

   3. Queue

   4. File systems

   5. Media storage

   6. Third-party tools

   7. Utility libraries

5. If any of the above tools and technologies differ from the existing systems, mention the reason for choosing them.

8. Business logic of the interfaces

Detail the business logic of each interface, starting from validation checks to when the output will be generated. This should be written in a line-by-line pseudo-code format. The idea is to have a good clarity of logic before we start translating that into code. Send this logic to your peers/managers for review to cross-check if you missed anything. You may create sub-documents of this document and mention the link if there are more than five interfaces. This will ensure that this document is cleaner and more readable.

9. Client-side logic and interactions

9.1 - Compare the interfaces with the design file

Do a final check to ensure that the interfaces designed earlier cater to all the user journeys created in the design file. This is a redundant but essential check to ensure that you have not missed anything.

9.2 - Check whether all the possible error cases have a corresponding UI/UX in the file.

Your API documentation should detail all the possible error responses, and the design file should have a UI/UX element for all these cases. Check the design file for the same.

9.3 - Check whether the analytics events (and other tracking tools) are available in the product requirements.

Analytics and other tracking requirements are often missed while writing PRDs or ERDs. These events can be triggered on both the client side and the server side. Ensure that they are clearly described with

9.4 - Detail the most critical client-side validations and security checks

In many cases, client-side validations and security checks play a crucial role in avoiding catastrophic security failures. If your use case has any of these cases, they should be mentioned in this required. Example - My application requires a client certificate security for a socket connection (handshake).

10. Application and infra-level logging and alerts

In this section, answer the following questions -

1. What things need to be logged at the application and infra level?

2. What things need to be alerted at the application and infra level?

Your final application code should factor in the application layer logging and alerts.

11. Single points of failure

Write a note about the single point of failure in the system. Discuss with the leads how critical these points of failure are and what steps we should take to ensure the disaster response and recovery process is fast.

11. Security consideration

1. Write down the possible security failures in the solution described above

2. Propose a solution for addressing them.

3. Figure out places that will need specialised security.

12. API documentation

Mention the links to all the relevant API and other interface documentation.

*************** template ends here ********************

Ending note

I hope this template helps you as much as it has helped me. If you have any questions, doubts or suggestions, please feel free to contact me on Twitter, Linkedin or Instagram. Do share this post with your friends and colleagues.