Development Flow
Development speed and quality are our top priorities. Although these two requirements sometimes contradict each other, they don't always have to. We are constantly experimenting and searching for solutions that bring the optimal balance of these two factors. Almost everything that's listed below contributes in one way or another to accelerating development or improving development quality.
Our approach is to get competent people involved and let them organize their work as they see fit. Several years ago, when SCRUM was at the height of fashion, we too were tempted to try it. We won't do that again. :) If you're curious why, check out this post - On Estimates.
All our developers can communicate with the customer directly. There are no games of telephone between layers of managers, analysts, etc. Although we have managers, their role is not to serve as mere vessels of information. Rather, our managers are there to solve various organizational issues and help developers give their projects their undivided attention, without distractions from irrelevant details. Of course, whenever communication problems do arise, the managers also help solve them.
When communicating with the customer, we try to keep our work as transparent as possible and demonstrate results frequently. In return for our openness, we ask that, whenever possible, our customers don't require time estimates of us. Estimates negatively affect the development process; they slow development down and distort how participants perceive the project's reality. When there's a deadline to meet, we try to follow this approach, Fix Time and Budget, Flex Scope.
We don't have any dedicated testers. Though we used to have them, we gradually came to a point where it became faster and more efficient to have developers be responsible for testing their own code. Naturally, we actively rely on automated testing. Recommended reading:
All the code is stored on GitHub. Well, not just the code: server configuration options, deployment scripts, documentation, and wikis are all here too.
Most of our projects are hosted by Amazon Web Services and we actively use the platform's features.
We aim for "ubiquitous automation", guided by the maxim "nothing manual on the server". Everything involved with how servers are set up and how code gets pushed to production is fully automated. We also require automatic backups of any valuable data that may be stored on our servers. This policy, combined with our use of cloud hosting, means that a failure of any of our servers is never a catastrophe ─ we can always re-create its fully functional copy within minutes.
Setup is fully automated based on Docker Compose and custom scripts. For development, we use the same image for the project environment as the one in production.
For all of our projects, we push code to production continuously. Commits to the master branch immediately initiate an automated Jenkins build that runs critical tests. If everything passes, the new version is immediately pushed to production.
Our deploy scripts are carefully configured to achieve Zero Downtime deploys. This allows us to push code to production frequently ─ several or even dozens of times a day ─ and do so constantly.
While well-configured deployment scripts help enormously with achieving Zero Downtime, there are still some things that developers need to keep an eye on. In particular, they need to carefully consider how they approach database migration. Here is a recommended article about how to migrate a production database without downtime:
Monitoring is an essential part of ensuring a stable operation. We use Sentry for error reporting, and Datadog and AWS CloudWatch for monitoring servers and applications.
Conventional industry practice creates a new branch whenever work begins on a large new feature, and merges the branch to master only once the feature is complete. In theory, such isolation prevents new code from disrupting user or developer activity. In practice, the merge typically brings an ensuing period of instability that lasts until the integration issues have been resolved.
However, we deploy continuously and release several times a day. As a result, the multi-branch approach to versioning is not a viable approach for us, for multiple reasons. First, the sudden appearance of a large block of new code in the master branch makes issues easier to miss and the code harder to review. Second, our projects advance at a velocity that would make staying isolated on one branch for any extended period of time very impractical for developers. By the time they start merging, the master may have become so different that integration will be labor-intensive and error-prone.
In rare cases, we create additional branches to the master, but we try to make sure that they don't live long and are removed as soon as possible. If you haven't mastered our development process very well yet, here's a rule of thumb: when determining whether you should create a new public branch, the answer is "no, no". Of course, the rule applies only to branches in the main Github repository. If for whatever reason you are comfortable working with branches in your local development environment, please go right ahead. But no branches should be in the main repository, and neither should any traces of branching (eg, "merge" commits).
Because it's carried out post facto, after the code hits master, our code review is asynchronous and does not block the development flow. In our experience, this provides the best balance of code quality, development speed, and developer comfort.
Reviewing all code before it reaches production creates a bottleneck in two directions. While developers wait (sometimes for several days) for their code to pass review, they switch to some other task, returning to the original one only when the review is finished. On the other side, trying to minimize delays caused by code reviews leads to frequent distractions for reviewers, as they are constantly pulled from their current work to review each new incoming commit.
The main argument for reviewing code before publishing is that doing so reduces the probability of introducing problematic code into the main branch. However, for our development practices, this approach would be ineffective. First, automated testing already heavily reduces the number of issues that could be detected in the code. The deploy flow automatically blocks unlinted code, and tests block many code regressions from ever passing. Second, our developers test their code themselves and we trust that it is very unlikely that they would introduce some critical problem into production. The typical problems we find in review are sub-optimal or inconsistent code implementations, and code that misses some edge cases. Fixing such problems post facto does not pose any serious risk to the overall stability of the system, and at the same time allows developers to deliver their code much faster and with minimal context switching.
English
Русский