Release Engineering at Openbravo

Archive for March 2010

Towards a better Openbravo ERP release process

Openbravo ERP 2.50 has a time based release plan. This helps us get bug fixes and new features into our user’s hands more quickly and improve our planning process.

Up to now we’ve been releasing Maintenance Packs on a monthly basis. However we have detected some improvement areas, mainly two:

  1. We want to release like a clock:  same time, same day, every month. Everyone gets benefited by this:
    • End users appreciate knowing the release dates without even reading a release calendar. They simply know that they’ll have an update at the end of every month.
    • Our engineering team can organize more effectively: developers, QA, Support, Release Management.
  2. Developers should know what commits they push go into what release. This way they can organize their work more efficiently.

So let’s start! We already have the list of desired improvements and we know that we need to set the key dates. So first we need to identify the different kind of events involved in the development process:

  • Regular commits (e.g. bugfixes or small features).
  • Large project merges (e.g. a new large feature).
  • QA automatic tests.
  • Feature freeze for the release.
  • Release packaging.
  • QA manual tests.
  • Publish the release.

Now we need to set some dates and deadlines. The QA manual work – bug found – fix it loop lasts around 10 days. So the freeze and packaging should be before the 20th. And we want to provide a guarantee to developers, so that they know which commits go into what release. Here’s a proposal:

  • All the commits pushed before the 15th of a month will go into that month’s release. We’ll make sure all the tests are passed, working together with the developers in case there’s a failure.
  • We’ll freeze the code the 18th and start the packaging right after this moment. This means we have 3 days to make the code pass all the automatic test suite. It should be more than enough. It also means that commits pushed between the 15th and the 18th might go into that month’s release. If it passes the tests of course.
  • From that moment on QA can start the manual work. If they find issues the affected developer will work on the fix and they’ll iterate on this process till QA happy is happy with the result.
  • Afterwards the Release Management team can publish the release.

And last but not least, project merges tend to create instabilities in the code main line, compared to having no reintegrations at all. So taking this into account the project merges must be done a specific time frame: from the 22nd of one month until the 5th of the next one. So that we give it enough room (10 days) to get the code stable for the freeze time.

We can see these policies summarized in the following graph:

Openbravo ERP 2.50 release timeline

Improving a release process is usually an evolution, not a revolution. And I believe this is the first step towards better timely releases.


Written by jpabloae

31/03/2010 at 16:21

Posted in openbravo

Tagged with ,

Amazon EC2: backup management tips

Cloud Computing services offer a great deal of advantages to end users and sysadmins. I’m sure most of us appreciate not taking care about the hardware, having agility to provision new resources, getting the ability to recover fast from disasters, taking benefit of a very high network bandwidth speed, and so forth. It’s a very good solution for many situations.

However managing a Cloud is different compared to the traditional server management. We have been using Amazon EC2 in Openbravo for the last couple of years, so I would like to share some tips with you with regards to backup management and recovery.

DNS management: Elastic IPs

Making backups is the first step towards a successful system recovery. But in order to take benefit of these backups it is also essential to react in a quick and easy manner when something goes wrong, so that we have the system is back up and running in a matter of minutes.  One of the biggest delays tied to the traditional server management comes with the DNS changes. That is, migrating to a new machine usually means having a new IP address. And therefore it forces you to update the DNS records, and this might take days to propagate.

Amazon has a nice feature called Elastic IP that allows you to forget about this. You basically allocate an IP address for you, so that you can assign it to any of your machines “on the fly”.

Think about the difference:

  1. Traditional DNS management: if a new server gives me a new IP I have to change the DNS records.
  2. Elastic IP DNS management: I can assign my IP address to any of my instances.

The first option might take several days to be spread in the entire world, while the second one is immediate.

EBS: persistent storage

Amazon EC2 has continuously improved the features they offer. One of the limitations it initially had consisted of the fact that the hard drives where not persistent. That is, if you shut down a machine you lose your data. In September 2008 Amazon introduced the EBS hard drives (Elastic Block Storage), which means persistent storage that essentially works like a normal hard drive that you can attach to any of your instances. However you could not run the entire operating system in one of these EBS drivers. In any case this was a big step forward, given that you could save your critical dynamic data (e.g. a database) in a persistent storage. Starting from December 2009 you can finally run an entire operating system in a EBS unit.

This feature is relatively new so most of the public AMIs are still based “instance-store”, instead of in “EBS-store”. The recommendation here is incremental:

  • Make sure you save your critical dynamic data in a EBS drive.
  • If possible, run the entire instance in a EBS drive.

I’ve just said that it works similar to a regular hard drive you can attach to any instance. This is not entirely true, it’s better: you can make snapshots of the entire disk. As many as you want, and they are incremental. You can of course restore them into a new drive any time you want.

So as a last tip with EBS do regular snapshots, and make sure you purge the old ones.

Off-site backups

There’s a golden rule with regards to backups: “Do not put all the eggs in the same basket”. It is very unlikely for Amazon EC2 to suffer a general downtime or disaster so that you would loose your data (e.g. a fire). They make sure this won’t happen and they do their backup homework as well. In any case it is generally a good idea to have a second recovery plan, physically isolated from your main backup location.

Amazon EC2 currently has two independent regions (US and EU), so the first option is to replicate the backups from region to another. However if a malicious user gets your EC2 credentials they might have temptations to wipe out all your data in both regions. To avoid this, as a recommendation, create an extra EC2/S3 account with complete read-only access to the first account. So that your backups cannot be compromised in that way.

If you are more paranoid than this, you can schedule weekly or biweekly backups to an external storage service provider.

Physical backups

It is sometimes very useful to have a recent backup available with you at your office. One option is to download it. But depending on the size of your backups and your network speed this might be prohibitive. Amazon has a nice feature called Import/Export that covers this need:

  • You send a hard drive to them with instructions.
  • They load the requested data into the hard drive.
  • They send you the hard drive back.

Openbravo ERP and EC2

OK, those tips sounds reasonable. So what should I specifically do with my Openbravo ERP instance in EC2? Stay tunned, a new post will be coming soon covering this topic.

Written by jpabloae

10/03/2010 at 16:11

Posted in openbravo

Tagged with ,