Openbravo ERP 2.40 continuous releases

Openbravo ERP has currently two major versions: 2.50 is the latest one and it’s actively developed. 2.40 is a maintenance release for Professional Edition users, and it mainly consists of bug fixes, with an average of 20 fixes per month.

These 2.40 updates are distributed as Maintenance Packs (MPs – a collection of bug fixes), with a frequency of one month during the first year and of two months during the second one (since November 2009). Starting from 2.40MP11 we are introducing a new concept of MPs. Instead of releasing bi-monthly updates we will release them on a bi-weekly basis, tested and verified by our Continuous Integration Framework. This is the workflow that this automated process follows:

• Mike reports an issue and it gets assigned to Ken, a Openbravo ERP developer.
• Ken fixes the issue and pushes the commit into the 2.40 Mercurial repository.
• Our build farm runs a set of tests on that changeset:
  • Incremental update test on PostgreSQL and Oracle. This is important to test updates to this changeset.
  • Full clean builds on PostgreSQL and Oracle. This is used to test new installations based on this new changeset.
  • Verify the database consistency. This verifies that the database model matches exactly its definition in the XML files.
  • Functional tests on PostgreSQL and Oracle. This runs a set of graphical on a web browser, powered by Selenium.
  • Other minor sanity tests, such as checking that no database data is entered using the reserved ID ranges.
• If all the tests pass successfully, we tag and sign the release in the Mercurial repository, and later on replicate it into our Subversion mirror. If one test fails the system waits for new changesets and starts the process again.
• Mike gets notified about the availability of a new MP that includes a fix for the issue he reported.

These kind of MPs are easily identifiable by it’s version numbering, following the 2.40MPX.Y schema. As an example, we currently have released four, called 2.40MP11.1, 2.40MP11.2, 2.40MP11.3 and 2.40MP11.4.

What does this mean for a 2.40 user?

• We are reducing the delivery time. You get maintenance updates every two weeks.
• You get updates of the fixes you reported faster than ever.
• You update your 2.40 installation using the usual method (Subversion), this does not change.

Note that we are also planning to release two more “traditional” MPs during the 2.40 life cycle, that will consist of additional manual verifications. They will be called 2.40MP12 and 2.40MP13, scheduled for February and August 2010 respectively. Here’s a table that summarizes the 2.40 MP frequencies:

Version	Release date
2.40MP11	November 2009 (already released)
2.40MP11.X	Bi-weekly
2.40MP12	February 2010
2.40MP12.X	Bi-weekly
2.40MP13	August 2010

RM updates: ERP 2.3x discontinued, graphical upgrade test, Issue Tracker and QA integration

These are the latest news from the Release Management Team:

2.3x discontinued

Important note! The 2.3x version of Openbravo ERP has been discontinued:

• We have removed all the continuous builds involving 2.3x from our build farm.
• It’s no longer possible to report bugs against this version in our issue tracker.
• The wiki documents related to 2.3x have been marked as deprecated.
• The Mercurial code repository is permanently frozen.
• The 2.3x installers have been removed from the SourceForge download area.

I would like remind you that if you’re running a Community Edition installation you should always be in the latest major release. Currently this means 2.50. Running an old major version is possible of course, but the voluntary support you get in the forums is provided on the latest major release. The professional support is offered in all the major releases: 2.3x, 2.40, 2.50 until now and 2.40, 2.50 after this 2.3x end of life.

So this means that if you’re running 2.3x you should seriously consider upgrading to 2.50 as soon as possible. If you have problems during this process you can either post your question in the forums, which is run by volunteers, or contact your closest partner to get professional support.

Continuous Integration: graphical upgrade test

We have a new job in our build farm that really makes a difference: perform an upgrade using the Module Management Console, starting from the last release Maintenance Pack and using to the latest daily Core OBX module. This test is now a prerequisite before merging changesets from pi to main.

It’s been developed using Selenium by the QA Team (thanks for that!) and we have adapted and integrated it into our continuous integration framework.

New issue tracker statuses

Soon we’ll have 2 new statuses in our issue tracker: Ready for integration and Ready for QA. They will be placed before the Resolved status. With this change we increase our compromise with the quality of the product. Because this means that we won’t consider an issue to be resolved until our Continuous Integration and QA Team have tested them. Check the rationale behind this decision or check the new workflow in the following graph:

Issue tracker and build farm integration

Every time a changeset is successfully promoted to erp/devel/main we automatically generate an Core OBX file out of it. Now we have integrated the issue tracker with our build farm so that a note is added with information regarding when this promotion happened, in what changeset and how to get the generated OBX file. See issue #11470 to see this in action.

For a complete list of the on-going stories we’ve been working on, check the Sprint 29 page of our Scrum spreadsheet.

RM updates: upgrade automation, documentation, appliance security

Following with our round of updates, here’s the latest news from the Release Management Team:

• Continuous Integration: there is a new job that tests upgrading from the last stable MP to the latest daily OBX file. This has been done using the command line. A graphical test is in progress.
• Live builds: the main page has been refactored and it now includes detailed build information as well as the runtime Tomcat log.
• Appliance security updates: previous versions of the appliance are vulnerable to a man-in-the-middle attack during TLS session renegotiation. This vulnerability has been addressed in this update. Check the full changelog at the newly created appliance release notes.
• Documentation: check the new document explaining our stack configuration in the appliances.
• Issue Tracker: we are working on upgrading to version 1.2.0, which includes new interesting features. Many things have changed in this version so this process will take longer than a regular update. A new testing server will be announced soon.
• Download area: the SourceForge download area has been updated to include only the latest 3 releases. The older ones have been moved to the 09-openbravo-old-releases directory.

For a complete list of the on-going stories we’ve been working on, check the Sprint 28 page of our Scrum spreadsheet.

ERP 2.50: experimenting with PostgreSQL 8.4 and OpenJDK 6

Currently Openbravo ERP 2.50 officially supports PostgreSQL 8.3 and Sun JDK 6. By the time when these versions were taken as our base, PostgreSQL 8.4 did not exist and OpenJDK 6 was an on-going project still not ready. One year later the situation is quite different:

• PostgreSQL 8.4.1 is the latest and greatest of the PostgreSQL releases.
• OpenJDK 6 is completed and a very real alternative to Sun’s JDK.

For those unfamiliar with OpenJDK, here’s a bit of history: on 2006 Sun decided to open their JDK and license it under the GPL. Some work was required for that, though: about 4% of the code was proprietary, dependent on 3rd parties who didn’t want to open those components. So the OpenJDK project started rewriting those parts and getting the rest ready. Now OpenJDK 6 has passed the Technology Compatibility Kit tests and claims to be a fully compatible Java 6 implementation.

Using OpenJDK will allow us to have a 100% open source and free software stack. Also,  as most modern Linux distributions include it as the default JDK, setting up Openbravo ERP will be easier. For example Ubuntu ships OpenJDK as the default one and in the next 10.04 LTS Sun’s JDK will be available in the multiverse repository or in none at all.

On the other hand supporting PostgreSQL 8.4 has the obvious benefits of enjoying the improvements of this new major version.

So let’s play a bit with them to see how they behave with our latest release, Openbravo ERP 2.50MP8.

PostgreSQL 8.4

I’ve run two tests using version 8.4.1:

• Full build of Openbravo ERP 2.50MP8 (ant install.source): no surprises, our code builds cleanly with this new major version. The build times are similar compared to PostgreSQL 8.3.8.
• Functional test (smoke test): this is usually a bit more tricky. But good news! No problems at all, it passes all the smoke test cleanly! Nice.

OpenJDK 6

Now the hard part. Let’s see how it goes:

• Full build of Openbravo ERP 2.50MP8 (ant install.source): ouch, it fails the first time when it minifies the JavaScript files using YUI Compressor. But no worries, it’s a known issue that has a simple workaround, acceptable for now. After deleting the conflictive file the build finishes successfully. So the first big test passed, this looks promising.
• Functional test (smoke test): so I set up Tomcat to run with OpenJDK, start the smoke tests on Openbravo ERP 2.50MP8 and 100% successfully completed! Yes, all of them.

Honestly, I’m quite impressed. I expected PostgreSQL 8.4 to work well, but I didn’t have as much faith on OpenJDK as my fellow Sree had. Congratulations to the OpenJDK team, you’ve done a great job.

Continuous Integration

You want proofs of all this? We’ve set up continuous builds and functional tests of our bleeding edge ERP code. Have a look at the experimental jobs in our build farm. This will help us detecting mismatches between our current stack and the new candidates.

The full builds will be run every 3 days and the smoke tests once per week. You can also check the trend and health of these builds.

Conclusions

My initial conclusions are clearly positive: the core of Openbravo ERP works well with PostgreSQL 8.4 and OpenJDK 6. It’s too early to say when we will officially support them, but this is an important milestone. The continuous builds and tests, as well as some manual QA and developer work will help us on taking the decision.

RM updates: home page face-lift and automation

Continuing with our promised updates about what has been done in the Release Management team during the last 2 weeks, other than the publication of the 2.50MP7 and 2.50MP8 releases.

Homepage face-lift

We have given a face-lift to the Release Management homepage, with the following major changes:

• Release schedules and security have now a greater weight.
• We have updated the our subprojects (what keeps us busy).
• The RM related documentation has been organized into 4 sections.

Automation

• New job to automatically create the API documentation (javadoc) for new maintenance packs.
• Huge progress towards achieving a total release automation.

For a complete list of the on-going stories we’ve been working on, check the Sprint 26 page of our Scrum spreadsheet.

Tip: Mercurial authentication comfort

Mercurial 1.3 has a nice feature that makes our coding lives easier. You can define your authentication credentials globally, so that it will remember your username, the password, or both in any repository cloned from code.openbravo.com. There’s therefore no need to define the credentials individually in every repository.

Example of the relevant section to be added to $HOME/.hgrc:

[auth]
ob.prefix = code.openbravo.com
ob.username = johndoe
ob.password = supersecret

If you only want it to remember the username, then remove the password line.

And if you still are using an older version (hg version), you can follow these instructions to update to 1.3.1.

For more information about this feature check the hgrc(5) man page.

RM updates: Continuous Integration, Ubuntu, Javadoc, Release Notes

Like in any other team, some of the tasks we do in the Release Management Team have consequences in users, other times in developers and some times even in other company staff. We feel it’s important to notify other teams about what has been done. Not the progress or the future plans, we have Scrum of Scrums for that. But about the actual specific results.

So this is a simple as this: whenever there’s something new we think it could useful to a specific groups of users we’ll write a short summary to them. This will happen every 2 weeks.

Continuous Integration

• There’s a new set of tests called erp_sanity_tests. This intends to collect various basic checks that are not worth to keep as a single Hudson job. As for now there’s one included, to Check if the primary keys of the database in erp/devel/pi that were present as columns in erp/stable/2.40 have a onCreateDefault value set.New tests are accepted either by describing them to us or by sending us the code to be run.
• The QuickStart and Module Installation jobs have been moved from slave1 to slave2. We currently have 4 machines for builds.openbravo.com, and with this change one 100% dedicated to smoke tests.This affect developers in the way that now smoke tests will be finished quicker without interruptions from other jobs.

Ubuntu package

There was a demand for instructions on how to install the Ubuntu package with Tomcat in one server and PostgreSQL in another. This steps are currently manual and they’ve been documented.

Release Notes

They have been refactored. Now there’s a main page where you can see all the versions.

Then you can access the specific summary notes for each release. Check 2.50MP6 for an example.

And if you’re interested in more details, the Changelog page shows the complete list of fixed issues.

The 3 pages are linked. And the idea is to replace the Changelog page at some point, by delegating in Mantis to do this automatically.

Openbravo ERP API Javadoc

The Openbravo ERP API is now available for online viewing.

This includes all the releases since 2.50, as well as docs for erp/devel/main and erp/devel/pi, which are automatically updated in each incremental build of our continous integration framework.

Ideas for Openbravo ERP development builds live testing

Many of our developers and power users are frequently requesting live installations where to test daily or even hourly builds of the development repositorios (pi, main, etc). We already had this internally in the headquarters of Pamplona (the so called tecnicia14 machine) and it was pretty popular among the local developers and even the consultants. In this post I collect some ideas and ask for feedback.

I’ve been thinking on why this is useful, who the target users are, how this fits with our current continuous integration framework (builds.openbravo.com), what the frequency should be, etc.

To begin with, adding this feature is on the second milestone in the Openbravo Continous Integration project.

Target audience

Firstly, I basically see three groups of users interested in this:

1. Developers and power users: it’s obviously interesting for them to test the latest versions with different databases and running on different environments.
2. Sporadic users who want to test the latest developments and see how the progress is going.
3. QA team: they need these environments all the time, all the day.

The QA team has one big restriction: they need to be the only ones accessing the ERP at that time. And vice versa, developers don’t want to share the environment with QA: imagine them testing the DeleteClient functionality.

So the proposal is to have two different environments:

• live.builds.openbravo.com: for developers, power users and sporadic curious users. Open to the world.
• liveqa.builds.openbravo.com: for the QA team. Restricted access.

Candidate jobs

Given that we use Hudson as our CI framework, these would naturally be two different Hudson nodes. Having a look at the list of builds I’ve detected the following candidates:

• erp_devel_pi-full-oracle: full periodic build of the erp/devel/pi repository using an Oracle 11g database (ant install.source).
• erp_devel_pi-full-pgsql: full periodic build of the erp/devel/pi repository using a PostgreSQL 8.3.x database (ant install.source).
• erp_devel_main-full-oracle: full periodic build of the erp/devel/main repository using an Oracle database (ant install.source).
• erp_devel_main-full-pgsql: full periodic build of the erp/devel/main repository using a PostgreSQL 8.3.x database (ant install.source).
• erp_stable_2.40-full-oracle: full periodic build of the erp/devel/2.40 repository using an Oracle 11g database (ant install.source).
• erp_stable_2.40-full-pgsql: full periodic build of the erp/devel/2.40 repository using a PostgreSQL 8.3.x database (ant install.source).
• erp_stable_2.3x-full-oracle: full periodic build of the erp/devel/2.3x repository using an Oracle 11g database (ant install.source).
• erp_stable_2.3x-full-pgsql: full periodic build of the erp/devel/2.3x repository using an Oracle 11g database (ant install.source).
• erp_devel_main-obquickstart (liveqa only): full periodic build of the latest erp/devel/main tag with the QuickStart template.

The incremental jobs are only useful for testing the upgrade process, but not for live testing. That’s why they’re not included here.

Implementation

• Every time one of the selected builds is successful we could create a tarball with a database dump and the WAR file, placing them in a known location. The sources could also be shipped, because some of the modularity features require this.
• Then, the every new Hudson job would take their correspondent files and reset the environments using this database dump and WAR file.

As the reset process would take 5 minutes at most, the frequency could be 1-4 times a day.

Resources

Three machines would be required for this:

• One for Oracle (shared). Because a bug in the ERP shouldn’t affect the daily builds.
• One for Tomcat and PostgreSQL (live.builds.openbravo.com).
• One for Tomcat and PostgreSQL (liveqa.builds.openbravo.com).

Open questions

Anything is open to questions, of course, but this is a list of topics I have brainstormed:

• Is there any other build you would include?
• Would you find useful to test the ERP in other operating systems and architectures other than Linux and x86/x86_64?
• What reset frequency would be reasonable?
• If QA needs a separate environment, do developers need it too?

What else other than listed above would be of interest for you in these live testing machines?

Openbravo ERP: SSL tips

Being Openbravo ERP a web based application, using SSL is currently a must for those who appreciate their privacy. Think about the kind of data that is transferred in an ERP: customer details, transactions with business partners, invoices, balance sheets, etc. Not using SSL basically exposes all this information to anyone around with a network sniffer.

This article assumes that you already have a working Openbravo ERP installation, using the following software stack:

• Apache Tomcat 6.0.x with the Tomcat Native libraries, version 1.1.x.
• Apache httpd web server 2.2.x with mod_ssl.
• The mod_jk 1.2.2x connector for the Tomcat and httpd integration.

If you try to use Openbravo ERP with the default configuration, then you will find some problems. Namely:

1. All the reports generated by Jasper Reports don’t work, displaying an error such as the following one:

09:23:59 [ajp-8009-3] WARN  org.openbravo.erpCommon.utility.ErrorTextParserPOSTGRE - did not find constraint name for error message: Error loading byte data : https://localhost:4443/openbravo/web/images/CompanyLogo_big.png

2. It’s slow. Noticeably slower than running it under plain HTTP, as if every request took an additional time of 100ms and 1 s, and as if nothing was being cached.

Needless to say that these are showstopper issues, because the first one prevents you from e.g. printing invoices and the second one makes it very unpleasant. Let’s fix these issues.

Reports problem

The first error is caused by the fact that the Java process that generates the reports requires some images located at a SSL protected URL of your server. And it doesn’t trust the provided certificate, so it fails. Actually this is supposed to be a feature, because it’s verifying that the site is really who it claims to be. There are two ways of solving this. The first one requires buying a SSL certificate from an approved provider and it’s appropriate for production servers. The second one doesn’t require to buy anything, and it’s suitable for testing servers.

For production servers:

1. Register a (sub)domain name for for your ERP, e.g. openbravoerp.mydomain.com
2. Buy a SSL certificate for openbravoerp.mydomain.com. You can find them for ~$70/year.
3. Register openbravoerp.mydomain.com in the internal DNS server of your LAN, in case the server is hosted in-house. If you don’t have one, it’s time to set it up. There are tiny DNS servers that take no more than 5 minutes to install and set up.

For testing servers:

1. If it’s going to be exposed to the Internet, register a (sub)domain name for the ERP in a free DNS service, such as DynDNS, e.g. erp-atlantis.dyndns.org
2. Generate s self-signed certificate using OpenSSL.
3. Import the SSL certitificate into the local JDK. Go to a command line terminal, download the InstallCert utility and run the following commands:

javac InstallCert.java
java InstallCert erp-atlantis.dyndns.org
cp jssecacerts $JAVA_HOME/jre/lib/security

4. Restart Tomcat to apply the changes of the previous step.
5. For using the ERP in your LAN, register erp-atlantis.dyndns.org in your internal DNS server.

Performance problem

There are two separate issues regarding slowness. First of all, the web browser starts a new SSL negotiation in every request, adding a 100ms-1s delay to every single request. To fix this, make sure your Apache httpd configuration has the KeepAlive option turned on:

KeepAlive On
MaxKeepAliveRequests 200

In this case we have also doubled the number of allowed alive requests, because it is expected that this numbers grows as we now allow persistent connections.

The second issue is related to the cache. SSL does not store cache in disk between sessions, for the sake of security. But there is a performance penalty. So this a trade-off you need to decide. To make Apache httpd save the cache between sessions, we need to set the Cache-Control header to Public. This can be achieved by using the mod_headers module:

Header unset Pragma
Header append Cache-Control "public"

I want to thank katratxo for finding the solution to the cache issue.

The new packaging of ERP 2.50

Openbravo ERP has replaced its installers with virtual appliances and source code tarballs. We believe this is a major step forward. Why? This post explains the rationale behind this decision.

In 2.1x and 2.2x Openbravo ERP was deployed using Ant Installer. In 2.3x and 2.40 we switched to a binary installer technology. Starting from 2.50, Openbravo ERP has changed its packaging formats.

I remember having a brainstorming session with a project colleague, analyzing what we could improve in the overall installation and release process. That is, from the moment Release Management takes the code from QA, till users install it in their computers. The discussion led to the following conclusions:

1. What’s the hardest part in the Openbravo ERP deployment? Certainly not the ERP, but all the software stack it relays on: Ant, JDK, Tomcat, PostgreSQL/Oracle and some extra adjustment in the operating system to make Openbravo ERP happy (system variables, permissions, etc).
2. Let’s think about average computer users, with experience as an ERP user. They have a good ERP functional knowledge, but they don’t necessarily know how to install, configure and optimize Apache Tomcat, PostgreSQL, Ant, JDK, system variables and user permissions. They simply are not interested nor have the time to deal with that. How would the installation process look like for these kind of users? They’ll most probably give up after the second or third problem.
3. Let’s think about advanced computer users. Does the installer help them deploy Openbravo ERP? The installer basically performs two tasks: double check that all the required component’s are sane and build the ERP from sources. All these checks were hidden behind the GUI and the 2.40 version did not have a command line equivalent. In 2.50 this can be summarized in 2 commands:

ant diagnostic
ant install.source

Advanced users know what they’re doing and they prefer running these 2 commands by hand. Hiding this behind a GUI does not help them at all. In fact they dislike it.

4. How fast is the 2.40 release process? The installers or any other packaging deliverables must not be the bottleneck. Ideally, if for some reason the QA process fails it must be because of an ERP bug and not because of a packaging problem. This was far from being truth. The release process was a pain and slow.

So having a look at the 4 points, we realized the installer failed in all of them. It doesn’t matter what installer technology we used, it just was not the right tool for us. Some of the problems of 1) and 2) could be solved by creating a one-click installer that deployed and configured all the technology stack plus Openbravo ERP. But this has mainly 2 problems:

• It’s not useful for production systems, where users want to use the Tomcat, Ant, JDK, PostgreSQL, etc. provided by their operating system’s package manager. This allows them to install and update those packages very easily. This is understandable and in fact recommended.
• We found out that virtual appliances do a better job in terms of ease of deployment and immediateness. And most importantly, we provide the users exactly the same environment we have tested. And not only that, it solved our 4th problem. The release process has been immensely simplified. QA does all the ERP testing in appliances, and when the product is ready Release Management creates the final appliances with the final ERP code. A smoke test gives the green light to publish the release.

So the virtual appliances solve the problems for the average computer users (points 1 and 2), and also the release problems (point 4). For advanced users (point 3) we offer them detailed installation instructions and the source code in two formats: tarballs and direct access to the main Mercurial repository.

Wondering how this affects the upgraders? Good news, starting from 2.50 upgrades are managed through the Module Management Console. In other words, Openbravo ERP is capable of updating itself and there is no need to have external upgraders.

Next steps? Package and include Openbravo ERP in all the possible Linux/BSD official repositories (Debian, Fedora/Centos/RedHat, FreeBSD, Gentoo, openSUSE, Ubuntu, etc).