Phase I - Performance Enhancements - Monthly Releases

Dateline: Roseville - May 27, 2008

This is the first news item in a series of articles discussing how we are improving performance of Escape Online 5. This article discusses Phase I - how we are incrementally improving performance on a monthly basis.

As you probably know, we track changes suggested by users and others in a system called OnTime, and the individual documents are called CR's - Change Requests. We are giving high priority to those CR's that refelect performance issues that affect users. With each release (like 8.04 here on the news page), we have a special "stamp" on those items which are performance issues. So you can see we are making progress on them.

Let me give you an example of how this plays out in real life. It was reported in April that certain processes in the Budget Entry activity had become very slow (CR 7037). Coincidentally, this is the first time the budget software has been used in a large scale, county-wide environment. Budget was used a year ago by several organizations without any problems.

Screen shot of budgeting within one account. The description column (blacked out) is the name of the employees that are being paid for by this account.

 

As we looked into this issue, we found that the database design relied on a unique (internal) reference number for each budget model item (i.e. a dollar amount within any given account). All of a sudden we had many users, across organizations, all performing the process. We had what is known as a "deadlock" issue where several processes are all waiting for the same (new) number. The design didn't allow for multiple requests all happening simultaneously, which is why our single district customers were unaffected. In a deadlock situation, the database just sort of sits there until the time out period (10 minutes in this case). Of course the end user is left hanging - ouch.

Fortunately, there is an easy answer. Rework the budget model item table such that the item number only has to be unique within that one model. Unfortunately, this also means changing programming in nine places, a couple days of work.

This is the kind of problem that doesn't rear its head until we have a slew of users doing it at the same time. There wasn't really a way for us to test a scenario like this. This problem definitely affected budget users at our County implementations, and for that we apologize. Fortunately, this is the kind of problem that, once resolved, is gone for good.

Since going live at Placer COE, we have seen several instances of this effect. We have something that worked fine until its first big test in the county setting. Usually we can correct these pretty quickly.

We have also been getting expert feedback from a consultant currently working at Ventura COE. He has made numerous helpful suggestions that we have implemented, especially around the types of "locks" that are possible in SQL Server. We really appreciate the help we are getting from our customer base, and their patience while we make these corrections. Of our four phase approach to increasing performance, this Phase, which is actually an ongoing process with releases each month, has the opportunity for the biggest "bang for the buck."

Comments? Send us an email and let us know what you think.