HP rushes out patch for DDX database bug

Click here for Robelle sponsor message

HP rushes out patch for DDX database bug

Repaired versions of TurboIMAGE offered as the only cure for potential data loss

HP has carried few reports of bugs in TurboIMAGE that can lead to data loss. That’s why the HP database lab raced through a three-week effort to repair a problem that lets dynamic detail expansion (DDX)-enabled databases lose detail data.

HP advised its customers in an Internet message that the bug deserves some serious attention, especially from sites with very large databases or ones running 24x7 operations. Only customers who have DDX turned on in their databases – a feature that lets TurboIMAGE dynamically allocate more space for detail datasets when a threshold is reached – are at risk. Anyone who uses DDX must apply HP’s official fix.

The patch, however, was still working its way out of beta test status as this issue of the NewsWire was going to press, causing more than a few sites to take the alternative route of turning off DDX altogether. The patch remained in beta testing through November because as Adager’s Rene Woc explained, “Anything that comes out of the lab two weeks after it’s been fixed has got to be beta. But it’s a well-localized patch, and very specific to the problem.” Testing was proceeding, with no published reports of problems with the patch.

While turning off DDX can stop any future data loss, it’s not a way to fix anything that may already have gone missing in a DDX database. Checking with a third-party database utility can verify whether a database has been bitten, or HP’s Response Center can lead users through the steps to check databases using QUERY and DBUTIL.



		Checking with a free third-party database utility can verify whether a database has been bitten, or HP’s Response Center can lead users through the steps to check databases using QUERY and DBUTIL

There are two levels of repair a site must employ if its database has been affected by the bug. The first level is to fix the damage in the database, using third-party database tools when possible. In some instances data can be recovered. The second level – really required for all sites with DDX-capable TurboIMAGE – is to replace TurboIMAGE with the fixed version.

HP said seven companies have reported the bug during the three years that DDX has been running in HP 3000 shops, but other sources say the problem is slightly more widespread. Working with third-party help can increase the possibility of recovering lost data.

HP’s Jon Bale, head of the HP IMAGE lab, assured customers in a rare posting from the R&D section of the database group that they can avoid any further problems if they apply HP’s remedy – patch TIXKX11 for MPE/iX 5.0 systems, or TIXKX13 for 5.5 systems that are new versions of TurboIMAGE (C.06.23 and C.07.07, respectively). DDX isn’t enabled for HP 3000s prior to MPE/iX 5.0.

Bale reported on the problem once his lab had developed a fix for it, an effort that couldn’t begin in earnest until someone helped track down the cause of the corruption. Adager’s support specialist Ken Paul doggedly traced a mysterious string of customer calls to assemble a pattern of why detail data turned up missing at a handful of customer sites. After Paul presented his case to CSY’s lab, the engineers went to work to develop a solution.

The patches are available through HP’s Response Center, but some customers reported that RC engineers have advised them not to install the patches unless customers are experiencing problems. Finding out if you’re experiencing problems can be a matter of using a third-party tool or letting the RC lead you through diagnostic tests on your databases. Since the problem is missing data, problems may not be apparent without such tests.

Customers who want the patch might do well to quote Bale’s post in getting it during the beta process. In reply to the question, “How can you avoid experiencing this problem on your IMAGE database?” Bale said, “The simplest answer, then, is: Install and use one of the TurboIMAGE beta patches mentioned above. This is the only option that I recommend.”

HP’s quick response to the problem doesn’t reflect on the quality of the fix, according to customers in the SIGIMAGE Executive Committee who have been helping to solve the problem. SIGIMAGE chairman Ken Sletten said the beta patch will be his path to repair.

“I'm confident about installing the patch on our production 959KS,” he said. “Even though the patches are still officially beta, everyone at the HP R&D Lab is fully aware that reliability and performance have been and are the key watchwords for IMAGE. I believe they were very careful in the construction and testing of these patches, and that they do completely close the tiny DDX window of vulnerability.”

If a site has a policy against using beta patches and needs to keep DDX running until the HP patch goes to General Release, Adager’s Paul says customers can limit their exposure by increasing the DDX expansion increment to a larger size.

“The DDX problem is caused by a potential timing problem between two processes,” Paul said. “Every time a DDX expansion occurs the ‘window’ is open. The more often that your datasets do DDX expansions the more ‘windows’ are open. By increasing the increment you are lowering the number of times that you are ‘opening the window’ and thus reducing your risk. The ‘window size’ has nothing to do with the ‘size’ of the increment.”

Paul said several users who experienced the problem had very active systems with people getting in and out of the databases, and most had increments in the 1000 to 5000 range. One client saw problems on three different occasions, he added. “You can also increase your current capacity to prevent DDX expansions from happening,” he said, “but this is similar to turning off DDX and requires the allocation of disc space now.”

Bale said the problem “relates to data stored in a dataset which has undergone some dynamic expansion – that is, it has ‘filled up’ and its capacity has been automatically increased,” Bale said. “Then some data that is subsequently written to the dataset is placed ‘beyond’ the end-of-file (EOF), and once the database is closed and reopened, that data is inaccessible. If a program attempts to read that data or add more data entries in the same area, it gets an error -212, ‘Database is Corrupt.’” The full text of Bale’s message is available at the Adager Web site.

Sites can use the DBUTIL program command SHOW DBNAME CAPACITY to find out if a database is DDX-enabled. The rightmost column (“Dyn Exp”) delivers the report on DDX: it says YES or NO. If you get an “invalid parameter” message, your version of IMAGE can’t use DDX, and you’re safe.

You can check for the corruption condition with a third-party database tool, even if you don’t own one. Adager was first to report that it’s making a limited-time version of its full utility available for free to anybody who needs to check for the bug. Contact them at 800.533.7346 (208.726.9100 outside North America) to get the copy of Adager that can find and fix the problem. Adager can deliver its tool over the Internet if you’re in a big hurry, employing private-URL technology that it uses to deliver secured versions of its products. If you’re already an Adager customer, the company reports that all models of Adager automatically flag the problem as part of the pre-processing consistency check. An anomaly like the DDX problem gets reported every time Adager opens any database for which you have READ access.

“You don't even have to be the database’s creator, and other processes may be accessing the database while you carry out this Adager checking,” Paul says. During its check, Adager analyzes all database structures and reports anomalies it finds. A discrepancy doesn’t always mean lost data.

But Paul added that if Adager reports a discrepancy and other users are accessing the database, “please do not get them out of the database. Contact Adager immediately and we will be delighted to guide you so that you can ‘gingerly’ get all of your users out of the database, hopefully without losing any data, if you catch the problem in time.” Paul says Adager’s therapy mode adjusts privileged database parameters, then can let database administrators see how much data, if any, they have lost.

One day later, Bradmark’s Jerry Fochtman reported his company was offering a free FIXDDX utility that can look for the DDX inconsistency and repair it. After a few days of testing, Bradmark (800.294.1251) also reported to us that its DBGENERAL utility can locate the DDX inconsistency. DBGENERAL standard diagnostics option 2.6 and option 2.3 will diagnose the DDX problem. Repair can be accomplished with option 2.3 (detail diagnose and repair) by following the repair prompts. In one case, while working on a database which had experienced the problem, Bradmark’s Tim Joseph said 2.3 repaired but left the dataset maximally expanded, and option 3.5 was required to adjust the set for further expansion. If you’re not a DBGENERAL customer you can use FIXDDX, which is downloadable from a link on the front page of Bradmark’s Web site .