As HP 3000 customers add more online maintenance capabilities, two suppliers of online backup applications are sending a new message to HP 3000 customers -- the competition's product may be hazardous to your databases.
HP has begun selling a 7x24 True-Online version of its TurboStore software with the advent of the 5.5 release of MPE/iX. HP's product, which isn't part of the operating system but a separate-charge item, is the first version of TurboStore that allows managers to back up production databases while users remain logged on. However, the method HP is using for its online backups looks as if it can introduce logical integrity problems, according to engineers working for competing firm Orbit Software as well an independent engineer.
Being established in the online backup business for several years, ORBiT has been sending an application note to prospects and customers taking HP to task for the design of its new True-Online version. After a few months, HP replied in a letter to some of its customers that Orbit's technical arguments about TurboStore are based on a different definition of a logical transaction. What's more, HP's letter states that Orbit's definition of a logical transaction "does not ensure database integrity."
With both suppliers working to educate users, it's appropriate to look at what's being debated. A logical transaction can span several screens, as a data entry staffer works to enter or retrieve a complete record. These kinds of transactions never had the potential for problems during backup in the past -- because the only way to back up databases was to get users out of them before backing up.
The miracle of online backup is it keeps track of those transactions being performed while the backup is underway, ensuring everything entered is on the backup. Online backup raises the awareness of logical transaction integrity on the HP 3000, something that's difficult to protect and even harder to verify.
A debate over logic
Orbit's David Merit says the HP product "can actually jeopardize data
integrity of systems that use this capability," basing his claim on the
idea that HP relies on explicit transaction defininitions for IMAGE
databases. Those are the DBBEGIN/DBEND or DBXBEGIN/DBXEND pairs written
directly into applications. Merit says that applications that don't use
this explicit definition at risk of having logical integrity corrupted, and
there are few that use it.
That kind of risk is even more insidious when a manager considers how tough it is to find logical integrity flaws in a database. There's currently no widely accepted method or tool to track such flaws. The only way a manager would be aware of problems is if report totals or other points of information aren't correct.
"IMAGE doesn't detect the logical corruption," Merit says. "Your application has to be smart enough to detect it. You'll start getting wrong reports or parts orders to manufacture something that are incomplete."
HP, in its letter responding to the Orbit application note, defends its online backup method as enabling customers recover to the most recently completed transaction. According to HP's engineers and product managers, TurboStore True-Online uses "a proprietary technology which can take a snapshot of an entire file set at a single point in time." HP says the secret technology "will ensure that all files in the backup are logically consistent with each other," according to software deisgn engineer Paul Nissen.
Alex Early, a program manager at CSY, was reluctant to explain in much detail how the snapshot process worked, saying "I don't want to give away more than we have to." He also said that HP's customers haven't asked about TurboStore's impact on logical integrity. Early said that the DBBEGIN/DBEND or DBXBEGIN/DBXEND pairs aren't integral to the way that TurboStore stores databases.
"We have an ability the third-party products don't have," Early said, "the ability to have an atomic operation that snapshots the entire file system at one time. By being able to have an atomic operation do this, we don't have to hold off any transactions. That's where we feel our approach is better."
Early said that the explicit BEGINs and ENDs aren't the issue, and went on to speculate about how the Orbit product works. "The way that Orbit quiesces the database is by stopping the terminal I/Os," he said. "By doing that they're saying terminal I/O is actually tied to the database transactions, and stopping one stops the other. That's simply not true. It's one way of doing it, and the percentage of error in that is probably pretty small."
The conflict lies in the fact that to observers both inside and outside Orbit's labs, it appears that the BEGINs and ENDs are required for TurboStore to quiesce a database. Orbit has used the Avatar Nugget program (now being offered by Lund Performance Solutions) as well as HP's DEBUG to monitor system activity during TurboStore's operations. According to the Orbit engineer designing and maintaining the Zero-Downtime portion of Backup+/iX, it's relatively simple to see what's essential to quiescing a database under TurboStore True-Online.
Orbit's engineers say they go to one terminal and run Query, and then type say TRANS BEGIN, which issues a DBBEGIN. Then they go to another terminal and tell TurboStore to STORE DATABASE. After five minutes, TurboStore comes back and reports it couldn't quiesce the database. During this five minutes, Turbostore is checking every second or so to see if the transaction which Query started is done. After five minutes, the transaction is still not complete (TRANSEND/DBEND has not been issued by QUERY), so Turbostore gives up and says the database can't be quiesced. Normally the transaction would be complete within a few seconds, unless the application was written with a terminal read in the middle of the transaction, and the user left their terminal.
Orbit says these tests appear to show that TurboStore's True-Online version "treats physical transactions as logical transactions. In cases in which DBBEGIN/DBEND or DBXBEGIN or DBXEND are not specified, TurboStore will sync between individual update operations, such as DBPUT, DBDELETE, or DBUPDATE."
If the snapshot that HP's Early described is the essence of TurboStore's control, then it might be possible that quiescing the database isn't a critical operation. "TurboStore itself doesn't look at transactions in the sense of DBBEGIN and DBEND," Early said. It's job is to go in and take a copy of the database that's at a quiesced point and store it off. It's job is not to say that this transaction or that transaction is complete."
Orbit points out that the snapshot and the quiescing are closely inter-related, not separate operations. The "atomic snapshot" process is also not the way a third party expert on databases understands the HP product works. Ken Paul is a member of the Adager support team who has contact with thousands of customer databases in the HP 3000 market. Paul believes that the TurboStore functions, as described in an HP database roundtable, hold the potential for logical integrity problems.
"There's a big difference between HP and Orbit. If you ever have to go to a [TurboStore True-Online] backup, there's a much greater chance you're not going to have a full logical transaction in your database," Paul said. "You'll have database integrity all the time, but you won't have logical integrity within your database, because a lot of people may not be using DBBEGINs and DBENDs."
Orbit's Merit said his company estimates that no more than 15 percent of applications use DBBEGIN and DBEND, which are defined as explicit logical transactions. Paul said that some customers may not have used the explicit transactions "because they misunderstood and thought you couldn't use them without IMAGE logging. People just don't use everything at their disposal when they're writing programs."
Paul added that the potential for logical integrity problems "doesn't hurt you, so long as you never have to go to your backups. There's nothing out there to check logical integrity. It just that things will stop balancing at the end of the month."
What to watch
Neither vendor could cite examples of any competitor's customers who'd
experienced logical integrity problems after online backups were put into
play. Given the nature of the problems stated above, however, customers
with problems may not be aware of them.
HP's Nissen said that "where we disagree with Orbit is in the implicit definition of a transaction." Orbit's product defines implicit as well as explicit transactions and tracks them all to quiesce the database before backing up. But Merit says that TurboStore quiesces files, and Backup/iX quiesces users.
"TurboStore is a dangerous product that has a high potential of with any warning introducing a logically corrupt database into production," Merit said, "by the simple act of restoring it from a backup."
Nissen said "the safe approach is to turn on the IMAGE logging process. TurboStore is integrated with the database such that the customer can not only recover from that log, but they can roll forward." Overhead of IMAGE logging isn't as great as it used to be, he added.