Delta can reduce the cost of confidence

ORBiT’s newest Backup+ option takes 3000 partial backups to new level

Review by Shawn Gordon

ORBiT has been around for many years, selling high-speed and online backup for HP systems. I’ve used its software myself for over 10 years now. For the last few years the company has been talking about a radical new approach to backups on the HP 3000 called Delta backup, and I was very excited to get my hands on it.

Delta backup takes the concept of a partial backup to a new level. Instead of storing the files that have been accessed since the last backup, it actually stores the pieces of the files that have changed, not the entire file. So in the case of a database, you could have a table with 20 million records in it — but if you only add 10,000 a day, and you do a Delta backup every day, then you will only store the 10,000 records.

This new functionality is an optional part of Backup+, so you can still do your online backups and zero-downtime backups and all the other great tricks that are part of Backup+. But Delta gives you a new weapon in your arsenal for improving uptime on the HP 3000, a system that lives on its uptime reputation.

How does it work?

The focus of this review is the Delta feature of Backup+, since that is the really new piece of the product. I’ll talk a bit about some of the other features, but Delta is the neatest bit. It takes the concept of a partial backup and makes it more granular: we are now talking about pages within files that have been touched, instead of whole files.

The first time you do a baseline backup for Delta it starts a file monitor that tracks the changes to the fileset that was used for the baseline. This is cumulative, so the pieces of files that were modified on Monday are also on Wednesday’s backup. This monitoring technology looked similar to what is used in the product’s Online backup, at least from what I could see. Because of this logging process, it means there can be no more than five Delta or Online (cumulative) backups going on at once.

Let me explain this a little more, since it can be confusing at first. Because of the logging that goes on, you could have done four different Delta baseline stores, for example, @.@.PROD, @.@.AR, @.@.DEV, and @.@.QA. Once you have done the baseline Delta backup, these logging processes stay live.

So say you want to go and do an ONLINE backup of some other fileset. This would still be acceptable, because you are now at the limit of five. But if you had two tape drives, and tried to do two online stores at the same time, this would exceed the threshold of available logging. To me this seems like a pretty reasonable limit, just something you’ll need to remember.

Features

Backup+ contains many features to allow for high speed, online backup in addition to its Delta backup option.. Options such as ONLINE and ZERODOWN let users continue to work during a backup, and once the synchronization point is reached, they will be suspended for a minute or so while the tape finishes writing.

Backup+ also has compression options to improve your tape usage. But if your tape drive supports hardware compression, you are better off using that, because it will be faster, typically, than software compression. You don’t want to use both, because that will slow down a backup and probably make it larger. The product offers a very robust file selection protocol for STORE and RESTORE that allows you to use a wide variety of file objects such as type and various dates to select files, rather like MPEX.

Backup+ also offers store-to-disk, a very cool option. I did a test storing a database to disk, and it was so fast and so small that I thought there must be a problem. I restored the database into another group, and it was fine. If you have the disk space for it, the store-to-disk is a great way to do your backup quickly, and then move the file off to tape.

There is only one thing that I don’t like about the Backup+ store-to-disk feature. I found the file type which it creates so bizarre that I couldn’t use any utility of recent vintage, like FTP, to move the file between systems. ORBiT told me that you can transfer this disk to disk backup to a target machine by using DSCOPY. But you need to do two of copies of each backup, one to copy the directory of the file, and the other to copy the data in the file. I would find this a very fast and convenient feature to get a fileset onto a shadow machine — if there was a better way to transfer the files.

A couple of options you can use are the Tape Manager and Librarian (TML) and Restore Wizard. TML is a very full-featured tape library and management system that lets you track everything, everywhere. It is extremely robust — but it has a pure command-line interface, which can make it a bit tedious to use, since you need to learn the commands. I was hard-pressed to find any features that TML was lacking, and overall it is well implemented.

One interesting part of TML is the Restore Wizard. As designed, it can be given a specific restore instruction and tell you exactly what tape or tapes to use. Of course, it relies on the fact that you are properly labeling your tapes as required — there is no way around this; there has to be some human intervention.

Virtually any type of storage device is supported by Backup+, including auto-changers and such. You can make even more frugal use of your tapes by using appended backups, although this might become confusing unless you carefully track what is where. There is a lot in Backup+, and while you can get by with using just a few of its features, you should at least browse what is available to see what else you might find useful.

One important note that people always wonder about when using third-party backup products: Backup+ uses the MPE STORE facility to put a small version of Backup+ at the front of a backup tape. This small version will restore whatever is on that tape, so you will always be able to restore your information without a problem.

Installation and Documentation

For the installation you restore a single file, put the tape back online, run the program you restored, answer a couple of questions. Then all the files are restored, directory structure adjusted as required, and installation programs are run.

I was rather disappointed with the documentation. It tops out at a hefty 500 pages, and I found a fair number of errors and duplicated information. It’s not really cohesive — and for this new user, I found it very daunting. I was also disappointed that it was only distributed as a PDF file on the HP 3000. You can get a hard copy of the manual from ORBiT , but this PDF format is the default.

The Test Drive

Well, the first thing to do to test Delta is to get a baseline store of your fileset. For my purposes I did an @.@.@ store of the whole system. On my box (a Series 957) and tape drive it took 114 minutes to store 1.2Gb of data. I then ran a process that touched most of the files on the system but only updated a couple of datasets and flat files.

Then, to test Delta, I ran the Delta backup specifying the prior baseline ID for the backup. This time the backup took two minutes, and only stored those pieces of the files I was concerned with.

During the process I encountered an error message on a database that Delta tried to backup. The database had worked fine before the backup. It turns out that this was a bug in Backup+ that was fixed just a week before I began to test the software. ORBiT said it had shipped a patch to fix this DBQUIESCE error, I just didn’t have it on my test system. I had no idea what the error message meant, although support said they could have explained it to me quickly. I think these error messages need to be more easily understood, without calling support.

The next step is the Restore process. Here is where I found the Delta backup process confusing, and it’s a good thing that that it is integrated with TML. I physically purged one of the dataset files that was modified between the baseline store and the Delta store. Then I wanted to restore the dataset file to its state after the Delta store, so this became a two-stage process. First I had to restore the file from the baseline store, then I had to take the tape with the Delta iteration that I was interested in and restore the file again from that tape.

While this might sound like a bit of extra work, just remember that you don’t restore files that often — it is typically for emergency purposes. One thing to keep in mind: the Delta backups are cumulative. They are much like an MPE Store partial backup, it’s just that we are cumulatively storing the pieces of the file. So if you have the need to get a file from its state on the third Delta of a set, you only have to restore the baseline and the third Delta. You don’t have to restore the first and second Deltas, which certainly reduces the confusion.

If you try to specify BASELINE on a restore from a tape that is a DELTA, I found that Backup+ told me very clearly that I had the wrong tape, and instructed me to put in the right one.

Conclusions

The Delta feature of Backup+ is fascinating, important and unique technology for HP e3000 customers. People now have enormous disk farms that have to be backed up, and systems require constant uptime. You can do things like buy a backup-only system, shadow to it and just back it up — but given the large amounts of data involved at some shops, that still might not be practical. Using something like the Delta feature in Backup+, in conjunction with the product’s Online and Zero Downtime options, can radically improve throughput in performance.

When using Delta, you will still need to do a new baseline backup every week or so — but it does dramatically improve your overall system availability and reduce your cost of ownership for backups. DLT tapes aren’t cheap, and neither are people who have to sit around and change tapes. Reducing all of that adds up over time.

While I wouldn’t say that the Delta option is something everyone needs, I can certainly recommend the Backup+ product to anyone that is doing HP e3000 backups. It’s certainly faster and more convenient than MPE’s Store. ORBiT has some of the brightest developers of any company in the 3000 community, a fact that has always given me a high confidence level in their software.