By Craig Solomon

This year’s HP World was abuzz with new products, technology, problems and issues, but the top question there seemed to clearly be “should we upgrade to MPE/iX 6.5?” HP’s street name for the 6.5 release is “The Amisys release,” because this version was designed and tuned for high-end systems running a complex and demanding environment like the Amisys HMO software.

Release 6.5, unlike any prior release, seems truly to be designed for supporting multiple-processor e3000 systems, which currently run up to 12 processors. MPE/iX 6.5 now supports 16Gb of main memory on the high-end systems (997), but only 8Gb of main memory in midrange systems (9x9KS)

Some of the new and unique features in MPE/iX 6.5 address the need for zero downtime, a benefit that HP CEO Carly Fiorina has begun to call “the Always On infrastructure.” The e3000’s new integrated transaction manager (XM), updated in 6.5, greatly improves the odds of recovering from any service interruptions without any loss to data.

Recovery and reboot are said to be much quicker while using 6.5 from failure to function, but I have not seen any actual numbers from real environments. One of the keys to this quick recovery is supposed to be the revamped HP AutoRestart/iX that quickly and automatically resets the e3000 during a system failure. AutoRestart/iX performs a dump to disk file and quickly restarts the box.

6.5 also offers also a new online dump facility called “SubSystem Dump.” I have been told it will allow a system manager to do a Key SubSystem Dump to disk while the system is still up and online. This allows HP to dial up and analyze the dump without having to reboot the e3000. I have not seen this happen in a production data center. In addition, HP has changed and reduced the number of errors that could trigger the system to abort, helping to insure higher uptime.

New features

Improvements and modifications in HP’s Workload Manager/iX — a separate purchase — make the tool PerfView more necessary then ever before. Adding PerfView could have its benefits if a 3000 site is in a multi-platform environment, since PerfView will have the same look and feel across HP’s Unix and NT systems.

Workload Manager/iX provides invaluable power. It goes beyond defined queue boundaries, along with the ability to allocate specific CPU resources to mission-critical processes in order to permit them to run when needed without being hindered by other applications.

I don’t know much about Threshold Manager; what I am told is that it’s a free monitoring tool that looks at crucial system tables. If Threshold Manager believes that a table will overflow, the software will notify the OP or SM and restrict system activity until the specific resource is cleared. If it truly works this well and this easy, it is certainly a nice bonus tool.

Performance issues

The original goal for 6.5 performance improvement by HP was to have a 30 percent increase on a MPE/iX 997/1200 over a 997/800 running MPE/iX 6.0. HP architect Kevin Cooper reports numbers which show a 44 percent increase between these two systems.

The 6.5 release features major changes in the Transaction Manager (XM). The new command “alterchkptpri” within the VOLUTIL utility allows the SM to change the checkpoint from 30 to 152.

Remember, as of MPE/iX 6.0, the XM log file can be resized within VOLUTIL. HP did release a patch for MPE/iX 5.5 to allow this to happen (MPEKXG8), but most users were unaware of it, and did not need this patch. I mention this under performance issues to illustrate the concern I have for sites which use this option and hope to improve performance — instead of using it as intended.

The HP 3000 transaction manager tracks system transactions, and when the XM log file reaches a 50 percent capacity, the XM flushes this modified /dirty data back to disk. On the larger, more intense e3000s, certain transactions were not completing prior to the XM log file reaching this point of saturation. By increasing the size of the log file, larger transactions can now complete before the XM needs to flush the log file back to disk.

There are other performance issues as well. If a site does not currently have any issues that require any XM log file modifications, and does modify the file, then performance bottlenecks could occur. If a site increases the site of the XM log file, then when the XM log file reaches the saturation point and needs to flush back to disk, this process most likely will cause a noticeable impede and pause users for several seconds while the XM performs this flush. This can have a direct, negative impact on both user response times and batch completion time.

However, if a site reduces the size of the XM log file, then these flushes most likely will happen more frequently — and although the impedes will be shorter in time, the frequency could be just as annoying and hinder performance on a regular basis.

HP offers this note on the new XM performance issues:

A large amount of internal data becomes irrelevant after completion of an extended transaction (any dbput/dbdelete/dbupdate is an extended transaction). This allows for more transactions even within the current 4Mb limit.”

Another enhancement to the XM is to increase the userlog. The current userlog size is 64Mb per volume set. Currently for each transaction the information that can be contained in the userlog is limited to 4Mb. Due to the increase in the transaction size for each process (to 32Mb), the total userlog file should also be increased. The userlog resides on the master volume of a volume set. A systems administrator can use VOLUTIL utility to increase the size of the userlog (the default value is 64Mb) with the following command from VOLUTIL: ALTUSERLOG <volset-name> <new size>

LDEV1 is the exception to the XM log file rule and should not be changed, and if changed should not be adjusted beyond 264256 sectors, or 64Mb. This is due to a specific reserved space. The XM log file cannot be resized beyond that point. Any changes made won’t occur until the next reboot, since XM log files cannot be modified while they are in use.

To see what size the XM log file is currently (in sectors), a user can enter SHOWUSERLOG <volumeset-name> from within VOLUTIL.

Overall, I believe MPE/iX 6.5 is designed for high-end e3000 systems. It is true that certain systems, mostly midrange and single processor small memory systems, could have some performance hits if they migrate to 6.5. HP claims that the new performance enhancements were designed to not degrade any size e3000. However, that does not address issues in performance utilizing other areas of the 6.5 release.

One of the best 6.5 enhancements to improve performance was to remove the bottleneck called “Spin Wait.” That’s a state where every CPU shares a single lock for all activity relating to the memory manager. If this resource was already locked by another process, the new request would be caught in a Spin Wait loop until the resource was freed. With the release of 6.5, MPE has a new locking mechanism inside the MPE Memory Manager that brings all locks down to a more detailed or granular level. This allows different CPUs to address memory management, at the same time greatly reducing or eliminating the chance of Spin Wait.

The final and most unique of all changes in 6.5 is its new Multiple Run Queues. This change is only activated on the 997/1000 and 997/1200. Spin Wait came into affect here as well. If a CPU attempted to lock this specific resource and the resource was already locked, then MPE once again went into a Spin Wait Loop. Now with the multiple run queues, each CPU has its own run queue. This release should be a significant performance boost for large Amisys and Smith-Gardner shops, especially where there is heavy batch activity. I also expect it to benefit sites where Netbase Clustering is utilized.

Craig Solomon (craig@craigs.com) began his consulting career 18 years ago and has worked and consulted for companies such as Kinko’s, Glaxo-Wellcome Pharmaceuticals UK, and Lund Performance Solutions. He is the founder of the IT Consulting Consortium, an HP e3000 performance and tuning expert, and an international speaker on IT solutions, performance and superior customer service.