Domino Diva

In working with a customer that is planning to upgrade their server this coming weekend to 8.5.1, there is some confusion regarding whether or not any hotfixes are required for 8.5.1 on the iSeries.  The answer is YES if you are planning to implement DAOS.  There is an issue with the catalog not remaining synchronized. 

The issue was written up in SPR #:
AHOE7W5RTL

There is also a knowledge base article detailing the issue, IBM i: DAOS catalog cannot stay synchronized

If you are implementing DAOS on Domino 8.5.1 on the iSeries, make sure you call IBM support and ask for this specific hotfix.

 I want to remind everyone that the Lotus Domino 8.5.x installations are saved to release V6R1 for the iSeries.  I recently worked with a customer to upgrade their environment to 8.5 and installation of the Domino server code took over 3 hours.  Their OS level is V5R4 and they have an older model 810. 

Once the installation was underway, the CPU utilization on the iSeries server hovered in the 70-95% range for the duration of the installation of the Domino 8.5 code.  There were 3 jobs consuming the majority of the CPU:

LDFX01
LDFX02
LDFX04

These LDFX jobs are the jobs associated with doing the program conversion.  Interestingly enough, they were only visible through the WRKSYSACT command, they did not show up in WRKACTJOB.  This is how the good old CFINIT jobs behave as well.  You have to have the performance tools loaded on your iSeries server to use the WRKSYSACT command.

Beware when installing Domino 8.5.x on a V5R4 system, the program conversions will increase installation time.  While my customer experienced an increase of 3 hours+, newer, faster systems will see shorter program conversion times.  One the newer hardware, the program conversions can be done in around 10 minutes.

Make sure you plan for the added time the installation will take due to the program conversion and ensure you don't decide to do the installation during production hours because of the significant increase in CPU consumption.  The multi-version capability has made this a regular practice in a number of iSeries shops.  While this normally has no impact on production applications, the program conversion can cause performance concerns to other applications running on the iSeries server.

The program conversion will also happen when installing 8.5.x hotfixes on a V5R4 system.

 In working with a customer recently, we were looking for ways to free up some disk space on their iSeries server.  This customer had this as their Domino upgrade path over the past several years:

6.5.0 -> 6.5.1 -> 6.5.2 -> 6.5.3 -> 8.0.1 -> 8.5

The older Domino code loads had all been removed, however the contents of the /QIBM/ProdData/Lotus/DominoXXX directories had not been cleaned up.  We found code in the /Domino650, /Domino 651, /Domino652, and Domino653 subdirectories.  The /Domino801 subdirectory was empty, however the subdirectory was still on the system. 

All of the programs, service programs, and jar files for a specific release of Domino are stored in the /DominoXXX subdirectory for that specific release of Domino server code.  With the actual licensed program product for the release of Domino removed from the server, there was no need to have these directory structures left on the server. 

In order to delete these directories and their contents, we used the Edit File utility (EDTF) which has an option to recursively delete the contents of the subdirectories and then the subdirectories themselves. Upon entering the command below, you will see all of the /DominoXXX subdirectories that exist on your system.  

If you have DAOS implemented on the IBM i (iSeries, AS/400) platform, you need to be aware of two issues you can run into.  Both issues are referenced in tech note 1398097. 

The first issue is related to the 'tell daosmgr prune x' command.  When this command is executed on the iSeries, it ignores the number of days passed in on the command and defaults to 0. 

The second issue affects reading of the daos.cfg file.  If there are more than 40,000 .NLO files in a single DAOS subdirectory, the daos.cfg file is not read correctly.

If you are using DAOS on the IBM i, you should contact support to obtain the proper hotfixes to avoid running into these two issues.

There are two new IBM i blogs I would like to share with you.  The first blog, You and i, is a blog where the IBM i product manager (Craig Johnson) and chief architect (Steve Will), share what is going on with IBM i along with the technical details on how it works and what sets it apart.  If you have ever wondered what Single Level Store is and why it makes the iSeries better, check out Steve's lastest blog post.

The other new blog, iCan, is written by Dawn May.  Dawn is the technical lead and business architect for IBM i.  The focus of her blog is to share features of the operating system.  I know from working with my customers that it's really hard for most customers to keep on top of all of the new enhancements added each release.  Dawn's blog will shed light on various features the system offers.

I hope you find these blogs helpful.

I was recently doing some performance work for a customer and noticed in analyzing their memory checks and memory dumps there was a memory leak.  The application that was causing the memory leak was using LS2J.  I contacted support, and they were able to confirm that the memory leak indeed existed when using LotusScript to call Java.  I checked with another customer that restarts a couple of their application servers every evening.  Sure enough, on those servers, they were executing LS2J code. 

Both of my customers are running on iSeries (AS/400, System i), and it appears this may be the only platform affected, although this is not 100% for sure.  If you use LS2J calls in your applications, I recommend you do memory dumps and memory checks to see if there is indeed a memory leak.  If you find a memory leak, here is the SPR number you can reference when calling the issue into support:

SPR #: AHOE7TPQKC  [SPR number has been corrected]
My customers that encountered the LS2J memory leak are running 7.0.3, 8.0.2, and 8.5.

In my most recent blog entry when I migrated a customer from Domino on Linux to Domino on iSeries ( Fun day tomorrow!  ) I noted there was an issue with the CFGDOMSVR command not taking the TIMEZONE parameter.  I did some checking and found this is a known issue with an open SPR.  This issue is planned to be fixed in release 8.0.3.

The TIMEZONE parameter is new in release 8 on iSeries (AS/400, System i, IBM i, ...).  The parameter allows a Domino server to adopt the timezone of the iSeries it is running on.  The value for this parameter can be set to *SYSVAL or a specific timezone setting.  If *SYSVAL is specified, the value in the iSeries system value, QTIMZON, will be used by the Domino server when setting it's timezone.  The problem is this parameter isn't working correctly.  The result when configuring a release 8.0, 8.0.1, or 8.0.2 server is that the Timezone variable in the notes.ini is set to 0.

Until this issue is addressed in release 8.0.3, your options are to either enter the correct timezone (GMT, CST, CET, ZE4, ...) when configuring the Domino server; or if you choose to specify the value *SYSVAL, you will need to edit the notes.ini of the Domino server and change the value of the Timezone variable as appropriate.  

Tomorrow I will be working with a customer to migrate their Domino environment from Linux to the iSeries.  Woo hoo!!

The Work with Domino Servers (WRKDOMSVR) command is a wonderful utility provided with the Lotus Domino server code for i5/OS.  It provides an interface to work with Domino servers in many ways.  The most commonly used features are starting, stopping, and displaying or working with the Domino console.  There are some options on the WRKDOMSVR command that are seldom used, however they provide some very helpful features.  These features aren't displayed on the main screen for this command, therefore I am finding few people know about them.

These hidden features are options 14, 15, and 16.  Option 14 is 'Work with support files'.  This option takes you directly into the IBM_TECHNICAL_SUPPORT subdirectory of the Domino server's data directory.  Option 15 is 'Work with NSDs' which takes you into the IBM_TECHNICAL_SUPPORT directory and only displays the NSD files.  Option 16 is 'Dump server call stacks'.  This option is especially helpful if you are experiencing a server hang condition or other critical issues with your Domino server.  By using option 16, call stacks are dumped for each active job running on the Domino server.  It used to be that you had to download a special utility to dump the call stacks, now it is as simple as specifying option 16 next to the Domino server!

There is one other option I would like to point out.  This one is shown on the main WRKDOMSVR screen, however I find it is not used as frequently as it could be.  This is option 9, 'Work server jobs'.  I find I use this option all the time.  I can quicly see which jobs are running on the Domino server, what their run state is, how much CPU they are consuming, and the number of threads that are active under the job.  When working with customers, I often see people switching to WRKACTJOB to see the jobs running under the Domino subsystem.  Once I show them how easy it is to get to the information they're looking for by simplying selecting option 9, they are quite pleased.

 I came upon a strange situation with my test Domino server I have on an iSeries (ok, IBM i) server.  When I configured this test server, I neglected to select the option to have the server automatically start when TCP/IP is started.  The iSeries was IPLed over the weekend for some maintenance work.  When I went to access my server today, I had to start it.  Right after starting it, I thought, "I need to change the autostart parameter so this situation doesn't happen in the future".

 

I issued a CHGDOMSVR command right after starting the server, which lead me to a really strange situation.  The server appeared as if it had started normally, however I was unable to access it.  Upon examining the messages on the console, I saw this message:

 

"Warning! Failed to allocate message queue resource; post-crash/fault recovery shutdown system disabled."

 

The server was hung, so I dumped the server call stacks, for futher debug purposes (from option 16 from the WRKDOMSVR command).  A consistent theme I saw in the call stacks for the threads that were hung was:

 

QDOMINO801   LIBNOTES     OSSEM        0000000001   OSLockReadFRWSemWithInfo                        
QDOMINO801   LIBNOTES     OSSEM        0000000032   OSLockReadFRWSemInt                             
QDOMINO801   LIBNOTES     OSSEM        0000000001   WaitOnNativeSemaphoreCounted                    
QDOMINO801   LIBNOTES     OSSEM        0000000025   WaitOnNativeSemaphore                           
QDOMINO801   LIBNOTES     OSSEM        0000000017   WaitForThreadSem                                
QSYS         QP0WPTHR     QP0WCOND     0000000049   pthread_cond_wait                               
QSYS         QP0WPINT     QP0WSCOND    0000000086   wait__20Qp0wPthreadConditionFP7Qp0wTcbP9Qp0wMutex

 

After dumping the call stacks, I ended the Domino server, *CNTRLD, of course.  This did nothing, the server was hung.  I next did a *IMMED end on my test server.  The interesting thing is that the status of the Domino server showed *ENDED, however when I used option 9 to look at the active server jobs, I saw that the Server job was stil active. 

 

The console was now showing "- Child stopped by signal -1". 

 

I tried ending the server with the *IMMED option, and received error LNT0922. 

 

Message: Error 11 has occured trying to end the Domino sever, DOMTEST1, the error code is 11.

Cause: Error trying to end server DOMTEST1 immediately.  Subsystem is QUSRNOTES/DOMTEST1.

Recovery: Save this job log and contact your service representative.

 

Well, that would be me!!  At this point I had to do an ENDJOBABN to get the Server job ended.  This did the trick. 

 

Interestingly enough, when I tried to do a CHGDOMSVR command on the server with the server and the subsystem ended, I received a CPD680B message.

 

Message: Error detected in prompt override program command string.

Cause: An error was detected in the command string created in the prompt override program QNNINPOP in QDOMINO801.  Processing of the command will not be affected by this error.

Recovery: After prompting is complete, see the previously listed messages in the job log to determine the cause of the error or see the CL topic in the Information Center book for more informaiton on prompt override programs.

 

Override what?!?  I checked the job log and saw these messages:

 

Character 's' not valid following string ' 'Kim'        '.

Character ' ' ' not valid following string 'Frankie      '.

A matching apostrophe not found.

A matching parenthesis not found.

Command not valid.

 

What it was complaining about was the title of my server, which is "Kim's Test Domino Server on Frankie".  Interestingly enough, the CFGDOMSVR command had no issue with me putting this in for the server's title when I configured it, however the CHGDOMSR command wouldn't accept it.

 

I was able to get the autostart parameter changed by qualifying the specific parameters I needed on the CHGDOMSVR command:

 

CHGDOMSVR SERVER(domtest1) AUTOSTART(*yes)

 

The moral of the story is never issue the CHGDOMSVR command against a Domino server when it's just starting.  The normal function of the CHGDOMSVR command is that it will let you change this paramter while the server is active.  The trick is that the server needs to be actually started when the command is issued.  There is a glitch when the server is starting where this command gets really messed up. 

 I received a call from a customer this morning saying noone could access their mail files.  The really strange thing about the situation was that the log.nsf, names.nsf, ... could be opened.  It was just a problem with the mail files.  When users tried to open their mail files, they received the following error message.

 

"Server Error: Device is Busy/In Use"

 

The console showed messages like this for mail files the router was trying to deliver mail to:

 

"Error delivering to servername/domain mail/xxxxxxxxxxxx.nsf; Device is Busy/In Use"

 

I dumped the server call stacks for all Domino jobs on that server and the job call stacks for any jobs I thought could be suspect.  I looked through the output and didn't find anything suspicious.  Next I invoked the QP0FPTOS api to identify if there were any IFS locks on the objects. The output showed there indeed were IFS locks on the files.  I issued the RLSIFSLCK command against a few mail files.  This did no good, the objects remained locked.  The only time the RLSIFSLCK command cannot release a lock is if it is a directory lock or if there is a save lock. 

 

This tipped me off to look at the job log for the save job.  This customer takes their Domino servers down every evening, saves the data directories, and then starts the servers after the save is complete.  The save output showed that the first Domino server had been saved fine, however the save job for the second server, the server we were having this issue with, was still active.

 

Once the save job was ended, the problem went away, no server restart was required.  The mail files are able to be accessed again and the customer is very happy.

 

Now it's time to get outside and enjoy this beautiful weather since it's the weekend!

 It's official, at the COMMON conference last week, Mark Shearer and Ross Mauri announced the new IBM Power Systems.  Both the System i and System p have shared the same hardware for a number of years.  What's great about this announcement is what Ross Mauri explained to the COMMON attendees at the IBM Town Hall session last Wednesday morning.  Ross Mauri is the General Manager for the Power Systems Division at IBM.

 

The big news Mr. Mauri shared with the crowd is that there will no longer be separate pricing for System i and System p hardware.  Wtih the new Power Systems announcement, the price i customers will pay will be same price p customers will pay for the hardware.  This announcement brought a loud applause from the crowd, as i customers have traditionally paid more for memory, disk, and other hardware components than p customers. 

 

I think this announcement is a very good thing for the System i.  Not only is there pricing equivalency between the hardware platforms, but now when IBM Power Systems is advertised, i will also be advertised.  This is something we have needed for a long, long time, more advertising.  Things are definitely looking up for the platform we now know as the former System i !!

I just tried to log into a System i server that is currently not available and had to laugh at the message I received from my Mochasoft client.

 

 

 

They can try to rename it, but it will alway be 'AS/400'  : ^)

 

Check out the title of this blog, notice it isn't AS/400?  Yeah, the '/' character isn't allowed in the title <grin>.

If you are looking for information related to the program conversion on System i that happens when upgrading to V6R1, you may want to check out some recent articles I wrote.  The intention was to have one article, but it ended up being too big to fit in the space available in the magazine.  The compromise was to split it into the main article included in the magazine and two web exclusives.  Here are links to all three articles:

 

"Preparing for V6R1, What to Expect During Your Conversion"

 

"The Impact on Domino"

 

"Tips for Working With the V6R1 Conversion Tool"

 

One of the biggest gotchas, IMO, is that the program conversion tool doesn't pick up the Domino libraries by default when estimating how long the program conversion times for libraries will take.  This would be a nasty one for customer to miss, as conversion of the Domino libraries can take over an hour in some worse case scenarios.  Even worse, if the customer doesn't realize that Domino gets converted because the tool doesn't report it by default, they may not find out about the program conversion until they start the Domino servers after the upgrade to V6R1.  Because of the time required to convert the Domino libraries, the server most likely won't start and could produce some strange errors.  This would not be a fun upgrade experience. 

 

Hopefully you will find these articles helpful in planning any upgrades to V6R1.

 In a conversation with a new customer this afternoon, I heard a common theme come through; multi-versioning is not well understood.  I have had similar conversations with a number of customers over the past few years. The net is that customers do not understand what multi-versioning is and how it can benefit their Domino installations.

 

Domino on the System i (iSeries, AS/400) has had multi-versioning since releases 6.0.3 and 6.5.0.  This support allows more than one version of Lotus Domino to be installed and running on your iSeries at the same time.  For example, a single iSeries box or iSeries LPAR could have releases 6.5.6, 7.0.2, 7.0.3, and 8.0.1 installed and running. The main areas of confusion I have found related to this support in my conversations with customers are related to these three areas:

 

1) how it affects the install process

2) how servers are upgraded to the new release after the new code installation

3) do all servers have to be upgraded at the same time

 

How it affects the install process

I like to use multi-versioning support to, at a minimum, install the new version of Lotus Domino.  With multi-versioning support, I can leave production servers active while installing the new version.  This saves about 1/2 hour of downtime for the customer.  It also allows the customer the flexibility of installing the new code version during production hours, so they are ready to update their Domino servers whenever is convenient for them. 

 

How do servers get updated after the installation

There is an UPDDOMSVR (Update Domino Server) command that allows you to specify which server you want to take to the new release level.  When you are ready to update a Domino server, simply end the server and issue the UPDDOMSVR command.  The parameters to the command are the name of the Domino server and the release level you would like to upgrade the server to.  It really is that simple.

 

Do all servers have to be upgraded

This is a very simple answer, no.  You can pick and choose which servers to upgrade.  For example, if a customer has three servers sitting at releast 6.5.6 and they want to upgrade one of the servers to release 7.0.3, they have that flexibility.  Better yet, if a customer wants to check out the new release 8.0.1 that came out yesterday, then can simply install the new code release, configure a test server with this new code release, while leaving their existing servers untouched.

 

Multi-versioning is really that simple!