Domino Diva

Failed to allocate message queue resource

Kim Greenekim@kimgreene.com  

 I came upon a strange situation with my test Domino server I have on an iSeries (ok, IBM i) server.  When I configured this test server, I neglected to select the option to have the server automatically start when TCP/IP is started.  The iSeries was IPLed over the weekend for some maintenance work.  When I went to access my server today, I had to start it.  Right after starting it, I thought, "I need to change the autostart parameter so this situation doesn't happen in the future".

I issued a CHGDOMSVR command right after starting the server, which lead me to a really strange situation.  The server appeared as if it had started normally, however I was unable to access it.  Upon examining the messages on the console, I saw this message:

"Warning! Failed to allocate message queue resource; post-crash/fault recovery shutdown system disabled."

The server was hung, so I dumped the server call stacks, for futher debug purposes (from option 16 from the WRKDOMSVR command).  A consistent theme I saw in the call stacks for the threads that were hung was:

QDOMINO801   LIBNOTES     OSSEM        0000000001   OSLockReadFRWSemWithInfo                        
QDOMINO801   LIBNOTES     OSSEM        0000000032   OSLockReadFRWSemInt                             
QDOMINO801   LIBNOTES     OSSEM        0000000001   WaitOnNativeSemaphoreCounted                    
QDOMINO801   LIBNOTES     OSSEM        0000000025   WaitOnNativeSemaphore                           
QDOMINO801   LIBNOTES     OSSEM        0000000017   WaitForThreadSem                                
QSYS         QP0WPTHR     QP0WCOND     0000000049   pthread_cond_wait                               
QSYS         QP0WPINT     QP0WSCOND    0000000086   wait__20Qp0wPthreadConditionFP7Qp0wTcbP9Qp0wMutex

After dumping the call stacks, I ended the Domino server, *CNTRLD, of course.  This did nothing, the server was hung.  I next did a *IMMED end on my test server.  The interesting thing is that the status of the Domino server showed *ENDED, however when I used option 9 to look at the active server jobs, I saw that the Server job was stil active. 

The console was now showing "- Child stopped by signal -1". 

I tried ending the server with the *IMMED option, and received error LNT0922. 

Message: Error 11 has occured trying to end the Domino sever, DOMTEST1, the error code is 11.

Cause: Error trying to end server DOMTEST1 immediately.  Subsystem is QUSRNOTES/DOMTEST1.

Recovery: Save this job log and contact your service representative.

Well, that would be me!!  At this point I had to do an ENDJOBABN to get the Server job ended.  This did the trick. 

Interestingly enough, when I tried to do a CHGDOMSVR command on the server with the server and the subsystem ended, I received a CPD680B message.

Message: Error detected in prompt override program command string.

Cause: An error was detected in the command string created in the prompt override program QNNINPOP in QDOMINO801.  Processing of the command will not be affected by this error.

Recovery: After prompting is complete, see the previously listed messages in the job log to determine the cause of the error or see the CL topic in the Information Center book for more informaiton on prompt override programs.

Override what?!?  I checked the job log and saw these messages:

Character 's' not valid following string ' 'Kim'        '.

Character ' ' ' not valid following string 'Frankie      '.

A matching apostrophe not found.

A matching parenthesis not found.

Command not valid.

What it was complaining about was the title of my server, which is "Kim's Test Domino Server on Frankie".  Interestingly enough, the CFGDOMSVR command had no issue with me putting this in for the server's title when I configured it, however the CHGDOMSR command wouldn't accept it.

I was able to get the autostart parameter changed by qualifying the specific parameters I needed on the CHGDOMSVR command:

CHGDOMSVR SERVER(domtest1) AUTOSTART(*yes)

The moral of the story is never issue the CHGDOMSVR command against a Domino server when it's just starting.  The normal function of the CHGDOMSVR command is that it will let you change this paramter while the server is active.  The trick is that the server needs to be actually started when the command is issued.  There is a glitch when the server is starting where this command gets really messed up. 

• Comments[0]

Server Error Device is Busy In Use

Kim Greenekim@kimgreene.com  

 I received a call from a customer this morning saying noone could access their mail files.  The really strange thing about the situation was that the log.nsf, names.nsf, ... could be opened.  It was just a problem with the mail files.  When users tried to open their mail files, they received the following error message.

"Server Error: Device is Busy/In Use"

The console showed messages like this for mail files the router was trying to deliver mail to:

"Error delivering to servername/domain mail/xxxxxxxxxxxx.nsf; Device is Busy/In Use"

I dumped the server call stacks for all Domino jobs on that server and the job call stacks for any jobs I thought could be suspect.  I looked through the output and didn't find anything suspicious.  Next I invoked the QP0FPTOS api to identify if there were any IFS locks on the objects. The output showed there indeed were IFS locks on the files.  I issued the RLSIFSLCK command against a few mail files.  This did no good, the objects remained locked.  The only time the RLSIFSLCK command cannot release a lock is if it is a directory lock or if there is a save lock. 

This tipped me off to look at the job log for the save job.  This customer takes their Domino servers down every evening, saves the data directories, and then starts the servers after the save is complete.  The save output showed that the first Domino server had been saved fine, however the save job for the second server, the server we were having this issue with, was still active.

Once the save job was ended, the problem went away, no server restart was required.  The mail files are able to be accessed again and the customer is very happy.

Now it's time to get outside and enjoy this beautiful weather since it's the weekend!

• Comments[3]

V6R1 Program Conversion on System i

Kim Greenekim@kimgreene.com  

If you are looking for information related to the program conversion on System i that happens when upgrading to V6R1, you may want to check out some recent articles I wrote.  The intention was to have one article, but it ended up being too big to fit in the space available in the magazine.  The compromise was to split it into the main article included in the magazine and two web exclusives.  Here are links to all three articles:

"Preparing for V6R1, What to Expect During Your Conversion"

"The Impact on Domino"

"Tips for Working With the V6R1 Conversion Tool"

One of the biggest gotchas, IMO, is that the program conversion tool doesn't pick up the Domino libraries by default when estimating how long the program conversion times for libraries will take.  This would be a nasty one for customer to miss, as conversion of the Domino libraries can take over an hour in some worse case scenarios.  Even worse, if the customer doesn't realize that Domino gets converted because the tool doesn't report it by default, they may not find out about the program conversion until they start the Domino servers after the upgrade to V6R1.  Because of the time required to convert the Domino libraries, the server most likely won't start and could produce some strange errors.  This would not be a fun upgrade experience. 

Hopefully you will find these articles helpful in planning any upgrades to V6R1.

• Comments[0]