Domino Diva

Last evening I read a wonderful article about women in IT (Making the Climb Women in IT Team Up To Succeed).  The article featured three different women and how they have been working to help other women succeed in IT.  The three women are Carolyn Leighton, Dee Dee Lear, and Sandy Carter.  Carolyn and Dee Dee made some great points that really struck me as I read the inserts about each of them.  Carolyn Leighton is the founder of Women in Technology International (WITI).  In the excerpt about her (Carolyn Leighton), she brought up a very critical point.  Carolyn states "today our biggest obstacles to having women as CEOs is how women view themselves".  This is so true.  I was told by my family I couldn't go to college because I was a girl and didn't need to.  I didn't believe that and therefore didn't let that stop me.  In general, I have never let other people's perceptions of what their role of women in society is stop me from doing what I wanted to do and was interested in doing.  Carolyn Leighton has started 6 companies, she clearly hasn't let her vision be stunted.

In the excerpt about Dee Dee Lear, she makes a very interesting observation.  She talks about young women graduating from college now and states "They have an untainted view and they don't see the glass ceiling.  It's the 'me generation'.  This generation doesn't let the outside world determine who they'll be and they aren't intimidated.  They've seen women as CEOs of large, worldwide Fortune 100 companies and women running for the highest political offices in their countries."

Sandy Carter has done some amazing things with the EXploring Interests in Technology and Engineering (EX.I.T.E) Camp for elementary girls.  She explains in the article that in the first camp, zero of the 100 girls in the camp were interested in IT at the start of the camp.  During the camp, the girls programmed a robot, created a Web site and pretended to be sales reps.  At the end of the camp, 98 out of the 100 girls wanted to pursue a career in IT.  This ties directly into Carolyn Leighton's point that the biggest obstacles women have today is how they view themselves.

I hope you enjoy reading the article as much as I did.

I hadn't defragged my hard drive for a while.  I used Power Defragmentor this evening (which I highly recommend, http://power-defragmenter-gui.en.softonic.com/).  It has been over a few weeks since I ran my last defrag, I know, bad!  Prior to running Power Defragmentor, I had 3.87 GB free on my hard drive.  After running Power Defragmentor, I had 8.15 GB free.  I really need to schedule this to run more frequently.  Happy Defragging!

Today is the 9th anniversary of starting my company.  I can't believe it's been that long!!

I have a customer that has about 550 users; of these 550 users, about 12 were reporting that they were receiving the OOO reminder email every morning, even though they had turned their OOO agents off.  The document properties showed the messages were received every morning at 3:59 AM.  About half of the users were using the iNotes6 template and the other half were using the Std6Mail template. The server level is 6.5.6 FP3. 

 In trying to figure out why these handful of users were receiving the reminder message to turn their OOO agents off, I did the following:

• checked that their OOO agents were not enabled
• ensured the users did not have local copies or replicas that may be causing issues
• made sure the users didn't have local archives with the OOO agent enabled
• checked for replicas on other servers in the domain
• checked the schedule of all agents on the server and din't see any OOO agents scheduled for these users nor were there any agents scheduled to run at 3:59 AM
• using NoteMan, looked at the contents of the OutOfOfficeProfile documents and ensured the documents were not corrupt in any way
  
• deleted the OutOfOficeProfile profile documents for the affected users
  
• deleted the form (OutOfOfficeProfile) and the agents OutOfOffice and (Edit Office Profile) from the affected mail files and then refreshed their design so these design elements were pulled from the template [prior to doing so, checked the templates to make sure the OOO agent wasn't enabled in the templates, everything checked out just fine]
• checked the document properties of the OOO agent reminder emails and looked for clues there, this turned up nothing as well
  

With all of these items turning up empty, I next enabled debug to capture agent execution details and then combed through the output.  This revealed the following message being generated for each of the affected users at 3:59 AM.

"AMgr: Received new mail event from Router for 'mail/xxxxx.nsf' "

So indeed the users were being sent the OOO remidner emails, but still no clues as to what was causing the reminder emails to be sent.

In looking at the tasks running on the server, I noticed the UPDATE task was not running.  The ServerTasks= line in the notes.ini had been edited and this task had been removed from the ServerTasks variable.  When inquiring about this, I found out this was a change a consultant that had previously been engaged had made.  There had been no specific reason given for removing the task.

I had the customer add the UPDATE task back to the ServerTasks= variable and then do a 'load update' at the console to get the task running right away.  This fixed the problem, no more OOO reminder messages were sent to the affected users.  I have to admit this is the strangest OOO agent issue I have worked on.

In my most recent blog entry when I migrated a customer from Domino on Linux to Domino on iSeries ( Fun day tomorrow!  ) I noted there was an issue with the CFGDOMSVR command not taking the TIMEZONE parameter.  I did some checking and found this is a known issue with an open SPR.  This issue is planned to be fixed in release 8.0.3.

The TIMEZONE parameter is new in release 8 on iSeries (AS/400, System i, IBM i, ...).  The parameter allows a Domino server to adopt the timezone of the iSeries it is running on.  The value for this parameter can be set to *SYSVAL or a specific timezone setting.  If *SYSVAL is specified, the value in the iSeries system value, QTIMZON, will be used by the Domino server when setting it's timezone.  The problem is this parameter isn't working correctly.  The result when configuring a release 8.0, 8.0.1, or 8.0.2 server is that the Timezone variable in the notes.ini is set to 0.

Until this issue is addressed in release 8.0.3, your options are to either enter the correct timezone (GMT, CST, CET, ZE4, ...) when configuring the Domino server; or if you choose to specify the value *SYSVAL, you will need to edit the notes.ini of the Domino server and change the value of the Timezone variable as appropriate.  

Tomorrow I will be working with a customer to migrate their Domino environment from Linux to the iSeries.  Woo hoo!!

In working with a customer to do some performance tuning of their Domino environment, it became very apparent that Domino release 8 requires less memory.  This particular customer is running 7 Domino servers on one LPAR on a 550 (iSeries).  Three of the Domino servers provide mail and calendaring services for about 1500 users, one server is a Sametime server providing instant messaging services, and the other three servers are used for development and testing of applications that integrate with back-end data. 

The customer was running Domino 6.5.5 prior to the upgrade.  They upgraded to release 8.0.1 on April 19.  The graph below shows a significant reduction in paging and faulting rates as soon as the upgrade happened.


The paging and faulting rates went from around 700-800 faults/second down to 300-400 faults/second.  There were no changes to the NSF buffer pool sizes for any of the servers, the only thing that changed was the release of Domino.  The graph shows pool 2, which is the base memory pool for i5/OS.  This is where all Domino servers run by default, so this was a pretty standard setup.

This significant reduction in paging and faulting would result for a customer with a similar configuration upgrading from release 7.  I have found there is very little different between release 6 and 7 servers from a memory management and consumption perspective.  Release 8 however is significantly more efficient when it comes to memory consumption. 

The Work with Domino Servers (WRKDOMSVR) command is a wonderful utility provided with the Lotus Domino server code for i5/OS.  It provides an interface to work with Domino servers in many ways.  The most commonly used features are starting, stopping, and displaying or working with the Domino console.  There are some options on the WRKDOMSVR command that are seldom used, however they provide some very helpful features.  These features aren't displayed on the main screen for this command, therefore I am finding few people know about them.

These hidden features are options 14, 15, and 16.  Option 14 is 'Work with support files'.  This option takes you directly into the IBM_TECHNICAL_SUPPORT subdirectory of the Domino server's data directory.  Option 15 is 'Work with NSDs' which takes you into the IBM_TECHNICAL_SUPPORT directory and only displays the NSD files.  Option 16 is 'Dump server call stacks'.  This option is especially helpful if you are experiencing a server hang condition or other critical issues with your Domino server.  By using option 16, call stacks are dumped for each active job running on the Domino server.  It used to be that you had to download a special utility to dump the call stacks, now it is as simple as specifying option 16 next to the Domino server!

There is one other option I would like to point out.  This one is shown on the main WRKDOMSVR screen, however I find it is not used as frequently as it could be.  This is option 9, 'Work server jobs'.  I find I use this option all the time.  I can quicly see which jobs are running on the Domino server, what their run state is, how much CPU they are consuming, and the number of threads that are active under the job.  When working with customers, I often see people switching to WRKACTJOB to see the jobs running under the Domino subsystem.  Once I show them how easy it is to get to the information they're looking for by simplying selecting option 9, they are quite pleased.

 I came upon a strange situation with my test Domino server I have on an iSeries (ok, IBM i) server.  When I configured this test server, I neglected to select the option to have the server automatically start when TCP/IP is started.  The iSeries was IPLed over the weekend for some maintenance work.  When I went to access my server today, I had to start it.  Right after starting it, I thought, "I need to change the autostart parameter so this situation doesn't happen in the future".

 

I issued a CHGDOMSVR command right after starting the server, which lead me to a really strange situation.  The server appeared as if it had started normally, however I was unable to access it.  Upon examining the messages on the console, I saw this message:

 

"Warning! Failed to allocate message queue resource; post-crash/fault recovery shutdown system disabled."

 

The server was hung, so I dumped the server call stacks, for futher debug purposes (from option 16 from the WRKDOMSVR command).  A consistent theme I saw in the call stacks for the threads that were hung was:

 

QDOMINO801   LIBNOTES     OSSEM        0000000001   OSLockReadFRWSemWithInfo                        
QDOMINO801   LIBNOTES     OSSEM        0000000032   OSLockReadFRWSemInt                             
QDOMINO801   LIBNOTES     OSSEM        0000000001   WaitOnNativeSemaphoreCounted                    
QDOMINO801   LIBNOTES     OSSEM        0000000025   WaitOnNativeSemaphore                           
QDOMINO801   LIBNOTES     OSSEM        0000000017   WaitForThreadSem                                
QSYS         QP0WPTHR     QP0WCOND     0000000049   pthread_cond_wait                               
QSYS         QP0WPINT     QP0WSCOND    0000000086   wait__20Qp0wPthreadConditionFP7Qp0wTcbP9Qp0wMutex

 

After dumping the call stacks, I ended the Domino server, *CNTRLD, of course.  This did nothing, the server was hung.  I next did a *IMMED end on my test server.  The interesting thing is that the status of the Domino server showed *ENDED, however when I used option 9 to look at the active server jobs, I saw that the Server job was stil active. 

 

The console was now showing "- Child stopped by signal -1". 

 

I tried ending the server with the *IMMED option, and received error LNT0922. 

 

Message: Error 11 has occured trying to end the Domino sever, DOMTEST1, the error code is 11.

Cause: Error trying to end server DOMTEST1 immediately.  Subsystem is QUSRNOTES/DOMTEST1.

Recovery: Save this job log and contact your service representative.

 

Well, that would be me!!  At this point I had to do an ENDJOBABN to get the Server job ended.  This did the trick. 

 

Interestingly enough, when I tried to do a CHGDOMSVR command on the server with the server and the subsystem ended, I received a CPD680B message.

 

Message: Error detected in prompt override program command string.

Cause: An error was detected in the command string created in the prompt override program QNNINPOP in QDOMINO801.  Processing of the command will not be affected by this error.

Recovery: After prompting is complete, see the previously listed messages in the job log to determine the cause of the error or see the CL topic in the Information Center book for more informaiton on prompt override programs.

 

Override what?!?  I checked the job log and saw these messages:

 

Character 's' not valid following string ' 'Kim'        '.

Character ' ' ' not valid following string 'Frankie      '.

A matching apostrophe not found.

A matching parenthesis not found.

Command not valid.

 

What it was complaining about was the title of my server, which is "Kim's Test Domino Server on Frankie".  Interestingly enough, the CFGDOMSVR command had no issue with me putting this in for the server's title when I configured it, however the CHGDOMSR command wouldn't accept it.

 

I was able to get the autostart parameter changed by qualifying the specific parameters I needed on the CHGDOMSVR command:

 

CHGDOMSVR SERVER(domtest1) AUTOSTART(*yes)

 

The moral of the story is never issue the CHGDOMSVR command against a Domino server when it's just starting.  The normal function of the CHGDOMSVR command is that it will let you change this paramter while the server is active.  The trick is that the server needs to be actually started when the command is issued.  There is a glitch when the server is starting where this command gets really messed up. 

 I received a call from a customer this morning saying noone could access their mail files.  The really strange thing about the situation was that the log.nsf, names.nsf, ... could be opened.  It was just a problem with the mail files.  When users tried to open their mail files, they received the following error message.

 

"Server Error: Device is Busy/In Use"

 

The console showed messages like this for mail files the router was trying to deliver mail to:

 

"Error delivering to servername/domain mail/xxxxxxxxxxxx.nsf; Device is Busy/In Use"

 

I dumped the server call stacks for all Domino jobs on that server and the job call stacks for any jobs I thought could be suspect.  I looked through the output and didn't find anything suspicious.  Next I invoked the QP0FPTOS api to identify if there were any IFS locks on the objects. The output showed there indeed were IFS locks on the files.  I issued the RLSIFSLCK command against a few mail files.  This did no good, the objects remained locked.  The only time the RLSIFSLCK command cannot release a lock is if it is a directory lock or if there is a save lock. 

 

This tipped me off to look at the job log for the save job.  This customer takes their Domino servers down every evening, saves the data directories, and then starts the servers after the save is complete.  The save output showed that the first Domino server had been saved fine, however the save job for the second server, the server we were having this issue with, was still active.

 

Once the save job was ended, the problem went away, no server restart was required.  The mail files are able to be accessed again and the customer is very happy.

 

Now it's time to get outside and enjoy this beautiful weather since it's the weekend!

This is one name change that went over extremely smoothly, in fact, I think it is the only name change for the operating system that has gone over so smoothly.  For a little background, the System i (AS/400, iSeries) has always had a different name for the operating system associated with the hardware platform.  When the System was known as the AS/400, the operating system was called OS/400.  The OS/400 name was kept when the system was renamed to iSeries.  Then along came the name change to System i.  At that time, the new hardware was Power 5, so the operating system was renamed to i5/OS.  We have had that name for the past few years, now along comes the new Power 6 hardware, uh, wait, guess we have to rename the operating system again.

 

Luckily they didn't decide to rename the OS to i6/OS, as you can guess what would happen when Power 7, 8, ... would come along.  Instead, the name has been shortened to simply 'i'.  Mark Shearer, VP of Marketing and Offerings for Business Systems, made this announcement last Wednesday morning at the IBM Town Hall session at the COMMON conference.  This is the first time I have seen an IBM executive get up in front of the COMMON crowd and announce a name change and actually get applause.  I think it helped that everyone knew it would be an endless cycle of renamings if the operating system name continued to follow the hardware version.  So now we have 'i' running on IBM Power Systems. 

 

The new name 'i', will also be known as 'i for Business', since this operating system is known for running business applications.  A positive that will come out of this announcement is that we will actually be seeing 'i for Business' in IBM advertistments, and no, it won't be hidden on some corner of the advertistment so tiny you can't read it!  Anytime the new Power Systems hardware is mentioned, all 3 operating systems that run on the hardware will also be advertised: i for Business, AIX, and Linux. 

 It's official, at the COMMON conference last week, Mark Shearer and Ross Mauri announced the new IBM Power Systems.  Both the System i and System p have shared the same hardware for a number of years.  What's great about this announcement is what Ross Mauri explained to the COMMON attendees at the IBM Town Hall session last Wednesday morning.  Ross Mauri is the General Manager for the Power Systems Division at IBM.

 

The big news Mr. Mauri shared with the crowd is that there will no longer be separate pricing for System i and System p hardware.  Wtih the new Power Systems announcement, the price i customers will pay will be same price p customers will pay for the hardware.  This announcement brought a loud applause from the crowd, as i customers have traditionally paid more for memory, disk, and other hardware components than p customers. 

 

I think this announcement is a very good thing for the System i.  Not only is there pricing equivalency between the hardware platforms, but now when IBM Power Systems is advertised, i will also be advertised.  This is something we have needed for a long, long time, more advertising.  Things are definitely looking up for the platform we now know as the former System i !!

 After seeing Michael Smelser and Karl-Henry Martinsson's travel maps, I decided to create my own. 

 

I just tried to log into a System i server that is currently not available and had to laugh at the message I received from my Mochasoft client.

 

 

 

They can try to rename it, but it will alway be 'AS/400'  : ^)

 

Check out the title of this blog, notice it isn't AS/400?  Yeah, the '/' character isn't allowed in the title <grin>.

 I am very impressed with the new database property 'compress database design'.  I recently upgraded my test Domino server from 7.0.3 to 8.0.1 and played around with the 'compress database design' database property.  After doing a compact -c on the database to switch it to the new ODS, level 48, I saw this on the console:

 

Compacted mail/dadminis.nsf, 9472K bytes recovered (54%)

 

Here is a bit of a precursor to explain my test.  The mail file (dadminis.nsf) was using the Standard Mail 7 template.  It was still at ODS 43 after the upgrade.  The mail file only had 4 documents in it, so it was primarily comprised of design elements.  I set the notes.ini parameter Create_R8_Database=1 and restarted the server.  I did a 'load compact -c mail/dadminis.nsf' at the console, and got the above results, the 54% reduction in file size.

 

The size of the mail file (still the Standard Mail 7 tempate) went from 18 MB down to 8192 Kb.  Talk about a nice improvement for a base mail file.  Thank you to the Lotus team for this improvement!

 

Next on my target list is how much the database property 'compress document data' pays off.