Lotus Nut

Tomorrow evening (Wednesday, June 24, 2009) we will have some downtime on almost all of the BleedYellow servers. This includes the Connections and Sametime servers. It's a planned, routine maintenance outage to load some fixes, etc...

Thanks!

As many of you may know, we have had a problem with Dogear here for a while. At one point, I started referring to it as Old Yeller (could not resist the dead dog/yellow reference - sorry). After doing a lot of upgrades (to IHS/WAS/Connections), DB troubleshooting, completely deleting the DB (which has since been restored), enabling advanced Windows logging (which did not help much), even moving to a fresh VM with a clean install of WAS, maintenance, etc..., nothing seemed to work. In all the crashes of the Dogear jvm, only once did a good Dr. Watson dump get sent out. But in one of them we noticed what looked like a strange search of the Dogear database. After further digging, I looked at this closer with IBM. Turns out we were seeing a lot of these in the IHS access.log:


It was not always the same search - it would be for different users, etc.... The IP address, though, is a Googlebot. So, in walks robots.txt to see if he can help! After adding it to the server, I wondered how long it would take for Google to honor the settings as we kept seeing the crashes. As part of the move to a new box with Deployment Manager, the Dogear jvm would automagically restart. So that helped uptime a bit. I noticed that the last crawl of Dogear by Google was around 8:25 PM on 6/12/09. And at 8:29 PM Dogear was last restarted. So it appears that Google took around 12 hours to start honoring the robots.txt.

After running now for almost a week, I think we are confident in saying that Dogear has issues with this type of string in the crawl. After some further data crunching from 6/12/09 (where we logged 7 Dogear crashes between 7:11 pm and 8:30 pm), here are some quick stats on when we saw Dogear get crawled with this string in the IHS access.log and when the server recovery message was logged in the WAS SystemOut.log:

the string "lang=en?ref=sex%" was found in IHS access.log
6/12 19:07
6/12 19:12
6/12 19:25
6/12 19:43
6/12 19:50, 19:52
6/12 20:06
6/12 20:25


Restarts in SystemOut.log
6/12 19:11
6/12 19:15
6/12 19:29
6/12 19:44
6/12 19:57
6/12 20:11
6/12 20:29

Since those times on 6/12, the crawl has not occurred any more and Dogear has not crashed. Here's to hoping this is finally resolved!

To all of our wonderful bloggers here, you are probably aware of the issues we have been working through with Dogear. This has the potential to make your blog appear unresponsive when Dogear hangs before automagically restarting itself. Before it would just die, now it restarts. If you have the "Dogear Bookmarks" available in your blog template, then this is what causes your blog to take a while to load sometimes. If this gets too irritating for you, you can modify your template to simply disable the Dogear integration (at least temporarily until we get this sorted out).

Thanks!

Or: Can I DAOSify a Server With < 8.5 Clients?

Recently I have had some colleagues and clients asking about updating a Domino Server to 8.5 and turning on DAOS while still running a mix of 6 and 7 Notes Clients. There is some confusion related to some upgrade seminars that are happening that appear to be using this slideshow. Here is slide 18 that is causing some confusion:


The last bullet is causing confusion. Of course "Domino 6 and 7 clients will not recognize DAOS" - even the 8.0.x client would fall into this category as it was released pre-DAOS - DAOS is a server feature. However, there are no known problems with using an older client against a DAOSified mail template. Here is why: the Domino Server is aware of DAOS. Any requests for an attachment from clients, servers, or services such as BES will be honored by the Domino server sending the attachment. The server will still have to get the nlo file and serve it to the requesting client (or server) over the wire. And, if you are running local replicas it does not matter either because the replica will be pulled down with all attachments inside the database.

Do I recommend a long-term strategy of having 6 and 7 clients with an 8.5 server? Negative. But you can immediately reap some benefits by upgrading your server and enabling DAOS while you are working on your client rollout strategy. And, as the slide above states, you'll need to wait until the clients are upgraded before you can update the template to mail85.ntf. Finally, DAOS is not just for mail databases. I have also sensed some confusion around this. Of course, in most environments, the mail store is where most databases would have duplicate attachments (either internal or shared) and most of DAOS's storage and i/o savings has been revolved around mail files.

Hopefully this helps!

In response to: FTOTM (first Tuesday of the month) = new rules

I never knew that you could actually disable xPages on the server completely. This is rather interesting. What's funnier is that the notes.ini that's used to do this is INOTESDISABLEXPAGECMD - iNotes and xPages are not even tied together at this point.

As many of you know, we have been having some issues with Dogear here and working with IBM. As we have upgraded, moved, and tried all kinds of different things, it appears to have been pointing to the database as the culprit. For now, I have created a new Dogear db to see if this is the problem. I have a full backup of the db from last night, and if it turns out that the DB was not the problem, then I will restore that backup. Sorry for the inconvenience - we have tried a lot of things for many many months to get this sorted out. Thanks for your continued patience on this. I am hoping that this will not be the issue and that we can restore it back, but after touching all of the other components (upgrades, reinstalls, moves, etc...) and running corrective actions on the db with no luck, this appears to be the next logical step.

Just thought I would remind everyone out there about the webadmin logo that IBM has been using over the years. Funny how close the 7.0 Beta 4 was to the actual Star Trek logo with the starship in the center. It was changed for the 7.0 GOLD release.

So, I wonder what IBM is trying to say about administrators?? By the way, I'm hoping to go see the new movie this week!

On April 16, 2009, IBM released version 1.4 of the DAOS estimator. I like some of the new options.

They are:

• -i option allows the estimator to run on an indirect file which contains a list of databases to run against.
• -c options causes the estimator to write attachment data to a CSV file for later processing. This is up to 65% faster to acquire data.
• -a option analyzes a CSV file created with the -c option. This option can also be used to analyze a set of CSV files using an '.ind' file to list the CSV files.
• -p option generates an estimate of the estimate by running against a percentage of databases and extrapolating the results.

Something that IBM hasn't pointed out in the technote is that it appears that using the -c option will allow for you to run multiple instances of the Estimator and then offload the analysis to a different server. Consider that you have multiple mail directories. If you run the estimator against each of them using the -c option, this will likely finish faster for acquiring the data. Then, you should be able to copy those CSV files to another box and use the -a -i option to analyze the multiple CSV files without hogging more CPU on your production mail server. Remember, it took Mitch Cohen around 5 days to run the estimator on a big server with 7,000 mail files. So, having that task not perform the analysis on a mail server would definitely help out!

By the way, indirect files are one of the most under-utilized options with Domino. Look into them and start using them!

Nope, I'm not talking about a blog for people into fast and furious 4 bangers (no offense to 4 banger drivers - like myself - who aren't fast or furious).

I'm talking about the new blog that IBMer Scott O'Keefe has started related to the Domino Configuration Tuner (DCT). You should go check it out at 'Tuner Blog. The first post is related to the "Check for Updates" option that's available with the template in which you can get the latest template from IBM as well as the latest updates to the rules. He also lists some of the new rules with the latest release.

You should really look into DCT if you haven't already. And with the 'Tuner Blog now active, I'm certain that Scott, et. al. will be looking for feedback!

In response to: Concerns over DAOS Estimator?

Just a note - I have seen what Vitor is referring to in his comment on my initial post and have verified that there are cases where if the file name is the same as well as the size reported in the DAOS Estimator report that the files are treated as the same object by the Estimator. But after enabling DAOS on the database and compacting it, the two attachments were treated as separate objects by DAOS even though the tool did not see them as different. This means that you may save less space than the Estimator thinks due to some edge cases where you may possibly have attachments with the same name and physical file size but with different internal data. In Vitor's case with auto-generated files, the results may be very skewed. Again, each environment is different and ymmv. IBM is already aware of these findings above.

Paul Mooney reminded us the other week that we should do this - and I agree! Over at Vaughan Rivett's blog, he's discussing some of the misunderstandings surrounding DAOS. He brings up several good points, but does question whether DAOS should be enabled on mail.box files. Of course, each situation may be a little different for every environment, so it's up to you to determine if you want to do this. But keep in mind that, like Vaughan states, DAOS is for any application - not just mail!

If I'm a daosified mail user and am using a CRM application to save a copy of an email message directly into the CRM, then if DAOS is enabled for the CRM application the attachment will be shared in DAOS! But if I want to send that email to a colleague and mail.box doesn't have DAOS enabled, then that attachment will be written to mail.box and then when it's delivered to the recipient's mail file DAOS would associate it as a duplicate from my mail file. But, if mail.box has DAOS enabled, then the attachment will never be pulled back out of DAOS (meaning there will not be the disk i/o associated with the transaction) and the router will know that it already existed in DAOS and not try to pull it back out. So, in most cases you will want to enable DAOS on your mail.box files. (Of course multiple server hops will not be aware of this currently, but for mail users on the same server it should be ok.)

Also remember that you'll need to enable transaction logging on your mail.box files. This may mean that you will need to disable the notes.ini settings for MailBoxDisableTXNLogging=1 and/or RM_NO_LOG_LARGE_OBJECTS=1. You will also need to manually enable transaction logging on the database with the router off by using load compact -T on each of your mail.box files.

Yesterday in the Twitterverse, there were some concerns over whether the tool is accurate. I decided to do a quick test. I did notice today that there was a v1.2 of the DAOS Estimator Tool released on February 19, 2009. So I ran the latest version of this tool on a single database and I can see where the concerns may be coming into play.

The concern was that the Estimator Tool was only looking at the file name to determine if the attachment was a duplicate. Here's what I did to test this concern:

• Created a new mail file
• Created 2 draft messages that had the same attachment (attach.ppt) that was 699Kb
• Created a 3rd draft message with an 880 Kb attachment (also named attach.ppt). This was a completely different file from the start - it just has the same name.

The results were right in the main file analysis section - you can see that it saw 3 files in the DB with 1 duplicated file resulting in 2 DAOS files. So far, so good. But the problem is with the section I have outlined in red below. As you can see, there are two lines that have the same text ("Total Duplicate Attachments found". This isn't right.



I went back to review some previous results (gathered from v1.1 of the Tool) and can see that the second line should really be "Total DAOS Eligible Attachments" (see image below). This explains some of the confusion with the above image. Using the wording from below in the results above, this is right. There's only 1 attachment that's duplicated but 3 total attachments that will be moved over into DAOS.



I've reported this to IBM, so hopefully this wording will get corrected in a v1.3. I think we can feel safe that the Tool is working just like DAOS will work in properly determining if an attachment is really a duplicate or if it just has the same file name. Though I certainly welcome any feedback otherwise!

Let's consider that for some reason your Connections server went down due to, say, a major power outage. Everything comes up ok except for Blogs (it could be another service as well, though) and you are getting the following error message:



When I was thinking about why this could be happening, I thought that either (a) there was a key file that was open but got corrupted, or (b) hopefully I just need to clear the Blogs cache directory because of corruption there. To make a long story short, it appears to have been a problem with the cache (whew...)!

The cache for a WAS server application is here:

[WAS_ROOT}\profiles\{Profile}\temp\{Node}\{application}

In this case it was:

C:\IBM\WebSphere\AppServer\profiles\myprofile\temp\mynode\BlogsServer

Basically, I shut down the server process, moved all of the files in the BlogsServer folder over to another location on the disk, and then started the server to let it create the temp files again. Thankfully, everything came back up alright. Oddly, the Homepage was rendering the blogs widget just fine - it was just the direct URL that wasn't working.

By default, BES debug logs are enabled at installation. They are not only enabled, but they have no roll-off time specified. I have seen this cause drive space issues (not to mention performance issues associated with so many heavily fragmented small files...)

To remedy this, you can open the BlackBerry Server Configuration and go to the Logging tab. From there, you will need enable the "Debug log auto-roll" and specify the number of days you wish to keep the logs in the "Debug log maximum daily file age" settings. In the example below, we will only be keeping 30 days worth of logs. After restarting the BES task and all of the BlackBerry Services, the files will be automatically purged.



Some notes:

You may just wish to restart the entire server instead of each of the individual services.

The daily folders will still exist, so do not panic if you still see them. They will exist because of three log files. By default, all BB phone calls are also logged. This is how your phone call logs get restored when you activate a new device. There are also some 2 SMS and PIN messaging log files. These are empty files by default, but administrators can enable this logging (without you being aware of it, by the way - you did know this, right?).

So go ahead and clean up your BES servers and try to remember to modify these logging parameters when you install the server (note to self...).

Kieran Reid has posted a great how-to on this. I just wanted to add something to what Kieran mentioned. For the MetaWeblog API url, you must use https in order for this to work. So, please head over to Kieran's blog and set this up for yourself. I just setup my blog with Scribefire, and that's where I'm posting from.

Cool UI thing in Scribefire: I copied the URL for the link above and when I clicked the "Add a link" button in Scribefire's editor, the URL was automatically pasted in. Nice touch. I think I'm going to like using this...