Dan's Blog

Recent efforts here to troubleshoot a unique issue with a user's Blackberry calendar edits made on the device, has flushed out a rather useful field in the mail file that will allow you to track changes to calendar and scheduling involving the BES and device, called "RIMCSTRACK".  This field seems to contain information on the past 3 changes to any calendar entry involving the device, and pending some specific feedback from IBM on exactly how that field is populated, and how the data in it is rotated (if it is), it can help greatly in associating what may have happened to a given calendar entry, when, and from where it may have been updated (device or server).  The field also contains an identifier, called "refid", that can be used to match the events in the mail file, to the BES messaging agent data contained in the "MAGT" log files, to reconstruct timelines and to determine where things may have gone awry.

 

In a manner of speaking, once a calendar edit involving the handheld/BES has been "committed" to the mail file on the domino, this field then appears in the mail file, and it shows the date, time and direction of when this edit occurred, so in our case, edits that never seem to make it to the mail file from the device for some unknown reason (the problem we are chasing with RIM and IBM), can be either confirmed or refuted as committed on the domino side, by examining the contents of this field.  (If the calendar entry has never involved BB/BES etc, this field is not present for that calendar doc).   It can be used in a view as well, and by adding this field to the "all calendar entries docs" view, (and adding that custom view to a user's mail file with an issue) can provide a nice quick troubleshooting mechanism to identify which calendar docs involved the Blackberry, and to determine what may have happened, as well as providing an export capability to use to match to the BES logs using the refid key.   This data will also show slow sync's, state database record creations, and other elements that may mean more to a person more familiar (than I) with the BES realm.

 

The example below shows that field's makeup, as well as the tracking of multiple changes to a test calendar document that I made from my device, while trying to reproduce the problem.  It appears that the "sending update" event is the BES sending the calendar entry down to the device, (server to client), and the other 3 "received update from device" are updates made from the device.  Upon export, this data is all appended together, (21/12/2009 20:02:54 refid-821122648: Sending update,21/12/2009 20:05:30 refid-821122648: Received update from device,21/12/2009 20:08:12 refid-821122648: Received update from device) without a consistent delimiter, so it takes a bit of work to parse out the data to one value per subfield of the entries (rimcstrack_date1, rimcstrack_time1, refid1,actiontaken1, etc)  for readability and matching the refid to BES logs, but once broken out it and easier to read (or match) it can start to tell a story about the timeline of changes, and from which direction they happened.

 

Field Name: RIMCSTrack

Data Type: Text List

Data Length: 253 bytes

Seq Num: 6

Dup Item ID: 0

Field Flags: SUMMARY

 

"21/12/2009 20:02:54 refid-821122648: Sending update"

"21/12/2009 20:05:30 refid-821122648: Received update from device"

"21/12/2009 20:08:12 refid-821122648: Received update from device"

"21/12/2009 20:10:51 refid-821122648: Received update from device"

 

It has allowed us to determine if a calendar doc edited on the Blackberry was ever committed to the Domino (by looking at this field alone and comparing timelines to the times when updates failed), and by joining this to the BES messaging agent logs (which also use the same refid key), we can identify all the activity relative to the attempted edit on that specific day, that may or may not have made it to the user's mail file, and get some inkling as to why this may have occurred based on the errors in the BES logs.

 

What we found appears to be an abnormal sequence of "rejected" device-side calendar updates in the BES logs, that for some reason (RIM is evaluating and trying to reproduce a possible bug) were never apparently committed to Domino, some of which were rejected for "conflicts", and others that were strangely rejected because "update not valid, newer update already received", despite no updates occurring from any direction for months prior in some cases.  We continue to work with IBM and RIM to understand this issue better, and to understand exactly how this field behaves on the Domino side, and hopefully get some kind of fix.

 

Hope this may help you if you find yourself troubleshooting odd issues with calendar entries and the BES/BB, it has certainly helped us in the past couple weeks.  The interaction between BES and Domino can fall into the fingerpointing realm between the 2 vendors, and this field seems to have helped lay things out in a manner which should help us, and the vendors, reconstruct what may have happened.

 

 

 

 

 

 

Looking to correspond with anyone who has had experience with the Kazeon legal case management tool running against a Domino messaging environment.  EMC recently purchased Kazeon, which was previously privately held.  Although not an archive solution, Kazeon's niche is to assist the legal teams with the onerous task of search and extraction aspects of legal case management, to offset the sizable workload involved with the Federal Rules for Civil Procedure.  It can handle at least email, pc files, file shares and sharepoint data sources.  We're looking to do an eval of the product including a sizing/pricing exercise and possibly a proof of concept test, and the email interaction aspect is key, given the lions share of ediscovery involves email.

Curious to know how this works in a proactive ("index now extract later") mode or reactive mode ("index and extract data case by case") while running against production Domino mail environment, and any other tips, hints or gotcha's anyone would be willing to share about this.   Happy to discuss here, offline from here, via phone, anything you may be comfortable with.

Thanks in advance.

Curious to know if anyone has any experience and feedback on the newer Notes Multilingual User Interface (MUI), and can share any tips, tricks or gotchas on that. We are exploring the use mainly for use with Windows 7 where such a capability exists on the o/s.

Thanks in advance for anything you may be able to share.

Dan

Through the extraordinary, timely and authoritative helpfulness of Steve Watts at IBM engineering, and some additional help from Symantec, the issues here with unexpected restore behaviors using archive-style TxL's 8.5 and Veritas net backup (so we can utilize DAOS at some  point) are much better understood, and have turned out to be issues on both sides of the vendor equation.  Steve indicated that issue had occurred for at least one other IBM customer, and a fix can be had by asking for a hot fix (spr SWAS7X5675), or when the fix is rolled into 8.5.2.  Symantec provided an updated binary which "ensures that the Domino transaction logs are left behind in NetBackup's temporary cache until the entire restore attempt is complete.  This would ensure that TXNs are not being pulled from tape over and over again".

I can't say enough good things about Steve Watt's monitoring of the community posts here and the notes.net forums and willingness to help out, respond to posts and coordinate getting the right nuggets to the right people.  This is the 2nd time in the past couple months where he went above and beyond for us, and provided true value beyond the normal support realm.

The net is we can again resume testing towards DAOS etc when time allows.

Thanks Steve!!

Is anyone successfully running 8.5/8.5.1 with TxL, DAOS and using Veritas Net Backup for backups and restores?

We are attempting to in DEV, but with archival TxL's, common, point in time restores are taking very, very long (20+ hours), and the specific behavior of rolling through the TxL's forwards then backwards to do the restore, has both IBM and Symantec (through an open ESR) scratching their heads and unable to explain.

If you are running this configuration successfully, and can do a point in time restore (from nightly incrementals with archival TxL's) please let us know if you have seen this issue, and if you have, how you may have overcome this issue, or not as the case may be.

Many thanks in advance.

As much as I respect most who post here and elsewhere around the Domino community, I've never really understood the continual buzz and apparent infatuation with the idea of "cloud computing".   Admittedly, as an in-house IT resource with what some might say are vested "job security" interests in keeping things in a private network with local data protection, local administration and control, things like this Not so much fun in this cloud article just seem to underscore the need to keep things private, and under the care, custody and control of someone or something with some internal accountability for security, business continuance, privacy, uptime, change control, and other things like that.

Whereas I get that "the cloud" may present great opportunities for SMB's to have quick capabilities they may not otherwise be positioned to provide to their own environments, this kind of thing with Sidekick and T-Mobile and Microsoft could well cripple an SMB, making the risk for smaller organizations even more alarming.  When those 2 large organizations, and Google for example has stated your data is not going to really be private, it still befuddles me that any organization of any size is willing to take those kinds of risks.

Talk is talk, marketing is marketing, and fads are fads, and for those of us still waiting for "thin clients" to replace the PC (for the 3rd or 4th time over the past 15 years), might we ought to know this sounds like another passing fad, at least in the way it is handled now, by providers like those mentioned above.
 

We, like many, are eagerly testing and evaluating Domino 8.5, DAOS and prerequisite transaction logging (TL).  We have not run TL before, mainly based on things like very long crash recovery times in the r5 days with TL..  We have an environment running Veritas Net Backup (NBU) 6.5.4, and significant business needs for point in time restores.  So far, those test point in time restores are behaving rather oddly, and taking forever to parse through the transaction logs and roll forward/roll back, one recent test for one restore took more than 20 hours, after "tuning".  Veritas, as per usual, is less than helpful, and the DAOS backup strategy (NSF backup and file system backup for the NLO's) will not be integrated into a coordinated offering from Veritas until 7.x of the NBU client for Lotus.   IBM support has punted, appropriately, in helping us understand the odd behavior with NBU.

It has me wondering.  Disk is cheap, cheaper all the time.  Although the advantages of saving disk space seem like good goals to shoot for, is the additional complexity of having to do transaction logging, and all the seeming complexity that comes with that for restores-- essentiually relying more heavily on your backup vendor to fully integrate and coordinate the aspects of recovering Domino data-- worth the effort?  Veritas cannot give any method in the current version to coordinate the 2 backups, so our Unix admin is scratching his head trying to figure out how to know when to schedule the file-system NLO backups, that need to occur after the NSF's are backed up.  All of this creates an increased dependency on backup vendors and products, simply to do the routine tasks like file restores that were pretty-much not dependent on them prior to DOAS and TL's, and in our case with Veritas, that is a dependency that is not attractive.

We're also starting to look at replacing NBU, possibly with something like Tivoli for backups, which as an IBM offering, is more integrated with DAOS. Now we're considering changing all of that, simply to de-dup attachments, which is another huge set of things to do.

We strive for simplicity and to avoid additional cost, complexity and increasing dependencies on "utility" software products and vendors to manage and leverage our Domino environments, and although decreasing storage seems great, I'm starting to think DOAS may not be worth adding more complexity, and headaches for doing simple things like point in time restores, and/or changing all the things that come with backups.

Interested in any comments you may have on this.  Thanks in advance.

When upper management or others ask you to participate in the process of evaluating why you should continue to run domino vs going to exchange, here are some key questions to ask Microsoft and why, that'll help you size, cost and specify a potential move to Exchange, and so those who may be footing the bill will know how much the MS path may cost.

 

1)  "Can you provide an Exchange messaging architect please?"  Do not allow yourself to be lulled into thinking you'll get difficult questions answered by the legions of pre-sales exchange people that Microsoft will make available to you.  They will burn your time regurgitating the sales hype about Exch 2007's improvements, and will not be able to answer detailed, technical questions without agreeing to follow up later, and will give you generic and/or wrong sizing information.  If the questions are difficult and the answers will portray MS in a less-than-ideal rosy light, if your experience is like mine, you'll never get answers from pre-sales people.   Insist on someone who understands the detailed architecture, sizing, protocols and associated services in depth.

 

2) "How do I size my Exchange 2007 environment in the event I want to use BES and Blackberry also, in terms of the relative load a Blackberry user will place on Exchange, using the load of one Outlook mapi user as the measuring stick?"  Ask the smart guy they might provide to you in #1 this question.  If the answer is that one BES user places the load of 4 Outlook/MAPI users, which added to his Outlook client means that one user places the load of 5 Outlook users, you've found an honest Exchange person. 

 

3) "How do I get the double-byte character set support for Asian/Pacific languages in Exchange 2007?"  The answer to this question is that you must pay the ongoing annual 33% costs for software assurance, for both the support on workstations and for support for Exchange in Asia pacific regions.   This expense is obviously a non-factor for Domino.  Microsoft has chosen to hold basic functionality required for a global company hostage, until the SA ransom is paid.  Aside from again reaffirming that MS is really just a marketing and sales engine, it just feels wrong to have to do this if you are a cost-driven entity.

 

4) "How many pieces of wintel hardware do I need to buy to replace my domino mail environment?"  Exchange requires 3 server roles (for non-edge services) to provide some of the functionality you can do today with domino, and in all but smaller environments, these will require 3 different pieces of hardware:  one for mailboxes, one for client access and one as an email relay that talks MAPI to the mailbox server.  Be sure to include your remote sites where you have one remote domino today.  If you are using partitioned domino servers today, this count of servers for the MS solution will be much significantly higher than the count of your host machines today.  If you are not using partitioned domino servers today, the count will still be higher, given you need 3 to do what one domino does. 

 

5) "Does Exchange 2007 support VMware or other virtualization?"  Although I've read that some have been able to get Exchange to run virtualized, Microsoft stated to us that it was not supported.  Domino has supported this via partitions for a long, long time.  This obviously increases cost, complexity, management and headaches with much more hardware required to do the work.

 

6) "Will Microsoft ever port Exchange to Unix, Linux or other more scalable o/s?  An obvious question and more obvious answer, but fun to see the look on the MS guy's face when you ask this.  It has further implications for long term cost, scalability, stability, security, etc, etc.

 

7) "I'm concerned about the security issues with the Microsoft web server, can I run OWA (Outlook web access) or other required web services for exchange with something like apache or other more secure web server?"  Another obvious answer, but a question that bears asking for due diligence, and to make your security people happier about Domino web services.

 

8) "How many additional tools will I need to buy to administer Exchange from the admin side?"  If you like being able to see what domino is doing in real time at the console view in the admin client, you'll not have that capability with Exchange.  Other admin functions that domino admins have had for many years will require add on tool from MS or 3rd parties.  Adds cost, complexity, etc.

 

9) "Our attorneys have advised us that due to the needs for litigation and e-discovery, we need to disable message recall.  How do you turn that off in Exchange?" It cannot be disabled in Exchange through the native admin capabilities.

 

10) "How many people will I need to manage Exchange for my environment with a given Exchange sizing?"  The answer to this one for many domino shops that are not overstaffed today, will be substantially more people to manage the servers and the clients (ie help desk) for the MS stack.  In our case this was "3 to 5 more help desk people and 1-3 more email admins".

 

The answers to these questions will help you begin to build a quantitative case to help choose a mail platform for your company, based on qualified, real information, vs using the typical pre-sales nonsense you'll get from MS.  In cases where you are already running on a more scalable platform, and you have domino virtualization already in place, and where up front cost and long-term TCO is important, Microsoft will have an uphill battle making a compelling case to change your environment.  If you add other aspects of drinking all the MS kool-aid to this, like having to buy server cals repeatedly with new versions, (ie a/d cal repurchases required to go to server 2008) the choice to go MS can be very expensive.