Enterprise Messaging Notes

Here  is a tool which provides a cost estimate for the costs of eDiscovery per GB, something like $5600 per GB of data captured for legal review, (or $5.6 million per TB!) from the folks at complexdiscovery.com.   These costs apply to any kind of EDiscovery data from any system, although the lion's share of eDiscovery involves email.   Although I've heard anecdotal impressions of these costs in the past, this is the first tool I've seen to allow IT and legal to start quantifying their specific exposures in dollars for the data sources they manage.

The eDiscovery disadvantages of Outlook/Exchange remain an important, and very costly, differentiator between Notes/Domino and the Microsoft stack.  So for shops running Exchange and Outlook with hundreds of gigs or more of  "fundamentally insecure" Outlook PST's lying around, which are typically asynchronous data versus server-based mailboxes, this is the legal aspect of the costs for normal discovery to go after that kind of data.  Because the security model for PST's is not extended to local storage (like it is with Notes replicas) forensic costs may also be added to this per GB cost ($600 per hour is typical) because anyone with access to the file systems they reside on may be able to modify them.  These legal costs are in addition to the internal costs to gather the distributed and voluminous PST files that pop up because users can and do create them at will in typical situations.

This is how the BP Oil Spill issue gets projected to be $100,000,000 in discovery costs.  Companies like Fiat who are looking to, or have, move(d) Chrysler off of Notes/Domino, may want to consider cost elements like this, in light of the other hefty legal costs Chrysler has been paying out to help work out the bankruptcy issue.  IBM should be looking to aggressively milk these kinds of TCO differentiators when competing for business with Microsoft.

 With all the somewhat grim news these days, thought we might share this unexpected mention and how Lotus Notes is seen to contribute to one of the 100 Best Places to work in IT.

Our company participates in the Computer World effort around best places to work.  This year, we were again included and were rated #62, and unbeknown to us the consolidation of the entire company onto Notes Domino was identified as one of the reasons we're perceived to be a great place to work.

By virtue of the great people I'm lucky enough to work with, who so expertly planned and implemented this large consolidation, we've been able to deliver more stability and business value using ND than some internal naysayers would have ever been able to predict as recently as a couple years ago when we were evaluating a move to Exchange.  Since 2004, our domino seat count has risen from about 6000, to about 18,000 today, including 13,000 notes clients, 3700 Mozilla IMAP clients, and about 1600 iNotes only clients.  Our senior leadership has some very aggressive yet very obtainable growth goals to achieve by 2016, the 150th anniversary of our company, and we continue to replace Outlook/Exchange and ISP-based services all over the world in favor of Notes/Domino.   The growth here will continue to be dramatic.

Recent discussions here and well-known factual observations about the dramatic cost-benefit disadvantages of Exchange/Outlook vs Notes/Domino in the context of eDiscovery means that our commitment to this technology will continue onward well into the future, and we will avoid millions of dollars of extra costs by doing so.  Those discussions, for the first time in my 6 years, also has decision makers discussing the need to possibly get back to ND app development for collaboration, given the commitments to the messaging platform, something I never thought I would hear.  In the past year, our internal partners have implemented a rock-solid sales force application using Blackberry, BES and also involving ND calendaring, which has proven to be robust, scalable, and very problem free.  The old paradigm whereby ND apps would make a move off of mail less likely, seems to have flipped 180 degrees in our case, which is a very encouraging trend that may provide a bit of a silver lining for some in the community.

Been reading the posts and responses over the past few days on the movement from Notes apps to things including Sharepoint, and the impact on many in this community by the decrease in app dev opportunities in many places, and having witnessed some of that cycle at our company, it has me wondering if the perceptions of the alternatives, and most importantly the actual use of the alternatives, are as poorly understood by the C-level types, and other IT managers, as the use and benefits of ND applications.   There might be a simple case here for folks to make, in some cases, that might add a bit of sanity to the marketing hype surrounding Sharepoint, by looking at the actual use of the product (vs hyped or C-level perceptions) and contrasting that to lower cost, lower complexity services which almost certainly already exist in most environments.

Here, well over 95% of all utilization of Sharepoint seems to be for file sharing only.  That file sharing capability, which already existed and still exists here on MS and Novell platform file shares, occurs with Sharepoint with the added non trivial requirements, dependencies and complexity of SQL db, web services, and yet another set of administrative responsibilities and overhead, which at the core is another duplication of services requiring more people and technology resources.  More complexity, cost, overhead and the duplication of existing services is not a value proposition of any kind, and in fact, is a move in the wrong direction.  C-level types should be able to grasp the idea that more dependencies, cost and complexity are not good things, and those educating them or selling to them should be able to make that point quite readily based on actual use observations (if your actual use experience is like ours).

The app developers in the community have listed many differences and net advantages of ND app development vs Sharepoint in a number of recent posts, so if you add those observations to the fact that Sharepoint in practice seems to be "FileSharepoint" only, it would seem reasonable that a simple enough case can be made, to not go there.  We have a few people doing very simple lists and a couple of calendars with Sharepoint, all of which are duplicated with previously existing default capabilities in ND, which as a committed ND messaging shop, are again duplication and  unnecessary.  Yes Sharepoint may offer more than that, but at the end of the day, the question remains as to who is really doing more with it, and how prevalent are those uses, and at what cost?  

Look around, ask your customers, see what people are really doing with the tool, and if others are doing what we are doing, "FileSharepoint" seems like a costly, complex and unnecessary additional service which may not become part of the move away from the ND RAD tools (and committed developers and admins) they already pay for.  Post what you find, educate decisions makers about what you find.  Maybe this can help in some situations.

We are looking very closely at Smart Upgrade to pilot for a segment of our international foot print, going from 6.5 to 8.5.1.  We are faced with working with a couple of different past problematic installation issues that I'm hoping someone may have come across and may have a solution for possibly.   Client upgrades are new to our team, our scope has grown to include the client upgrades recently for a number of reasons (long stories....), so please forgive the newbie kind of questions.  We have read the docs, and used SU here in small context but we do not have broad experience with it yet.

We understand we need to build different kits for specific "past to present upgrade" scenarios, but some of them seem to require some manual intervention or desk visits that are not attractive options.

1) Past errant multi-user install that we want to move to single user:  With data directories in the user's profile, is there any supported way to upgrade to 8.5.1 and place a single user install in place WITHOUT the need to uninstall the old multi-user install first?  (We realize we have to move the data directory back to the std single-user location via scripts etc).  The Wolcott ADT tool seems to offer the capability to standardize install locations and realign stuff to standard locations.

2) Past errant "all client" (install type=2) install that we want to change to regular Notes client only install.  Same question, is there a way for Smart Upgrade to do that without the requirement to uninstall the old versions (designer, admin, notes) first?

We can script a number of things to work outside of Smart Upgrade to accomplish some/all of these changes, just curious to know if SU can natively do these steps for us somehow.

Thanks in advance for any insight and experience you may be willing to share on this.

Microsoft is marketing "Exchange 2010 Archiving" as new technology that'll make your ability to respond to eDiscovery much better, as some sort of  "new" capability designed to reduce the need for a 3rd party archive.  A recent Ferris Research group study pertaining to email retention, touts Exchange 2010 archiving as "Emerging Technology", in contrast to similar capabilities in ND that have existed for many years (although not marketed as such by IBM).

A decent overview from one MS centric site here describes some of the features, advantages and some disadvantages of it.

The substantial disadvantages include:

1) Requires an expensive and cost-prohibitive enterprise CAL, which may exceed the cost of 3rd party solution (of course it does....)
2) It archives the data to the same Exchange database and storage group (which may be a distributed data store), which does nothing in and of itself, to decrease the size of the operational mail store
3) The "multi-mailbox search capability" requires full text indexing be enabled, for the entire data store, a costly, expensive and potentially big performance hit (see here ) that is typically not done
4) Requires upgrade to Outlook 2010 and Exchange 2010, which as we all know, essentially requires a "upgrade migration", a well known set of difficult and time-intensive problems with Exchange
5) The search feature copies results to another mailbox, PRODUCTION of data still requires a unsecurable PST be created and all results dragged to it (lengthy process and introduces chain-of-custody issues)
6) Does nothing to address the largest liability for eDiscovery in a MS context, local Outlook PST files, and local Outlook OST (sync) files

Notes and Domino have provided similar, yet better, basic capabilities for a very long time, differentiated below, point by point:

1) No additional licensing required
2) Server-based archive servers and mailboxes can be implemented apart from your on-line operational mailboxes.  Policies exist to force this if desired
3) Individual archives, mailboxes etc in ND can be individually indexed vs the requirement to index and entire environment (huge space savings possible that way vs MS)
4) Archiving has been supported for older releases in various situations for many years, no upgrade-migrate scenario required to leverage ND capabilties in this realm
5) Domino search results can be copied to server-based repositories, that can be produced to 3rd parties inside a securable NSF file, that would remain in the care, custody and control of the domino server security environment until time of the actual production
6) Notes/Domino, in practical terms, avoids the PST/OST conundrum via the typical configuration of synchronous replicas, for which security is integrated locally, and can be kept consistent with server based ACL rights

Obviously dedicated archiving products will extend and even further improve both products capabilities around email preservation for eDiscovery, but the relative disadvantages of the "new" MS solution suggest shops running Exchange and Outlook may benefit even more from a 3rd-party archive than ND shops, even with the Exchange 2010 Archiving capability. 

Recent efforts here to troubleshoot a unique issue with a user's Blackberry calendar edits made on the device, has flushed out a rather useful field in the mail file that will allow you to track changes to calendar and scheduling involving the BES and device, called "RIMCSTRACK".  This field seems to contain information on the past 3 changes to any calendar entry involving the device, and pending some specific feedback from IBM on exactly how that field is populated, and how the data in it is rotated (if it is), it can help greatly in associating what may have happened to a given calendar entry, when, and from where it may have been updated (device or server).  The field also contains an identifier, called "refid", that can be used to match the events in the mail file, to the BES messaging agent data contained in the "MAGT" log files, to reconstruct timelines and to determine where things may have gone awry.

 

In a manner of speaking, once a calendar edit involving the handheld/BES has been "committed" to the mail file on the domino, this field then appears in the mail file, and it shows the date, time and direction of when this edit occurred, so in our case, edits that never seem to make it to the mail file from the device for some unknown reason (the problem we are chasing with RIM and IBM), can be either confirmed or refuted as committed on the domino side, by examining the contents of this field.  (If the calendar entry has never involved BB/BES etc, this field is not present for that calendar doc).   It can be used in a view as well, and by adding this field to the "all calendar entries docs" view, (and adding that custom view to a user's mail file with an issue) can provide a nice quick troubleshooting mechanism to identify which calendar docs involved the Blackberry, and to determine what may have happened, as well as providing an export capability to use to match to the BES logs using the refid key.   This data will also show slow sync's, state database record creations, and other elements that may mean more to a person more familiar (than I) with the BES realm.

 

The example below shows that field's makeup, as well as the tracking of multiple changes to a test calendar document that I made from my device, while trying to reproduce the problem.  It appears that the "sending update" event is the BES sending the calendar entry down to the device, (server to client), and the other 3 "received update from device" are updates made from the device.  Upon export, this data is all appended together, (21/12/2009 20:02:54 refid-821122648: Sending update,21/12/2009 20:05:30 refid-821122648: Received update from device,21/12/2009 20:08:12 refid-821122648: Received update from device) without a consistent delimiter, so it takes a bit of work to parse out the data to one value per subfield of the entries (rimcstrack_date1, rimcstrack_time1, refid1,actiontaken1, etc)  for readability and matching the refid to BES logs, but once broken out it and easier to read (or match) it can start to tell a story about the timeline of changes, and from which direction they happened.

 

Field Name: RIMCSTrack

Data Type: Text List

Data Length: 253 bytes

Seq Num: 6

Dup Item ID: 0

Field Flags: SUMMARY

 

"21/12/2009 20:02:54 refid-821122648: Sending update"

"21/12/2009 20:05:30 refid-821122648: Received update from device"

"21/12/2009 20:08:12 refid-821122648: Received update from device"

"21/12/2009 20:10:51 refid-821122648: Received update from device"

 

It has allowed us to determine if a calendar doc edited on the Blackberry was ever committed to the Domino (by looking at this field alone and comparing timelines to the times when updates failed), and by joining this to the BES messaging agent logs (which also use the same refid key), we can identify all the activity relative to the attempted edit on that specific day, that may or may not have made it to the user's mail file, and get some inkling as to why this may have occurred based on the errors in the BES logs.

 

What we found appears to be an abnormal sequence of "rejected" device-side calendar updates in the BES logs, that for some reason (RIM is evaluating and trying to reproduce a possible bug) were never apparently committed to Domino, some of which were rejected for "conflicts", and others that were strangely rejected because "update not valid, newer update already received", despite no updates occurring from any direction for months prior in some cases.  We continue to work with IBM and RIM to understand this issue better, and to understand exactly how this field behaves on the Domino side, and hopefully get some kind of fix.

 

Hope this may help you if you find yourself troubleshooting odd issues with calendar entries and the BES/BB, it has certainly helped us in the past couple weeks.  The interaction between BES and Domino can fall into the fingerpointing realm between the 2 vendors, and this field seems to have helped lay things out in a manner which should help us, and the vendors, reconstruct what may have happened.

 

 

 

 

 

 

Looking to correspond with anyone who has had experience with the Kazeon legal case management tool running against a Domino messaging environment.  EMC recently purchased Kazeon, which was previously privately held.  Although not an archive solution, Kazeon's niche is to assist the legal teams with the onerous task of search and extraction aspects of legal case management, to offset the sizable workload involved with the Federal Rules for Civil Procedure.  It can handle at least email, pc files, file shares and sharepoint data sources.  We're looking to do an eval of the product including a sizing/pricing exercise and possibly a proof of concept test, and the email interaction aspect is key, given the lions share of ediscovery involves email.

Curious to know how this works in a proactive ("index now extract later") mode or reactive mode ("index and extract data case by case") while running against production Domino mail environment, and any other tips, hints or gotcha's anyone would be willing to share about this.   Happy to discuss here, offline from here, via phone, anything you may be comfortable with.

Thanks in advance.

Curious to know if anyone has any experience and feedback on the newer Notes Multilingual User Interface (MUI), and can share any tips, tricks or gotchas on that. We are exploring the use mainly for use with Windows 7 where such a capability exists on the o/s.

Thanks in advance for anything you may be able to share.

Dan

Through the extraordinary, timely and authoritative helpfulness of Steve Watts at IBM engineering, and some additional help from Symantec, the issues here with unexpected restore behaviors using archive-style TxL's 8.5 and Veritas net backup (so we can utilize DAOS at some  point) are much better understood, and have turned out to be issues on both sides of the vendor equation.  Steve indicated that issue had occurred for at least one other IBM customer, and a fix can be had by asking for a hot fix (spr SWAS7X5675), or when the fix is rolled into 8.5.2.  Symantec provided an updated binary which "ensures that the Domino transaction logs are left behind in NetBackup's temporary cache until the entire restore attempt is complete.  This would ensure that TXNs are not being pulled from tape over and over again".

I can't say enough good things about Steve Watt's monitoring of the community posts here and the notes.net forums and willingness to help out, respond to posts and coordinate getting the right nuggets to the right people.  This is the 2nd time in the past couple months where he went above and beyond for us, and provided true value beyond the normal support realm.

The net is we can again resume testing towards DAOS etc when time allows.

Thanks Steve!!

Is anyone successfully running 8.5/8.5.1 with TxL, DAOS and using Veritas Net Backup for backups and restores?

We are attempting to in DEV, but with archival TxL's, common, point in time restores are taking very, very long (20+ hours), and the specific behavior of rolling through the TxL's forwards then backwards to do the restore, has both IBM and Symantec (through an open ESR) scratching their heads and unable to explain.

If you are running this configuration successfully, and can do a point in time restore (from nightly incrementals with archival TxL's) please let us know if you have seen this issue, and if you have, how you may have overcome this issue, or not as the case may be.

Many thanks in advance.

We, like many, are eagerly testing and evaluating Domino 8.5, DAOS and prerequisite transaction logging (TL).  We have not run TL before, mainly based on things like very long crash recovery times in the r5 days with TL..  We have an environment running Veritas Net Backup (NBU) 6.5.4, and significant business needs for point in time restores.  So far, those test point in time restores are behaving rather oddly, and taking forever to parse through the transaction logs and roll forward/roll back, one recent test for one restore took more than 20 hours, after "tuning".  Veritas, as per usual, is less than helpful, and the DAOS backup strategy (NSF backup and file system backup for the NLO's) will not be integrated into a coordinated offering from Veritas until 7.x of the NBU client for Lotus.   IBM support has punted, appropriately, in helping us understand the odd behavior with NBU.

It has me wondering.  Disk is cheap, cheaper all the time.  Although the advantages of saving disk space seem like good goals to shoot for, is the additional complexity of having to do transaction logging, and all the seeming complexity that comes with that for restores-- essentiually relying more heavily on your backup vendor to fully integrate and coordinate the aspects of recovering Domino data-- worth the effort?  Veritas cannot give any method in the current version to coordinate the 2 backups, so our Unix admin is scratching his head trying to figure out how to know when to schedule the file-system NLO backups, that need to occur after the NSF's are backed up.  All of this creates an increased dependency on backup vendors and products, simply to do the routine tasks like file restores that were pretty-much not dependent on them prior to DOAS and TL's, and in our case with Veritas, that is a dependency that is not attractive.

We're also starting to look at replacing NBU, possibly with something like Tivoli for backups, which as an IBM offering, is more integrated with DAOS. Now we're considering changing all of that, simply to de-dup attachments, which is another huge set of things to do.

We strive for simplicity and to avoid additional cost, complexity and increasing dependencies on "utility" software products and vendors to manage and leverage our Domino environments, and although decreasing storage seems great, I'm starting to think DOAS may not be worth adding more complexity, and headaches for doing simple things like point in time restores, and/or changing all the things that come with backups.

Interested in any comments you may have on this.  Thanks in advance.