Erik Brooks

 
Wow... there's been nearly 4,000 hits to my blog entries regarding the GetDocumentByKey bug.
 
A lot of you have been asking me about an update so here we go:
 
The Short Story
 

The issue's reportedly been fixed. There is a technote where you can contact IBM for a fix.a technote where you can contact IBM for a fix.
 
As a nice side effect the original bug they were trying to fix all along (the "infinite loop" bug) is now truly fixed. Your apps that perform any GetXXXByKey calls will likely scale much better under load than they ever have.

 
8.5.1 is great. Really great. Except when it decides to make your code no longer work. And especially when that code involves NotesView.GetDocumentByKey, NotesView.GetAllDocumentsByKey, NotesView.GetEntryByKey or NotesView.GetAllEntriesByKey.
 
If you do Notes/Domino dev you've probably written something like this before:
 
Dim v as NotesView
Dim d as NotesDocument
Set v = GetView("myView")
Set d = v.GetDocumentByKey("something")
 
There's probably a trillion lines of this type of code in production all over the world, and it has worked for years. Something close to 15 years (Domino 4.6?), if I recall.  However in 7.0.4, 8.5FP1 and 8.5.1 and beyond this code may arbitrarily return the error "The collection has become invalid."
 
What I typed above is LotusScript but Java (and potentially SSJS) are all affected by what I'm about to describe on both client-run and server-side code and agents.
 
Here's what happens in all prior versions:
 

When the lookup executes and a doc is found it checks to see if there is a document that was modified after the view's index was last updated. If so then the collection is technically out-of-date and the lookup is re-executed again and again until it is considered up-to-date.
 
You can change this behavior with NotesView.AutoUpdate=False but that will cause any other code (e.g. agents) running to pile up waiting for your original code to finish so the index can be updated. Beside, most code (even long-running agents) actually want the most up-to-date information for each lookup anyway.
 

Here's what happens now:
 

Everything happens the same as it used to except instead of executing the lookup again and again there is a limit of 10 tries. After 10 tries Notes/Domino throws its hands up in the air and returns the error, "The collection has become invalid."
 

So your code that used to run fine in prior versions might now throw an error - sometimes immediately, sometimes after thousands of successful calls for days on end. All it takes is a single doc to be updated around the time your lookup occurs. Obviously the busier your database the higher the chance you'll encounter this new error. In a nutshell, NotesView.AutoUpdate=True (the default behavior) doesn't work any more.
 
IBM/Lotus support says the "workaround" is to trap the error and call a NotesView.Refresh() and try your lookup again. Nevermind that you've probably got 488.223E+309 lines of this code across every app you've ever written, you get to work around this mess just to have your code work the way it used to. And nevermind the fact that additional error catching is always fairly expensive performance-wise.
 
This change was introduced by two "fixes":  SPRs #AHOE7JEKWY and #AJMO7LHMK9. They are both mislabeled in the Fix List Database as "This fix prevents functions from potentially going into an infinite loop when future TimeDates exist in views and folders." Apparently what happens is you might have a document that was created in the future (caused by a very rare Domino "time creep" bug when massive amounts of new docs are created rapidly.) Then your GetDocumentByKey call would loop and loop until the server's time passed whatever the latest doc's time was at which point the view would be considered "up-to-date". So your call would eventually complete but it might take a few seconds/minutes depending on how much time creep your database experienced. If you had a doc created hours in the future then that could be seen as an "infinite" loop and is apparently why IBM decided to "fix" this problem by instead only checking 10 times and then throwing an error.
 
No, there's no documentation update mentioning this.
No, there's nothing in the Readme file about this.
No, there's no INI setting to revert this new behavior back to the way things used to work. This is "how it should have worked all along" according to IBM.
 
I asked for an enhancement request to be created to add an INI setting to control this behavior and apparently there's already other enhancement requests that are similar (including one that has Notes/Domino just do a Refresh() and attempt the lookup again). We're not the only people who have hit this timebomb and as people upgrade I'm sure we won't be the last.
 
That error number is 4678, by the way. Have fun trapping.
 

 
I haven't seen anything regarding a release date for 8.5.2 yet except for "mid-year". But according to the Fix List database here we can expect it Q3 2010.
 
I'm part of the managed beta (go join if you want to take part also) so I can't comment about it yet, but as Tim Tripcony blogged it's going to be "cuh-ray-zee".
 

 
We were downloading in London from a Domino server in Florida. Domino was serving the file at a whopping 65K/sec to London, despite being on a connection with about 8 MB upstream available and the London connection having tons of available downstream bandwidth. Running an FTP server on the same machines allowed the same file to be transferred at about 600K/sec.
 
HTTP transfers are slower than FTP but not that much slower. So we opened a PMR. The answer is this SPR on Windows:
 

SPR # MKIN7TGMQV    Fixed in release: 8.0.2 FP3, 8.5.1
Product Area: Server  Technical Area: Web Server  Platform: Windows
Lotus Customer Support APAR:  LO45289  What is an APAR?

SPR# MKIN7TGMQV - Fix for Windows, where we had  performance problem on high latency networks. We added/changed code to reduce latencies by changing how the winsock layer is used and by increasing the buffer size of network writes.

 
Apparently Microsoft changed the Winsock layer interface when they released Windows Server 2003 but didn't tell anybody. The old interface (which IBM was using) became worse while the new one was the latest-and-greatest (which IIS, surprise surprise, was using.) The end result is that a user's transfer speed was heavily influenced by their latency.  E.g. with 60ms of latency you might get 240K/sec.  With 80ms of latency you might get 180K/sec. And in our case (160ms or so) we were getting 65K/sec.
 
Installing a hotfix for this SPR brought that server's HTTP transfer speed up to around 400-450K/sec to London.
 
While this fix is great and all, the scary bit here for me is that IBM didn't notice this until 8.5.1, so sometime in 2009... about 6 years after Windows 2003 shipped. I hope that a high-latency network model is part of their test suite now.
 

 
If you missed it, you MUST  check out Niklas Hiedloff's blog video. It showcases some projects on OpenNTF, and most importantly shows products on the Notes/Domino platform. And since they're on OpenNTF they're freely available.
 
If you sell Notes/Domino, this video covers in 8 minutes a ton of stuff that - if I were a prospective customer - would make me salivate. It shows off the technical prowess/flexibility of N/D without getting techy, and when you say "yeah there's this entire website of stuff like this" and "yes you can deploy these easily within your organization" and "yes it integrates automagically with your corporate security hierarchy" and "yes lots of this works in your browser too" it's powerful stuff.
 
This specific video looks like it's on its way to replacing the "Nifty Fifty" or whatever it was called from back in the day. If you were to glue a few key LotusKnows video snippets (e.g. LiveText) to the front of this video you would have a 10-12 minute quick intro to Notes capabilities that would blow peoples' minds.
 
Thanks, Niklas. Your work and your videos rock, and you've managed to showcase more of the product in 8 minutes than IBM has for years (LotusKnows not withstanding).
 
Spread the word about this video - this simply can't stay buried on the OpenNTF website.
 

 

 
So a few weeks ago I blogged about the Dojo website getting a facelift. It was much cleaner than before, a bit more organized, though less eye-catching.
 
Well apparently they decided to step it up a notch with an entirely new site design. Click here to see it. I have to say I like it.
 
It looks like they've really put a lot of effort into documentation, too. Click the "Learning is Easy" in the lower-right.
 
And I didn't know Cisco was using Dojo also. Cool stuff.
 

 
So today I was doing some Yellowsphere community service. I decided to hop on over to IdeaJam and start looking for ideas to comment on. I clicked on "Popular" where you see a list of all of the most highly-promoted items. Some of these have been delivered in recent versions (GO IBM!) and others are coming soon. But they're still in the list, and there's no way to tell that some of them have been addressed. I posted "hey, this is done" in the comments of some of them, but it'd be great if IdeaJam had a way to flag something as "RESOLVED" possibly with a note, like "addressed in version XXXX".
 
But there's something else IdeaJam could use: an SPR tracking field. Some of these ideas are already tracked via SPR numbers inside of IBM. Having the SPR number visible on the Idea would allow us to open PMRs to have our customer weight added to the SPR, therefore helping to add weight to the Idea. If two Ideas have a similar vote count, but one has 200,000 seats of customers attached to it and the other 350, that probably makes it obvious which one is more important. Of course if those 200,000 seats are distributed amongst 50 SMBs, that's even more useful data for IBM.
 
It would also help IBM -- who does look at IdeaJam for input -- to relate an Idea to their own internal tracking of the SPR (which has internal notes, insight from development, etc). This would help reduce effort on their end in determining the effort needed to implement an Idea.
 
Whad'ya think, Bruce? Anybody else got any input about this too?
 

A couple of days ago I asked IBM two questions about Lotus App Dev.
 
I started this blog entry by typing up a big article explaining in great detail my reasons for asking, but I junked it. I simply started searching Google, thinking "somebody has to be thinking what I'm thinking."
 
Yup. Buried on Peter Presnell's blog was a great dialogue where the infamous Nathan brings up the exact point I'm thinking (albeit 4 months ago).


Excerpt #1(bold emphasis is mine)
 

"For the record, I think the application development strategy coming from the Notes/Domino team right now is a mess. The APIs that we have today are consistent only in their incompleteness. And while I'm bound by NDA not to discuss future plans in public, I can offer an opinion about those plans, and it's simply that I don't know of any efforts going on that make me feel confident we'll see a better overall app dev strategy from Lotus any time soon.

Obviously, I think that's a great tragedy."

"Unfortunately, there seems to be pretty much zero support for strategic-level thinking around Domino as an app dev platform. IBM has accomplished some great things with N/D technology over the last 3-4 years, but the empowering of developers have been largely a consequence of IBM either scratching it's own itch (the entire Eclipse container in 8) or technical fetish experiments from motivated individual engineers (the Java UI API.) "

 
You can add Domino HTTP GZIP compression to the "technical fetish experiments" list. That feature, though not formally released, was first introduced several years ago via hacking your NAB  -- work on it started during the Dark Years(tm). Enabling it crashed your server horribly though, and it was shelved. That feature sounds like an absolute no-brainer given the recent effort to bolster Domino as a web platform, but apparently it took an individual developer championing for it inside of IBM for it to make it on the 8.5.1 dev list.
 
To that I can only say: "What? Really?"
 
 
Excerpt #2

"The basic problem with App Dev on the ND platform at the moment is that there's no individual IN CHARGE of it. It's more spread out across the people that are responsible for the Domino server, the Notes client, and the overall Expeditor clients. All of these people are talented, focused engineers that bring a huge amount of direction to their teams. But none of them are specifically charged with making Notes/Domino a more successful application development platform. The people that are responsible for app dev can only ASK for resources and direction on this front. They don't get to TELL."

"Right now, you have competing engineering visions of development on Lotus platforms. N/D is by far the most prolific dev platform in Lotus -- at least by number of developers and number of applications. (Portal *might* have more consumers of the actual apps -- it's difficult to say.) But the programmability needs of the Domino server team are addressed by one person, while the Notes client is addressed by another, while Expeditor itself is set by still another -- and that's leaving out all the direction being set by Sametime, Portal, Mashups, Forms, Connections and Quickr in various groups."

"I mean, if IBM *did* put out a development vision statement, and that vision was "Java everywhere," which meant that you'd be able to write Java code everywhere you can currently write LS and @formula, while those two would be deprecated -- would you really be happy?

Maybe you would. I certainly would. But I'd also know that it was a BIG job and that it was going to take 2 or 3 major versions to get there. Whoever set that direction wouldn't be popular in the process.

I don't think we should be sad about that. I think we should be enthusiastic. I think we should smile at IBM and cheer them and say "YES, we're willing to take some pain to get to a proper vision for the platform."

 
The entire N/D process is simply fragmented as it stands now.

If you're a new N/D developer, you need to learn 4 languages: @Formulas for views, Javascript and SSJS for Xpages, and Java for agents. Each language has pros and cons, but the simple fact that the language you choose depends on the run-time context is what I consider to be the second most frustrating thing about developing on N/D.  The most frustrating thing is the fact that the greatest New Thing (XPages) requires a Microsoft-style rip-and-replace to get it to work with the greatest Old Thing (legacy LotusScript apps).
 
And arguably the most powerful language in the Domino arsenal, Java, forces you to manage your own garbage collection via .recycle() -- or else a simple doc-looping agent may bring down your entire server! You also can't use it to code to the Notes UI (though this might be coming in 8.5.2 as the result of a "technical fetish project").
 
Nathan is frustrated with the same problems:
 
 
Excerpt #3:
 

"Off the top of my head...

1) No migration path from existing code (no J2LS bridge)
2) No commitment to making Java a first class language in all code locations (UI elements, actions, view columns, etc)
3) No commitment to resolving the mess stemming from .recycle()
4) C/C++ still primary language for server platform development"

 
Getting back to my original two questions: Why did I ask them?

I asked them because I don't see a clear direction for Notes/Domino app dev. Instead I see 3-4 different ones. I don't know who (if anybody) is *in charge* of Notes/Domino app dev, the person who should obviously be tasked with clearing up some of these things and spelling out a development and application migration roadmap.  I haven't seen any formal discussions around these topics either, as we did when Bob Balaban was still at IBM. All I see as of late is the code that's been shipped. And while it's some great code, it leaves the direction of the platform in a very confusing state.
 

So some of us remember the hard-hitting air cover from the R5 ad campaign featuring the infamous Dennis Leary.  Apparently some marketing agency thought he was doing a good job, as he's gotten some similar work lately:  If you haven't seen it, check out the jabbing commercial for Hulu.

Ok, this blog entry had me REALLY curious.

I just had to go find this technote.  But... it's gone!  "Error: Document not found."  So I search the technote db for "Russ", "8.5 panic", etc.  No go.  So I hit the web and start searching.  Still nothing.

Fortunately, there's Google cache to the rescue!

Edit: Looks like the cache expired.  Click here to issue another search, and click the "Cached" button for the first link.

So I'm cracking up laughing at this point.  This screams of internal debate...

Developer: "Hey Russ, what do we do if somebody hits this particular code fork?"

Russ: "Well, nobody should ever hit that.  Ever."

Developer: "Yeah, but what if somebody does?"

Russ: "Well then you can have them talk to me, because I'll want to see that first hand.  That'd be like winning the lottery."

Developer: "Um, ok..."

Oh, and I know *who* Russ is, though we've never talked or met personally (Russ, if you're reading this, I honestly mean no offense).

Then I get to my favorite bits of the technote:

- Symptom: "Panic message is unusual and not descriptive."

Ya think?  But wait, it gets better...

- Resolving the problem: "...currently there are no plans to address the issue."

Bwahahahaha!  Now, I'm sure they're not just going to leave this error message around saying "talk to Russ" (which is probably why they removed the technote), but the fact that somebody typed up this technote, logged it, got it marked as "no plans to fix" and updated and published it is cracking me up.  Throughout the comedy, multiple somebodies at Lotus Support were being diligent and doing their damndest to make sure that -- come hell or high water -- this thing got documented, reviewed, and published.

I haven't seen this exact crash yet, but I expect to as we start stress-testing our massively complex Domino web app on 8.5.  If I see it, I'll take a screenshot for prosperity.  While I'm not exactly glad to see a crash inside of ServerReadEntries() (we do, um, a bazillion of those a day), it tickles me to no end to see a flash of humanity leak out of the classically buttoned-down IBM online support database, and Domino 8.5 Gold.