Erik Brooks

UPDATE: Here's a link to this idea on IdeaJam.

So XPages are cool.  VERY cool.  And since they can be bolted onto an existing Notes app you should be really, REALLY tempted to upgrade your existing Domino web apps with screen-by-screen updates.

Simply rewrite a form/view/etc. as an XPage and leverage all of the CSS and JSF goodness, give your users a better experience, give your app better performance, etc.  And you can finally code stuff where it's supposed to be, without having to use all of the hack-tastic legacy Domino web techniques.

But there's one big hurdle here:  XPages don't work with LotusScript.  At all.  So those legacy apps, if they use WebQueryOpen or WebQuerySave agents, are looking for a forklift-style upgrade where you get to rewrite your agents and script libraries in Javascript and/or Java.

So a cool new technology requires a massive rewrite?  Hmm.... sounds kinda like Microsoft.

But wait, it gets worse.  The Notes client doesn't support XPages (yet).  Oh, I'm sorry - did you need to use that app on the web AND in the Notes client?  Well now you get to maintain your Lotusscript libraries for the Notes client, all while maintaining new Javascript/Java libraries for XPages on the web, right?  Certainly you're ready to double your dev and testing efforts.

I know, I know - Notes 8.5.1 will get XPage support in the Notes client.  This may mean we finally have a single dev road where we can simply write an app in XPages and it Just Works(tm) in browser or client.  And that would be really, really fantastic.  But (as far as I know) Xpages still aren't going to support LotusScript.  So, again, any legacy apps will need major rewrites to take advantage of the new front-end.

I'm sure Lotus development saw this coming a mile away as a shortcoming of XPages.  It was asked about in forums for the Notes/Domino 8.5 Beta, and also the Gold release.  There's even a Wiki article that tells you how you can have an XPage run an agent (which could be an LS agent) as a pseudo-band-aid to try to help on some things.  Unfortunately that approach is riddled with shortcomings, such as requiring that you save your doc(s) prior to calling the agent (even if you're not certain you want to do so yet) and foregoing all XPage error encapsulation.  All your XPage can do is say "Hey man, I kicked off the agent, ok?"  And recreating the functionality of a WebQueryOpen agent? That's even more fun.

Naturally, I opened a PMR for this, asking to be added to the enhancement request to have XPages consume Lotusscript.  But you know what?  THERE WASN'T ONE.  Not a single SPR relating to this.

Now, certainly Lotus development has talked about this.  Certainly Design Partners have mentioned this on conference calls.  Certainly there was feedback about this in the private Betas.  I'm sure Nathan's shouted loudly that this is a huge barrier for legacy applications.  But for general customers this is the only formal means at our disposal to ask for this type of thing.

So if you too would like to see Lotusscript support in XPages, open a PMR and ask to be added to SPR# PALT7SVH29 .

So some of us remember the hard-hitting air cover from the R5 ad campaign featuring the infamous Dennis Leary.  Apparently some marketing agency thought he was doing a good job, as he's gotten some similar work lately:  If you haven't seen it, check out the jabbing commercial for Hulu.

This blog post is part of a series that focuses on some possibilities (some mine, some from inside IBM themselves) that could be a big deal to anybody who cares about view performance.  And if you had high hopes for DB2NSF, there's definitely some carryover here as well.

Note: Stephan Wissel beat me to the punch (I've been slowly working on this blog post for a few weeks) and posted a great article explaining how readers fields work within views.  Click here to read it for a little more preliminary background (he's got a great picture, as well).

If you've read his article, some of this post will be redundant.  But please keep reading, there's some other ideas here as well.

Readers fields!

We all love them.  Readers and authors fields are one of NSF's greatest strengths.  Unfortunately when it comes to view performance they are also one of NSF's greatest weaknesses.  One large view with reader's fields can bring an application (and even an entire server) to its knees.

 

A little background:

When a user opens a view (whether in Notes or on the web) a "page" of the view is transferred.  The page contains a certain number of rows determined by several factors.  I'm not going to get into what those factors are here but the point is: the entire view isn't simply transferred to the client (or browser).

That's why as you scroll through a view you'll see network traffic from the little lightning bolt icon.  After scrolling for a certain number of rows, more of the view needs to be loaded, so another "page" is transferred.

When a user requests a page, Domino scans the view for documents they can access.  If the user has access, the doc is added to the page.  Only when Domino has enough rows to fill a page (or it hits the bottom of the view) does it then transfer the page to the user.

 

The Problem:

Imagine a scenario where a user opens a view.  Their client or web browser requests a view "page" of 100 rows.

How quickly does the user get their data?  That depends...

The Good Scenario - The user has access to all of the docs in the first 100 rows.

In this case Domino scans the first 100 rows, and - great - they all go into the page.  The page is shipped off to the client/browser, and everybody's happy.

The Not-So-Good Scenario - The user has access to all of the docs in the first 99 rows.

In this case Domino scans the first 99 rows, and - great - they all go into the page.  But the user needs 1 more row to fill the page to 100 rows, so Domino is going to keep looking.  So the server keeps scanning the view.

If the user can see another doc say, 50 rows down, then Domino will scan those 50 rows, find the next doc, and add it to the page.  Great!  Now there's 100 rows in the page, so the page is sent down to the user.

It Gets Worse...

What if that 100th doc wasn't a mere 50 rows down?  What if it was 5000 rows down?

What if there is no "100th" doc because the user simply doesn't have readers' access to any more docs in the view?

Domino doesn't know.  It will scan and scan and scan until it finds a 100th doc or it reaches the end of the view.  During that time server cpu will be completely maxed out until the scan is done.  In a large view this can easily take several seconds or more.  There will also be some disk i/o for reading the view index (though compared to the cpu usage this is minimal).

If you've got an 8-cpu-core server all you need is 8 users to open this view within several seconds of each other and ALL of your server's cpu will be pegged at 100% for quite awhile.

And Worse...

The results of the scan aren't stored on the server anywhere after the scan is complete.  So if the same user comes back in an hour and opens the view, the entire process is repeated - even though the view might not have changed at all.

And Even Worse...

During the scan, that view's index can't be updated.  That means that if a doc was created or edited during the scan and then somebody else tried to open the view, that second user's scan can't even start until the first user's scan is done AND the view index is updated.

If your database is a bit busy with writes this can quickly cause a massive snowball effect as users "wait in line".  Your app (and even the entire server) can quickly become very slow or completely unusable.

And Even Worse than THAT...

This doesn't just affect a user opening a view.  This problem affects ANY program code that uses view lookups, as well.  @DbColumn, in particular, is extremely vulnerable to this problem because there's no "page" involved - the entire view must be scanned, each and every time.

Workarounds / How Can This Be Fixed?

I would bet that many "Notes doesn't scale" cases would be crushed if this bottleneck was addressed.  It's also a very common problem that bites new Notes developers, as their application that performed quickly for a small number of docs and users in testing suddenly becomes unusable as they start simulating a large production setting.

IBM might be able to better address this problem.  More on that in a minute.  In the meantime you can try to avoid this bottleneck if you're willing to make some tradeoffs:

1. Don't use readers/authors fields if you don't have to.

If you don't need to use them then you can avoid this problem entirely.

2. Set your view index refresh to something besides "Automatic".

Changing this setting might not even be an option if your application always requires current information to be shown in the view.  But if you set the view's refresh to "Manual" or "Auto, at most every X hours" then this can help avoid the "wait in line" snowball problem above.  This won't speed up the scanning process, though, so it's still possible for a small number of users to consume massive server-side resources, and even a single user will be waiting for a page much longer than desired.

3. Add a category column to the front of the view.  Set it to show the docs' readers.  Only use the view as an embedded view.

By making the view an embedded view you can embed it and use the "restrict to category" feature to restrict by @Username or @Userroles.  This will cause a somewhat-optimized scenario during the scan, since all of each user's readable docs will all be adjacent in the view.  Page scans will complete quickly.  Stephan Wissel covers this workaround in his blog (referenced at the top of this entry).

Unfortunately this means that you give up all resorting capabilities of the view for your users, your view index will get much larger (especially if there are multiple readers on the docs as each doc will be repeated across many categories), and you probably won't be able to use it for lookups anymore (but you had a separate view for that anyway, right?)  @DbColumn performance would be much worse since the view is even larger and the scanning process would take even longer for that function.  Assuming you don't run into the 64K @DbColumn limit beforehand.

But most importantly, this workaround doesn't work under some complex scenarios, e.g.: if you're using roles and a user has access to multiple roles with docs in them.  To handle that you've got to jump through some code hoops, essentially hand-crafting your own dataset.  And if you need it sorted ACROSS those separate roles, Domino isn't storing that index for you.

A Real Solution

Many of this blog's readers will have already guessed the "correct" way to fix this bottleneck:

A view needs to track multiple indexes, ideally one per each reader entry found in the set of docs.

WHAT? WHOA, THAT'S A LOT OF INDEXES!   Yes, it is.  Potentially.  Obviously if all of our view index sizes increased by 1000-fold we'd all be a bit concerned.

But it might not need to be *that* bad.  We don't need an index of each of the combinations of readers -- that would grow exponentially (the famous O^n in computer science lingo) and is therefore a no-go.  But if there was a single index for each reader in the view it wouldn't be hard for Domino to simply cycle through a user's multiple roles and piece together the multiple indices on-the-fly (similar to workaround #3 above).  That algorithm is fairly straightforward, and fairly quick -- YEARS quicker than the way things work now.  And for Domino to run it at the compiled source level would be much faster than us having to "roll our own" workaround as per #3 above.

Another possible optimization technique is to have Domino perform and on-the-fly build-and-store of those readers-aware indexes.  I.E, Domino builds its standard view index as it does now.  Then, on first access, Domino could perform its page-scanning technique as it does now, and then store the results that it did calculate in a specific index for that user, helping to avoid future recalculations for that user.  The indexer could then update those per-user indexes as it updates the view's main index.  The per-user-generated indexes could follow the "Discard index" setting from the view.  E.g. if a per-user index wasn't used for X days, delete it.

Final Thoughts

This idea is currently SPRed within IBM as SPR #SVRO44BU6Y.  Open a PMR and reference that SPR# if you're interested.

I don't, however, believe that the increased view index size and -- most importantly -- increased NSF write contention would keep performance acceptable as long as view indexes are still stored in the NSF.  They'll have to be moved to somewhere on the general file system to make this workable from a storage and performance standpoint.  If you've been following this series, though, you'll remember that was the topic of my previous blog entry in this series.

Ok, this blog entry had me REALLY curious.

I just had to go find this technote.  But... it's gone!  "Error: Document not found."  So I search the technote db for "Russ", "8.5 panic", etc.  No go.  So I hit the web and start searching.  Still nothing.

Fortunately, there's Google cache to the rescue!

Edit: Looks like the cache expired.  Click here to issue another search, and click the "Cached" button for the first link.

So I'm cracking up laughing at this point.  This screams of internal debate...

Developer: "Hey Russ, what do we do if somebody hits this particular code fork?"

Russ: "Well, nobody should ever hit that.  Ever."

Developer: "Yeah, but what if somebody does?"

Russ: "Well then you can have them talk to me, because I'll want to see that first hand.  That'd be like winning the lottery."

Developer: "Um, ok..."

Oh, and I know *who* Russ is, though we've never talked or met personally (Russ, if you're reading this, I honestly mean no offense).

Then I get to my favorite bits of the technote:

- Symptom: "Panic message is unusual and not descriptive."

Ya think?  But wait, it gets better...

- Resolving the problem: "...currently there are no plans to address the issue."

Bwahahahaha!  Now, I'm sure they're not just going to leave this error message around saying "talk to Russ" (which is probably why they removed the technote), but the fact that somebody typed up this technote, logged it, got it marked as "no plans to fix" and updated and published it is cracking me up.  Throughout the comedy, multiple somebodies at Lotus Support were being diligent and doing their damndest to make sure that -- come hell or high water -- this thing got documented, reviewed, and published.

I haven't seen this exact crash yet, but I expect to as we start stress-testing our massively complex Domino web app on 8.5.  If I see it, I'll take a screenshot for prosperity.  While I'm not exactly glad to see a crash inside of ServerReadEntries() (we do, um, a bazillion of those a day), it tickles me to no end to see a flash of humanity leak out of the classically buttoned-down IBM online support database, and Domino 8.5 Gold.

So apparently in Lotusphere 2001, when "RNext" was ND6, IBM/Lotus had plans for developers to be able to start/commit/rollback transactions to NSF.
 
Supposedly it was in the R6 Beta.  Where did this go?  Was it thrown to the side as we stepped onto the Workplace path?  Information here is sparse.
 
This is a common feature that RDBMSs support that people have been asking for NSF to support for years.  Anybody know anything about it?  Some more searching uncovered another article from R5 mentioning it.

So, I'm reading blogs this morning and on Ed's blog I see the following Twitter:

"Google announces offline mail support for gMail.  Wow, Domino Offline Services was only 10 years ahead of its time!"

True.  But: How many people out there have used DOLS?  How many people reading this blog have used DOLS?  I have.  I'd bet large sums of money that I have one of the most complex, multi-NSF Domino web apps in history running on DOLS, with quite a bunch of distributed users.  And based on a couple of DOLS-related support tickets that I've had to open, I'm pushing the envelope with it.

 

In my opinion DOLS is (and has been) the greatest unknown secret of Notes/Domino.  DOLS can allow users to access their mail offline in their browser, yes.  But it can do that with ANY OTHER Notes/Domino databases, too.  It's fairly easy to set up and administer (though it could be easier), is fairly easy for users to work with (though that could be much easier, and *this* is where Google will likely leapfrog Lotus in the future), and although it has its share of problems it is an incredible tool.

 

Simply put, DOLS is a packaged-installation of the Domino HTTP server and a small Windows-desktop app for launching said HTTP server to do work offline.  It also has a mechanism for replicating NSFs back to the corporate office.  Your users work in a browser as they would online; they're just connecting to the local HTTP server on the laptop instead of your corporate server.  It's all core Notes, through-and-through, and is a brilliant re-use of existing technology.  Click here to read Damien Katz's similar comments.    If you've got a Domino web app that you want to enable for offline use, it's literally only a couple of hours to put DOLS support into it and try it out.

 

So is IBM continuing the advance of this fantastic add-on?  It's a bit uncertain. They've kept re-packaging it with each release since its launch (back in R 5.x!) But it has had only minor bug fixes since then, and the release of Notes/Domino 8.5 has a fairly glaring omission: DOLS doesn't currently support xPages offline. 

 

I wouldn't be surprised if this was a decision of "we gotta ship 8.5, we'll do it in 8.5.x".  I also wouldn't be surprised if it was simply overlooked - the DOLS packaging is so straightforward it probably easily builds automatically whenever a new update is released.  It should be the case that the xPages engine just needs to get packaged in with everything else, and it would Just Work (tm).

 

With xPages as the obvious future of Domino web dev, DOLS now runs ths risk of becoming a mail-only utility unless it's brought up to the present with xPage support.  If it can't run the new wikis, blogs, discussion databases, custom apps, etc. that IBM and the community are creating its usefulness will be severely diminished.  Then the question becomes: How long will it take Google to extend their offline functionality to other areas, e.g. Google Apps?

 

The good news, though, is that adding xPages support should be fairly simple for IBM/Lotus to do:  they're already packaging the full HTTP server as it is.  Can you imagine: xPages-enabled Domino apps running offline?  Mmmmmm....

 

If you haven't yet tried DOLS, check it out.  You will be surprised.

Well, it's coming at least.  Probably.  From the Fix List database:

 

SPR # DMAT72JLWW Fixed in release: 7.0.4 

Product Area: Designer  Technical Area: LotusScript  Platform: Cross Platform

Lotus Customer Support APAR:  LO28240  What is an APAR?

SPR# DMAT72JLWW - Added NotesDatabase.CreateDocumentCollection method to create empty doc collection.

 

So it sounds like we'll see this in 7.0.4, due in March.  Which means it will likely make it into 8.0.3 and/or 8.5.1.  I guess we can all get ready to cross another one off of the IdeaJam list!

This blog post is the first of a 3-part series that focuses on some possibilities (some mine, some from inside IBM themselves) that could be a big deal to anybody who cares about view performance. 

 

Views.

 

They're the core component of any Domino database.  You may argue that forms are more important, but with xPages you don't even technically need those anymore either.

 

Views are the only real-time-capable index you can create.  If you need to sort or categorize data with anything remotely resembling scalability or speed, you need views.  As a result we all want them to be fast.  REAL fast.  And we want lots of them, with all the sortable options enabled, and with as many different columns showing as many different pieces of data as possible.

 

But views are expensive.

  

One of the best things IBM could do for views (besides adding JOIN capabilities) would be to store view indexes outside of NSFs.

A view "index" is essentially two items, $Collation and $Collection, that gets stored on the view design docs.  Go pull up the document properties on a view design doc in Designer, and you'll see them.  But you won't see the sizes.  For that, you'll need to use the Administrator client.  Simply open the "Files" tab, right-click on your database and select "Manage Views".

 

 

If these fields were moved to be outside of the NSFs, then we'd realize some significant benefits:

 

1. Many NSF sizes would be drastically reduced -- many NSFs have over half of their size allocated to view indexes.  And smaller NSFs perform much faster than larger ones.

 

2. View indexing could be offloaded to separate drives from the NSFs altogether -- even more speed!

 

3. Notes queues writes.  A db is completely write-locked with *any* write that occurs to an NSF.  So while a view index is being written to inside an NSF a doc cannot be simultaneously added to the db, user activity can't be recorded, etc.  With this change this bottleneck could be removed - the view index could be written to externally while a doc is added to the NSF.                            

  
Here's the kicker:  With DAOS in 8.5 I'd bet money that the groundwork is likely already there for abstracting an item out to the filesystem.  View indexes aren't replicated anyway - why store them in the view design doc at all? You could argue that for security reasons they should stay in the NSF, but that's easily addressable by encryption similar to how DAOS encrypts its files.

 

Sound interesting?  If you're interested, there's two ways to let IBM know that you care: 

 

1. Open a support ticket with IBM, asking to be added to SPR#: DCOE7N32FM / APAR#:  LO36534

 

2. Go vote on the topic on IdeaJam: http://www.ideajam.net/IdeaJam/P/ij.nsf/0/BA723B6332ED0C67862573B4003E9AE2?OpenDocument

 

Thanks for reading.  There's more fun stuff to come, including an IBM patent that will make some of us Domino people say "WHAT? WHEN?"

This is a follow-up to a discussion on Volker's blog.  Things got a bit technical (and lengthy) where I shared my thoughts on possible future directions for NSF so I decided to bring it here.  Besides, my blog's been fairly empty and it needs some content.

Here's the original discussion:  Volker's blog

First, my original post:

Technically speaking, there's not much more IBM needs to do to bring Domino up to the next level DB-wise.

 

Domino 8.0 finally got rid of the last of the 16-bit memory handles. If IBM continues to push into the 64-bit realm with Domino, one would assume they'll push the file handles also, and therefore we should see the expansion of DB sizes past that 64GB limit.

 

All of the latest compression options obviously help to reduce size. DAOS helps move attachments out of an NSF, which is a huge help in many cases. But more importantly, the abstraction introduced with DAOS lays the groundwork for other abstraction.

 

For example, getting the $Collation item on view design docs (i.e. a view index) OUT of the database would be a huge help. It's not replicated, so why even store it there? Many complex Notes apps have over 50% of their space consumed in view indexes.

 

And with complex indexes comes a serious performance hit. Domino locks an entire NSF any time ANY write is performed to it. This includes updating view indexes, a doc update, etc.

 

As a result, getting view indexes out of the NSF files themselves would shrink the filesize further, likely allow for faster simultaneous index updating, and possibly allow for truly complex JOIN indexes to be stored on-disk.

 

Throw in the capabilities NSF currently has (solid replication, readers/authors security, an ultra-flexible storage model), and it's not far from being the obvious db of choice for 90% of most modern applications, provided that somebody knows the query language (@Formula).

 

After that, flesh out the SQL querying capabilities and it's a slam-dunk.

 

But let's hope IBM wants to do this work.

 

Then the question was asked: "why can't Notes just store its data in a relational db?"  Here's my response:

 

You *can* store your data in an SQL database, but you lose the flexibility of the Notes model.

 

Each record  (document) is self-defining, and fields that don't exist on a record (document) don't consume any additional space.  Each document is also completely encapsulated, so when you hit Mail -> Forward on a document it's easily wrapped up and shipped off.  You can shove it into another Notes db with no prior prep work on the destination database.  Take all of those capabilities and add multi-value fields, field-level security and field-level replication, it really becomes something that is quite difficult to reproduce in a relational db.

 

Approaching it from the other direction, it would probably be (or have been) easier for IBM to try to squeeze more SQL-like capabilities into NSF:

 

1.  SQL queries - This has been somewhat doable for years with NotesSQL, and now with NSFDB2.  It needs to become a core part of the product.

 

2.  JOIN capabilities - This is partially addressed with #1, and you can write backend code or use XPages in 8.5 to do the UI-equivalent but again, indexing speed is key.  There is currently NO way to *store* the index that results of a JOIN -- at least not without NSFDB2 and again, it needs to become a core part of the product.  The lack of JOIN capabilities is really the main architectural disadvantage of NSF (besides just getting big as we've discussed).  There has been a *tiny* bit of progress here in 8.0 with the new NotesDocumentCollection.Intersect() methods and few others.

 

3.  Enforcing Referential integrity - If a doc is "dependent" on another doc, not much is keeping you from deleting the 2nd doc, therefore causing an invalid relationship in the database.  This would likely be a lot of work, and when taking replication and doc security into account I'm not even sure it's completely doable.  But for many applications this is a "nice to have" that can be addressed in other ways and not an absolute requirement at the db level.

 

4.  Transactional model - IMHO, NSF can't really go there while maintaining it's current capabilities, and this would be the #1 reason to stick with a relational database.  People don't enjoy replication/save conflicts in their payment systems. 

Well, you blog-readers have probably seen me posting all over various blogs in the community, but I guess a formal introduction is in order:

At the time of this posting I'm in my early 30's, am entering my 10th year of Domino experience, have been coding since I was 6, and have a formal computer science background.

I've been working on what is probably one of the most complex Domino web apps in the world for the past decade - my servers run HTTP, Amgr, and Update.  I've pushed the web limits to the extreme, breathe DXL, started building AJAX tools back in 2003, and integrated Dojo quite awhile ago. 

Stay tuned for some juicy web stuff, including some info on the web-related SPRs that some of you know I've been working.

So I've been hem-and-hawing about starting a blog... and ever since I posted the "300" reference on Ed's blog I had been thinking of sniping Planet Lotus' 300 spot as a goal to finally make me get off my butt and do it.

Did I make it?