Erik Brooks

A couple of days ago I asked IBM two questions about Lotus App Dev.
 
I started this blog entry by typing up a big article explaining in great detail my reasons for asking, but I junked it. I simply started searching Google, thinking "somebody has to be thinking what I'm thinking."
 
Yup. Buried on Peter Presnell's blog was a great dialogue where the infamous Nathan brings up the exact point I'm thinking (albeit 4 months ago).


Excerpt #1(bold emphasis is mine)
 

"For the record, I think the application development strategy coming from the Notes/Domino team right now is a mess. The APIs that we have today are consistent only in their incompleteness. And while I'm bound by NDA not to discuss future plans in public, I can offer an opinion about those plans, and it's simply that I don't know of any efforts going on that make me feel confident we'll see a better overall app dev strategy from Lotus any time soon.

Obviously, I think that's a great tragedy."

"Unfortunately, there seems to be pretty much zero support for strategic-level thinking around Domino as an app dev platform. IBM has accomplished some great things with N/D technology over the last 3-4 years, but the empowering of developers have been largely a consequence of IBM either scratching it's own itch (the entire Eclipse container in 8) or technical fetish experiments from motivated individual engineers (the Java UI API.) "

 
You can add Domino HTTP GZIP compression to the "technical fetish experiments" list. That feature, though not formally released, was first introduced several years ago via hacking your NAB  -- work on it started during the Dark Years(tm). Enabling it crashed your server horribly though, and it was shelved. That feature sounds like an absolute no-brainer given the recent effort to bolster Domino as a web platform, but apparently it took an individual developer championing for it inside of IBM for it to make it on the 8.5.1 dev list.
 
To that I can only say: "What? Really?"
 
 
Excerpt #2

"The basic problem with App Dev on the ND platform at the moment is that there's no individual IN CHARGE of it. It's more spread out across the people that are responsible for the Domino server, the Notes client, and the overall Expeditor clients. All of these people are talented, focused engineers that bring a huge amount of direction to their teams. But none of them are specifically charged with making Notes/Domino a more successful application development platform. The people that are responsible for app dev can only ASK for resources and direction on this front. They don't get to TELL."

"Right now, you have competing engineering visions of development on Lotus platforms. N/D is by far the most prolific dev platform in Lotus -- at least by number of developers and number of applications. (Portal *might* have more consumers of the actual apps -- it's difficult to say.) But the programmability needs of the Domino server team are addressed by one person, while the Notes client is addressed by another, while Expeditor itself is set by still another -- and that's leaving out all the direction being set by Sametime, Portal, Mashups, Forms, Connections and Quickr in various groups."

"I mean, if IBM *did* put out a development vision statement, and that vision was "Java everywhere," which meant that you'd be able to write Java code everywhere you can currently write LS and @formula, while those two would be deprecated -- would you really be happy?

Maybe you would. I certainly would. But I'd also know that it was a BIG job and that it was going to take 2 or 3 major versions to get there. Whoever set that direction wouldn't be popular in the process.

I don't think we should be sad about that. I think we should be enthusiastic. I think we should smile at IBM and cheer them and say "YES, we're willing to take some pain to get to a proper vision for the platform."

 
The entire N/D process is simply fragmented as it stands now.

If you're a new N/D developer, you need to learn 4 languages: @Formulas for views, Javascript and SSJS for Xpages, and Java for agents. Each language has pros and cons, but the simple fact that the language you choose depends on the run-time context is what I consider to be the second most frustrating thing about developing on N/D.  The most frustrating thing is the fact that the greatest New Thing (XPages) requires a Microsoft-style rip-and-replace to get it to work with the greatest Old Thing (legacy LotusScript apps).
 
And arguably the most powerful language in the Domino arsenal, Java, forces you to manage your own garbage collection via .recycle() -- or else a simple doc-looping agent may bring down your entire server! You also can't use it to code to the Notes UI (though this might be coming in 8.5.2 as the result of a "technical fetish project").
 
Nathan is frustrated with the same problems:
 
 
Excerpt #3:
 

"Off the top of my head...

1) No migration path from existing code (no J2LS bridge)
2) No commitment to making Java a first class language in all code locations (UI elements, actions, view columns, etc)
3) No commitment to resolving the mess stemming from .recycle()
4) C/C++ still primary language for server platform development"

 
Getting back to my original two questions: Why did I ask them?

I asked them because I don't see a clear direction for Notes/Domino app dev. Instead I see 3-4 different ones. I don't know who (if anybody) is *in charge* of Notes/Domino app dev, the person who should obviously be tasked with clearing up some of these things and spelling out a development and application migration roadmap.  I haven't seen any formal discussions around these topics either, as we did when Bob Balaban was still at IBM. All I see as of late is the code that's been shipped. And while it's some great code, it leaves the direction of the platform in a very confusing state.
 

XPages are gaining traction, and this is a Good Thing (tm).  IBM is obviously behind the concept, as 8.5.1 has some new goodies, and it's no doubt that 8.5.2 will also.  The Dojo community is addressing IBM's needs (and vice-versa), and we're seeing a bit of fresh Domino/XPages developer blood pop up in the 8.5 forums as a result.
 
And a really, really awesome thing is happening: People (including IBM) are contributing sidebar widgets and XPage custom controls to the community. This is huge, and hopefully I'll be donating some nice ones in awhile.
 
XPages are truly revolutionizing Domino development. Aside from a couple of glaring barriers-to-entry (inability to consume LotusScript, complaints of inadequate documentation, lack of user-versus-signer ACL control, inability to "partial refresh" multiple controls) they really are pretty darn amazing. They let you do things with the interface that would simply be cost-prohibitive on legacy Domino.
 
One of the design paradigms that is becoming prevalent in XPage development is a de-normalization of Domino data.  I.E. you don't use list fields (text list, number list, etc.) as often. Many pieces of XPages are built around handling a collection of documents, and to leverage that integration you're better off using individual documents for some things.
 
As a case-in-point here are two of my favorite examples of reusable controls -- both are File Upload managers:
 
Here's one.
 
Here's the other.
 
They both let users upload attachments. Each attachment becomes a new document, with a few pieces of additional information (including the UNID of the "main" document).
 
Cool right? I thought so too. The interface is definitely waaaay better than the old "File Upload Control" from legacy Domino dev. But then I thought about it a bit, and realized that there's a big problem with this new paradigm: indexing.
 
You've got a collection of "main" documents, and a bunch of "file upload" documents. But as soon as you've decoupled your data in this way, many things get much more difficult.
 
Try this: Get a collection of all "main" documents that have more than 2 files uploaded.
 
Now, if instead of XPages you were working with classic Domino dev you would be storing your attachments in the "main" docs themselves. And you would simply type this in your view selection formula: @Attachments > 2
 
But in this new decoupled paradigm of XPages where your data spans multiple docs, what do you do? You either get to hack in some sort of incrementing code to write to a field on the main document (and decrementing when you go to delete) and search on that, or you get to hack in a loop routine to do your own pseudo-JOIN and return the collection of docs that matches your criteria. The first would be potentially open to sync problems, the second would not scale as your dataset grew. A third possible option would be to manipulate the file after it's uploaded to shoehorn it back into the "main" doc, but that's got its own set of problems. And all of these approaches require a bunch of extra code.
 
Ouch.  Score one for legacy Domino dev.
 
I think that XPages are going to quickly bring to the forefront the #1 weakness of NSF - views.  Don't get me wrong - views are still fairly powerful, and there's a lot you can do with them. But they haven't gotten any true indexing enhancements in years. Well, there's the "defer index build" feature which is more of an admin/disk space help (8.0), and also the feature where view index builds are integrated with the transaction log (6.0 I believe). But as far as helping you search/group/sort/evaluate your data, things haven't changed in a decade.
 
The list of potential enhancements is huge, some of which we've discussed here:
 

• store readers' field info as part of the view index to help address the "many readers and docs" problem
• get view indexes out of the NSF, on an external path similar to DAOS' functionality
• implement IBM's patent from 2002  (which pertains to Notes/Domino specifically) to allow users to resort while inside of a category, or while using restrict-to-category
• sorting on category-row totals
  
• cross-column totaling (e.g. a category row has a 12 in column A and 15 in column B.  Column C shows 12/15)
• allow developers to specify fields to be indexed at the NSF level, similar to what "Optimize Document Table Map" does for the Form field (though that feature got broke in 6.0).
• add additional column totaling algorithms (min, max, stddev, "custom" with an @Formula would also be nice)

 
...and of course, the classic problem I've exemplified in this blog post: JOINs. Part of me thinks that, with NSFDB2 being scrapped, IBM's got to at least be aware of the need for those. Hopefully there's work being done. With just a bit more enhancements on views, there's really no stopping XPages.

So the video link in my last blog entry about SSDs might have been bad for some people.

Here's an updated link for those of you who want your faces rocked off.

 
 If you haven't been paying attention to SSDs, prepare to have your face rocked off.
 
As SSDs get cheaper, they will usher in a new era of Domino/NSF scalability. 99% of all Domino implementations are I/O-bottlenecked.
 
With DAOS, you can move your attachments to a separate SAN or a less-expensive array of SAS/SATA drives, keeping your NSF data on SSDs.
 
With the level of write performance SSDs offer, NSF's full-database-locked-while-writing contention won't matter anywhere near as much as it does today.  Perhaps we'll see JOINs in views finally?
 
I'm hoping to have Domino running on SSD soon - I'll post benchmarks when I have them.