Erik Brooks

A couple of days ago I asked IBM two questions about Lotus App Dev.
 
I started this blog entry by typing up a big article explaining in great detail my reasons for asking, but I junked it. I simply started searching Google, thinking "somebody has to be thinking what I'm thinking."
 
Yup. Buried on Peter Presnell's blog was a great dialogue where the infamous Nathan brings up the exact point I'm thinking (albeit 4 months ago).


Excerpt #1(bold emphasis is mine)
 

"For the record, I think the application development strategy coming from the Notes/Domino team right now is a mess. The APIs that we have today are consistent only in their incompleteness. And while I'm bound by NDA not to discuss future plans in public, I can offer an opinion about those plans, and it's simply that I don't know of any efforts going on that make me feel confident we'll see a better overall app dev strategy from Lotus any time soon.

Obviously, I think that's a great tragedy."

"Unfortunately, there seems to be pretty much zero support for strategic-level thinking around Domino as an app dev platform. IBM has accomplished some great things with N/D technology over the last 3-4 years, but the empowering of developers have been largely a consequence of IBM either scratching it's own itch (the entire Eclipse container in 8) or technical fetish experiments from motivated individual engineers (the Java UI API.) "

 
You can add Domino HTTP GZIP compression to the "technical fetish experiments" list. That feature, though not formally released, was first introduced several years ago via hacking your NAB  -- work on it started during the Dark Years(tm). Enabling it crashed your server horribly though, and it was shelved. That feature sounds like an absolute no-brainer given the recent effort to bolster Domino as a web platform, but apparently it took an individual developer championing for it inside of IBM for it to make it on the 8.5.1 dev list.
 
To that I can only say: "What? Really?"
 
 
Excerpt #2

"The basic problem with App Dev on the ND platform at the moment is that there's no individual IN CHARGE of it. It's more spread out across the people that are responsible for the Domino server, the Notes client, and the overall Expeditor clients. All of these people are talented, focused engineers that bring a huge amount of direction to their teams. But none of them are specifically charged with making Notes/Domino a more successful application development platform. The people that are responsible for app dev can only ASK for resources and direction on this front. They don't get to TELL."

"Right now, you have competing engineering visions of development on Lotus platforms. N/D is by far the most prolific dev platform in Lotus -- at least by number of developers and number of applications. (Portal *might* have more consumers of the actual apps -- it's difficult to say.) But the programmability needs of the Domino server team are addressed by one person, while the Notes client is addressed by another, while Expeditor itself is set by still another -- and that's leaving out all the direction being set by Sametime, Portal, Mashups, Forms, Connections and Quickr in various groups."

"I mean, if IBM *did* put out a development vision statement, and that vision was "Java everywhere," which meant that you'd be able to write Java code everywhere you can currently write LS and @formula, while those two would be deprecated -- would you really be happy?

Maybe you would. I certainly would. But I'd also know that it was a BIG job and that it was going to take 2 or 3 major versions to get there. Whoever set that direction wouldn't be popular in the process.

I don't think we should be sad about that. I think we should be enthusiastic. I think we should smile at IBM and cheer them and say "YES, we're willing to take some pain to get to a proper vision for the platform."

 
The entire N/D process is simply fragmented as it stands now.

If you're a new N/D developer, you need to learn 4 languages: @Formulas for views, Javascript and SSJS for Xpages, and Java for agents. Each language has pros and cons, but the simple fact that the language you choose depends on the run-time context is what I consider to be the second most frustrating thing about developing on N/D.  The most frustrating thing is the fact that the greatest New Thing (XPages) requires a Microsoft-style rip-and-replace to get it to work with the greatest Old Thing (legacy LotusScript apps).
 
And arguably the most powerful language in the Domino arsenal, Java, forces you to manage your own garbage collection via .recycle() -- or else a simple doc-looping agent may bring down your entire server! You also can't use it to code to the Notes UI (though this might be coming in 8.5.2 as the result of a "technical fetish project").
 
Nathan is frustrated with the same problems:
 
 
Excerpt #3:
 

"Off the top of my head...

1) No migration path from existing code (no J2LS bridge)
2) No commitment to making Java a first class language in all code locations (UI elements, actions, view columns, etc)
3) No commitment to resolving the mess stemming from .recycle()
4) C/C++ still primary language for server platform development"

 
Getting back to my original two questions: Why did I ask them?

I asked them because I don't see a clear direction for Notes/Domino app dev. Instead I see 3-4 different ones. I don't know who (if anybody) is *in charge* of Notes/Domino app dev, the person who should obviously be tasked with clearing up some of these things and spelling out a development and application migration roadmap.  I haven't seen any formal discussions around these topics either, as we did when Bob Balaban was still at IBM. All I see as of late is the code that's been shipped. And while it's some great code, it leaves the direction of the platform in a very confusing state.
 

XPages are gaining traction, and this is a Good Thing (tm).  IBM is obviously behind the concept, as 8.5.1 has some new goodies, and it's no doubt that 8.5.2 will also.  The Dojo community is addressing IBM's needs (and vice-versa), and we're seeing a bit of fresh Domino/XPages developer blood pop up in the 8.5 forums as a result.
 
And a really, really awesome thing is happening: People (including IBM) are contributing sidebar widgets and XPage custom controls to the community. This is huge, and hopefully I'll be donating some nice ones in awhile.
 
XPages are truly revolutionizing Domino development. Aside from a couple of glaring barriers-to-entry (inability to consume LotusScript, complaints of inadequate documentation, lack of user-versus-signer ACL control, inability to "partial refresh" multiple controls) they really are pretty darn amazing. They let you do things with the interface that would simply be cost-prohibitive on legacy Domino.
 
One of the design paradigms that is becoming prevalent in XPage development is a de-normalization of Domino data.  I.E. you don't use list fields (text list, number list, etc.) as often. Many pieces of XPages are built around handling a collection of documents, and to leverage that integration you're better off using individual documents for some things.
 
As a case-in-point here are two of my favorite examples of reusable controls -- both are File Upload managers:
 
Here's one.
 
Here's the other.
 
They both let users upload attachments. Each attachment becomes a new document, with a few pieces of additional information (including the UNID of the "main" document).
 
Cool right? I thought so too. The interface is definitely waaaay better than the old "File Upload Control" from legacy Domino dev. But then I thought about it a bit, and realized that there's a big problem with this new paradigm: indexing.
 
You've got a collection of "main" documents, and a bunch of "file upload" documents. But as soon as you've decoupled your data in this way, many things get much more difficult.
 
Try this: Get a collection of all "main" documents that have more than 2 files uploaded.
 
Now, if instead of XPages you were working with classic Domino dev you would be storing your attachments in the "main" docs themselves. And you would simply type this in your view selection formula: @Attachments > 2
 
But in this new decoupled paradigm of XPages where your data spans multiple docs, what do you do? You either get to hack in some sort of incrementing code to write to a field on the main document (and decrementing when you go to delete) and search on that, or you get to hack in a loop routine to do your own pseudo-JOIN and return the collection of docs that matches your criteria. The first would be potentially open to sync problems, the second would not scale as your dataset grew. A third possible option would be to manipulate the file after it's uploaded to shoehorn it back into the "main" doc, but that's got its own set of problems. And all of these approaches require a bunch of extra code.
 
Ouch.  Score one for legacy Domino dev.
 
I think that XPages are going to quickly bring to the forefront the #1 weakness of NSF - views.  Don't get me wrong - views are still fairly powerful, and there's a lot you can do with them. But they haven't gotten any true indexing enhancements in years. Well, there's the "defer index build" feature which is more of an admin/disk space help (8.0), and also the feature where view index builds are integrated with the transaction log (6.0 I believe). But as far as helping you search/group/sort/evaluate your data, things haven't changed in a decade.
 
The list of potential enhancements is huge, some of which we've discussed here:
 

• store readers' field info as part of the view index to help address the "many readers and docs" problem
• get view indexes out of the NSF, on an external path similar to DAOS' functionality
• implement IBM's patent from 2002  (which pertains to Notes/Domino specifically) to allow users to resort while inside of a category, or while using restrict-to-category
• sorting on category-row totals
  
• cross-column totaling (e.g. a category row has a 12 in column A and 15 in column B.  Column C shows 12/15)
• allow developers to specify fields to be indexed at the NSF level, similar to what "Optimize Document Table Map" does for the Form field (though that feature got broke in 6.0).
• add additional column totaling algorithms (min, max, stddev, "custom" with an @Formula would also be nice)

 
...and of course, the classic problem I've exemplified in this blog post: JOINs. Part of me thinks that, with NSFDB2 being scrapped, IBM's got to at least be aware of the need for those. Hopefully there's work being done. With just a bit more enhancements on views, there's really no stopping XPages.

So the video link in my last blog entry about SSDs might have been bad for some people.

Here's an updated link for those of you who want your faces rocked off.

 
 If you haven't been paying attention to SSDs, prepare to have your face rocked off.
 
As SSDs get cheaper, they will usher in a new era of Domino/NSF scalability. 99% of all Domino implementations are I/O-bottlenecked.
 
With DAOS, you can move your attachments to a separate SAN or a less-expensive array of SAS/SATA drives, keeping your NSF data on SSDs.
 
With the level of write performance SSDs offer, NSF's full-database-locked-while-writing contention won't matter anywhere near as much as it does today.  Perhaps we'll see JOINs in views finally?
 
I'm hoping to have Domino running on SSD soon - I'll post benchmarks when I have them.

Do you work with Notes/Domino? If you're reading this, probably. I've got two questions for you about my favorite topic: VIEWS.

1. Have you ever wanted to offer your users resorting capabilities while *inside* of a category?

2. How about resorting on a view column while using "Restrict To Category"?

Yes, we all have wanted to offer these things since the R4/R5 and R6/R7 days. You can add sorting to your view columns for scenario #1, but as soon as the user clicks on them the categorization is lost, which makes them useless in 99% of scenarios. For #2 you've got nothing.

Over the weekend I'm tinkering around with all of the available ways to "roll your own" to allow such capability, and hitting (as anticipated) scalability problems.

Sometime around  4am in a "can't sleep" stupor of tiredness and computer-science theory, I dream up an elaborate (and very hack-ish) way to potentially solve this problem. So I hit up Google to see if anybody else in the blog arena has ever written about the method I was considering. And then I find this patent that IBM submitted back in 2001.

Holy crap! That's it! And it specifically mentions Notes/Domino as the context, so somebody was actively working on this back then. So... why hasn't this made it into the product? I'm going to cross my fingers that this simply fell by the wayside during the Dark Years, similar to this feature. After all, such sorting is trivial on DB2, so it would have made sense to write-off this effort back then.

So - if anybody at IBM/Lotus is reading this: please check out this patent.

The only real architectural concern I see here is that your view index will get larger -- as always happens with adding sorting to a view. But you will likely reduce the need for other views in the database, and (as I've blogged before) the size concern could be further mitigated if view indexes were stored outside of the NSF.

There's a growing number of fixes listed for 8.5.1.

And even one for 8.5.2!

Admins everywhere can rejoice!

I'm crossing my fingers that I'm right (and this isn't just something that leaked that shouldn't have.) The Fix List database has an SPR listed for 8.5.1:

Fix List Database

Oh, and for the record, 8.5.1 is currently in BETA, might be subject to change prior to release, etc. etc.

I found this interesting... it looks like Google is taking out ads on SearchDomino.com now.  Emphasis is mine, in RED:


::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
SearchDomino.com: Domino Advisor:
Notes/Domino Development edition
September 02, 2009
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
ISSUE SPONSORED BY: Google
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::



::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
IN THIS ISSUE
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
-- Three steps to trap and handle save conflicts with LotusScript
-- Submit your best tip to win our summer contest
-- Trap JavaScript runtime errors in Domino Web apps
-- Create a dynamic fluid image gallery with Lotus Notes views


::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
HEADLINES
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
THREE STEPS TO TRAP AND HANDLE SAVE CONFLICTS WITH LOTUSSCRIPT
Jack Swalwell, SearchDomino.com member

Save conflicts can cause problems in Notes/Domino databases when one
or more users open a document with a link. That document then opens
several times in the Notes client. Use this LotusScript code to trap
and handle save-conflict issues before they reach the Notes automatic
handler.
http://go.techtarget.com/r/9124131/

MORE INFO:

Troubleshoot agents by displaying LotusScript variables online
http://go.techtarget.com/r/9124132/123456

LotusScript sorts lists alphabetically
http://go.techtarget.com/r/9124134/123456

Soft-code item names to facilitate LotusScript management
http://go.techtarget.com/r/9124136/123456


:::::::::::::::::::::::::: ADVERTISEMENT :::::::::::::::::::::::::::

Google

Lotus Notes users are switching to Google Apps.
More and more Lotus Notes users are seeing how they can lower costs,
reduce IT workload and increase employee satisfaction by switching to
Google Apps - the enterprise-ready suite of email, calendar, IM, and
collaboration tools that can be accessed securely from any browser.

Learn how Google Apps can help your company.
http://go.techtarget.com/r/9124137/123456

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::



::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
SUBMIT YOUR BEST NOTES/DOMINO ADMIN OR DEVELOPER TIP TO WIN OUR
SUMMER CONTEST
SearchDomino.com
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

There's only a month left in our summer contest, but we're still
looking for more great tips from our user community! Take a look at
what we're offering, an ioSafe Solo - 500 GB external hard drive.
This external hard drive is waterproof, fireproof and could be yours
if you submit a top Notes/Domino development or administration tip.
http://go.techtarget.com/r/9124138/123456

We'll publish your tips in our weekly newsletters, so that you can
share your helpful tips and tricks with the rest of the
SearchDomino.com community. Send in your best tips now to have a
chance to win.
http://go.techtarget.com/r/9124139/123456

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
ALSO ON SEARCHDOMINO.COM
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
TRAP JAVASCRIPT RUNTIME ERRORS IN DOMINO WEB APPS

Domino developers know that errors can and do occur during
application development and testing as well as during production.
This JavaScript method traps errors in Domino applications --
providing two advantages over the browser's built-in error-handling
method.


CREATE A DYNAMIC FLUID IMAGE GALLERY WITH LOTUS NOTES VIEWS

Publishing Lotus Notes views on the Web is easy, but the usual method
doesn't work as well when publishing image galleries. By creating
dynamic fluid image galleries, users can view images -- no matter
what their screen resolutions are.
http://go.techtarget.com/r/9124143/123456


::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
CONTACT US:
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
Sales
For sales inquiries, please contact us at:
mailto:plarkin@techtarget.com

Editorial
For feedback about any of our articles or to send us your article
ideas, please contact us at:
mailto:editor@searchdomino.com

 



::::::::::::::::::::: ABOUT THIS E NEWSLETTER ::::::::::::::::::::::
This e-newsletter is published by SearchDomino.com, part of the
TechTarget network. TechTarget provides IT professionals with the
resources they need to perform their jobs: Web sites, newsletters,
forums, blogs, white papers, webcasts, events and more. Copyright
2009 TechTarget. All rights reserved. Designated trademarks and
brands are the property of their respective owners.

____________________________________________________________________

To unsubscribe from "Domino Advisor:Notes/Domino Development
edition":

Go to unsubscribe: http://SearchDomino.techtarget.com/

Please note, unsubscribe requests may take up to 24 hours to process;
you may receive additional mailings during that time. A confirmation
e-mail will be sent when your request has been successfully
processed.

Contact us:
SearchDomino
Member Services
117 Kendrick Street, Suite 800
Needham, MA 02494

I'm sure some people are jumping right in here with a raised eyebrow so let me say this up front:  I'm not tired of hearing about new features coming in 8.5.1.
 
What I'm tired of is finding and hearing about all of the things that are still broken in 8.5 FP1 that won't be fixed until 8.5.1 ships. Some of which have apparently been "fixed" since March but still didn't make it into 8.5 FP1.
 
DDE is pig-dog slow, glitchy, and buggy. And riddled with stuff that still hasn't been fixed since the 2nd Beta for 8.5.0.
 
According to some forum posts many of these things are "fixed in 8.5.1" or "much improved in 8.5.1".  Good thing we read the forum. Once again I ask: what has happened to the Fix List database? It's a ghost town.
 
8.5 shipped with enough regression bugs to sink a ship. Heck, it had a Critical Fix released the same day as it shipped. I've been working with Lotus Support on some 8.5.0 and 8.5 FP1 crashes for 2+ months revolving around the NotesViewNavigator class, which was proven buggy as far back as January. That class has been around since R5, but apparently was changed in 8.5 (for XPages, if I had to guess). Logic would dictate that IBM would have unit tested that backwards and forwards, but it obviously wasn't. This is Software Quality 101 stuff here, guys.
 
8.5.1 is supposedly on track for an October release. TEN months for a dot-one update? I think that's a new Lotus record.  Heck, 8.5's first Fix Pack was a 7-month wait. Keep in mind that there is no 8.0.3 upcoming, and there will only be minor Fix Packs to 7.0.x from here on out.
 
Perhaps 8.5.1 is so far behind the usual schedule because it's got so many wonderful new things in it. That's nice, but it's got me worried. If there are more regressions is it going to be another 7 months before 8.5.1's first Fix Pack? If this is the new Lotus release schedule, the quality better be way up there. 8.5.0 simply doesn't cut it.
 
Keep in mind that I wouldn't be so irate if it wasn't for all the good stuff that's so close, yet out-of-reach: XPages, DAOS, etc...Hopefully 8.5.1 will be the "real" 8.5.0.
 
</rant>
 

Those of you who follow the 8.5 forum have probably noticed me fussing at IBM about the lack of fixes/SPRs/etc. listed for 8.5. The Fix List database has been a ghost town since 8.0.2 FP1, complete with tumbleweeds.  Only a few weeks ago was a *partial* list of 8.5 FP1 fixes finally listed in an obscure corner of the notes.net website. Even when 8.5 FP1 was released last week, no formal list of fixes was to be found.

Finally, earlier this week, Ed twittered that there was a full list of fixes posted online.  It's still buried in the same obscure corner of the notes.net website, but here's a direct link to the full list.

If you haven't seen it yet, check it out.  There's some great fixes there, including client fixes that fix some big issues (e.g. "scrollbar quits working").  Many fixes are UI-related that should make things feel snappier and more reliable, and may also fix some of the DDE flakiness that 8.5.0 has had.

There's still nothing listed in the Fix List database, though, and still no word on fixes in 8.5.1 (other than forum posts) which is a shame.  Perhaps IBM is truly embracing the open source movement and they're simply too busy writing code to bother writing documentation?  ;-)

UPDATE: Here's a link to this idea on IdeaJam.

So XPages are cool.  VERY cool.  And since they can be bolted onto an existing Notes app you should be really, REALLY tempted to upgrade your existing Domino web apps with screen-by-screen updates.

Simply rewrite a form/view/etc. as an XPage and leverage all of the CSS and JSF goodness, give your users a better experience, give your app better performance, etc.  And you can finally code stuff where it's supposed to be, without having to use all of the hack-tastic legacy Domino web techniques.

But there's one big hurdle here:  XPages don't work with LotusScript.  At all.  So those legacy apps, if they use WebQueryOpen or WebQuerySave agents, are looking for a forklift-style upgrade where you get to rewrite your agents and script libraries in Javascript and/or Java.

So a cool new technology requires a massive rewrite?  Hmm.... sounds kinda like Microsoft.

But wait, it gets worse.  The Notes client doesn't support XPages (yet).  Oh, I'm sorry - did you need to use that app on the web AND in the Notes client?  Well now you get to maintain your Lotusscript libraries for the Notes client, all while maintaining new Javascript/Java libraries for XPages on the web, right?  Certainly you're ready to double your dev and testing efforts.

I know, I know - Notes 8.5.1 will get XPage support in the Notes client.  This may mean we finally have a single dev road where we can simply write an app in XPages and it Just Works(tm) in browser or client.  And that would be really, really fantastic.  But (as far as I know) Xpages still aren't going to support LotusScript.  So, again, any legacy apps will need major rewrites to take advantage of the new front-end.

I'm sure Lotus development saw this coming a mile away as a shortcoming of XPages.  It was asked about in forums for the Notes/Domino 8.5 Beta, and also the Gold release.  There's even a Wiki article that tells you how you can have an XPage run an agent (which could be an LS agent) as a pseudo-band-aid to try to help on some things.  Unfortunately that approach is riddled with shortcomings, such as requiring that you save your doc(s) prior to calling the agent (even if you're not certain you want to do so yet) and foregoing all XPage error encapsulation.  All your XPage can do is say "Hey man, I kicked off the agent, ok?"  And recreating the functionality of a WebQueryOpen agent? That's even more fun.

Naturally, I opened a PMR for this, asking to be added to the enhancement request to have XPages consume Lotusscript.  But you know what?  THERE WASN'T ONE.  Not a single SPR relating to this.

Now, certainly Lotus development has talked about this.  Certainly Design Partners have mentioned this on conference calls.  Certainly there was feedback about this in the private Betas.  I'm sure Nathan's shouted loudly that this is a huge barrier for legacy applications.  But for general customers this is the only formal means at our disposal to ask for this type of thing.

So if you too would like to see Lotusscript support in XPages, open a PMR and ask to be added to SPR# PALT7SVH29 .