I have never had the good fortune to attend LotusSphere but if there was one session that might tempt me more than any other to attend it would be the following:-

INV100 Notes/Domino Development Strategy
An outline of IBM's 5-year plan for Notes/Domino as an application development platform.  The session will outline our multi-client strategy for developing applications that run on the Notes client, Web client and/or mobile devices.  The strategy behind Xpages will be made clear and what that means for the Notes Classic design elements.  We will outline our intentions for Xpages as a modern Web development environment  and how it will be positioned against non-IBM Web development platforms such as .Net, Java, PHP etc.  The future for each of the programming languages (@Language, LS, Java, JS, and SSJS) will be clarified.  We will cover planned enhancements to styles/themes, including oneUI and adding styles to Notes classic design elements.  Hear about our plans for JavaScript libraries (e.g. Dojo) and extensions into SSJS.  We will also be covering XPages for the Notes client and our plans for new Rich Client controls. And finally we will wrap this session up with details of how DDE will be improved to allow all the above to be developed as part of a rapid application development environment for which Lotus Notes has always been known.  Expect to come away with a clear road-map of what we would like to achieve over the next five years and how you, the Notes development community, will have a chance to shape that future.

For a longest time now I have been kind of curious to know just how many more weird people there were like me in the yellowverse preferring to develop applications using Lotus Notes rather than some of those other languages.  In my last blog I amused myself with a definition of all things yellow.  So I then decided it was time to do a little bit of research to see if there was any way to measure the size of the yellowverse and the various water coolers dispensing the yellow kool-ade.    It seems there are not a lot of hard numbers available in the public domain so I lined up a few of the key players to conduct a straw poll to at least put a number out there for others to refine.  Many thanks to Ed Brill, Bruce Elgort, Nathan T Freeman, and Chris Whisonant who kindly offered some of their insight.  The following are collection of stats, best guesses, (and not so good guesses) about our amazing yellowverse.....

134,000,000 - Number of Lotus Notes users
405,000 - Postings in Notes 6/7 Discussion forum
300,000 - Yellowverse - Those that have had cause to take the shrink wrap off their copies of either Domino Designer or Domino Admionistrator clients (est.)
100,000 - Yelow Core - Those who use either Domino Designer or Domino Admionistrator clients as a major part of their daily job (est.)
62,000 - Number of votes cast at Idea Jam
60,000 - Number of registered users at openNTF
20,000 - Number of people who have asked questions and/or posted answers in the Notes 6/7 Discussion forum
10,000 - Members of a Chinese Notes User Group
10,000 - Estimated attendees at Lotusphere 2001 (before enrollments were forced to close)
7,000 - Estimated attendees at Lotusphere 2009
7,000 -Comments added at IdeaJam
5,600 - Registered members at bleedyellow.com
2,500 - Total number of ideas posted at Idea Jam
1,500 - Yellow Bubble - active members of Notes community (est.)
1,500 - Highest known hit count for a blog entry at bleedyellow.com
840 - Highest known hit count for a blog entry via Planet Lotus
350 - Number of blogs found at Planet Lotus
200 - Typical number of yellowbleeders online at any time (Sametime group)
1 - The number of Lotus Notes developers required to build the average Notes application
0 - The actual number of other weird people found to be just like me.  (yes we in the yellowverse are all different in our own special ways)

Please feel free to add to or correct any of the above numbers.

I have often asked myself similar questions?  At one of my previous clients an audit was done on Notes applications and it was found that there were 70,000 Notes databases!  This led everyone to start asking - where the **** did all these applications come from?  During my 16+ years involvement with Lotus Notes I have observed the following key sources for Notes applications:-

1. Non-IT Solutions: My biggest beef with IT departments has been that we often make it harder, rather than easier,  for people to get applications developed/enhanced.  The response to this is typically universal - finding a way to do it themselves. (that's how I started doing Notes)  So when a business process becomes too large or complex to be handled manually or via MS Office, Notes often provides a way for an enterprising business professional to automate that process.  This is based upon the speed with which a Notes application can evolve, the relative simplicity of its programming model, and the absolute ease in which Notes applications can be deployed (including security and replication). These are usually the most common form of Notes applications (in sheer numbers).  Many of these applications I never get to see as a Notes developer.  Those that I do are usually because there is a need for a change that cannot be implemented by the inventor.   Often because they are no longer in that role. These applications are highly effective but often not the prettiest, most efficient, or best engineered solutions. The dark forces that often line up against Notes use these examples to paint a picture of Notes as an inferior application development platform without truly understanding the unique nature and characteristics of such applications.  I am sure Java programs would look/behave just as bad if non-programmers could ever figure out how to use Java!
2. RAD:  One of the key reasons I get asked as a Notes developer to build a new Notes application is the speed with which the solution can be built and significantly lower cost structure this entails.  This is perhaps best demonstrated by one IT project which was intending to replace a series of existing Notes applications with a new Java/Oracle application.  As is often the case, design alternatives were requested and a Notes alternative outlined.  The proposed Java approach would cost over $1 million.  The Notes alternative was costed at less than $250K.  A greater focus on cost reduction helped sway that decision to the obvious solution - build it in Notes.
3. Building A Brand: Once a particular business area has one or more professionally built Notes applications the customer is more inclined to specifically request future needs be met by Notes-based solutions or they direct their requests to the Notes development team that have been so responsive in the past.  The only time these requests seem to be intercepted is when the project is big enough or strategic enough to gain senior management attention or the organizational structure is one in which a direct superior of the Notes development group does not (yet) bleed yellow.  Many Domino Web applications seem to get developed because of this predisposition to use Notes and/or a Notes development team.
4. Simply Too Expensive To Replace:  I have never yet had the pleasure of working in an IT department in which Notes was considered strategic. (That would be a dream contract).   But despite the efforts of many non-believers, Notes applications continue to flourish because of the above.  Too often it is simply too expensive to replace existing Notes applications with a "strategic" alternative.  e.g. I once was involved with the support of a Notes application that started as (1), became a (2) before evolving to a (3).  The database had grown to over 1 million records presenting a few challenges on the performance side.  When I was first asked to do some performance tuning for this application the brief I was given was  to give the application legs for a few more months as it was being replaced.  Four years later I left that contract without seeing that application replaced.  The project to replace the application had become a huge money pit.  As the business needs constantly changed we were able to quickly enhance the Notes application while the alternative (C#/SQL Server) simply couldn't keep up with the changes.  It was too big and was never completed.

There are other reasons but I believe the above account for the majority of Notes applications that get built.  What you do not see there is Notes applications being chosen as part of a deliberate long term strategy or because Notes had specific features and capabilities that alternatives did not.  They were chosen largely because they were quick, low-cost, and very effective.  To me this is what makes Notes Application development yellow and differentiates it from most other products.  It is this aspect of Notes I am hoping we don't loose in the efforts by big blue to align its products.  Just as we are starting to realize that our long term future on this planet will require us reverse some of our non-green habits there is a risk that by becoming too blue and less yellow we may also cause an imbalance in the Notes eco-system.

News Release: Yellow Blob Machines (YBM) Announce New Developer Tools

Jan 19 2011: As part of the release of Lotus Notes X announced at Yellowsphere 2010 today, collaboration software giant YBM announced two new development products that seem certain to send shock-waves through the development community and place collaboration rivals Microsoft (SharePoint)  & Google (Gears) on the back foot.  Rather than trying to be all thing to all people (developers) with a single development platform, YBM will now market two separate products for Notes development.  DDXpress and DDProfessional.

Domino Designer XPress:

Requiring little or no training, DDXpress is targeted at the growing DIY programming market of business professionals and academics wanting to implement collaboration solutions without the need for advanced programming skills.  YBM Sale Manager Mr Ed explained this had been a core market for Lotus Notes since its inception but many customers had complained that the power and complexity of Domino Designer for Eclipse was proving to be too confusing and intimidating for novice and part time developers.  Rather than leave this important sector of our market behind we decided to develop a specific development environment that would better meet their needs and allow them to grow over time.   DDExpress will be made available free of charge with Lotus Notes X and the new Lotus Notes XPress clients in a bold bid to encourage computer users to customize and develop applications for it thick client platform.  DDXpress will also allow customers to develop advanced features for its Lotus Symphony product.

DDXpress is based upon Domino Designer 8.0 but goes to even greater lengths to simplify the experience.  This is not your fathers Domino Designer Mr Ed added.  The interface provides true WYSIWYG capabilities to application design and add drag/drop capabilities for an expanded set of controls that can be quickly added to each design element.  Some design elements found in Designer 8.0 such as Navigators, Java Libraries, Web Services, and Styles Sheets have been removed.  DDXpress is designed for Notes Client applications.

Simplifed Programming Choices:
DDXpress will provide three levels of programming, allowing novice developers to slowly graduate as they become more familiar with Notes client programming.

1. Simple Actions:  At the simplest level Notes applications can be constructed using a wide range of common tasks known as simple actions.  YBM have added 150+ simple actions that can now be easily added to Notes applications.
   
2. @Language: The existing @Command and @Formulae language has been updated and expanded from Notes 9.0
   
3. LotusScript: For full programing capabilities, users of DDExpress will be provided with access to the latest version of its LotusScript programming language.  It has been a long time coming but YBM finally decided there was a need to modernize this language that the yellow blob has so strongly supported.  We tried to get these people to budge several times but they just wouldn't Mr Ed admitted.
   

New Application Templates
In collaboration with OpenNTF, YMB have announced a greatly expanded set of application templates for home and office productivity and collaboration.  These new templates will form part of a new Notes XPress Client, a scaled down version of the powerful Notes client targeted at Home & SMB markets that can operate in standalone without the need for a Domino server.  A total of 50 templates are being made available covering a wide range of common applications at both home and work.

Domino Designer Professional

For professional developers YBM have expanded it Domino Designer Eclipse (DDE) product into an exciting new offering that leapfrogs it past Microsoft's Visual Studio as the clear leader in professional application development. Domino Designer Professional provides access to the same design elements supported in Domino Designer Express but adds additional capabilities based upon  it XPage technology introduced with 8.5.

Programming Choices Expanded:
Similar to Microsoft .Net, YBM have announced that it will be fully supporting multiple (three) languages:-

1. LotusScript has been redeveloped  as a full-featured Object Oriented Programming language similar to VB.Net.  This includes support for abstract, partial, and sealed classes, interfaces, casting and the ability for the first time to extend product classes.  Methods can now be overloaded allowing the number and type of each parameter to vary.  An new try/catch error handling schema has also been added.
   
2. SSJS has been expanded to bring it line with the capabilities of LotusScript.
   
3. Java has also been updated to bring it in line with the capabilities of LotusScript.
   

New User Interface Options
YBM sale Manager Mr Ed announced a new alliance with telerik that was crucial in developing a series of 30 new powerful controls for the Notes client.  These controls include date/time picker, name picker, rich text editor, image editor, tag cloud, gauge, slider, action bar, carousel, scheduler, tabstrip, outline, treeview, menu, sliding panel, data grid, graphing, report generator, file exporter, and media player.

1.  Web versions of the controls have integrated AJAX support.  These new controls all render brilliantly in the Web client rivaling Microsoft's Silverlight technology. 
   
2. Notes client versions of these controls add additional capabilities beyond the capabilities of Web browsers.
3. A third, lighter, version of these controls is being supported on leading mobiles devices such as Blackberry and iPhone.

Enhanced Editors
Editors for LotusScript, Java, and SSJS have been enhanced considerably.  Powerful new features have been added for code refactoring.   Integration with the Rational Suite also allowed classes for all programming languages to be built from data models generated during Systems Analysis.  Integrated collaboration connects the developer to a much wider range of on-line documentation, wikis, and new discussion forums that include sametime chat sessions between developers around the globe.

Powerful New Data Sources
DDProfessional will now provide access to a wide range of relational databases using the Xpage design element.  Now for the first time Rapid Application Development and true collaboration can be quickly and easily added to front-end data stored in a wide range of RDMS such as Oracle, SQL Server, Sybase and BD2.
  
Multiple Client Support
The true power of  Notes application development is the speed with which a single application can be developed and deployed for use by Web browsers, mobile devices and the power Notes client.   The enhanced version of XPages now renders almost identically on all clients.  The only exception being the richness of the controls is determined by the capabilities of the client and/or device.

Migration Tools
YBM also announced completion of a project that allowed all existing Notes code and data to be represented in XML.  This means for the first time that both code and data can be exported as XML, translated using XLST and reliably imported back into the some, or another Notes database.  This same capability has lead to YBM developing  a series of migration tools for taking application developed in DDXpress and importing them into DDProfessional.  Mr Ed explained this was another capability high on the wish list of the Notes community since the release of Notes 8.5.

Integrated Code Control:
DDProfessional will include a license for a new version of TeamStudio CIAO allowing for the first integrated code control in a team environment.

New Collaboration Features
DDProfessional will also contain new document management and workflow capabilities taken from its Domino.Doc and Lotus Wo0/rkflow products.  Key components of Lotus Connections and Quickr are being made available to bring many capabilities of these products to the Notes client via Pages.  This move signals that YBM have finally decided to use its Notes product to take Microsoft Sharepoint head-on.

Full Cluster And Load Balancing
In a related announcement YBM also announced that its server products Domino X would now provide better support for application clustering, load balancing, and state management.  Clustering and load balancing now allow a home server to be designated for Notes applications.   Clustering and  load balancing can now be enabled for Notes Client and Web client access ensuring users are constantly being directed to an operating server best placed to service the clients needs.  YBM's powerful replication and clustering technology is also being used to actively replicate state data between servers to ensure no information is lost when it becomes necessary to redirect an active session to an alternate server.


Commenting on this announcement , Notes guru Mr NTF indicated that YBM had finally found a way to exploit JFace technology and JavaBytes to bacback-end SSJS via RPC calls that could elevate dilithium xpage plamsa to an elevated state whereby hypnotic carousel trances could be induced.  As usual no-one else in the yellowverse would admit they had no idea what he was saying but all were still very impressed that he alone seemed to understood how Lotus Notes really worked and that this would go a long way to ward off his arch-nemisis voldemort.

The traditional way to approach formal application development is to create a project, draw up a list of requirements to be included, estimate the effort, commit resources, prepare a project plan, get approval, and then try as hard as possible to deliver the requested changes within the agreed time frame.  This approach I call "fixed content" development and forms the basis of most waterfall SDLC's.
The approach works well in many situations but does have its problems:-

1. Scope Creep: Whenever someone decides to change the requirements this usually has an impact on the project causing delays in delivery and cost overruns.  Participants in the project often have conflicting priorities when trying to resolve these changes.
   
2. Delays: If the project stops going to plan for any reason (including scope creep) there can be a ripple effect across this project and onto other projects that are interrelated because the applications interact in some way or because resources are shared across multiple projects.  This often results in changes being required to project plans for one or more projects and can lead to issues with resource availability.  The most like change is to make the project late which then can lead to other projects being delayed.

In  many organizations Lotus Notes tends to be implemented as a solution for small-medium sized applications.  Notes application development is often way faster than other development platforms.  As a result, the typical Notes developer is often working on a much larger number of projects in a given timeframe than say a Java or C# developer.  Notes developers are therefore more likely to have to deal with the issues that arise whenever one or more of they projects fall behind schedule.  Many SDLCs simply do not apply well to Notes development.

In order to address these issues I have been slowly trying to adapt my approach to demand/project management by focusing more on what I call "fixed time" development.  This is an adaptation of some of the newer agile methodologies, in particular SCRUM.  The following is a brief outline of how this works:-

1. Projects are now broken down into fixed time cycles/units (aka Sprint).  In my case I find a week is a workable unit for the Notes applications I develop/support.  In a given week I can typically spend 60-80% of my time on "new development".  The remainder goes in to administrative tasks, addressing QA defects, and performing releases etc.  So a weekly cycle roughly equates to 25-30 hours per week.  (It is charged as 40 hours to cover those other costs).
   
2. When a project is assigned we allocate a fixed number of cycles and we provide an assessment of how many requests can be completed in that cycle.  Note: For any one application there is often more requested features than can be delivered so these are added to a backlog queue.
3. Now when development starts we either add or subtract requests based upon the developers burn rate during the course of the week.  No matter what development for that cycle is completed at the end of the cycle (week).
4. If we are falling behind schedule we also have the option of adding additional resources and/or increase the hours worked for that cycle.  This is not required but an option that can suit some circumstances.
   
5. If a project consumes more than one cycle then QA can start with the output from the first cycle so that QA issues can be addressed as part of the next cycle.  If is suits the project then the output from each cycle can be released into production so that users of the application can take advantage of the added features without having to wait for them all to be completed.  Note: This is not a requirement, but an option where it makes sense.
6. For really really small projects, two (or more) could be scheduled in a single cycle.
   

The advantages of the above approach include:-

1. Because we always hit our delivery time the organization does not consume significant amounts of time changing project plans and working out how to juggle projects/resources that are downstream from this project.
2. Projects almost always come in on time and on budget!  Sometime we may not have all the features we had originally hoped, but sometime we have more.
3. If a customer/analyst chooses to make late changes to requirements they do so knowing that it will probably come at the cost of something else getting bumped.  Its surprising how this seems to moderate the number of changes requested and sets more balanced expectations!
   
4. Life is so much easier to plan in advance when every project is running in fixed units.  e.g. Meetings etc can be booked well in advance because we have a greater degree of certainty of exactly when things will happen.  A meeting room can be reserved for the same day of every week.
5. My future availability is so much easier to represent as I have weekly cycles into which sprints are allocated.  Rescheduling involves juggling my weeks, not reworking milestones for each and every project I am involved.
   
6. Applications are typically enhanced by smaller more frequent releases.  Users are more likely to be more flexible with what they require because they know if it is not done now it will be in the next release in the not too distant future.  Feedback from smaller releases can be applied before we develop the functionality of the next release.  With organizations constantly in change we are less likely to deliver features that are no longer relevant.

The above is not suited to all projects or even all developers.  For this Notes developer it does seem to work, making me more productive and reducing stress from trying to constantly juggle the many conflicting demands from various stakeholders in the large number of projects I am assigned.

Note: I presently have a new OpenNTF project called .SCRUM that aims to provide a Notes application capable of assisting in the management of projects using a modified form of SCRUM such as the one above.  It will be a Notes 8.5.1 application and based upon the .Domino Framework 2.0 project I am kicking off July 1.  For now I am working on refining the business process before I write too much of the code.  Anyone who wishes to share there own experiences please do as I am sure there are a great many tricks/techniques that can be accommodated within .SCRUM as it is built.

Let me first start by saying I am not a design partner and I do not have any inside contacts at IBM, so I have absolutely no inside information as to what IBM's plans are for the future of Notes programming.  But because I am not subject to any non-disclosure agreements I am free to speculate and say whatever I like. If I start to take a closer look at the tea-leaves now forming at the bottom of my yellow mug I wonder if there may be a pattern forming I had not seen before....

 

I have been taking a much closer look at X-Pages, the new design element  that is part of Notes 8.5.  This design element looks nothing like any other existing Notes design element.  And not just because of the new Eclipse editor.  I see a whole bunch of new controls that remind me more of my time as an ASP.Net developer than they do of Notes development.  I see the ability to bind these controls to data sources that include not just Notes documents and Notes views but also to SQL-databases and XML.  Again, this is not unlike ASP.Net development (except for the ability to natively link to Notes databases).  I also see that as I design an X-Page, it is building XML code which I can not only see but edit directly.... 

 

And when you look at the functionality of X-Pages, they seem to be able to do pretty much everything that many of the existing UI elements of Notes (Forms, Subforms, View, and Pages) can do.  Except that I can now combine them in ways that were never possible with these other design elements.  A number of other bloggers are starting to rave about the really cool stuff they can do using X-Pages.

 

I believe IBM are on record as saying they intend to make X-Pages available for the Notes client in a later release.  So what does that mean?  Unless X-Pages for Notes has a different set of controls that mirror more closely the existing set of controls I am probably going to be soon building new applications for the Notes client in a completely new way.  And thanks to X-pages these applications will look and behave the same on a Web client..  If X-Pages provides the functionality of other design elements such as forms why would I use both in an application?  Perhaps I am not supposed to?  Perhaps in a post Notes 8.5 world I am now supposed to do most of my UI design using ONLY these new design elements such as X-Pages?

 

Perhaps round-tripping of DXL is not an issue for IBM because they  see a future for us developers where we will be able to edit the XML from inside Notes.  Perthaps IBM are not making a major push to develop Eclipse editors for Forms, Pages, and Views because they see these design elements going the way of the Navigator (i.e. no new features added in future releases).  Perhaps there is no need to extend other Notes design elements such as views to support style sheets for the same reason.

 

It is not immediately clear If LotusScript will form a key component of the brave new world.  I am hoping so, because one of the reasons I believe the world has as many Notes applications as it does is that non-programmers (and many programmers) are often better able to teach themselves to program in BASIC-based programming languages much more so than they have ever been able to with Java, C, or JavaScript like languages...

 

The tea-leaves are still swirling around and so the future may show that I am completely wrong.  (But I do hope the dead fly floating at the top is Sharepoint!).  But just in case I am not completely wrong , I would suggest Notes Developers take a very close look at X-Pages - even if you do not do a lot of Web development.  X-Pages (and custom controls) just might provide an insight into how you will be building Notes client and dual client applications in the years to come.  Oh... and read the blogs of those that are design partners.  They may not be able to say a lot just yet... but look at what they don't say!