Beyond The Yellow Bubble

From time to time I get requests to create a view in a Notes applications that shows the first or first x documents (only) for each category in the view.  Out of the box there does not appear to be a simple way to do this as the selection formula for Notes views is executed against the document data and not against the view data itself.  It would be kind of nice if one day it became possible in Notes to add view filtering that ran against the view results allowing us Notes developers to do such things as only show those categories where the document count is greater than 1, only show categories where the average is within a certain range, or only show the first x for each category.  With SQL this is a breeze.

Last week I presented the concept of a building datasets (Part 1, Part 2, Part 3) and how this can be used to create a runtime version of a view that can get rid of some of the limitations commonly encountered with standard views.  The components to build this are being included in the .Domino Framework.  I have now extended this concept in the .Domino Framework to allow for the creation of a dataset in which it is possible to specify that the dataset only contain the top x documents for each category.  The resulting dataset can then be saved in a folder allowing the results to be displayed as part of the application.  With one application I have the data in the folder can refreshed on a daily basis using an agent.  With another I have set the QueryOpen event of the folder to refresh each time the folder is opened.  I am working on one final tweak to be included with the 1.0 release of the .Domino Framework.  This provides the ability to nominate if you wish to display the top x per lop level category or top x per subcategory.

In Part 1 and Part 2 I presented the opportunity to add functionality to applications by generating a collection of data (dataset) at run-time  and presenting this data within a Notes application as a datagrid.  Using this technique it becomes possible to do such things as embed multiple category views, display the top 5 x by category, or generate dynamic columns based upon logic built into a form.  The components to implement this in the .Domino Framework are based upon Notes 6.0 (Form, View, Folder, LS Class).

In July I will be starting work on .Domino Framewok 2.0.  This new version of the framework will be targeted specifically at Notes 8.5.1.  (I previously blogged about bypassing 8.5.0 and whilst a year has now passsed my views havent changed about this).  One of the first things I want to do with 2.0 is extend the concept of datasets/datagrids even further....

In .Domino Framework 2.0 the process for building a dataset will be converted to a Web service.  The Web service will then return a Notes URL that correspons to either an existing view/folder that can be used to view this dataset or a folder that has been dynamically generated by the Web service.  This takes the task of (potentially) building a document collection away from the Notes client and places it on the Domino server.  What it also means is that I now have a simple way to build a dataset in one Notes database and pass it to another Notes application where it can potentially be embedded in a frame, form, page, or XPage.  This may not always be the most efficient way to achieve the end result, but in implementing an SOA approach, it eliminates the need to customize views in application A to meet the needs of Application B.

In part 1I discussed the concept of a dataset and how it can be applied to Notes application.  While Notes views are a representation of a dataset I wanted to focus on the opportunities of generating the dataset at run time.  e.g. it becomes possible to take input from the application/user to build the data collection.

Internally, the easiest way to represent a dataset is a NotesDocumentCollection.  One of the issues with this class is there is presently no easy way to store such a dataset, even for the duration of a session.  Nor is there an easy way to display its contents within an application.  Notes provides views/folders as the only effective way of displaying a collection of documents.

1. Views have a number of issues..  The contents of a view are based upon a specified selection formula.  Once created a large amount of server resources are devoted to manitaining the contents of views each and every time the contents of database change.  For this reason the number of views in an application (especially large applications) should always be kept to a practical minimum.
2. Folders seems a much better match for a dynamic run-time dataset as they provide the same basic look/feel but do not carry an overhead to maintan their content.  The process of taking a document collection and adding it toi a view is a very simple one.

The .Domino Framework's DominoDataset class provides a Rebuild method which takes a dataset and gives it a phyiscal form using folders.  We tare then able to mimic the functionality of a datagrid control using the Notes folder control which can be displayed directly or embedded on a form or page.  To maximize flexibility options exist to:-

1. Recognize a dataset that already exists as a view
2. Allow a specific (existing) folder design element to be used.
3. Allow an existing view/folder to be used as a template for a new folder
4. Build a new folder from scratch based upon the field defined in the dataset

 '/**
 ' *  Place the dataset into a folder so that is able to be displayed
 ' */  
 Function Rebuild() As NotesView
  
  Dim Collection As NotesDocumentCollection
  Dim Column  As NotesViewColumn
  Dim Entries  As NotesViewEntryCollection  
  Dim FieldIndex As Integer
  Dim Dataset  As NotesView
  
' Locate exisging view/folder
  
  Set Dataset = iDocument.ParentDatabase.GetView(Me.Foldername$)
  
' For folders, empty the content
  
  If (Not Dataset Is Nothing) Then
   If (Dataset.IsFolder) Then
    Set Entries = Dataset.AllEntries
    Call Entries.RemoveAllFromFolder(Me.Foldername$)
   End If
  End If   
  
' Add documents to folder
  
  If (Dataset.IsFolder) Then
   Set Collection = Me.AllDocuments
   Call Collection.PutAllInFolder(Me.FolderName$)
   Set Dataset = iDocument.ParentDatabase.GetView(Me.Foldername$)
   
' Set folder name and alias
   
   If (Dataset.Aliases(0) <> Me.FolderName$) Then Dataset.Aliases = Me.FolderName$
   If (Dataset.Name <> Me.Title) Then Dataset.Name = Me.Title$
   
' For dynamic folders, refresh design of folder.
'  If a template view is provided, refersh from that else build the folder design based upon the dataset definition.
   
   If (Me.StoreData$ = ENUM_STORE_DATA__DYNAMIC_FOLDER) Then
    If (Me.TemplateView$ = "") Then
     If (Not Isempty(Dataset.Columns)) Then
      Forall ExistingColumn In Dataset.Columns
       Call Dataset.RemoveColumn(ExistingColumn)
      End Forall
     End If
     Forall NewColumn In Me.Fields
      Set Column = Dataset.CreateColumn(Dataset.ColumnCount%+1,Cstr(NewColumn),Cstr(NewColumn))
      Column.Title$ = Me.Label(FieldIndex%)
      If Not (Isnull(Arraygetindex(Me.SortBy,NewColumn))) Then Column.IsSorted = True
      FieldIndex% = FieldIndex% + 1
     End Forall
    Else
     Evaluate(|@UpdateViewDesign("| + Me.FolderName$   + |";"| + Me.TemplateView$ + |")|)
     Call Dataset.Refresh()
    End If
   End If
  End If
  
' Refresh the view/folder
  
  Call Dataset.Refresh()
  Set Rebuild = Dataset
  
 End Function

The following are some of the features that can be added to an application using the dataset/datagrid approach:-

1. Provide the functional equivalence of emedding multiple categories of a view
2. Embed a single category view in which it is still possible to do collapse all/expand all
3. Provide the ability to export data from an application in which a predefined set of rows (documents) & columns (fields) are exported.  The user can also have the option of selecting which of these columns they wish to export.
4. Provide an alternative to views without the overhead of maintaining view contents.  For relative static data folder contents can be rebuilt via a scheduled agent.  Datasets used infrequently can have their content built as part of the folders query open event.
5. Provide a dynamic report buiulding capability
6. Provides an alternate approach to search building in which the use can even define the columns they wish to see in their search results
   

In the non-Notes world of SQL-based development, a dataset  is commonly used to represent data as a two-dimenional collection of rows and columns generated via a SQL query which is then often displayed on a page using a control such as a datagrid.  Because the dataset is generated at run-time its contents can be based upon parameters provided by the application (user).  This includes the ability to display a subset of rows held in a table (view), and/or presenting a subset of the columns in a table, or even bringing together data from multiple tables.  Notes development usually relies on views which are built and maintained in advanced and then displayed in a view control or embedded on a form.  The only options presently available in Notes for filtering the data is to display a single category using an embedded control or @SetViewInfo.

I am presently working on a generic export capability for the .Domino Framework and wanted to provide an option in which users could dynamically select the columns they wished to export.  I wanted to do this in a way that I could restrict the choices of fields to be exported and I also wanted to provide more meaningful names (labels) for some of the fields.  In trying to solvie this need I stumbled across the idea of allowing a Notes application to define datasets.

The business logic for a dataset is represented by the new DominoDataset class, which will be included as part of the .Domino Framework 1.0 planned for release at the end of this month.  In addition to the class we need a form to define the dataset's content and a view to display the available datasets.  The DominoDataset class builds a NotesDocumentCollection based upon a definition which includes data source (database or view), the ability to select specific categories (View only), and/or a selection formula.  The dataset definition also allows the definition of the columns, including options to sort the dataset.

The logic for building the dataset can be found in the AllDocuments properties:-

 '/**
 ' *  Returns a collection of the documents based upon the dataset's definition.
 ' */  
 Property Get AllDocuments As NotesDocumentCollection
  Dim SourceView  As NotesView
  Dim DomView  As baseDominoView
  Dim Collection  As baseDominoDocumentCollection
  
' Generate document collecton from designated source
  
  Set AllDocuments = iDocument.ParentDatabase.GetProfileDocCollection("something really stupid")
  Select Case Me.Source$
  Case ENUM_DATASET_SOURCE__DATABASE
   Set Collection = New baseDominoDocumentCollection(iDocument.ParentDatabase.AllDocuments)
  Case ENUM_DATASET_SOURCE__VIEW
   If (Me.View$ <> "") Then
    Set SourceView = iDocument.ParentDatabase.GetView(Me.View$)
    If (SourceView Is Nothing) Then Exit Property
    Set DomView = New baseDominoView(SourceView)
    If Me.SelectCategory(0) = "" Then
     Set Collection = New baseDominoDocumentCollection(DomView.AllDocuments)
    Else
     Forall Category In Me.SelectCategory
      If (Collection Is Nothing) Then 
       Set Collection = New baseDominoDocumentCollection(DomView.GetCategory(Cstr(Category)))
      Else
       Call Collection.AddDocuments(DomView.GetCategory(Cstr(Category)))
      End If
     End Forall
    End If
   End If
  End Select
  
' Apply any filters to document collection
  
  If (Me.SelectionCriteria$ <> "") Then Call Collection.Documents.FTSearch(Me.SelectionCriteria$,0)
  If (Not Collection Is Nothing) Then Set AllDocuments = Collection.Documents
  
 End Property

Note: A "dataset" does not need to be a new document collection.  It can represent documentation for data found in an existing view/folder within the application.  Datasets define the collections (existing or new) that we chose to expose within an application with the option of generating the data at runtime rather than ahead of time (as is the case with Notes views).

In Part 2 I will explain how it is then possible to take the dataset's contents and place it in a folder allowing us to represent the dataset's in an application using the existing view and embedded view controls.