Mindoo Blog - Cutting edge technologies - About Java, Lotus Notes and iPhone

  • New Domino JNA version available with LOTS of new features

    Karsten Lehmann  September 19 2019 17:23:43
    It's been a long time since the last release of our Domino JNA project. Version 0.9.21 is now available for download as XPages Extensibility Plugin and on its way to Maven Central (takes a few hours to appear).

    Here are a some highlights of the new version:

    • New method to read item definition table of a database
    • Added function to get the template db path by the template name
    • Support for profile note reading and writing
    • Added API to export mails as EML files on client and server
    • Added API to read client and server statistics
    • Tested in Notes/Domino 11 beta 1
    • Added method to harvest DB design for DQL speedup
    • DQL query builder updated for new Domino 11 "contains" terms (FT search)
    • New method to send server console commands
    • Improved richtext to HTML conversion sample to extract files as well
    • Added convenience functions NotesDatabase.toUnid / toNoteId
    • Added NotesDatabase.getAllCollections() to read infos about all views
    • NotesTimeDate.toString method with date/time/timezone formatting options
    • Added NotesIntlFormat argument to NotesTimeDate.fromString
    • Added setter to change timezone of NotesTimeDate
    • New methods to read ACL entries and all ACL roles
    • New methods in NotesCollection to read view design properties, e.g. .isConflict(), .isCollapsed(), isGotoTopOnOpen() etc.
    • Added method to check if $file item belongs to a TYPE_MIME_PART item
    • Added method NotesItem.copyToNote() with parameter to rename the item
    • Added check method to large summary buffer support on DB
    • Added checks for ERR 1028 when openings docs to return null if not found
    • Improved reading of data below view categories
    • Added method to convert RFC822 items to native Domino format
    • NotesDatabase.openMailDatabase(), hasFullAccess() and runWithDbLock()
    • Changed flag to find view in design, did not find all available views
    • Flags to update note without triggering replication (used by replicator)
    • Added experimental code to set sequence number and time
    • Added more HTML conversion options
    • More FTSearch options for NotesDatabase searches (e.g. get result sorted by created/modified/score)
    • Additional methods to write Notes.ini variables (e.g. OSGI_HTTP_DYNAMIC_BUNDLES)
    • Added getter to read TIMEDATE value from Notes.ini with testcase
    • NotesCollection column title/name no longer converted to lowercase
    • Removed ICU4J dependency, now using undocumented method OSTranslate32 to translate between LMBCS and Java strings with support for long texts (length > WORD size that OSTranslate supports)
    • Performance optimizations, fixed memory handle leaks

    Advanced view lookup strategies with Domino JNA for small view index sizes and dynamic filtering and sorting

    Karsten Lehmann  March 13 2019 00:09:50
    This might be interesting for some of you, a pattern how I am using Domino JNA in a recent customer project to speed up view lookups and reduce overall view index size.

    I am using at least three views to produce the content for a data table in the web application:
    • one or more key lookup views
    • one view for the sorting
    • and the final one to read all required view columns.

    1. Key lookup views
    The key lookup views has the minimum required columns for the lookup, e.g. just the sorted columns containing the lookup key(s) and I use NotesCollection.getAllIdsByKey(EnumSet findFlags, Object... keys) to collect the note ids of all documents matching my lookup criteria(s) and the com.mindoo.domino.jna.utils.SetUtil class to AND/OR multiple note id sets.

    The first view might even contain less lookup columns (e.g. just the default index position column) if I am just interested in all documents that match the view selection formula. In that case I am using the view as a stored database search result.

    You can read the note ids of all documents in the view with NotesCollection.getAllIds(Navigate navigator, boolean filterTable, NotesIDTable idTable), which is EXTREMELY fast with parameters navigator=Navigate.NEXT and filterTable=false, as long as the flag "show response hierarchy" is NOT set in the view design. Then NIF just copies an internal index into the IDTable and is done (=>no b-tree traversion and reader list check for the current user).

    Unfortunately, AFAIK, "show response hierarchy" is set by default when you create new views. This lets NIF use a secondary index to search for responses for all view rows, which is slow.

    I first thought that this response hierarchy flag would be required to find conflict documents in the view (as they are response documents), but this is not the case. My tests have shown that this information is still there, although there was a "bug" in Domino JNA which declared each row as conflict because I was reading the conflict flag as it is documented in the C API toolkit.
    I fixed that recently with a special case for "show response hierarchy"==false (details: https://github.com/klehmann/domino-jna/commit/cdfbc6f8e3087eed1eb8328341451f4f0ffbc7dd).

    Additional note id sets could be retrieved from fulltext searches (NotesDatabase.ftSearch(String query, short limit, NotesIDTable filterIDTable)).

    With Domino 10, a DQL search could be used to collect the relevant note ids as well, in Domino JNA: NotesDatabase.query(DQLTerm query, EnumSet flags, int maxDocsScanned, int maxEntriesScanned, int maxMsecs).

    2. Sort view
    The sort view may be identical to one of the key lookup views. I am using this view to find all note ids of my key lookups that are visible in the requested page in the web datatable (=> offset / count received from the browser) and get them returned in view sorting.

    I call NotesCollection.select(Collection noteIds, boolean clearPrevSelection) with clearPrevSelection=true to select all relevant note ids in the view and then call
    NotesCollection.getAllEntries(final String startPosStr, int skipCount, EnumSet returnNav,int preloadEntryCount, EnumSet returnMask, ViewLookupCallback callback) with the following parameters:
    • startPosStr "0" => start at the beginning of the view
    • skipCount = offset+1 =>skip rows based on paging parameters received from browser, "+1" to go from row "0" (which is one row above the first row) to the first
    • returnNav = EnumSet.of(Navigate.NEXT_SELECTED)        => only return previously selected rows
    • preloadEntryCount        => count parameter from browser
    • returnMask = EnumSet.of(ReadMask.NOTEID)        => just read the note ids (in view sorting)
    • callback = a ViewLookupCallback implementation similar to the one I am using for the getAllIds methods internally (https://github.com/klehmann/domino-jna/blob/master/domino-jna/src/main/java/com/mindoo/domino/jna/NotesCollection.java#L996) which returns a LinkedHashSet with the note ids in view sorting, but stops after "count" entries.

    3. Data lookup view
    This "masterdata" view contains all the view columns required to fill the web datatable columns. It only has one fixed sorting (here: by creation date) and no resortable columns.
    As discussed before, "show response hierarchy" is not set here as well.

    Once again I call NotesCollection.select(Collection noteIds, boolean clearPrevSelection) with clearPrevSelection=true to set the view selection, but in this case I select just the note ids read from the sort view in step 2.
    So I read the data columns for all rows in the visible web datatable page, but as they are returned in the wrong sorting (creation date), I need to hash them by note id and reorder them based on the sort view sorting (not a problem, because my page size of 400 entries is quite small).

    To read the view rows, I use NotesCollection.getAllEntries(final String startPosStr, int skipCount, EnumSet returnNav,int preloadEntryCount, EnumSet returnMask, ViewLookupCallback callback) as in step 2, but this time I want to read the UNID and column values for the rows as well:
    • returnMask = EnumSet.of(ReadMask.NOTEID, ReadMask.SUMMARYVALUES, ReadMask.NOTEUNID)
    and get a List returned by the callback:
    • callback = new NotesCollection.EntriesAsListCallback(count)

    Benefits
    Sounds difficult? Yes, I agree, this still needs some convenience methods to make the code more readable.

    The benefit is that I only have one stable view with all the data, resulting in a large view index size and slow initial index time (here 800 MB for 300.000 documents).

    The key lookup and sort views are very small and fast to build (e.g. 60 MB each). So adding another lookup criteria or result sorting later is cheap, which would not be the case if I had to touch my data lookup view and add another resortable view column.

    And even though I collect data from three views, the lookup performance is very fast, almost instant.







    New Domino JNA version with support for Domino Query Language (DQL) in Domino V10

    Karsten Lehmann  October 12 2018 00:07:25
    Today I released version 0.9.20 of our Domino JNA project. This version contains the following new features:

    • Added support for Domino Query Language (DQL) on Domino V10
    • Added basic support to read MIME items (NotesMIMEPart) via getItemValue
    • Added NotesNote.attachFile with in-memory file generation
    • Added NotesDatabase.isDesignHidden()
    • Added utility class to read and write Out-out-office information
    • Bugfixing

    The version is available on Maven Central and as an XPages extensibility API in the releases area of the project on Github.

    Now that Domino V10 is out, we can leverage new functionality, e.g. DQL, but the new functionality has been added in a way that the remaining project still runs on Domino R9.0.1 servers.



    Domino Query Language (DQL)


    Domino V10 provides a new facility to run adhoc searches against Notes databases, using the new Domino Query Language, with a much better performance compared to a classic database search via formula language.


    At CollabSphere 2018, John Curtis, software architect and lead on the DQL project at HCL, gave an introduction to the available search options, how queries are planned and executed and how search performance can be improved by creating lookup views that the facility then uses for efficient data lookups.


    You can find a session recording here
    :
    Demo and Deep Dive: Domino General Query Facility and DominoDB NPM

    and here are links to the Domino Query Language documentation and a technote describing the
    DQL Restrictions for Domino 10.0.0.

    One of the main restrictions in Domino 10.0.0 is that DQL searches can only be run via the new AppDev Pack for Node.js (beta).

    There is no support yet to use DQL in LotusScript and the Notes.jar Java API. This will be added in 10.0.1, planned for the end of the year.

    Domino JNA to the rescue! :-)

    For our Domino JNA project, there is no such restriction, as we are calling the DQL C API method directly.
    So this gives you a chance to play with DQL a bit ealier.

    To prevent malformed queries and malicous code injection, we provide some tooling to build queries in code instead of composing the query string manually.

    There is a test case on Github that demonstrates all the available search options.

    A DQL search returns a set of note ids for matching documents, in the Domino C API called an IDTable (in Domino JNA: NotesIDTable).
    By nature, note ids in an IDTable are sorted in ascending note id order, because that is the way they are written in memory (compressed to reduce memory consumption, see this technote for details).

    To display DQL search results in a web application, a note id sort order is not really useful, that's why the test case demonstrates a technique to cast the IDTable onto a Domino view.

    In short, this means that you can efficiently filter a view down to only return view rows with note ids from that IDTable.

    Blog series about Domino JNA

    Another topic that I would like to draw your attention to is a great blog series about Domino JNA, written by Mark Leusink.

    The first two articles are already available and demonstrate how to implement faceted searches:

    Query Domino data and faceted search with Domino JNA (part 1): setup, sorting and pagination

    Query Domino data and faceted search with Domino JNA (part 2): filtering

    I am really glad and thankful that he is working on this since my free time is still very limited.


    Display all allowed readers of documents in a view column

    Karsten Lehmann  March 6 2018 16:19:03
    By setting the programmatic column name and formula value of a view column to "$C1$", the column automatically displays the content of all reader items in the document.

    If there are any readers, the column also container the author item values, since these are also allowed to see read-restricted documents.

    Image:Display all allowed readers of documents in a view column

    Image:Display all allowed readers of documents in a view column

    "$C1$" is a special column value that the NIF indexer sends to the NSF Search command in the C API to let it compute read access. This information is stored in the view index and compared to the current user's NAMES_LIST (list of name wildcards, groups and roles) on view access (case-insensitive).
    To use this in your own C code, some more work is required. You don't pass a compiled selection formula to NSFSearch, but a view formula, which is the selection formula merged with all view column formulas (NSFFormulaSummaryItem).

    Of course this is not documented anywhere in the C API, that such a view formula can be passed to NSFSearch, same for the special value "$C1$". ;-)

    In addition, you need to pass the constant

    SEARCH_NOITEMNAMES 0x0020

    to the NSF Search, which tells it to not return the whole note summary buffer for each search match (ITEM_TABLE structure), but just the values specified as columns in the view formula in their order, which makes the search a lot faster.


    Too difficult? No problem. Domino JNA to the rescue. This NSF search is the base of our incremental Domino data sync.

    So effectively, we do (almost) index Domino data like NIF, but with different target system. If not all information is available via summary items, the sync code can provide access to the document as well.

    There's code to sync Domino data with the CQEngine indexer and an SQLite database. The two projects contain testcases to demonstrate the syntax.

    Major rewrite of Domino JNA for improved performance, now with incremental indexing of Domino data

    Karsten Lehmann  January 8 2018 11:14:30
    I spend a few days during the Christmas holidays and first week of January to work on our Domino JNA project. The result is version 0.9.11, which is now on its way to Maven Central and available for download as an OSGi plugin for XPages developers.

    Here are the three main features that I have been working on:

    Improved performance
    The project source code has been completely rewritten to use JNA direct mapping. This significantly improves performance in areas with lots of native method invocations like document item access or when adding note ids to ID tables.
    Other performance work has been done to speed up LMBCS-Java-String conversion and to lazily convert summary buffer values when doing a database search (the old version always converted the whole buffer).

    Incremental data indexing
    As I recently wrote on Twitter, I do not know how many code I have written in the past to sync Domino data to external databases or indexers, because they are more powerful doing adhoc-queries than Domino. The good news is that IBM is actively looking into this topic for Domino 10.

    So here is another approach, this time using C API calls, incrementally searching for database changes filtered by an @-formula and not requiring any lookup views.

    The key to getting it implemented this way was the discovery that NSFSearchExtended3, which I am using to search for formula matches, can not only run incrementally by passing a "since" value and returns all changed notes matching the formula, but also calls the callback for all changed and deleted notes not matching the formula.

    This way my code knows what to add/update in the external db or index and also knows what to remove if it existed in the index before.

    To make the algorithm easily reusable, I haven't hard coded a specific sync target. The whole sync process is running against a simple Java interface.

    A sample project is already available with code that synchronized Domino data with the Java based CQEngine indexer. The project contains a testcase that indexes the fakenames database and makes sure everything is working as expected.
    My plan is to create more implementations, e.g. for SQLite or Lucene, but no promises :-).

    The generic sync process handles replication against multiple replicas of the same database (e.g. when replicating with a Domino cluster) and changing the selection formula between syncs, where we do a performance improved comparison what needs to be added to the target and what does not match the new selection formula anymore and gets purged.

    Better testcases
    This is far from being finished, but I started reworking the available testcases so that they do not any longer only "work on my machine", because the sample database that is available for download does not contain some newly added lookup views or documents.

    The plan is to automatically create the sample database with the required structure the first time the test cases are executed, based on a Domino address book template, creating lookup views in code and 50.000 fake person documents.


    For other changes, here are the release notes posted  on Github:
    • Complete project rewrite, now using JNA direct mapping to improve native method invocation performance
    • other performance improvement: faster LMBCS->Java String conversion, lazy conversion of summary value items in NotesSearch.search(...)
    • Improved database search (NotesSearch.search) that now also optionally returns non-matches and deletions when searching incrementally with a "since" date (see callback class com.mindoo.domino.jna.NotesSearch.SearchCallback)
    • New generic class com.mindoo.domino.jna.sync.SyncUtil to incrementally read Domino data for indexing and migration purpose
    • Sample implementation for SyncUtil that indexes data using CQEngine to index Domino data is available as separate project (to reduce the Domino JNA dependencies to a minimum)
    • New methods NotesNote.hasReadersField() and NotesNote.getReadersFields() to get reader fields of a note using an optimized C call
    • New methods for NotesDatabase:
      • getTitle() / setTitle(String title)
      • getCategories() / setCategories(String cat)
      • getTemplateName() / setTemplateName(String name)
      • getDesignTemplateName() / setDesignTemplateName(String name)
      • refreshDesign(String server)
    • added method DirectoryScanner.scan(String formula) to filter directory entries using Domino formula language
    • Bugfixing and package refactoring to hide internal code (struct package moved to com.mindoo.domino.jna.internal)
    • Removed the unfinished com.mindoo.domino.jna.queries package;
      will probably not continue this path, incremental indexing is the way to go


    Bleedyellow.com has a new home

    Karsten Lehmann  November 11 2017 12:16:44
    It took some time and the domain transfer is not 100% complete, but bleedyellow.com is on it's way from René Winkelmeyer to Mindoo.

    Our blog is currently configured to be the default website on
     

    We do not have any plans yet for the domain, but did not want to let it expire. If you have any suggestions, feel free to comment.

    Life sign / Major update for Domino JNA on Github

    Karsten Lehmann  November 10 2017 17:25:35
    Wow - I haven't written a blog post in the last 9 months! It's just so much easier to write a Tweet than to produce long articles, when I split the week between three customers and my family with two children.

    Just wanted to let you know that we are alive and kicking and still doing very much Domino related work! :-)

    Technologies have as always shifted a bit. Most of the time we create pure JavaEE projects now for our customer web apps, developed in Eclipse and not Domino Designer (ok, using it to create lookup views, still working on that part ;-) ).
    A Maven build then collects all moving parts and generates an OSGi update site that can be deployed on the Domino server. So our NSFs contain no code and no further design elements except those lookup views.

    For the frontend, we are mostly using a mix of Bootstrap, React, jQuery and a lot of open source widgets and libraries that we find on the Internet. Some other areas of interest include using ES5/ES6 JavaScript features like async/await and as always exploring what other database systems have to offer.

    The backend of our apps is more and more powered by our Domino JNA project to get the most out of Domino. The project is now used in 7 of our customer web apps and services, of course running on production servers for months.

    Since my last post in January, a lot of work has been done to make the project more powerful and more robust.

    I am happy to announce that version 0.9.9 is now available on Maven Central and as an XPages Extension Library.

    Here are some of the new features:
    • APIs to convert a single richtext item or a whole document to HTML with access to embedded images - NotesNote.convertItemToHtml / NotesNote.convertNoteToHtml()
    • APIs to stop long running code, get progress information and replication progress/results - SignalHandlerUtil
    • APIs to read/write Notes.ini values - NotesIniUtils
    • Utility function to compare Notes names with wildcards (*/Company) - NotesNamingUtils.nameMatchesWildcard()
    • Support to pass open flags when opening a database, e.g. for cluster failover - OpenDatabase enum
    • Method to replicate a database with options and detailed replication stats - NotesDatabase.replicateWithServer()
    • Method NotesNote.setUNID() to change the UNID of a document
    • Utility class to get names of cluster members of a Domino server - ServerUtils

    Performance improvements:
    • Now reading view column names/sortings/formulas using C API in NotesCollection, not having to reopen the DB/View with legacy API (Notes.jar) anymore
    • Improved LMBCS string conversion performance (less copy operations for pure ASCII strings)

    Breaking changes:
    • Removed lotus.domino.Session argument from some of the NotesDatabase constructors (not needed anymore)
    • Using NoteClass enum instead of noteclass bitmask, e.g. for database searches

    I'd love to write more about all the features or product some slides and (re-)appear at conferences, but unfortunately time is very limited. So I see what I can do to at least produce more test cases or make the existing ones work.

    I should have a bit more time now, since there are not many prio 1 topics left on my list that I need to add to the project. :-)



    New APIs for Domino JNA project, now available for XPages development

    Karsten Lehmann  January 16 2017 08:21:26
    It took some time (mainly because I was very busy and my main use case for the project is not XPages), but now the Domino JNA project is available for XPages development.

    That means that there is an XPages Extensibility API plugin available in the project's release section on Github that can be installed both in Domino Designer and on the Domino R9 server to use the API in your own applications.

    After installation, the API classes are available in SSJS and Java code including API source code.
    Since they are integrated with the JSF lifecycle, memory management is automatically handled by the plugin. That means that code using the API does not have to be wrapped in NotesGC.runWithAutoGC() blocks to automatically recycle allocated C handles and memory after a HTTP request.

    If you want to build your own release of the project (or of your own fork), please take a look at the section "Using Domino JNA in XPages applications". The Maven build has been tested on macOS and Windows.

    New features

    Tons of new features have been added to the project during the last months. It's difficult to keep up with test cases and extend code documentation.

    Here are a few highlights:
    • new APIs to use native NSF transactions (atomically create/modify/delete multiple documents in a database, see the Transactions class for known issues/limitations)
    • new APIs to read the ECL for a specific user
    • API to let NSF quickly filter a number of documents with a formula (not just the whole database as in IBM's Java API): NotesIDTable.filter(NotesDatabase db, String formula)
    • support added to direct attachment streaming to quickly skip a number of bytes (can be used to implement partial downloads via Accept-Ranges HTTP header, restricted to uncompressed attachments)
    • APIs to read/write database options with more than 100 available option constants (IBM's Java API contains only 13 of them)
    • Reading selected view rows specified as an ID table now also works on remote servers (use NotesCollection.updateFilters() to push your SelectedList ID table changes to remote servers before reading data)
    • Performance enhancements when inserting many IDs to a NotesIDTable

    To achieve the new functionality, lots of undocumented C API methods have been used that IBM should really add to the public API:
    • NIFReadEntriesExt
    • NIFIsNoteInView
    • NIFIsUpdateInProgress
    • NIFGetIDTableExtended
    • NIFCollectionUpToDate
    • NIFUpdateFilters
    • NIFIsTimeVariantView
    • IDScanBack
    • IDInsertRange
    • IDTableDifferences
    • IDTableReplaceExtended
    • ECLGetListCapabilities
    • NSFSearchExtended3
    • NSFDbGetOptionsExt
    • NSFDbSetOptionsExt
    • NSFTransactionBegin
    • NSFTransactionCommit
    • NSFTransactionRollback
    • extended FIND_XXX flags e.g. FIND_CATEGORY_MATCH to look up category rows in views
    • more than 100 DBOPTBIT_XXX constants to read/write database options

    Please let me know if there are any problems using the API.

    My latest wish list for the Domino Java API

    Karsten Lehmann  November 3 2016 11:10:49
    Last week I sent an email with enhancement ideas for the Domino Java API to IBM development. Most of the stuff listed here (except new @-functions) can be implemented by wrapping existing C API functions or simple copy&paste of already existing code.
    We already did this using our Domino JNA project, but I thought it would be a good idea to add this functionality to the standard API and provide full platform support.

    I don't really have unrealistic expectations whether this will ever be added to the product, given the latest speed new features got in, but I haven't given up hope yet.

    I am posting this here in the blog in case it is interesting for anyone.


    1. ViewNavigator View.createViewNavForSelection(int[] noteIds)
    Filter a view navigator to only return entries with specified note ids, e.g. received via previous lookups.

    Using the C API, this can be done by opening the view with NIFOpenCollection and populating the ID table "rethSelectedList":

    STATUS LNPUBLIC NIFOpenCollection(
    DBHANDLE hViewDB,
    DBHANDLE hDataDB,
    NOTEID ViewNoteID,
    WORD OpenFlags,
    HANDLE hUnreadList,
    HCOLLECTION far *rethCollection,
    NOTEHANDLE far *rethViewNote,
    UNID far *retViewUNID,
    HANDLE far *rethCollapsedList,
    HANDLE far *rethSelectedList);


    To navigate the view, NIFReadEntries would be called with NAVIGATE_NEXT_SELECTED / NAVIGATE_PREV_SELECTED flag.

    2. int[] View.getAllIdsByKey(Object key, boolean exact)
    Convenience function to only read the note ids of view entries matching the lookup key for better performance (less data in the summary buffer returned from NIFReadEntries, so more rows read with one C API call).

    Using the C API, this can be done with NIFReadEntries calls where only READ_MASK_NOTEID is used as return data mask.

    3. View.getViewEntryByKeyWithOptions(Object startKey, int searchOptions)

    This function is already available in the XPages runtime and would need to be copied over to the Domino Java API. The AbstractDominoViewPickerData class in the Extension Library uses it to locate view entries with a search key greater or equal "X":
    https://github.com/the-ntf/xspstarterkit/blob/master/extlib/com.ibm.xsp.extlib.domino/src/com/ibm/xsp/extlib/component/picker/data/AbstractDominoViewPickerData.java#L221

    ViewEntry ve=null;
    String startKey = options.getStartKey();
    if(startKey!=null) {
            int searchOptions = DominoUtils.FIND_GREATER_THAN|DominoUtils.FIND_EQUAL|DominoUtils.FIND_PARTIAL|DominoUtils.FIND_CASE_INSENSITIVE;
            ve = DominoUtils.getViewEntryByKeyWithOptions(view, startKey, searchOptions);
    } else {
            ve=nav.getCurrent();
    }


    So this gives us a way to do inequality searches for Strings and Numbers in views and decide if keys need to be case sensitive.



    These functions can be combined to implement powerful view lookups, e.g. filter view entries by multiple column values and sort the result dynamically.


    Here is an example:

    View myView = db.getView("MyView");
    myView.setAutoUpdate(false);

    //resort view by first sort column
    myView.resortView("firstsortcolumn", true);

    //get matches for first search key
    int[] noteIdsFirstKey = myView.getAllIdsByKey(firstSearchKey, false);

    //resort view by second sort column
    myView.resortView("secondsortcolumn", true);

    //get matches for second search key
    int[] noteIdsSecondKey = myView.getAllIdsByKey(secondSearchKey, false);

    //intersect both note ids lists: list1 AND list2 using some tool class
    int[] noteIdsFirstAndSecondKey = IDUtil.and(noteIdsFirstKey, noteIdsSecondKey);

    //resort view by third sort column
    myView.resortView("thirdsortcolumn", true);

    //get matches for third search key
    int noteIdsThirdKey = myView.getAllIdsByKey(thirdSearchKey, false);

    //merge intersection of first/second key matches with third: (list1 AND list2) OR list3
    int[] noteIdsFirstAndSecondKeyOrThirdKey = IDUtil.or(noteIdsFirstAndSecondKey, noteIdsThirdKey);

    //and now we produce the final sorting:
    myView.resortView("finalsortcolumn", true);

    //we would need this API to reduce the view to entries with specified note ids with max performance (let NIF skip irrelevant collection entries for us)
    ViewNavigator nav = myView.createViewNavForSelection(noteIdsFirstAndSecondKeyOrThirdKey);
    nav.setCacheGuidance(400, ViewNavigator.VN_CACHEGUIDANCE_READSELECTIVE);

    //skip some view entries to support paging
    nav.skip(start);

    //write requested page data (start/count) in JSON format:
    ViewEntry veCurrent = nav.getCurrent();
    while (veCurrent!=null && count>0) {
            if (veCurrent.isValid()) {
                    writeAsJson(veCurrent);
                    count--;
            }

            //read next view entry matching our selection list
            ViewEntry veNext = nav.getNext();
            veCurrent.recycle();
            veCurrent = veNext;
    }



    This view lookup technique is already being used by us at a big Domino customer in Germany using C calls (Domino JNA project), but platform support is currently limited to those supported by JNA and this should really be part of the standard API.



    Other useful APIs to add:

    4. @ModifiedInThisFile / @AddedToThisFile (posted in the Design Partner Forum in January 2014)
    Two new @-functions to sort view entries by the date their corresponding documents got added to/modified in the database. Required for synchronization of Domino data with external databases. Using @modified instead would not cover changes coming in via replication that happened before the cutoff date for the current data synchronization. Response to DP forum article in 2014 said the feature will be in 9.0.2, so it seems to be already implemented and would need to be added to a feature pack.

    5. NoteCollection.add(int[] noteIds)
    Fast bulk add function to add many note ids at once to a NoteCollection for better performance.

    6. int[] DocumentCollection.getNoteIDs()
    Fast method to get all note ids stored in a DocumentCollection as an int array. If DocumentCollection is sorted internally, the array should have the same sorting.
    Could be used in combination with "DocumentCollection Database.FTSearch()" to quickly get a note id list of documents matching a fulltext search.

    7. int[] Database.getDeletedDocuments(DateTime since, int noteClass)
    Method to get a list of note ids for documents deleted since the specified datetime. C API function NSFDbGetModifiedNoteTable already returns deleted notes ORed with RRV_DELETED flag. So this function would simply copy this list to an int array.
    Already available method Database.getModifiedDocuments(DateTime since, int noteClass) is no help because it filters out deleted documents.

    Use case is data synchronisation between Domino databases and with external systems.


    Big update for Domino JNA project on Github

    Karsten Lehmann  July 11 2016 23:33:05
    Last weekend, I committed a big update for the




    Here is what's new:

    Note (Document) APIs
    The last weeks since creating the project, I added basic APIs to work with database notes (documents). The main focus was to be able to read the classic note item value types, e.g. String, Number and Date/Time with single or multiple values per item, handle a few rarely known edge cases (e.g. multiline text values that Domino stores with delimiter \0 between the lines instead of platform dependent \n or \r\n), do proper LMBCS/Unicode conversion routines for special characters like Umlauts and provide access to item metadata like the sequence number or the last modified date of an item.

    While reading the note flag documentation, I discovered NOTE_FLAG_READONLY, which Domino automatically adds to the note info area. This note flag can be used to quickly check whether the current Domino user has edit access to a document without all the work of traversing all author items and manually compare the values to @UserNamesList. So that's what is being queried now when you call NotesNote.isReadOnly().

    Attachment streaming
    Another thing I added was support for note attachment streaming. You probably know that IBM's (poor) implementation of EmbeddedObject.getInputStream() internally first extracts the file to a temp directory and then returns an InputStream to this temporary file. Depending on the file size, that may ruin application performance and fill disk space. So the Domino JNA way of reading attachment is to manually decode the $File items and read the file content in small chunks of 16K, the buffer size of the C function to read attachment data.

    The Domino C API provides some easy methods to set item values for selected data types (e.g. single values for text, number and Date/Time). I added them to the Domino JNA as well, but there is still work to do to write multiple values.

    A few other things that are new:

    User spoofing - reader items on steroids
    A NAMES_LIST data structure is what Domino returns when you call NSFBuildNamesList with a Domino username. This NAMES_LIST contains all name variants/wildcards and groups/nested groups on the server for the specified username.
    It is passed as a parameter when you open a database or a view so that Domino can compute document visibility (reader items) and write access rights.

    Ever since writing the code to decode this simple memory data structure, I was asking myself whether it can be built programatically, because it's just the number of names and the null terminated LMBCS encoded list of names.

    And since I need that functionality for an upcoming customer project (Google+ like application with very flexible read access rights for postings), I tried it and it worked like a charm. :-)

    So now you can build your own Domino users at runtime that do not have to be in a Domino directory. Just pass a Java List with name variants in the NotesDatabase constructor and all operations are executed on behalf of that user.

    Unification of view column value reading
    Domino has two ways to read view column data: READ_MASK_SUMMARY which returns programmatic column names and their values in the summary buffer and READ_MASK_SUMMARYVALUES that just returns the column values.

    The latter leaves more space in the summary buffer for data, as the column names never change between view entries. Unfortunately, one new optimized lookup call that IBM added in R9 to atomically look up a key and start reading values, currently only works with READ_MASK_SUMMARY.

    I don't think that developers should care about this difference, so I now handle all that internally in the NotesViewEntryData object and you simply call NotesViewEntryData.get("Firstname") to read a column value.

    Here is a test case to see some sample code for view data reading (the API has changed compared to the initial release).

    Performance optimizations
    I did some profiler sessions and optimized the API for performance. The main optimization was to add an LMBCS string decoder cache that reduces the string conversion operations and to lazily convert strings when they are first needed (e.g. when reading only a few view column values).


    The code on Github was tested on Mac 64 bit and Windows 32 bit clients so far. The next days, I will do further tests with Windows 64 bit servers since we are using the API in multiple customer projects.

    The repository contains various test cases to get started.