Mindoo Blog - Cutting edge technologies - About Java, Lotus Notes and iPhone

  • XPages series #15: Free FTP server on top of Domino’s OSGi framework

    Karsten Lehmann  November 3 2012 14:14:02
    Last week we had a conference call with IBM. They provide free VMware images of several IBM products for business partners to be used for demo and development purpose, including IBM Connections 4, already set up for the well known Renovations company.
    Those images very pretty large, with several GB's of data and the fastest and easiest way to get our hands on the images was to set up an FTP server and have the IBM'ers upload the files.

    That was a good occasion to build a small proof-of-concept that I had already thought of 1-2 years ago:

    An FTP server running on top of Domino


    The implementation is actually quite easy: We simply take an existing FTP server and integrate it into the OSGi framework of the Domino server as a plug-in.
    I selected the Apache FtpServer for this purpose, because it's available as open source with Apache License and easy to set up.

    The first task was to find all necessary libraries to run the server. The FtpServer project is available via Apache's Maven repositories. Maven is a powerful build system that takes care of all code dependencies.
    Although Maven can often be a real PITA, creating a build means basically just to download the source code and use the command

    mvn clean install

    or

    mvn clean install -DskipTests

    The option "-DskipTests" skips the automatic test case execution during the build process and reduces build time.

    As tweeted a few months ago, there is a way to tweak Maven's pom.xml file to let it generate all required Jar files and copy them to a target directory.
    You just need to add this snippet to the pom.xml file:

    <plugin>
          <artifactId>maven-dependency-plugin</artifactId>
          <executions>
                  <execution>
                          <phase>install</phase>
                          <goals>
                                  <goal>copy-dependencies</goal>
                          </goals>
                          <configuration>
                                  <outputDirectory>${project.build.directory}/lib</outputDirectory>
                          </configuration>
                  </execution>
          </executions>
    </plugin>


    I then copied all generated Jar files into a new plug-in:

    Image:XPages series #15: Free FTP server on top of Domino’s OSGi framework




    and exported the relevant packages in the plug-in's manifest file:

    Image:XPages series #15: Free FTP server on top of Domino’s OSGi framework



    Next, I created a plug-in "com.mindoo.ftp" that uses the code from "org.apache.ftpserver" and creates the actual FTP server instance.
    It reads the server configuration and user account data from a small Notes database and writes log documents into another one.

    For now, the FTP server reads and writes its files from disk and not from document attachments in NSF databases, although that would be technically possible and might be an option for the future.

    I decided not to use NSF for the files, because a) this is just a proof-of-concept and b) the FTP server should be able to resume uploaded and downloaded files. The standard Notes Java API does not provide any real streaming functionality (EmbeddedObject.getInputStream() internally just writes the file to disk first, which would kill performance).

    Streaming of attachments out of NSF is possible via NSFNoteExtractWithCallback in the C API (we used this in the MindPlan application to let the user cancel attachment extraction) and there may be a similar method to import attachments via stream.

    Server Configuration


    Our configuration database looks like this - nothing fancy yet:

    Image:XPages series #15: Free FTP server on top of Domino’s OSGi framework



    I added fields for the most relevant server settings to the configuration form, e.g. the server's TCP/IP port, a blacklist to block IP addresses and subnets as well as parameters to use the server behind a NAT router (I tried running it from inside a VMware image).

    Image:XPages series #15: Free FTP server on top of Domino’s OSGi framework



    For users, we can configure their home directory, set a password (we only save a one-way hash generated by @Password) and set a few options to limit the user's access to the server, e.g. define an access period and remove write permissions.

    Image:XPages series #15: Free FTP server on top of Domino’s OSGi framework


    Running the server


    The FTP server plugins should be deployed into the Domino server's HTTP task OSGi framework and start when the HTTP server gets started.
    Unfortunately, in 8.5.3 this is not that easy. There is no such extension point that can be used to automatically start a plug-in on HTTP task startup, like org.eclipse.ui.startup for RCP applications (we have an open feature request for 8.5.4). Everything is started lazily when it is used for the first time. Bad for a server that needs to listen for incoming connections on a TCP/IP port.

    However, the XPages runtime provides one extension point that we can use:

    <plugin>
           <extension
           point="com.ibm.commons.Extension">
                   <service
                   class="com.mindoo.ftp.startup.StartupHandler"
                   type="com.ibm.designer.runtime.extensions.RuntimeInitializationEvent">
                   </service>
           </extension>
    </plugin>


    This way, our plugin gets started the first time an XPage is called. That's a first step, but we need to automate this XPage invocation somehow, to be sure that the server is running after the HTTP task or the whole server is restarted.

    Fortunately, tweeting about this issue helped. Thanks to René Winkelmeyer, I could come up with a solution that leverages the XPages preload feature of Domino 8.5.3 to automatically start the FTP server plugin.
    All you have to do is add the following lines to the server's Notes.ini:

    XPagesPreload=1
    XPagesPreloadDB=mindoo-ftp-config.nsf/ftplauncher.xsp


    "ftplauncher.xsp" is a small dummy XPage in our configuration database. You can use any other XPage instead, since the purpose is just to start the OSGi framework.

    DOTS


    There is an alternative OSGi implementation that can run as part of the Domino server installation: DOTS, also known as Domino Tasklet Service for IBM Lotus Domino.
    In constrast to the HTTP OSGi framework, DOTS contains an extension point to launch plug-ins on framework startup.

    I did not choose DOTS to run the FTP server, because as of 8.5.3, DOTS is not part of the Domino server and needs to be deployed manually. In addition, by using the HTTP OSGi framework instead of DOTS, we can directly call plug-in methods from XPages applications, which might be useful for some applications.

    Server console commands


    I added three server console commands to control the FTP server from the server console:

    tell http osgi mftp restart  -  stops and starts the server
    tell http osgi mftp stop  -  stops the server
    tell http osgi mftp start  -  starts the server


    And of course, the FTP server gets restarted on

    restart task http

    which restarts the whole HTTP task.

    Adding and modifying users does not require a server restart. The lists of blocked IP's and subnets in the server configurations does also get automatically re-read when somebody tries to log into the server.
    All other options currently require a restart of the FTP server.

    Free as in beer


    This proof-of-concept is made available for free. I am sure there are other cool use cases for OSGi based background tasks in Domino. In particular, a scheduler plug-in might be a good thing.

    Setup instructions


    The download archive contains an update site with the plugin code that needs to be deployed onto the Domino server just like the XPages Extension Library : create an NSF update site database, import the plug-ins and features, configure the Notes.ini variable OSGI_HTTP_DYNAMIC_BUNDLES and restart the HTTP task.

    The default paths for the configuration database is "mindoo-ftp-config.nsf" and "mindoo-ftp-config-log.nsf" for the log database.

    An alternative configuration database path can be configured via the Notes.ini variable "$mndFTPConfig=configdb.nsf". For the log database, the variable is called "$mndFTPLog=logdb.nsf".

    And finally, the base directory of the FTP server can be set via "$mndFTPBaseDir=c:\temp\mftp".

    Hope this tool is useful!

    Here is the download link:
    mftp.zip


    Comments

    1Philippe Riand  11/05/2012 2:54:00  XPages series #15: Free FTP server on top of Domino’s OSGi framework

    This is a great post and a great project.

    Unfortunately, people a bit concerned by legal constraints won't be able to use it, as it is not covered by any license. For sure, we cannot at IBM. To make it more consumable to potential users, I would advise you to release it on OpenNTF under the Apache 2.0 license. If you need help, you can contact Niklas.

    Again, this is a great project and it deserves to be shared!

    2Stephan H. Wissel  11/05/2012 3:45:06  Nice one, now repeat

    ... with a websocket server. Would { Link } fit the bill? It is LGPL, so that could work.

    3Karsten Lehmann  11/05/2012 6:35:22  XPages series #15: Free FTP server on top of Domino’s OSGi framework

    @Phil:

    I will put it on OpenNTF under Apache license. Just wanted this to be out and documented before the next week full of work starts :-).

    4Niklas Heidloff  11/05/2012 6:40:42  XPages series #15: Free FTP server on top of Domino’s OSGi framework

    Great project! As Phil said it would be great to share this on OpenNTF. If you made the config an XPages UI it would also qualify for the contest.

    5Karsten Lehmann  11/05/2012 7:15:00  XPages series #15: Free FTP server on top of Domino’s OSGi framework

    @Stephan:

    We already embedded a Jetty servlet engine this way to work around limitations in the file upload process of Domino's HTTP task (no chunked file upload that we needed to stream through an attachment uploaded on a Websphere portlet that gets its data from Domino).

    Websockets should work, but we need to find a solution for authentication to make the server read/write data on behalf on the web user. By default, it uses the server session to access data, which may be a bit too much. :-)

    @Niklas:

    Will take a look into that 'XPages thing'! :-)

    We normally develop for Dojo 1.8 and AMD, so that would have meant to add a bit too much stuff from our framework to make it Apache licensed. Will see what I can produce with this old Dojo 1.6 stuff that Domino is using. ;-)

    6Per Henrik Lausten  11/05/2012 8:38:20  XPages series #15: Free FTP server on top of Domino’s OSGi framework

    Great work (and great detailed blog post too), Karsten :-)

    7Mark Barton  11/05/2012 12:44:22  XPages series #15: Free FTP server on top of Domino’s OSGi framework

    Karsten thanks for this - any recommendations for the best documentation for building OSGI plugins for Domino?

    Currently using a presentation PDF by David Taieb as my source of information (which is good) and got me so far but I can see looking at your code theres a lot I need to learn.

    Thanks

    Mark

    8Karsten Lehmann  11/05/2012 13:06:21  XPages series #15: Free FTP server on top of Domino’s OSGi framework

    Mark, I don't really have a good recommendation.

    For Extlib extension points, you might take a look into the Extlib source code.

    The other stuff is more or less standard OSGi development, like the server console commands via Equinox command provider:

    { Link }

    or the background log thread that gets its log messages via a BlockingQueue of Java's concurrency API:

    { Link }

    9Karsten Lehmann  11/05/2012 13:14:10  XPages series #15: Free FTP server on top of Domino’s OSGi framework

    Here is another link about the BlockingQueue:

    { Link }

    They even have a similar example for a logger thread:

    { Link }

    10Mark Barton  11/05/2012 14:56:38  XPages series #15: Free FTP server on top of Domino’s OSGi framework

    Thanks Karsten I will check them out.

    11Karsten Lehmann  06/24/2013 11:26:00  XPages series #15: Free FTP server on top of Domino’s OSGi framework

    The FTP Server project is now available on OpenNTF:

    { Link }