DAOS and Lotus Quickr: The Right Way

QuickrLogoTinyThis entry is an excerpt of a report prepared for an enterprise client of IBM Lotus Quickr, running several clusters in staging, development and production. The project was completed a year ago and the client has seen many benefits to enabling DAOS. However, as the documentation on DAOS specific to Quickr is nonexistent, we learned to take into account several other factors…

Introduction

Domino Attachment Object Store is a low-level option on Domino servers (beginning with version 8.5) that allows files to be saved to the file system and referenced seamlessly as if they were still in the Domino nsf structure. Two advantages of DAOS in a traditional environment are lower disk space usage and improved backup time.

In a Quickr environment, not much disk will be saved overall, as most of the attachments are different and de-duplication of files is the primary reason for this savings. However, as most attachments are stored on the file system and only changed when the attachment is changed from the web UI, incremental backups will be significantly faster due to the much smaller size of nsfs and <1% of files changed in a day.

My experience from an enterprise client with fourteen Quickr servers has been phenomenal savings in backup time – more than 85% reduction from what used to be approaching a 24 hour cycle across production.

When DAOS is enabled, attachments over a specified size are saved to disk as “nlo” files. These are the attachments and are – speaking in binary terms – identical to their more friendly-named files such as Word, PDF, Excel documents. “nlo” stands for Notes Linked Object.

It is important to note that DAOS operates on a single server level. Normally this means that nlo files are encrypted with a specific server’s ID, and not shared among clusters. If this is the case, a place cannot be moved from one server to another when there are DAOS files detached from the nsf. However, in any Quickr environment, I recommend that we disable encryption of the attachments on the file system by implementing the notes.ini variable “DAOS_Encrypt_NLO=0”. This must be done on each server prior to enabling DAOS. This will allow us to reconstitute a Quickr place on another server for disaster recovery, testing, development, etc.

Quickr Templates manual step (pre-DAOS)

Quickr databases are based on templates stored in the directory <domino data>\LotusQuickr\Areatypes. When DAOS is first enabled, the PlaceTypes have DAOS turned on. However, some of the Quickr “.ntf” files need to have this turned on manually. To do so, the database property should be checked: “Use Domino Attachment Object Store” on the databases PageLibrary.ntf and MeetingRoom.ntf.

Moving a Place

In order to move a place temporarily from one server to another for testing, we must “un-DAOS” the place. This activity puts all of the attachments back into the place. Testing has revealed that although not specifically documented, this work on both a single NSF and on a directory full of them – which is how Quickr places are organized.

To prepare a place for moving, the following command should be initiated from Domino Administrator, RECON, the Domino server console, or from the Java console:

>load compact –C –daos off lotusquickr\placename

This will a) turn off DAOS on the database properties and b) reattach the attachments with their original names. Note that the NLO attachments will remain on the file system for a default 30 days unless they are pruned.

After performing this command, it is important to review the log.nsf file (or if the place has few rooms, the console) to determine if there were any errors during the compact operation. One error that was seen once during testing, actually happened while turning DAOS on and relates to attachments failing to detach – hence not an impact in this case.

After the log is reviewed, the place may be archived (qptool archive), or the server downed and the place copied to its new location.

Once in its new location, if the target server has DAOS enabled, the place may be “re-DAOS” enabled by using the compact command again with the place directory as an argument:

>load compact –C –daos on lotusquickr\placename

Performance Notes and Caveats

It is not recommended by IBM that the DAOS base path is on the same drive as the Domino Data directory, for disk performance reasons. However, this recommendation is based on the preponderance of Domino MAIL servers using DAOS and the high transaction rate of mail servers. In our case, Quickr attachments will be detached once, then a miniscule percentage of changes made to them in a given day. That coupled with the disk space available on other volumes often being scarce, and the significant backup benefits, there is enough reason to not heed this recommendation.

It should be noted for the future that transaction logging, which is on, is a prerequisite for DAOS, so should not be turned off while DAOS is enabled.

At Lotusphere 2012, it was recommended that we do NOT decrease the minimum object size to 4000 bytes as sometimes discussed, but for management purposes and performance of the DAOS catalog keep it at the higher 64000 byte level. This means we will also only require one pass per server for DAOS enablement.

This should be enough to get you started. In addition to the above, we conducted numerous tests and published results, developed a Production Plan, identified five Scenarios for disaster recovery and restoration procedures, and built a DAOS utility to gather NLO file references that was a necessity for Quickr. If you are interested in this additional analysis and how it may apply to your situation, please let me know in the comments. I can’t send it out as is, but am happy to answer questions you have about DAOS in general or Quickr in specific!

4 thoughts on “DAOS and Lotus Quickr: The Right Way

  1. Mark Gesick

    Thanks Rob,
    Funny this is on my to list …
    What’s the best way to to turn DAOS on for the entire Lotusquickr directory? Since each place has at least 3 .nsf but only main.nsf will/can take advantage of DAOS. Will running the Compact DAOS on break or effect the others?

    Reply
    1. Rob Novak Post author

      Hi Mark,
      Nope, I run it against the entire lotusquickr directory. Turning it on wont hurt databases that don’t take advantage of it. Feel free to run your plan by me — you know my email!
      -Rob

      Reply
  2. bu11frogg

    We’ve had DAOS enabled on our Quickr server for a couple years now and it’s been a big benefit in terms of performance in the browser opening documents and other attachments.

    One of the problems with DAOS is having a proper backup strategy. I was told to delete a place over 18 months ago that “would never be needed again.” And now it is needed…of course. We restored the NSFs from backup but then discovered the DAOS files hadn’t been retained…long story short, it’s been a disaster. What I do now is Archive a place and have the server team back that up somewhere special; it can be easily restored and re-DAOSified if necessary.

    Reply
    1. Rob Novak Post author

      Ah bummer. Yes, of course the NLO files would need to be retained “forever”.

      The application we wrote was part of this equation. When you need to restore Quickr, it could be tens or hundreds of databases as you know. With the “tell daosmgr listnlo” command only taking a single database as an argument, it becomes a nightmare to collect NLO files for even a single Quickr place. We built a mechanism with Notes to list them all, gather them up into a text file, and attach it to a Notes document that is then forwarded to the backup team for file retrieval.

      Anyway, yes qptool archive is also a valid option if they can be managed one by one on retirement…

      -Rob

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *