As a consultant, I work on a variety of projects for a variety of clients in a variety of locations. Sometimes I need to look back and determine where I was on a certain day, or I may need to know how long it has been since I visited a given client. Sometimes I need to look at data from several years ago. I can usually figure this out by looking through my email or calender history. But what I wanted was an easy way to log my daily location.
At Aptera, we use Lync as our office communication tool. Lync knows my location, and once I’ve given a name to a location, it will remember it. Lync also has a great API, so I can get at this information programmatically. The only question that remained was how to store the information.
Each record in the location log simply contains a date and a string to store the location name. Because the data is not relational, I figured I would use a document database like MongoDB or RavenDB. I was able to quickly write code to store the location data in Mongo. But this was overkill. I had to have Mongo running whenever I wanted to log a location or view the location history.
Storing Knowledge in Plain Text
I decided that I wanted to store the data in plain text. In the fantastic book The Pragmatic Programmer, authors Andrew Hunt and David Thomas expound on the virtues of storing long-living data as plain text.
“And we believe that the best format for storing knowledge persistently is plain text.”
Plain text does not replace the need for a database, relational or otherwise. Text is not as performant as a database. It cannot model data in the sophisticated ways that a database can. But it removes the dependency on database, or even the application itself.
“With plain text, however, you can achieve a self-describing data stream that is independent of the application that created it.”
Because your data is not dependent on another application or library, it is less likely to become obsolete. In my career I have seen multiple instances of data being stored in database backups, but at some later date when the data is needed the backup format is incompatible with the latest version of the database. The older version of the database has to be found and installed in order to get the data. If the data is in plain text, I know that I’ll be able to read it many years from now.
The other big advantage of plain text is that it can be opened with a huge variety of tools.
“Virtually every tool in the computing universe, from source code management systems to compiler environments and stand-alone filters, can operate on plain text.”
If the data is stored in plain text, I can view it quickly in notepad if I need to. It will show up in file system search results. I edit it with my favorite text editor.
I’m certainly not saying that all data should be stored in plain text. In most cases, data belongs in a database. But for some data, particularly data that can have a long lifetime. Plain text is the best option.
Adding Structure to Plain Text
Even though I decided that I wanted to store the data in plain text, I still wanted a way to query it. That is when I found Biggy. Biggy is a C# open source project by Rob Conery. It provides an implementation of ICollection that has automatic persistent storage, including an option to store the collection in JSON. This provided the killer combo of using plain text for storage while still having the power of Linq to manipulate the data in code.
Using Biggy is easy. Here is all the code you need to read from and write to the JSON file.
Creating the instance of the BiggyList will create the JSON file if it doesn’t exist. There are optional constructor parameters that allow you to specify the directory and file name. Every time the collection is altered the changes are written to the file. It uses the Newtonsoft Json.NET library, so the JSON is standard.
There are many benefits to using Biggy.
- Because the BiggyList implements IEnumerable, you can query it with the full power of Linq to objects. There is no new API to learn.
- As long as the BiggyList is in scope, the entire list is in memory. This makes it very fast.
- If there is a future need to scale, Biggy also supports MongoDB and PostgreSQL.
The Results
Biggy is great for my location logger. In code it is simple to write data, and it is simple to query it. This would allow me to generate reports. It would be easy to write an app to determine when I was with specific clients. But if I want to quickly glance at the log, I can open it in notepad. The best part is that if I lose my application, and I need to get at the data years from now, it will be just as easy to read it.
