
                             The Objectify Project

                                    History

  This project started out with several ideas I had about developing a new
operating system.  Back in the late '70s a friend and I started a company to
develop a computer system and I started to write an operating system for it.
Then IBM came out with the PC and Apple came out with the Macintosh and we 
couldn't see getting into that fray.

  Twenty years later, in April 2000, while trying to change something on my 
cell phone, I was digging around in it's 100 page manual, and getting really,
really frustrated.  Why is something that should be so simple is so difficult.

  That started a train of thought about how technology and especially computers
are so frustrating.  It seems to me that in many cases the way computers work
now exasperate ?? my weaknesses.

  For example: my memory was never very good and as I've gotten older it has 
gotten worse.  And given that I am usually working on several different 
projects at any given time, very often on different computers with completely
different software.  I have four different computers sitting within 3 feet of
me right now.  A couple more in the next room and of course a laptop.  Many of
them have several different operating systems with different file systems on
them.  At one time I had an NFS file system that most of my machines at home 
were connected to.  That didn't help much with my laptop and inevitbly I would
leave some file on just one machine and then have to search all over for it.

  While this may not be the best example, my point is simply that it very
often seems like computers play on my weaknesses.  I think that, wherever and
whenever possible, they should try to make up for them.  So I decided to 
resurrect the operating system project and see if I could do anything
differently to make it easier for myself.

  I started a project EasyOS (1) on SourceForge to develop an OS based upon the
best parts of Linux, FreeBSD, OpenBSD, etc.  I messed around with some ideas
for a couple of years and finally determined that I needed to stop trying to
base it upon existing operating systems and rethink all of the concepts about 
how an OS works.  I started another project, "New World OS" 
http://sourceforge.net/projects/nwos on SourceForge.

  In December 2004, I wrote an article for OS News about some of my ideas:
http://www.osnews.com/story/9192 and started writing some experimental code.
I debated whether I should really start completely from scratch and develop it
from the talking to the hardware level and work my way up.  Or if I should
start by testing some of my ideas on an existing system and see if they were
feasible at all before dumping years worth of time into them, just to discover
that I had overlooked some fundamental flaw.  I decided it made the most sense
to test out some ideas on top of another OS first.

  At some point I decided that it would be cool to just run my project on top
of another operating system.  That way I could start using it now instead of 
waiting for what could be years and years.  So I started yet another project
"Objectify" http://sourceforge.net/projects/objectify on SourceForge that I 
want to move the current code that runs on top of an existing operating system
into.  However, I have not yet completed the move so the CVS repository and the
released packages are still under the New World OS project.  The only thing I
have in the Objectify project so far are the Feature Requests.


                          The Current State of Affairs

  First I have to say that this project is taking far longer than I ever 
expected.  I never thought it would be quick or easy, but I think my 
expectations were off by an order of magnitude or maybe even two. :(  While on
the one hand my using the system at the same time I'm developing it helps me 
find problems, it has also caused some grief.  If I make a change to the code 
that breaks it completely, everything else grinds to a halt until I can get it
functioning again.

  Second, one of the fundamental ideas I had was to eliminate the concept of
storing data in files.  My theory was that data should be stored on disk in the 
same object format that it used in memory.  That data would only be converted 
to a file format when one needed to exchange information with a different type
of device or system.  So it is ironic that the most developed feature on my
project at the present time is storing files.  And I am sad to say that
presently it does not interpret the data in the files in any way.  The files
are just stored in their raw format.

  So in this article I thought I would just give a brief overview of how the
system can be used to store files and what my plans are for the near future.

  Please keep in mind that this software is really just a prototype at this
point.  Just about anything is subject to change at this time.  While I have 
thousands of my files stored in it, it still needs a lot of work.

One good thing about it being an Open Source project, even if something were to
happen to me, you should always be able to access your data.  (That is given 
that you have your pass phrase or phrases.  If you lose or forget those, well, 
it is unlikely that you will be able to recover your data.)


                                    Storage

  One of the first things I decided upon is that every piece of data would be
stored as an "object" similar to the objects you are probably familiar with in
Object Oriented programming languages like C++, Eiffel, Java, etc.  I mention
Eiffel http://www.eiffel.com because it was my primary influence.

  In Eiffel (and other languages) one object references another object using a
"reference", so I had to figure how an object in my system was going to
reference another object.  This has been one of the things that I have wrestled
with over and over since I started working on this project.  It has changed at
least a couple of dozen times since I started.  When I first started I finally
decided to start with a 4 byte ID (32 bits) just to get going.

  I wrote some preliminary code in late 2004 using the 4 byte ID and got it
working.  Then I decided to scrap that whole thing and start on the second
generation.  I worked on the second generation for several months and it's
reference ID got up to 16 bytes (128 bits).  I finally ran into a chicken and
egg problem with the second generation and decided it would be many more months
before I could have anything working.

  I went back to the first generation and decided since some of it was working
to experiment with it some more.  I have found over the years that it is
easier to have some working code and morph it into what you want, than to try
to get code that doen't work at all working.  There are many things in the 
second generation that I still want to incorporate, but the current code base 
has been derived from the first generation and so it currently has 4 byte IDs.
This will likely be changing to 5 bytes in the near future and eventually it
will likely get back to 16 bytes (2).  I have also toyed with the concept of
having variable length references, but that has some difficulties too.

  In the beginning I was storing each object in a separate file on the host OS.
I cannot remember now exactly the problems with that.  I cannot remember now
all of the things I tried for storage.  I remember I tried storing all of the
objects in a "sparse" file, with the ID of the block being the offset into the
file.  The ratio of the file size to the data stored proved to be horrible, on
order of hundreds to one, if I remember correctly.  I.E. after storing
kilobytes of data the sparse file was megabytes.

  I believe at that point I decided that I would just get a separate hard drive
and have my system use a whole partition or the while drive for storage.  Later
I changed it so that you could create an ordinary (non-sparse) file and use it
for storage.  I should add that Linux is the only system where using a disk
partition or an entire disk drive works.  I have tried it on OpenBSD and it did
not work, I had to use a file for storage.  I think I tried it on FreeBSD as
well and it did not work, but I can't remember for sure.

  The current system uses 256 byte blocks to store each object.  Linux seems to
be happy letting me read and write 256 byte blocks to and from a disk drive or
partition.  I suspect the other OSes weren't happy with non-sector size writes.
The block size will likely have to be increased to 512 or 1024 byte blocks when
the reference IDs change to 16 bytes.


                                  Encryption

  One of the first things I wanted to do was to make sure that if a copy of my
data fell into the wrong hands, that it would be extremely difficult for the 
thief to read the data.

  The encryption is fairly simple at this time.  The first 8 bytes of each 256
byte object is the reference ID and is not encrypted.  The other 248 bytes are
then scrambled and run through the Blowfish encryption from the OpenSSL library
(libcrypto).


         1         2         3         4         5         6         7         8
12345678901234567890123456789012345678901234567890123456789012345678901234567890


List OS's that it works on.

Notes:

(1) In early 2005 someone else wanted to use the EasyOS moniker on SourceForge.
Since I was concentrating on New World OS by that time, I let them take it over.
So the EasyOS project now on SourceForge has no relation to my project.

(2) The reason for having 16 bytes references is because I have the concept of
public and private objects and there would have to be quadrillions of IDs
available for all of the public data.  I was not going to go into the whole
public data concept for this article.

