
PLEASE NOTE: This software is really just a prototype.  Use it at your
own risk.  It is licenced free under the GPL, see the file LICENSE.
If you did not receive the license you can write to:

      Free Software Foundation, Inc.
      59 Temple Place - Suite 330
      Boston, MA 02111-1307, USA.

Or view online at: http://www.gnu.org/copyleft/gpl.html

                              *******************
                              *  PLEASE READ!!! *
  *****************************                 ******************************
  *                                                                          *
  * Versions Alpha_23 through Alpha_26 don't have the capability to expand   *
  * the object storage.  In Alpha_26.6 I fixed it so that it will now        *
  * allocate new chunks as needed.  It will still only allocate them after   *
  * the first chunk so it may not allocate all of the available chunks.  I   *
  * hope to have this fixed soon, but I still haven't gotten to it.  I am    *
  * sorry for the inconvience!!                                              *
  *                                                                          *
  * ALSO PLEASE NOTE: I recommend using the import_file --recursive (-r)     *
  * option in this version with caution, especially if you have have backup  *
  * (.dif) files enabled.  In this version it only creates one backup (.dif) *
  * file per session, even if you are only importing small files, they all   *
  * go into one .dif file and they will add up.  The * larger the .dif file  *
  * the longer it takes to sort.  I recommend if you are going to be         *
  * importing a large amount of files, to do a backup of the whole archive   *
  * (with compress_sparse), disable backup, import the files, do another     *
  * backup, and then re-enable the backups.                                  *
  *                                                                          *
  ****************************************************************************


****************
  Installation
****************

There is no installation at this point in the project.  Although it now uses
GNU autoconf to configure some parameters, some things must still be configured
by hand, by editing the /etc/objectify.conf file.

There really are only three functions that are working in this version
(Alpha_26):

The first function is simply storing information about a person, first 
name, middle name, last name, date of birth, and mobile phone number.

The second function is storage of regular computer files.  They can be
imported, listed, verified and exported.

The third function is disc logging.  It can record the MD5 and SHA1
checksums of the files on a disc and log copies of that disc.
NOTE: it can only deal with files in the root directory of a disc and
CANNOT handle sub-directories at this time.

In this version the person information is not linked to the files in
any way.  Although the plan is to connect them up, they are completely 
separate in this version.


**************************************
 Step 1: Download the public objects
**************************************

The first thing that you must do is download the latest Public Objects.
The code cannot be compiled or installed without the public objects file.
It can be downloaded from the sourceforge.net/projects/nwos web page, under
the "Public Objects" package.  You should just get the latest one.

After you have done that, you need to unzip it:

	gunzip public-080314-0027.gz


***************************************
 Step 2: Configure the storage medium
***************************************

There are two possible ways to store the data for this system.  The first is
to allocate an entire disk or disk partition for storage.  The second is to
allocate a normal file for storage.

As of version 26.6 the default path for the private.obj is:

  ~/.objectify/private.obj

This is the safest thing so that each user has her or his own storage and
there is no risk of two users accessing the storage at the same time.  If
you want to use a different file for storage see the instructions below.

As of version 26.4 the disk and file paths can be configured using the file
/etc/objectify.conf.  There is no need to edit the config.h.in file any more.
Copy the objectify.conf file to /etc:

  sudo cp -pi objectify.conf /etc

To use an entire hard drive you can edit the /etc/objectify.conf file and set
the private variable, for example, as follows:

  private /dev/hdb

To use an entire disk partition you can edit the /etc/objectify.conf file and
set the variable, for example, as follows:

  private /dev/sda8

To use a file you can edit the /etc/objectify file and set the values, for
example, as follows:

  /var/lib/objectify/private.obj

Note: although it says "Sparse_File" it is actually just a regular file.  I
found out that using a sparse file for this was extremely inefficient.  To
store 100 megabytes in a sparse file used gigabytes of disk space.  The
system now does not work with a "sparse" file.

NOTE: I have tested this system on OpenBSD and found that it will *NOT* work
correctly using a partition or an entire disk.  So on OpenBSD the only option 
is to use a file at the present time.  I have successfully used a file on
FreeBSD systems, I have not tried using a disk partition for storage.  And I
have not tried NetBSD or any other system yet.  I suspect most other systems
will not allow a disk drive or a disk partition to be used for storage.
To date the only systems I have successfully used the partition and entire
disk options on, are the various GNU/Linux distributions.

As of version Alpha_22 (0022) the public objects are not stored in the same
storage as the private objects.  Instead the public object file is just
read directly.  So you also need to point to the public objects file in the
/etc/objectify.conf file as well (the following is the default):

  public /usr/local/share/objectify/public.obj

You will need a symbolic link (or just put the public objects file in the 
/etc/objectify.conf file above).  To create a symbolic link to the public 
objects file downloaded in step 1:

  sudo ln -s /wherever/public-080315-0027 /usr/share/objectify/public.obj

Or you could just point it directly to the file:

  public /wherever/public-080315-0027

If you don't want to edit the /etc/objectify.conf file, I have set the
config.h.in to "~/.objectify/private.obj".

In version Alpha_26.2 I added a new feature which creates a backup (.dif) file
each time additions are made to the system.  To activate this feature you have
to edit the /etc/objectify.conf file and define the "backup" variable to the
directory where you want the backup files written.  For example, one possible
way to use it is to have it write the backup files to a USB stick.  Another way
would be to just have it write the backup files to local directory and then
each time you are finished using the system copy the files to a remote machine.

To store the files on a USB stick change the "backup" variable in
/etc/objectify.conf to:

  backup /media/usbdrive/

Please note that if you do this, the USB stick will need to be in place before
inputting any information to the system.

Also note that when using backup files it creates a new separate backup file 
each time the system is started.  The system has to sort the blocks stored in
the backup file into order and so I don't recommend importing many large sized
files at the same time.  I recommend keeping the size of imports to less than
maybe 20 megabytes per import.  When I have imported groups of files larger
than that is sometimes takes several minutes to do the sort.


*****************************
 Step 4: Build the programs
*****************************

Note: on some systems you will need to make sure that the header files and
libraries for openssl are installed.  For example on Ubuntu/Kubuntu Linux
you need to install the libssl-dev package before doing the following.


Then you should be able to just do the usual configure and make:

  ./configure
  make

If you want a debug version you can do the make as follows:

  make type=DEBUG

NOTE: now that ./configure is used the big endian thing should happen
automatically.


***********************************************
 Step 5: Install in /usr/local/bin (OPTIONAL)
***********************************************

You can install the programs in the /usr/local/bin directory with the following
command:

  sudo make install

The following steps assume you have done so.  If you don't install them in
/usr/local/bin you will need to prefix the commands with an absolute path
for example: /usr/src/objectify-XXX/prep_disk.


************************************************************************
 Step 6a: Prepare the storage (when using a disk partition for storage)
************************************************************************

If you have selected (in Step 2) to use a disk partition or an entire disk for 
storage you can prepare the disk as follows:

As of version Alpha_26.6 you can just do this:

  prep_disk


**************************************************************
 Step 6a: Prepare the storage (when using a file for storage)
**************************************************************

If you selected (in Step 2) a file as the storage you will need to specify
the amount of space to reserve for private objects as well:

As of version Alpha_26.6 you can just do this:

  prep_disk 1G

This will create a file of 1024 megabytes for storage.  Note: that 16 megabytes
of whatever you specify is used for the system and is not used for storage of
any data.  So the minimum you can specify is 32M, which allocates 16 megabytes
for the system and 16 megabytes for storage.

********************************************************************
  Step 6b: When using either a disk partition or a file for storage
********************************************************************

When it has formatted the storage it will ask for a pass phrase.  I currently
don't try to hide the pass phrase, so it will print out to the screen.  This
is because I was initially using only very long pass phrases that proved to
be difficult to enter in reliably when you couldn't see what you were typing.

Since then I have modified it so you can use shorter pass phrases, but I have
not changed the program to mask them yet.

On the topic of pass phrases.  I believe a longer one (34 characters or more) 
will be slightly more secure than a shorter one (minimum is 10 characters).
Although I cannot say for sure.  I should also mention here that I cannot
guarantee that your data is stored secure.  It is encrypted and scrambled so
I believe that it is secure, but again, use this software at your own risk.

NOTE: if your storage is large it can take a significant amount of time to
prepare it.  For example, a 200 gigabyte drive can take almost 2 hours to
complete.


************
 Upgrading 
************

I should have done a better job of keeping this section up to date.  These
are the last few that I can remember.  If you have an archive older than
these, please e-mail me at j.scott.edwards.nwos@gmail.com and I will try
to help you upgrade.

Upgrade 0021 to 0022:

	compress_sparse compressed-070610-0021
	upgrade to version 22
	expand_sparse compressed-070610-0022

Upgrade 0022 to 0023:

	compress_sparse compressed-070620-0022
	upgrade to version 23
	convert_0022_compressed_to_0023 compressed-070620-0022 compressed-070620-0023
	expand_sparse compressed-070620-0023

Upgrade 0023, 0024, 0025, 0026 or 0027 to 0028:

	compress_sparse compressed-080604-0027
	upgrade software to version 28
	expand_sparse compressed-080604-0027
        update_files_0028      <-- If you have multiple pass phrases this must be run
                                   for each one.

    NOTE: there is no difference in the headers or disk layout  between 
    0024 and 0025, the version was changed to insure that the public 
    objects and the code matched up correctly.  In fact the compressed
    versions of 0023, 0024, 0025, 0026, 0027 and 0028 are all compatible.


***********************************************************
 Using the "computer" program to store person information 
***********************************************************

After installation you can run the "computer" program to create a person 
and add a mobile phone for a person.  It can also describe a person or state 
or return a person's phone number.  

The "computer" command can be run in two ways.  It can execute one command
from the command line as follows:

    > computer what is george\'s phone number

Note: it has to have a back slash before the apostrophe or the shell will
want a matching closing quote.

Alternatively, "computer" can be run interactively by just typing:

    > computer

It will then prompt you for commands.  The commands "computer" accepts in 
this version are:

   "add person" - adds a new person to the database.

   "add city" - adds a new city to the database 
                (not of much use in this version).

   "describe <name>" - describes any person(s) with the name <name> in 
                       the database.

   "list cities in <state>" - lists the cities in the database for state 
                              <state>.

   "<name>'s mobile phone is <number>" - adds the phone number <number> 
                                         to the person <name>.

   "what is <name>'s number" - prints out the phone number for the person 
                               <name>.


***************************************
 Storing computer files in the system
***************************************

You can store files encrypted in the system as follows:

  import_file *.jpg

This will store all of the jpg files in the current directory to the system.

As of version Alpha_28 you can import files recursively using the -r or
--recursive option.  This command will import all of the files in the xyzzy
subdirectory:

  import_file -r xyzzy

Please note: it will not accept absolute paths, for example "/home/user" or
paths relative to the parent directory, for example "../whatever".

PLEASE ALSO READ THE WARNING ABOUT IMPORTING RECURSIVELY WITH BACKUP ENABLED AT
THE BEGINNING OF THIS FILE!!

You can adjust the security vs. speed using the following options:

    --minimal-security

         This will give maximum speed for storing and reading the file,
         but because it is stored in almost sequential blocks on the
         disk it may be easier for a cracker to decode.

    --very-low-security

         This will give slightly reduced speed, but spread the file out
         further on the disk to make it a little harder to find the
         pieces.

    --low-security (default)

         This spreads the file out even more on the disk, but is a somewhat
         slower.

    --medium-security

         Spreads the file out even more in storage, quite a bit slower.

    --extreme-security

         Stores each block of the file in random places in storage.  This
         provides the most difficutly in finding the pieces of the file,
         but is extremely slow.  I would not recommend using this for large
         files, it could take hours to read it from the disk. 

To retrieve the files from the system you can use the following command:

   export_file <file names>

You can export all the file stored as follows:

   export_file "*"   

You can verify files using the following command:

   verify_file <file names>

You can verify and then delete the files with the remove option:

   verify_file --remove <file names>

NOTE: file globbing doesn't work correctly in most of these programs yet.

Globbing should work in the list_files program:

   list_files "*.jpg"

You can also get list_files to print the MD5 or SHA1 sums (but not at
the same time):

   list_files --md5 my_file.txt



***********************************
 Keeping a record of backup disks
***********************************

There are some extra programs to log discs burned.  This version cannot deal
with subdirectories and will likely crash and burn in some horrible way.

The way I use this feature is as follows:

  I make a back CD or DVD in the normal way, with all of the tarballs for
  example in the directory /tmp/backup.  I first create a disc list in the
  system as follows:

     cd /tmp/backup
     disc_list *

  At this point it will ask for a label, which must be 12 characters.  I use
  the date and three digits sequential number for the label such as:

     20070413-001

  It will then record the file name, file size, and MD5, SHA1, SHA256, and
  SHA512 sums of all the files.  My logic is that even if all of those are
  compromised in the future having all of them is still pretty safe because I
  can't imagine that it would be possible for one cracked file to still match
  all 4 of them at the same time.

  Then after I burn a disk I mount it and do the following:

     cd /mount/cdrom0
     log_disc *

  It will read all the files from the disc and if it matches the disc list
  created above, it will ask which copy of the disk it is, and then where
  it is stored.

  You can see what discs you have stored with the following command:

     list_discs

  And if you want to see what files are on the discs:

     list_discs --contents

  NOTES:

    * As of release Alpha_28.4 it can deal with subdirectories on the disc.
      In prior versions all of the files had to be in the root directory of the
      disc.

    * Currently, the system cannot store information about empty directories or
      files.  In version Alpha_29, two options (--ignore-empty-directories
      and --ignore-empty-files) were added to allow the disc_list and log_disc
      to allow a disc with empty files and/or empty directories to be logged.


*********************
 Backup and Restore
*********************

There are a couple of programs to allow an Objectify archive to be compressed
and later restored.

To save the data use the following:

   compress_sparse <compressed file name>

NOTE: 'sparse' is a holdover from when I was using sparse files for storage.
The compress_sparse program is used to save the objects stored on any of the 
types of storage.

To restore storage from a compressed file, use the following commands for a
partition or entire disk:

   prep_disk --no-private
   expand_sparse <compressed file name>

Or for a file: 

   prep_disk --no-private 2G
   expand_sparse <compressed file name>

Also note that restoring the file with 'expand_sparse' is very slow to a large
disc drive (can take hours).

Finally, there is a program 'diff_compressed' that compares two compressed 
files (the first file must be an older version of the second file):

   diff_compressed compressed-070401 compressed-070408

This will output to stdout all of the blocks that changed or were added.  It
cannot handle deleted blocks!

This command will output all of the changed and added blocks to a file:

   diff_compressed compressed-070401 compressed-070408 diff-070401-070408

And then you can use that diff file to "patch" a copy of the earlier file to
create the newer file as follows:

   patch_compressed compressed-070401 diff-070401-070408 compressed-070408


In version Alpha_26.2 I added the capability to create backup files as
information is added to the system (see: Step 2: Configure the storage medium).

The backup files are exactly the same as a diff file created by the
"diff_compressed" program, so it can be used with "patch_compressed" the same 
way:

   patch_compressed compressed-071130 objectify-20071202100443.dif compressed-071202
   patch_compressed compressed-071202 objectify-20071203083127.dif compressed-071203

Since it could take a significant amount of time to apply each backup file 
separately, I have added a program to combine the patch files into one big
dif file:

   combine_backup_files *.dif combined-071203.dif

Which can then be used to apply all of the backup files at the same time:

   patch_compressed compressed-071130 combined-071203.dif compressed-071204



*****************************
 Thanks for trying it out!!
*****************************

I have added a couple of documents on how I use the system in the "docs"
directory.

Please send me your suggestions, comments, etc. at j.scott.edwards.nwos@gmail.com

