                             hsh Developer Manual

Copyright (C) 2010 Alexander Taler (dissent@0--0.org)

This file is part of hsh.

hsh is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

hsh is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with hsh.  If not, see 
http://www.gnu.org/licenses/

Contents
--------

  Overview
      Job Manager
      Command Text
      Job
      Curses Display
      Curses Job Display
      Curses List Display
      Curses Input
  Guidelines
  Internals
      Curses
      Jobs
      Command
      Content
      Config
  Build
      Source Directory Structure
      Packaging
  Development Plans
      Handling Terminal Using Programs
      Improving Display Appearance
      Upgrades and Supporting Multiple Versions
      Command Line Enhancements
      Workspaces
      Miscellaneous
  Justifications
  Reference

Overview
--------

hsh is implemented in Python 2.5.  I chose 2.5 because that's what my Debian
system had installed, and it seemed a reasonable choice for accessibility.
Also for accessibility reasons it doesn't depend on any non-stock Python
libraries.

The code has been divided into curses-specific parts and generic parts, to
allow for the future possibility of a non-curses interface.

The main objects of hsh internals are illustrated in this diagram:

"Internals.png"

The curses-specific objects are shown above the dotted-line, and the generic
ones below.  Curses Display and Job Manager are the two central singletons.
Communication from the generic objects to the curses objects is achieved using
listener interfaces, to preserve the curses-independence of the layer.

  Job Manager
  -----------

This singleton instantiates itself on load.  It maintains a list of all
existing jobs, and takes responsibility for searching the disk for jobs, as
well as assigning ids.  A JobManagerListener interface is defined, allowing
objects to be notified of any events around creation and destruction of jobs.
Additionally, some specific job management functions are implemented.

  Command Text
  ------------

This object represents the command text entered by the user, parsed, and
expanded to a program and its parameters.  The program can be an external
program or a built-in routine.  The parse tree is described in further detail
below, and is fairly complex due to supporting dynamic updates.

  Job
  ---

Job is the basic object of hsh, consisting of the command text, an execution
context and optionally the results of execution.

The execution context is the current working directory, the user, the host, the
environment, and the associated terminal.  In the future there will be
mechanisms for changing them all, although currently only working directory
changes are supported.

A job can be executed, causing the specified program to be run, in which case
additional information may be available, including process id, process state,
and process output.

There are two subclasses, ExternalJob is for an external program being run in a
different process, and BuiltInJob for a Python function being run in the same
process, although possibly in a separate thread.  ExternalJobs have extra
threads for reading the output of the other process.

  Curses Display
  --------------

This singleton is the central point for the curses display and input code.  It
is responsible for reading configuration and instantiating the other curses
objects, including the iniatialization of the curses library itself.  It
contains the main loop for the input thread, as well as launching a separate
display thread.

The curses views refer to the display to locate other views, particularly
whichever one is visible.

  Curses Job Display
  ------------------

This class is used for displaying a job's output and other info in a curses
window.  It also has implementation of any user functions specific to
displaying a job.

Its instances exist in a one to one relationship with Jobs, and created as
needed when the job needs to be displayed.  They are stored in a registry
within the Curses Display singleton - although perhaps this would be better
done in the Job Display class itself.

  Curses List Display
  -------------------

This class is used for displaying information about a list of jobs.
Additionally it has implementation of any user functions specific to a list of
jobs.

The class is configurable, and has three instances for the Session List,
Command History and Running Jobs List.  The configuration for these instances
is in config.py, and they are created by the CursesDisplay during its
instantiation.

  Curses Input
  ------------

This class is for accepting user input and displaying status of the parsed
command.  It is the one class which uses the CommandText, and contains
specialized code for displaying command related information.

Guidelines
----------

  Documentation
  -------------

User documentation and high level internal documentation is maintained in
html/text format.  Detailed internal documentation is maintained in the Python
source code itself.  PyDoc is likely to be used in the future.

  Testing
  -------

Automated testing scripts are maintained for those areas which can be easily
tested.  Coverage needs to be increased in this area.

Internals
---------

  Curses
  ------

The CursesDisplay singleton is the main point of interaction with the curses
library, as well as a central point for the curses View objects.  The
CursesDisplayThread is a separate thread responsible for performing the actual
painting of the terminal.

View is a superclass for anything which can be drawn in the curses window and
receive input.  They include a draw() method which is called by the display
thread, as well as set_dirty() for other threads to mark them for redraw.

Keystrokes in curses are funny things, and not everything you'd like to
recognize is possible.  Ctrl-Tab for instance is not available.  The
curses_display/keybinds.py file holds the default keybindings and sets the
template for how to define future keys.

  Jobs
  ----

The current Job interface is simple and has plenty of potential for performance
difficulties.  Information about each job is maintained on disk within the
~/.hsh directory.  More advanced caching facilities may be needed in future.
Process management is also very simple, and no attempt is made to locate
processes belonging to other hsh instances.

  Command
  -------

The user's command-line input is parsed on the fly in order to provide constant
feedback to the user.  The parsed command-line is maintained as a tree of
tokens, with a CommandText object at the top of the tree, containing a list of
Prog, Arg, and ArgSeparator objects, each of which contains the other tokens.
The class inheritance hierarchy is quite different, with a generic Token class
at the top, with an AggregateToken and LeafToken below it, and several more
specialized Tokens below those.

[ Need some hierarchy diagrams here. ]

  Content
  -------

Content is used to represent text to be displayed, in a non-curses-specific
way.  It is essentially an array of '\n' terminated strings.  Additionally each
line is split into separate regions to which can be attached arbitrary
information.  Jobs attach information about the origin of the text to each
region, and the curses implementation attaches display face information to
regions.

  Config
  ------

The beginnings of a user accessible configuration system exist, in the
config.py file.  It contains configuration of face colours, display headers,
and job list configurations, in addition to some more minor options.

  Build
  -----

  Source Directory Structure
  --------------------------

  doc/
    Documentation

  src/
    Source code

      hsh/
    Source code.  Subdirectory used to make Python imports clearer.

          curses_display/
    Code for the hsh curses display.

          test/
    Automated test code.


  Packaging
  ---------

  bin/make_pkg.sh

  Sign the package.  gpg --detach-sign hsh-*.tar.gz

  Upload to savannah download area.  sftp dissent@dl.sv.nongnu.org Verify that
  it's arrived at the master site:

  Post news on Savannah, Freshmeat.

  Increase version number in bin/make_pkg.sh


Development Plans
-------------------

  Handling Terminal Using Programs
  --------------------------------

  Make entering full-screen mode for jobs an explicit operation, rather
  than happening just when the job receives focus.

  Friendly messages for "full-screen" processes.  One to say "this is
  full-screen, switch focus (M-o) to use it", another to say "this full-screen
  app has terminted, hit Return to return to hsh" (Perhaps try curses.ungetch()
  to push back the input.)

  Readline type editing for line-oriented job input.

  Status of #exclusive jobs is not returned correctly.  Could this be an issue
  with Python's poll() method?

  Run vi in #piped mode, in a terminal larger than 25 rows, then suspend it
  with ^Z Enter.  Hsh is left in a weird state where the screen is drawn offset
  by one row.  Calling curses.nonl() resolves the drawing issue, but disables
  the enter key as well.  Resizing the terminal fixes the problem.

  Run vi in #piped mode, if the first keystroke is an arrow key it complains
  about input of characters greater than 255.  The issue is logged and ignored
  in the code.

  Change the controlling terminal for processes in a pseudo terminal.

  Hsh should check for when a full-screen job has stopped, and take control of
  the terminal back, rather than waiting for the Ctrl-Z Enter keystrokes.

  Improving Display Appearance
  ----------------------------

  Create a session log (or adapt session list) to give a similar experience to
  the traditional shell, where the output of multiple jobs can be seen.

  Include visible cues to differentiate the end of output from blank lines. a
  la vi.

  Add a little spinny thing to the header to identify running jobs more easily.

  Add way to toggle job headers between long and short mode.

  More output modes: Hex Output, character sets?  Line numbers, timestamps.

  Tab spacing isn't working as some process may expect.  Try 'ls #pty'.

  Make alert messages disappear after a while.

  Alert message when a job in the background finishes.

  Upgrades and Supporting Multiple Versions
  --------------------------------


  Command Line Enhancements
  --------------------------------

  Add #secret directive so that no output or input will be logged.

  Implement ** syntax for globbing.

  Bind | within job listing views to edit command line piping output from job.

  Do we want to support boolean operators or ; for multiple processes?

  Highlight warnings in the command-line (broken env variables, dodgy control
  characters, etc.)

  Add #detach directive, which closes fds and detaches the process from the
  shell's process group.  Used for daemonization - anything else?

  Workspaces
  ----------

A longer term development goal is to add the concept of a workspace, allowing
the user to segregate their different work areas.  Within a workspace the
various histories will be restricted to showing only commands, jobs and
contexts which belong to that workspace.

Some ideas for implementing workspaces is to have tags on jobs, and only show
ones matching active workspaces.  It seems intuitive to restrict to having a
single active workspace, but perhaps permitting multiple could have some
benefit.  There will need to be a default or no-workspace mode which shows
everything.

Users could use workspaces to segregate different projects they work on, or
different tasks such as working on graphics and systems administration.

  Miscellaneous
  -------------

  Implement options/config like aliases.

  A help line in the headers of each view.

  A glob symbol in the prog raises an exception, but should just generate
  a warning.

  Warning when trying to restart a running job.

  Colours for cucumber?

  Feature to recalculate all jobids to make them smaller.

  Implement a specialized JobOutput class:

class JobOutput(object):

    # Place holder for a future class representing the output of a job.  
    # Class for storing output of a job, can be backed by disk or populated
    # from a running process.  Will store output to disk.

    # Provides an array interface for ease of use by displayable objects.

    # Each line of output is annotated with a timestamp, fd, line num, and line
    # num from that fd.
    pass

  Do we want to use ! as a "fix current job" option, allowing editing of the
  command line and reexecution, but preserving job id.  Binding Alt-E for this
  operation is good too.

  Job sort order cannot be changed.

  Numeric arguments to keystrokes.

  Movement by word

  Built-in editor accessible via $EDITOR.  Alternatively, allow command line
  bitties to be expanded to a multiline edit mode.

  Less cumbersome mechanism for accesing sh, than calling sh -c. 

  Memory management - automatically forget old jobs

  Nicer way than kill -STOP to suspend running jobs. 

  sftp works, but the prompt doesn't appear until after the command and newline
  is entered.

  Exception trace dump corrupting console when running process terminates.

  Prevent deletion of jobs belonging to other instances.  Probably implement
  session stuff.

  When showing directory name, abbreviate ~s as possible.

  Output for jobs from other instances is not read from disk.

  Command history shouldn't pick stuff up from other instances.

  running it over with stderr.

  Use the distutils.  There are a few things to overcome though:
      Automate running of tests
      Automate copying of version into code.
      Processing and inclusion of docs.
      Other stuff from make_pkg.sh and Makefile
  Here's an initial pass at the setup.py:

import distutils.core

distutils.core.setup(
    name='hsh',
    version='1.-2.1',
    description='The {Happy|Human|History} Shell',
    author='Alexander Taler',
    author_email='dissent@0--0.org',
    url='http://www.nongnu.org/hsh/',
    package_dir={'':'src'},
    packages=['hsh', 'hsh/curses_display'],
    package_data={'hsh':['../../doc/*.html','../../doc/*.txt','../../doc/*.png']},
    scripts=['bin/hsh'],
)

Justifications
--------------

  Why reimplement shell parsing and command line stuff?

hsh is not built atop existing shells because of the difficulty of managing
output of background jobs.

  Why use curses when it's so ugly and clunky?

I think there is room for both a curses front-end and a windowey front-end.  I
chose curses first because the typical shell user is comfortable in a terminal.
Furthermore, I expect the effort of supporting other curses apps (notably vi)
within the terminal will be less than in a windowey front-end.

Reference
---------

The TTY demystified
   http://www.linusakesson.net/programming/tty/index.php

Also refer to the libc documentation about job control.


