                             hsh Developer Manual

Copyright (C) 2010 Alexander Taler (dissent@0--0.org)

This file is part of hsh.

hsh is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

hsh is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with hsh.  If not, see 
http://www.gnu.org/licenses/

Contents
--------

  Overview
  Guidelines
  Internals
  Source Directory Structure
  Near Term Tasks
  Justifications

Overview
--------

hsh is implemented in Python 2.5.  I chose 2.5 because that's what my Debian
system had installed, and it seemed a reasonable choice for accessibility.
Also for accessibility reasons it doesn't depend on any non-stock Python
libraries.

The code has been divided into curses-specific parts and generic parts, to
allow for the future possibility of a non-curses interface.

The main objects of hsh internals are illustrated in this diagram:

"Internals.png"

The curses-specific objects are shown above the dotted-line, and the generic
ones below.  Curses Display and Job Manager are the two central singletons.
Communication from the generic objects to the curses objects is achieved using
listener interfaces, to preserve the curses-independence of the layer.

  Job Manager
  -----------

This singleton instantiates itself on load.  It maintains a list of all
existing jobs, and takes responsibility for searching the disk for jobs, as
well as assigning ids.  A JobManagerListener interface is defined, allowing
objects to be notified of any events around creation and destruction of jobs.
Additionally, some specific job management functions are implemented.

  Command Text
  ------------

This object represents the command text entered by the user, parsed, and
expanded to a program and its parameters.  The program can be an external
program or a built-in routine.  The parse tree is described in further detail
below, and is fairly complex due to supporting dynamic updates.

  Job
  ---

Job is the basic object of hsh, consisting of the command text, an execution
context and optionally the results of execution.

The execution context is the current working directory, the user, the host, the
environment, and the associated terminal.  In the future there will be
mechanisms for changing them all, although currently only working directory
changes are supported.

A job can be executed, causing the specified program to be run, in which case
additional information may be available, including process id, process state,
and process output.

There are two subclasses, ExternalJob is for an external program being run in a
different process, and BuiltInJob for a Python function being run in the same
process, although possibly in a separate thread.  ExternalJobs have extra
threads for reading the output of the other process.

  Curses Display
  --------------

This singleton contains the main run method and the top-level input loop.  It
reads the config and instantiates the other curses objects.  It also initiates
the curses library and maintains references to its datastructures.

The curses views refer to the display to locate other views, particularly
whichever one is visible.

  Curses Job Display
  ------------------

This class is used for displaying a job's output and other info in a curses
window.  It also has implementation of any user functions specific to
displaying a job.

Its instances exist in a one to one relationship with Jobs, and created as
needed when the job needs to be displayed.  They are stored in a registry
within the Curses Display singleton - although perhaps this would be better
done in the Job Display class itself.

  Curses List Display
  -------------------

This class is used for displaying information about a list of jobs.
Additionally it has implementation of any user functions specific to a list of
jobs.

The class is configurable, and has three instances for the Session List,
Command History and Running Jobs List.  The configuration for these instances
is in config.py, and they are created by the CursesDisplay during its
instantiation.

  Curses Input
  ------------

This class is for accepting user input and displaying status of the parsed
command.  It is the one class which uses the CommandText, and contains
specialized code for displaying command related information.

Guidelines
----------

  Documentation
  -------------

User documentation and high level internal documentation is maintained in
html/text format.  Detailed internal documentation is maintained in the Python
source code itself.  PyDoc is likely to be used in the future.

  Testing
  -------

Automated testing scripts are maintained for those areas which can be easily
tested.  Coverage needs to be increased in this area.

Internals
---------

  Curses
  ------

The CursesDisplay singleton is the main point of contact for the curses
library.  It maintains the reference to the main curses screen, allocates
curses windows to the displayable objects, and handles curses input and resize
events.  CursesDisplayable is a superclass for anything which is displayable
within curses.  It maintains a reference to an assigned curses window and
includes some shared drawing code.

Keystrokes in curses are funny things, and not everything you'd like to
recognize is possible.  Ctrl-Tab for instance is not available.  The
curses_display/keybinds.py file holds the default keybindings and sets the
template for how to define future keys.

  Jobs
  ----

The current Job interface is simple and has plenty of potential for performance
difficulties.  Information about each job is maintained on disk within the
~/.hsh directory.  More advanced caching facilities may be needed in future.
Process management is also very simple, and no attempt is made to locate
processes belonging to other hsh instances.

  Command
  -------

The user's command-line input is parsed on the fly in order to provide constant
feedback to the user.  The parsed command-line is maintained as a tree of
tokens, with a CommandText object at the top of the tree, containing a list of
Prog, Arg, and ArgSeparator objects, each of which contains the other tokens.
The class inheritance hierarchy is quite different, with a generic Token class
at the top, with an AggregateToken and LeafToken below it, and several more
specialized Tokens below those.

[ Need some hierarchy diagrams here. ]

  Content
  -------

Content is used to represent text to be displayed, in a non-curses-specific
way.  It is essentially an array of '\n' terminated strings.  Additionally each
line is split into separate regions to which can be attached arbitrary
information.  Jobs attach information about the origin of the text to each
region, and the curses implementation attaches display face information to
regions.

  Config
  ------

The beginnings of a user accessible configuration system exist, in the
config.py file.  It contains configuration of face colours, display headers,
and job list configurations, in addition to some more minor options.

Source Directory Structure
--------------------------

  doc/
    Documentation

  src/
    Source code

      hsh/
    Source code.  Subdirectory used to make Python imports clearer.

          curses_display/
    Code for the hsh curses display.

          test/
    Automated test code.



Near Term Tasks
---------------

  Add a warning if an external command appears multiple times on the path.

  Implement ** syntax for globbing.

  Make entering exclusive input mode for jobs an explicit operation, rather
  than happening just when the job receives focus.

  Feature to recalculate all jobids to make them smaller.

  Implement a specialized JobOutput class:

class JobOutput(object):

    # Place holder for a future class representing the output of a job.  
    # Class for storing output of a job, can be backed by disk or populated
    # from a running process.  Will store output to disk.

    # Provides an array interface for ease of use by displayable objects.

    # Each line of output is annotated with a timestamp, fd, line num, and line
    # num from that fd.
    pass

  Bind | within job listing views to edit command line piping output from job.

  Do we want to use ! as a "fix current job" option, allowing editing of the
  command line and reexecution, but preserving job id.

  Job sort order cannot be changed.

  Do we want to support boolean operators or ; for multiple processes?

  Friendly messages for "full-screen" processes.  One to say "this is
  full-screen, switch focus (M-o) to use it", another to say "this full-screen
  app has terminted, hit Return to return to hsh" (Perhaps try curses.ungetch()
  to push back the input.)

  Nicer indicators for top and bottom of output, sides and line-folding
  ^~ for empty lines like vi  (inverse ^D?)  Different faces a possibility, or
  extra columns like in Emacs.

  More output modes: Hex Output, character sets?  Line numbers, timestamps.

  Highlight warnings in the command-line (broken env variables, dodgy control
  characters, etc.)

  Split main views into two curses windows - one for header and one for main
  bit so they can be updated separately and hopefully reduce flicker.

  Make use of cd.win thread-safe, since it is regularly updated from output
  threads.

  Numeric arguments to keystrokes.

  Movement by word

  Built-in editor accessible via $EDITOR.  Alternatively, allow command line
  bitties to be expanded to a multiline edit mode.

  Replace job function - edits the jobs command line and reruns it.

  Less cumbersome mechanism for accesing sh, than calling sh -c. 

  Memory management - automatically forget old jobs

  Nicer way than kill -STOP to suspend running jobs. 

  Define "programs/themes/tasks"(?) which can be switched and have restricted
  memories for contexts, command and job history.  Jobs are "tagged" with a
  theme.

  

  


Justifications
--------------

  Why reimplement shell parsing and command line stuff?

hsh is not built atop existing shells because of the difficulty of managing
output of background jobs.

  Why uses curses when it's so ugly and clunky?

I think there is room for both a curses front-end and a windowey front-end.  I
chose curses first because the typical shell user is comfortable in a terminal.
Furthermore, I expect the effort of supporting other curses apps (notably vi)
within the terminal will be less than in a windowey front-end.

