                             hsh Developer Manual

Copyright (C) 2010 Alexander Taler (dissent@0--0.org)

This file is part of hsh.

hsh is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

hsh is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with hsh.  If not, see 
http://www.gnu.org/licenses/

Contents
--------

  System Overview
      Job Manager
      Command Text
      Job
      Curses Display
      Job View
      List View
      Input View
  Guidelines
      Dev Environment
      Documentation
      Testing
  Internals
      Curses
      Jobs
      Command
      Content
      Config
  Build
      Source Directory Structure
      Packaging
  Development Plans
      Handling Terminal Using Programs
      Improving Display Appearance
      Upgrades and Supporting Multiple Versions
      Command Line Enhancements
      Workspaces
      Miscellaneous
  Justifications
  Reference

System Overview
---------------

hsh is implemented in Python 2.5.  I chose 2.5 because that's what my Debian
system had installed, and it seemed a reasonable choice for accessibility.
Also for accessibility reasons it doesn't depend on any non-stock Python
libraries.

The code has been divided into curses-specific parts and generic parts, to
allow for the future possibility of a non-curses interface.

The main objects of hsh internals are illustrated in this diagram:

"Internals.png"

The curses-specific objects are shown above the dotted-line, and the generic
ones below.  Curses Display and Job Manager are the two central singletons.
Communication from the generic objects to the curses objects is achieved using
listener interfaces, to preserve the curses-independence of the layer.

  Job Manager
  -----------

This singleton instantiates itself on load.  It maintains a list of all
existing jobs, and takes responsibility for searching the disk for jobs, as
well as assigning ids.  A JobManagerListener interface is defined, allowing
objects to be notified of any events around creation and destruction of jobs.
Additionally, some specific job management functions are implemented.

  Command Text
  ------------

This object represents the command text entered by the user, parsed, and
expanded to a program and its parameters.  The program can be an external
program, a built-in routine or an alias.  The parse tree is described in
further detail below, and is fairly complex due to supporting dynamic updates.

  Job
  ---

Job is the basic object of hsh, consisting of the command text, an execution
context and optionally the results of execution.

The execution context is the current working directory, the user, the host, the
environment, and the associated terminal.  The future intention is to record
all of these, and make them modifiable, although currently only the environment
and working directory are modifiable, and only working directory is recorded.

A job can be executed, causing the specified program to be run, in which case
additional information may be available, including process id, process state,
and process output.

There are two subclasses, ExternalJob is for an external program being run in a
different process, and BuiltInJob for a Python function being run in the same
process, although possibly in a separate thread.  ExternalJobs have extra
threads for reading the output of the other process.  (Aliases are not jobs,
they are managed in the CommandText layer.)

  Curses Display
  --------------

This singleton is the central point for the curses display and input code.  It
is responsible for reading configuration and instantiating the other curses
objects, including the initialization of the curses library itself.  It
contains the main loop for the input thread, as well as launching a separate
display thread.

It maintains a dict of other curses views, and tracks which view is currently
visible in the main area.  The input view is always visible.

  Job View
  ------------------

This class is used for displaying a job's output and other info in a curses
window.  It also has implementation of any user functions specific to
displaying a job.

Its instances exist in a one to one relationship with Jobs, and are created as
needed when the job needs to be displayed.  They are stored in a registry
within the Curses Display singleton, although storing them in the Job View
class itself would be more flexible if multiple displays ever needed to be
supported.

  List View
  -------------------

This class is used for displaying information about a list of jobs.
Additionally it has implementation of any user functions specific to a list of
jobs.

The class is configurable, and has multiple instances, including the Session
List, Command History and Running Jobs List.  The configuration for these
instances is in config.py, and they are created by the CursesDisplay during its
instantiation.

  Input View
  ------------

This class is for accepting user input and displaying status of the parsed
command.  It is the one class which uses the CommandText, and contains
specialized code for displaying command related information.

Guidelines
----------

  Dev Environment
  ---------------

  I suggest having the latest stable hsh and your local development version
  both available on the path, I call them 'hsh' and 'hshx' (for experimental).
  This means you'll always be able to use hsh (I am becoming dependant on it)
  even when your development one is broken.  It also has the nice benefit of
  running older and newer versions alongside each other in order to catch
  potential upgrade problems.

  Documentation
  -------------

User documentation and high level internal documentation is maintained in
html/text format.  Detailed internal documentation is maintained in the Python
source code itself, and presented as generated html through Pydoc.

  Testing
  -------

Automated testing scripts are maintained for those areas which can be easily
tested.  Coverage needs to be increased in this area.  The test scripts are in
the src/hsh/test directory, and can be executed using src/Makefile.

Internals
---------

  Curses
  ------

The CursesDisplay singleton is the main point of interaction with the curses
library, as well as a central point for the curses View objects.  The
CursesDisplayThread is a separate thread responsible for performing the actual
painting of the terminal.

View is a superclass for anything which can be drawn in the curses window and
receive input.  They include a draw() method which is called by the display
thread, as well as set_dirty() for other threads to mark them for redraw.

Keystrokes in curses are funny things, and not everything you'd like to
recognize is possible.  Ctrl-Tab for instance is not available.  The
curses_display/keybinds.py file holds the default keybindings and sets the
template for how to define future keys.

  Jobs
  ----

The current Job interface is simple and has plenty of potential for performance
difficulties.  Information about each job is maintained on disk within the
~/.hsh directory.  More advanced caching facilities may be needed in future.
Process management is also very simple, and no attempt is made to locate
processes belonging to other hsh instances.

  Command
  -------

The user's command-line input is parsed on the fly in order to provide constant
feedback to the user.  The parsed command-line is maintained as a tree of
tokens, with a CommandText object at the top of the tree, containing a list of
Prog, Arg, and ArgSeparator objects, each of which contains the other tokens.
The class inheritance hierarchy is quite different, with a generic Token class
at the top, with an AggregateToken and LeafToken below it, and several more
specialized Tokens below those.

"CommandObjects.png"

  Content
  -------

Content is the module for content related code, restricted to text content for
the moment.  The Text object presents a basic interface as an array of '\n'
terminated strings.  Subclasses provide additional functionality, importantly
MutableText whose elements are mutable arrays of characters, with regions that
can be tagged with information such as face, origin and line number details.

The relationship between the content and curses modules is:

"ContentCurses.png"

  Text:
Basic interface to contain a list of \n terminated strings.

  TextFormatter:
For formatting Text for fixed-width display.

  TextViewport:
Extends TextFormatter adding tracking of a display position and height.

  ContentView:
A superclass for the various views which display basic text based content.  It
utilizes the TextViewport and Text classes directly.

  JobOutput &amp; JobView:
Specializations of Text and ContentView for handling of job output.  The
JobOutput class doesn't actually exist yet, but is a necessity for better
handling of job output.

  CommandText &amp; InputView:
Specializations of Text and ContentView for handling of the input.  CommandText
doesn't actually exist yet, MutableText is used instead, it would just be an
interface layer between the Command and Text objects.

  TextList &amp; MessageView:
TextList is a simple list of strings, appropriate for MessageView which
displays a mostly static output.

  Config
  ------

The beginnings of a user accessible configuration system exist, in the
config.py file.  It contains configuration of face colours, display headers,
and job list configurations, in addition to some more minor options.

  Build
  -----

  Source Directory Structure
  --------------------------

  doc/
    Documentation

  src/
    Source code

      hsh/
    Source code.  Subdirectory used to make Python imports clearer.

          curses_display/
    Code for the hsh curses display.

          test/
    Automated test code.


  Packaging
  ---------

  make pkg

  Follow emitted instructions (sign package, upload to Savannah, tag).

  Verify that package has arrived at the master site:

  Post news on Savannah, Freshmeat.

  Send an email to hsh-announce@nongnu.org

  Increase version number in bin/make_pkg.sh


Development Plans
-------------------

  Next Features
  -------------

  Output Search

  Overview:
    M-s starts incremental search in any view.  InputView hands this
    off to the main view.

    Additionally, this will be available with / in List View and Job View while
    in silent input mode.

    Incremental search mode shows a new one line window between the main
    view and the input view.

    Regular editing keys affect the incremental search window.

    All strings matching the search pattern are highlighted.  Display can be
    moved so that a string match is visible using M-s and M-S.

    Other keys do:
      M-C Close the incremental search window
      M-o Change focus
      C-c clear the search string
      M-r toggle regexp and plain text search
      UpArrow goes to previous search pattern
      DownArrow goes to next search pattern

      M-s search forward
      M-S search backward

    When the search view is opened it has the pattern last used in that main
    view or last used at all?  -- config option

  Implementation Details:

    1] Search highlighting is done as part of the draw() methods.
    2] Search movement is done using a new move_search() method of View objects.

  Multiline search patterns are tricky because regexp library requires a string
  interface.  Hopefully this can be effectively faked.

  Movement by word.

  Handling Terminal Using Programs
  --------------------------------

  Handling of non-printable characters.  First off, curl uses CR characters to
  rewrite the current line.  Next, there may be other characters which curses
  intereprets in special ways.  Finally, arbitrary binary data causes
  exceptions.  cat /vmlinuz

  info #pty doesn't display a cursor, making it rather difficult to use.

  Make entering full-screen mode for jobs an explicit operation, rather
  than happening just when the job receives focus.

  Friendly messages for "full-screen" processes.  One to say "this is
  full-screen, switch focus (M-o) to use it", another to say "this full-screen
  app has terminted, hit Return to return to hsh" (Perhaps try curses.ungetch()
  to push back the input.)

  Readline type editing for line-oriented job input.

  Run vi in #piped mode, in a terminal larger than 25 rows, then suspend it
  with ^Z Enter.  Hsh is left in a weird state where the screen is drawn offset
  by one row.  Calling curses.nonl() resolves the drawing issue, but disables
  the enter key as well.  Resizing the terminal fixes the problem.

  Run vi in #piped mode, if the first keystroke is an arrow key it complains
  about input of characters greater than 255.  The issue is logged and ignored
  in the code.

  Change the controlling terminal for processes in a pseudo terminal.

  Hsh should check for when a full-screen job has stopped, and take control of
  the terminal back, rather than waiting for the Ctrl-Z Enter keystrokes.

  What can be done about #exclusive jobs that terminate quickly, so their
  output is hidden?  A pause for short-lived programs is one option, but rather
  unsatisfactory in many ways.  How about a builtin/keystroke for turning off
  curses so the user can browse through the hidden terminal output, since it's
  typically preserved?

  Improving Display Appearance
  ----------------------------

  Named directories feature.  They are defined and edited in the same way as
  env and alias, with a builtin called nameddir, and a KeyValue view.  They are
  displayed anywhere that directory names are drawn using a leading ~.

  Provide way to switch off line wrap indicators, they mess with cut'n'paste.

  Include visible cues to differentiate the end of output from blank lines. a
  la vi.

  Add a little spinny thing to the header to identify running jobs more easily.

  Add way to toggle job headers between long and short mode.

  More output modes: Hex Output, character sets?  Line numbers, timestamps.

  Make alert messages disappear after a while.

  Alert message when a job in the background finishes.

  Packaging, Upgrades and Supporting Multiple Versions
  ----------------------------------------------------

  Use the distutils.  There are a few things to overcome though:
      Automate running of tests
      Automate copying of version into code.
      Processing and inclusion of docs.
      Other stuff from make_pkg.sh and Makefile
  Here's an initial pass at the setup.py:

import distutils.core

distutils.core.setup(
    name='hsh',
    version='1.-2.1',
    description='The {Happy|Human|History} Shell',
    author='Alexander Taler',
    author_email='dissent@0--0.org',
    url='http://www.nongnu.org/hsh/',
    package_dir={'':'src'},
    packages=['hsh', 'hsh/curses_display'],
    package_data={'hsh':['../../doc/*.html','../../doc/*.txt','../../doc/*.png']},
    scripts=['bin/hsh'],
)

  Command Line Enhancements
  --------------------------------

  Add #secret directive so that no output or input will be logged.

  Implement ** syntax for globbing.

  Bind | within job listing views to edit command line piping output from job.

  Do we want to support boolean operators or ; for multiple processes?

  Highlight warnings in the command-line (broken env variables, dodgy control
  characters, etc.)

  Add #detach directive, which closes fds and detaches the process from the
  shell's process group.  Used for daemonization - anything else?

  Workspaces
  ----------

A longer term development goal is to add the concept of a workspace, allowing
the user to segregate their different work areas.  Within a workspace the
various histories will be restricted to showing only commands, jobs and
contexts which belong to that workspace.

Some ideas for implementing workspaces is to have tags on jobs, and only show
ones matching active workspaces.  It seems intuitive to restrict to having a
single active workspace, but perhaps permitting multiple could have some
benefit.  There will need to be a default or no-workspace mode which shows
everything.

Users could use workspaces to segregate different projects they work on, or
different tasks such as working on graphics and systems administration.

  Miscellaneous
  -------------

  Keystrokes are getting lost.  Sometimes M-o to list view, then M-w to toggle wrap and
  nothing happens.

  Using replace on commands with %to means changing current job to the right
  one.

  Make wrap, clip, trim, and truncation markers part of formatter config.

  TextFormatter should probably cache results of face lookups.

  Create a CommandText class which provides a Text interface to the
  CommandToken.  This simplifies InputView, but opens up the difficulty of
  dealing with array interfaces across two classes.  It is appropriate as part
  of a multiline command input feature.  Add a clear() method to Text for this
  scenario.

  Rename ContentView to TextView

  new class: EditableTextView which extends TextView, and has the editing
  commands and cursor management.  Can remove the self.editable flag.  Can it
  contain align_display(), move_left() and move_start() from
  SearchView/InputView?

  %f* syntax should highlight as an error if the relevant file is still open.

  Add #exec directive or exec builtin.

  Rerunning a job doesn't remove standard error output from the display.
  Happened with tar output from dbackup script.  

  Implement options/config like aliases.

  A help line in the headers of each view.

  A glob symbol in the prog raises an exception, but should just generate
  a warning.

  Warning when trying to restart a running job.

  Colours for cucumber?

  Feature to recalculate all jobids to make them smaller.

  Implement a specialized JobOutput class:

class JobOutput(object):

    # Place holder for a future class representing the output of a job.  
    # Class for storing output of a job, can be backed by disk or populated
    # from a running process.  Will store output to disk.

    # Provides an array interface for ease of use by displayable objects.

    # Each line of output is annotated with a timestamp, fd, line num, and line
    # num from that fd.
    pass

  Job sort order cannot be changed.

  Numeric arguments to keystrokes.

  Movement by word

  Built-in editor accessible via $EDITOR.  Alternatively, allow command line
  bitties to be expanded to a multiline edit mode.

  Less cumbersome mechanism for accesing sh, than calling sh -c. 

  Memory management - automatically forget old jobs

  Nicer way than kill -STOP to suspend running jobs. 

  sftp works, but the prompt doesn't appear until after the command and newline
  is entered.

  Prevent deletion of jobs belonging to other instances.  Probably implement
  session stuff.

  Output for jobs from other instances is not read from disk.

  Command history shouldn't pick stuff up from other instances.

  CommandHistory current job and the state of history in InputView are linked.
  SessionList current job and the JobView are linked.  Can some code be shared?

Justifications
--------------

  Why reimplement shell parsing and command line stuff?

hsh is not built atop existing shells because of the difficulty of managing
output of background jobs.

  Why use curses when it's so ugly and clunky?

I think there is room for both a curses front-end and a windowey front-end.  I
chose curses first because the typical shell user is comfortable in a terminal.
Furthermore, I expect the effort of supporting other curses apps (notably vi)
within the terminal will be less than in a windowey front-end.

Reference
---------

The TTY demystified
   http://www.linusakesson.net/programming/tty/index.php

Also refer to the libc documentation about job control.


