PLAN for HarvestMan 2.0
-----------------------

This document captures the list of TBD and planned dates for HarvestMan
and Hget for the next release (2.0).


HarvestMan TBD
--------------

Most of the changes are shared between HarvestMan and
EIAOHarvestMan, but a few are specific to HarvestMan.
The changes specific to HarvestMan are marked with an
asterik (*).

Completed

1. New program exit logic => Completed
2. New control params for controllong multimedia, movies,
etc => Done.
3. Signal management - Added support for SIGINT, is enough for now, Done.
4. Integration with sgmlop - Done.
5. Config file split - Done.
6. Datastructure enhancements - Mostly completed.

Pending major tasks

1. Database integration - Storing program statistics, session information,
downloaded data etc into an sqlite database - High priority, design to 
be done.

Pending minor tasks

1. Localization fixes
2. Option for urlfilter to filter URLs only for download.


Hget TBD
--------

+---+-----------------------+---------------+---------------+---------------------------------------+
| NO|       Feature/Fix     | STATUS        |    Priority   |       Description                     |
+---+-----------------------+---------------+---------------+---------------------------------------+
|   | Automatic thread      |               |   Desirable   | When program finds that the user has  |
| 1 | reduction control     |               |               | split the download to "n", but either |
|   |                       |  Lower        |               | it cannot get "n" mirrors or the host |
|   |                       | Priority/ Drop|               | is not responding to "n" threads (in  |
|   |                       |               |               | case of same host downloads), use a   |
|   |                       |               |               | logic to automatically reduce the     |
|   |                       |               |               | load on the host/mirrors by reducing  |
|   |                       |               |               | the thread count adaptively and       |
|   |                       |               |               | recomputing the file piece size per   |
|   |                       |               |               | thread, thereby ensuring that the     |
|   |                       |               |               | download is completed. The user can   |
|   |                       |               |               | set a minimum value for this, the     |
|   |                       |               |               | default being 1.                      |
+---+-----------------------+---------------+---------------+---------------------------------------+
|   |Mirror selection       |               |  Normal       |When failing mirrors, put mirrors that |
| 2 |improvements           |               |               |fail with non-fatal errors in a retry  |
|   |                       |      Done     |               |cache, so that when mirror list is     |
|   |                       |               |               |exhausted,we can try these again.      |
+---+-----------------------+---------------+---------------+---------------------------------------+
|   |Mirror search          |               |  High         |Add support for searching mirrors      |
| 3 |engine                 |   Done        |               |in mirror search engines and dynamicall|
|   |                       |               |               |y obtain mirrors for URLs.             |
+---+-----------------------+---------------+---------------+---------------------------------------+
|   |FTP resume/multipart   |               |               |                                       |
| 4 |support                |               |  Highest      |So far, supports only HTTP byte-range  |
|   |                       |      TODO     |               |and resume. Need to add same for FTP   |
|   |                       |               |               |for supporting most of the mirrors out |
|   |                       |               |               |there.                                 |
+---+-----------------------+---------------+---------------+---------------------------------------+
|   |Adaptive learning      |               | Desirable     |Rank mirrors based on historical       |
| 5 |and mirror prioritizing|   Lower       |               |performance and automatically select   |
|   |                       |Priority/Drop  |               |the best ones. (Get logic from SMART)  |
+---+-----------------------+---------------+---------------+---------------------------------------+
|   |Better failover        |               |  Normal       |Detect threads which hang and perform  |
| 6 |                       |       TODO    |               |check-pointing and migration etc to    |
|   |                       |               |               |increase robustness.                   |
+---+-----------------------+---------------+---------------+---------------------------------------+

