SHORT TERM:

FIXME: handle HTTP/1.1 100 continuations, resetting the HTTP expiration
timer.  Just continue reading headers, and set a timeout to 2 minutes after
each 100.  Otherwise, handle like a "503 Queued".  Advertise support
with a X-Features: http_100

FIXME: in clock.c, record the last 100 IP that were used to adjust the clock,
and don't ever use them as a source for computing the delta whilst they are
in the hash.  This should yield a better and more stable offset.

FIXME: check hostiles.txt with BearShare's latest version.

FIXME: Icons not seen from .deb.?  [A mystery, to doublecheck before releasing]

FIXME: Fix "make dist" to include ALL sources, irrelevant of the GTK package.

FIXME: with PARQ, it's important to auto-ban web browsers.

FIXME: enhance zalloc tracker with __builtin_return_address(int frame).

FIXME: BUG: saw an endless request / sinking / request /sinking loop.

FIXME: when QUEUE is being sent, and the remote end closes the connection
after QUEUE was sent twice in a row, discard the entry.

FIXME: use the special "min speed" tagging in queries now to mark when
we are firewalled?

FIXME: A good measure of whether we can be a valid UP is the time we keep our
UP peers.  If we can't manage to keep one longer than 30 minutes after a
period of 2 hours, then we might consider going back to leaf mode.
If user has its PARQ queue filling (say more than 5 entries), then force
leaf mode to maximize throughput (user has popular/desired content).
OR: the measure of the round-trip times.

Monitor queries from our leaves.  If we see the same query string "too often",
then drop it, instead of propagating it.  No more than 1 requery every 1/2
hour for a given query string.

Drop SHA1 queries.  They are being abused.  Remove support for them in search?
Remove them from the QRP tables?

Allow leaves to give a list of SHA1 they are interested in.  Inspect all
hits and propagate the relevant ones to leaves.  Only forward the GGEP
extensions we know about.  Automatically gather the SHA1 they are interested
in by the SHA1 queries they make.  Remember the hits we sent to a node for
1 hour at least, using 2 hash tables switched every hour.

Throttle query traffic: recall per connection the (query, hops, ttl) received
during the last 5 seconds, rotate the hash table every 5 seconds.  Count
the throttled queries.  We recall the TTL to allow selective querying to
work: same host requeries with a greater TTL.

Add compression support for uploads / downloads.

"Sign" query hits to make them more authentic.  An unsigned GTKG hit is
now converted into a FAKE hit.  A signed GTKG hit with an unmatching
signature is now flagged as ALTR hit.  Those flags appear in the search
window for now, and the packet is not physically altered for now.

Do not advertise our IP to GWC if we're in "forced ultrapeer" mode.

server-side PFSP:
* pfsp_server props says whether it's enabled.
* pfsp_first_chunk is the size of the first contig. chunk.  Otherwise,
  random (line 'n' has "1/(n+1)" chances of superseding current).  Then
  random start in the selected chunk.
* when serving, maintain a table of SHA1 -> reqfile structs.
* if PFSP-enabled, send dmesh entries for the file (N2R form).
* Do a 404 when pfsp_server is disabled, even if we have the file there.
* Only allow /uri-res/N2R? requests for those.
* need to support X-Alternate-Partial-Location for partial locations, so dmesh
  must remember which location is full/partial.  And serve full first, and if
  not enough, partial, or a mix.

Vendor-Msg: WARN, QRT-SHA1(send) QRT-SHA1(ack)

Add/parse X-Try and X-Try-Ultrapeer to BYE 503 packets, since we now
use them extensively.

Verify mesh entries of files we OWN in a background task, sending the
HEAD requests to other sources when they expire.  Entries are either
removed or added to the mesh depending on the result.The bgtask never
stops and iterates over all the mesh entries of files we OWN.  Needs a
bg_task_delay() to suspend scheduling of the task for some defined time,
and a bg_task_goto().  "active grooming".  Maybe not a bgtask, but an
explicit timer callback!

"passive grooming": only propagate verified sources during the download
process.

------------------------------------------

BUG: some queued downloads stay in "Suspended (SHA1 Checking)" even when done.

BUG:

Dans ~/.gtk-gnutella/downloads:

lost_in_space.avi
367442432, 996:00000000000000000000000000000000, 12.254.208.139:6340
2YLZ3CH34567NHSBBKEWAJTFDEUMZ2VR

lost_in_space.avi
367442432, 996:00000000000000000000000000000000, 12.254.208.139:6340
2YLZ3CH34567NHSBBKEWAJTFDEUMZ2VR

Dans fileinfo:

# refcount 2
NAME lost_in_space.avi
PATH /ext/tmp2/ram/gnutella/
GENR 0
ALIA lost_in_space.avi
SHA1 2YLZ3CH34567NHSBBKEWAJTFDEUMZ2VR
SIZE 367442432
DONE 0
TIME 1032933773
SWRM 1
CHNK 0 367442432 0

A la sortie:

freeing remaining String atom 0xa9232cc, refcnt=1: "lost_in_space.avi"

------------------------------------------

[GURUZ: guruz.dyn.bawue.de:6346]

* upload_add -- only first line read, need to handle headers before doing
  the upload. [DONE]

[test this -- we shall accept incoming 0.6, but still connect at 0.4]
[make sure uploads work]

* parse handshake ack/replies for codes other than 2xx.
* flag HEAD/GET correctly in uploads.
* add header parsing in download, for their replies (needs errcode parsing).
* Send 10 fresh pongs to old 0.4 servents to whom we deny the connection.
* Push-reply handling (reception of GIV, and sending)
* if "push" flag is on, start immediately with a PUSH.

[make sure we can download]

* tests ping/pong reduction
* remove all routing information for pongs
  (will need to think/rewrite node_parse() to not route_message() before
   handling the ping/pong -- also could separate the handle_it/drop_it
   function from the route-to-whom function, and differ the actual routing
   after the processing: would allow to drop invalid search requests with
   no NULL in them).
* change 10 fresh pong sending: wait for first incoming PING message, and
  reuse the same GUID for the reply.  Kill "Ponging" connection after timeout
  or if first message is not a PING.

* add connection at the 0.6 level, falling back to 0.4 if they close.

[test we can still connect to the network]

* Generate X-Try header, and parse the ones we may receive.
* Add parameter and logic to avoid sending a push if too old a record.
  Note, when re-issuing, update the "last search" like we do for the indices,
  meaning the route is "fresh".  [Implemented as "Route Lost"]

* apply the "double-click" patch
* apply the filtering patch
* URL (un)escaping (GIV strings as well? no need)
* Node: header emission and parsing, along with X-My-Address
* Fix queue_remove_all_named() to remove all those in "Retry in 10s" state.
* Deal with GIV in unexpected state or unexpected GIV more gracefully.
  Those are really precious.

* implement Bye! (will need to read data on write error, not close at once)
. for nodes, set SO_SNDBUF to 8K (for instance), and on bye, raise it back
  to the size of the last unsent packet + sizeof bye to make sure we can
  write the whole thing to the kernel without blocking.
. add node_timer(), download_timer() routines... to remove all this crap
  from main.c's main_timer().
* Support upper end on Range requests, including Range: bytes=-10.
* Fix BUG: rx_dropped can be incremented twice for same message if TTL reached
  0 and it's a ping and we throttle it, for instance.
  -- I've seen RX=1 and Drop(RX=3456) [when FC on]
* collect user-agent info on neighbouring node, parsing query hits if
  necessary to collect vendor code when nothing is in the handshaking headers.
* Add setlocal(LC_TIME, "C") to ensure dates are shown uniformly;
* Bandwidth management
* Add Date and Last-Modified headers to HTTP upload replies.
* Add TCP_CORK on uploads?
* Use sendfile(2) (not very portable)?
* Investigate routing bug on my own messages (core in ~/dbg) -- Not a bug!
* Raise receive buffer for downloads?
* Handle "alive pings" as highly prioritary: enqueue message at pri=1.
  (modify msg comparison routine to handle priority as well first)
* Bug de mbuf->data: verifier que le node est present dans la boucle,
  en testant n->membuf.
* Patches de guruz...
* Sticky GUID for servents behind a firewall.
* Add lockfile. (no multiple runs on same machine)
* When killing all downloads named..., only do so with those whose size is
  lower than the finished one.  If a bigger exists, need to continue download,
  so move back file to working directory.
* Investigate download_push_insert() bug. -- found it, I think (13/03/2002)
* Fix Route Lost for downloads that have only fallback to push.
* Generate a config.new, renamed as config, and move config as config.orig
* Ensure minimum value for node_sendqueue_size (1.5 * max packet size).

>>> for 0.85 release

* Try to return "412 Request URI too large" for large HTTP requests. [Check!]
* Re-write README
* Review man page, sort options alphabetically for reference.
* Ensure no option is missing.
* Document screen interface, explain columns.
* Write a FEATURES section/file.

>>> for 0.90

* force_ip logic change: don't auto-update if forced.  But change
  the local_ip on Remote-IP if no force_ip.
* download_remove_file() and download_file_exists().
* Check for new "download_delete_aborted" condition in download_abort() to
  remove file when it is set.
* Fix SIGFXSZ by ignoring and stopping downloads on E2BIG (see =soft/gtkg).
* Make search queue and persistent searches.
* Mark downloads moved out of queue: do a link() instead of renaming.
  Then remove the links (if > 1) when finishing the download because
  there's no more to get, or when completing it (moving to target done dir
  with same inode and linknum > 1).
  [Fixed by moving out only if the file is smaller than what we download]
* Use link() instead of rename() when moving from done->tmp.  When moving
  back to done, if nlinks > 1, then check for the file with same name in done,
  and if same inode and same filesystem, then simply unlink. [NO, fixed]
* BUG fix: local_ip is persistent, ok for local default, but must determine
  it at least once on new launches, in case it changed...
* Ignore size mismatch if greater than what we thought, and rely on range
  checking to resume safely. [REJECTED, see comment in downloads.c]
* Content-Encoding: deflate [DONE]
* apply SHA1 patch
* apply __attribute__ patch. / SOCKS message change patch.
* Ensure SHA1 computed during library rebuilt is dumped out to the cache
  ASAP in case of crash later on, or quit!  Maybe add boolean/flag "on_disk"
  when loaded from disk or saved, and force append to disk cache when we
  have a new computation not on disk already. [FIXED differently: was a BUG]
* Make sure qrp_finalize_computation() is done as a coroutine, one step
  at a time every second.
* String atoms in atoms.c.  -> streq() can become == between atoms. [DONE]
* Change library: atomize the full path (shared with HUGE anyway) and
  have the filename point rigth into this full string. [DONE]
* Add 301 Location redirect on non-matching SHA1 for /get which we have. [DONE]
* Avoid 409 if we have the filename, and it's unique, for old servents. [DONE]
* Factorize SHA1 validation!
* Extract SHA1 in downloads and searches, and make them persistent.
* Make persistent results filters. [DONE, bluefire]
* Also detect non-support of /uri-res if EOF during headers? [DONE]
* Propage stamp to downloads, enter in mesh what we download, remove it
  when we get a fatal error from host (404, etc..) or can't connect to it.
  Don't insert if PUSH, of course! [DONE]
* Ensure we don't insert private IPs in mesh.
* Fix qrt_compact(), which starts from the end, abd puts LAST slot
  in bit 7!  This seems to contradict the logic of the patching routines. [DONE]
* Memory leak on "clear searches": does not free all the rs attached, only
  frees the clist.  (in search.c and gui.c) [DONE]
* Probe for max #fd, and reserve only MIN(maxfds/5, 100)
* urn:sha1: searches. [DONE]
* When removing done downloads, remove by name identical and by SHA1 of the
  completed download. [DONE]
* Add configure check for zlib. [DONE]
* When closing a search, clear the search queue. [DONE]
* If query sent on connection with TTL=1 and we get hops>0 for reply, it's
  probably an invalid hopcount. [NO: there could be re-routing]
* Create alive.c, handling alive pings and replies.  Maintains a list of
  MUID/gettimeofday, appended.  If maxsize reached, timeout.  On ACK, trim.
  Keep stats of min_rt, max_rt_, avg_rt (EMA), last_rt. (in ms). [DONE]
* Separate version stable / dev.  Show date in messages. [DONE]
* If we don't get any incoming connection after 60 minutes, act as if we
  were firewalled again, by calling pcache_port_updated(). [DONE]
* BUG: Misparses Shareaza's alternate locations: [FIXED]
	http://63.227.56.41:6346/get/151/xxxx,http://24.243.150.166:6346/get/12/xxx
* GWebCache: http://www.gnucleus.net/gcache/gcache.php [DONE]
* Have a file holding the SHA1 of all the complete things we own (but do not
  necessarily share).  When we download, check against this list to avoid
  duplicates.  Even possibly filter them out.  The file would be reloaded
  when its timestamp changes. [DONE, somehow]
* Only emit /uri-res/N2R? Alt-Locs if there was a SHA1 in the query hit
  [DONE, but only after successful 503/2xx return code from server on a
   /uri-res/N2R?urn:sha1: request will we convert the download and insert
   the /uri-res form in the mesh]
* Validate all the /uri-res/N2R? we get as Alt-Locs. [DONE]
* It is imperative to measure the amount of time we spend in `f' of `h' state
  (if remote is a not a leaf), and if we spend more than 50% of our time over
  5 minutes, drop connection. [DONE]
* Connection burst: do 10-15 more than what is needed, and prune afterwards?
  [DONE, thanks to node_can_accept_connection()]
* Expire 'u' and other non-stable versions after 60 days. [DONE]
* Defeat Shareaza's user-agent filtering feature: on a "404 Please Share"
  or a "401 Please Share", record that the server is doing user-agent filtering
  and serve it back its own ID at the next request! [DONE]
* Compute our offset to GMT based on the other timestamps we see.  Allow a prop
  for "I have NTP, so my clock is accurate". [DONE]


[TODO BEFORE 0.92 stable]

 1. Handle "H" GGEP in query hits and queries. [NOT DONE FOR QUERIES]
>2. Handle multiple SHA1 queries gracefully.
>3. Flag spurious separators between extensions -- drop those (query) packets? 
>4. Find memory leaks.
>5. Split our query hits in smaller ~2KB packets.
>6. No longer suffix queries with urn:\0
>7. Add ~5 minute delay on retry for urlfile if we are connected.

>>> for 0.92

>>> NEXT STEP...

* Change GIV processing: only care about the GUID now.  Then select one of
  the downloads for this host.  No longer rely on the index, which may change.
* Add/process:
	Content-Disposition: inline; filename="xxx
  for /uri-res requests (unless we already replied recently, and only when
  replying with 200/206, of course).

* IDEA: exchange Web-Cache: http:// at the 3rd handshake, if OK, to give
  remote our own web-cache if we have one?  Or echange that via a vendor-spec
  message once connected?


* BEWARE: some people want to say "http://...","http://....". URL mesh
  parsing will have to account for "" and remove them first.  Any "," in
  a quoted string MUST be ignored, and all quotes should be properly closed.

* BUG: says: cannot parse Alternate-Location URL:
"http://64.228.175.12:6346/uri-res/N2R?urn:sha1:WXZXUWTHYF5C65ROULOAW6OHFMZRFZCZ"
This should be perfectly parsed!

* BUG: when receiving a GIV for a download in state 11, this should be
  top priority: move a non-push download back to queue, and activate it.
* BUG: when aborting downloads from the queue, they do not appear as Aborted
  in the top window, preventing resuming if something incorrect has been
  aborted.
* BUG: queue scheduling should select PUSH downloads in the queue FIRST!
  Before the route expires!  Should probably mark host with A_PUSH attribute.

* BUG: has:
	http://62.128.212.57:6980/uri-res/N2R?File With Space 2002-07-06T15:12:08Z
in the download mesh, with an N2R??
* BUG: on BYE sent, don't display any other error message than the BYE.
  It's OK if they EOF, or reset the connection.
* Implement Gnet ban for BYE'ed clients.  This will need restructuring of
  ban.c and a slightly different interface since here we're explicitly banning,
  and do not wait an inordinate amount of requests.
* On /uri-res request to gtk-gnutella, or any servent known to support /uri-res
  incrementally and persistently, honour 404, 403, etc... [DONE for GTKG only]
* Make header maxline/maxsize configurable on a per-header structure basis
  to allow larger HTTP headers and smaller Gnet handshaking for instance.
* Validate query hits:
	hops=0 && !firewalled => IP address valid and matches.
	hops>0 && !firewalled => IP address != servent's one
	hops=0 && vendor => always same vendor code for hops=0 hits
	hops=0 => always same servent GUID in hits
	hops>0 => GUID different than servent's own (if known).
    always: GUID different from our's.
* On 401, ban IP for servent type, forever.
* Add "reservation" of upload slot for 60 seconds from same IP and same file.
  This is poor man's persistent connections...
* Add safeguard to dynamic upload slots so that we do not fall below 256 bytes/s
  of bandwidth, otherwise TCP overhead is too large!
* Count retries/timeout on a server level, not at the download level. Eject
  host and all its downloads after that many unsuccessful retries.
* Develop a preference for Accept-Encoding: deflate clients: on incoming,
  prefer them, like we prefer local nodes at times.
* Write hostcache bootstrap servents in a file, so they may add/remove entries
* Don't send back to host the Alt-Locs it just sent to us!
* Compare downloads on IP address first, then on GUID if equal.
* Break down query hits to 4 KB entities.  Will require changing the
  buffering and changing mechanism so that buffering can flush when ready
  and start a new packet.
* Add the following to downloads: timestamp (start), timestamp (last connect),
  user-agent, flags (attributes, status) 
* Make sure huge_init() does not load SHA1 cache.  Do that incrementally,
  whilst GUI is start up but before scanning library.
* Change Push broadcasting: send them all at once, even if multiple files.
  Then, when get a hook with a 200/206, cancel the others.
* Use SHORTEST route for query hits, assuming we saw the same queries with
  different TTLs along different nodes.
* Regulate HTTP uploads: keep track of who. when, which file, and send
  the full headers only once, then send minimal ones.  If too aggressive,
  ban the user.
* Likewise, when making Outgoing connections and waiting, if we get an
  incoming, randomly drop an outgoing to avoid replying Busy to the incoming
  yet get failed/timeout outgoing.
* Count replies we get from each TTL=2 probe (that we don't send today), and
  display replies.
* Look at node inactivity timeout.  Must be on last RX.

* See message about HTTP status: reject Range requests that have ",".
* Fix the hops++ in route_message.  This must be done when resending packet!
  (remember to change all tests for e.g. hops==1 in code to hops==0)
* Broadcast Pushes to ALL known routes, to maximize chances.  Introduce a new
  ROUTE_LIST with a dyn-allocated list of nodes to send the message to.
  (will need to change interface of forward_message())
* Keep indices persistent (assuming basename is unique, keep a table),
  then allocate indices sequentially (keeping a list of "free slots"?) to
  ensure consecutive rebuiilds and re-runs don't perturb indices.
* Allocate routing table dynamically (list of fixed chunks) based on minimal
  amount of cycling time.  Use this opportunity to separate queries and Qhits
  into two separate tables with possibly different lifetimes.  Keep list of
  messages seen separately to avoid dups the longest time possible at little
  memory cost?
  Keep 2 buffers, and start the first, expand it, then after the time limit,
  switch to a new circular buffer, but keep old data.  When the time limit
  expires, go back to the first and sequentially replace entries -> typical
  lifetime will be around 2*limit.
* Clean traffic: remove bad queries, query hits with 0 files, etc...
* Randomize search results when limit is reached?
* If we get a non-/uri-res alt-loc, check whether the N2R? is valid by
  issuing a HEAD request.
* Add size information to the download mesh, and validate all new alt-locs
  in the background via an HEAD request.
* Keep track of the last IP:port we had over the last week (lifetime of the
  dmesh) and always remove all alt-locs for files we own that are located at
  those IP:ports.
* Save HTTP/Gnutella headers exchanged during a connect session and have the
  GUI display them upon request.
* Add counters for of # of busy sources in queue (count all entries from all
  servents), # of distinct hosts, # of busy hosts, and # of Actively queued
  sources in the active pane.
* Measure and display traffic per message type, to know how many bytes/sec
  are used for queries, qhits, pongs, etc...


[test we can still connect to the network]
[test for memory leaks]
[release]

* Implement upload queue (mike's version, with variations).
* Generate pre-pended comment with date in all files generated by autogen.sh
  so they can be committed back (without changing timestamps).
* Use getrusage() to monitor CPU usage on query processing?  Flow control
  the link between Gnetd and the query processors.

* Embed Perl, add Perl->C gateway to filtering.  Or allow a PERL call rule,
  giving it record/result-set information.  NB: adding a perl call to every
  rc that comes is going to be costly?
* Use Perl module Video::Info to collect meta-data information from library
  files.

[rest a little]

<ram> Also the average size of each packet (i.e. sum the amount of
traffic per packet type).
<ram> Also of interest would be the amount
of packets dropped, by type.  Dropped because of TTL limit reached,
dropped because of flow-control, and dropped because of size limits.

[we can parse the thing below, but shall we emit it as well?]
<guruz> X-Try-Ultrapeers: 68.63.90.223:6060
	2002-11-16T16:05Z,131.252.204.56:6346 2002-11-16T16:05Z,195.194.39.52:6346
	2002-11-16T16:02Z,172.195.224.77:6346 2002-11-16T16:05Z,64.175.69.144:9438
	2002-11-16T16:05Z,24.239.83.18:9061 2002-11-16T16:05Z,4.64.0.133:9161
	2002-11-16T16:05Z,134.53.169.161:9057 2002-11-16T16:05Z,152.30.103.10:6346
	2002-11-16T16:05Z,156.12.134.50:5456 2002-11-16T16:11Z

<guruz> thats the formnat in case you are interesed..
