
UNICAST3: syncing a receiver with a sender
==========================================
Author: Bela Ban
Date:   Sept 2014
JIRA:   https://issues.jboss.org/browse/JGRP-1875


Issue
-----
When we have sender A and receiver B, and B removes A from its view, but not vice versa (asymmetric split),
then B will close its receive window for A (but A will keep its send window for B).
Setting a conn_close_timeout in UNICAST3 mitigates this (contrary to UNICAST2, which immediately closes the window),
but this scenario can still happen, even in UNICAST3.

When the network heals, and B receives messages from A again, the SEND-FIRST-SEQNO protocol starts,
asking A for the lowest seqno and for retransmission of missing messages.

There are 2 deficiencies with SEND-FIRST-SEQNO:
- It generates a lot of traffic (see https://issues.jboss.org/browse/JGRP-1874 for details)
- A delayed ACK message sent by B to A can cause A to purge messages that haven't yet been seen by B
  (see https://issues.jboss.org/browse/JGRP-1873 and https://issues.jboss.org/browse/JGRP-1875)

The goals are therefore:
#1 Reduce traffic needed by a receiver to sync its receive window (JGRP-1874)
#2 Make syncing of the receiver reliable, so that JGRP-1873 doesn't happen (leading to endless retransmissions)



Solution
--------
The solution is to (1) time stamp SEND-FIRST-SEQNO and ACK messages, to prevent spurious purges of sender windows when
receiving delayed (and therefore out-of-order) ACKs and (2) to reduce traffic caused by redundant SEND-FIRST-SEQNO
messages, as detailed in JGRP-1874.


Design
------
- When receiver B receives a message from sender A and it doesn't have a receive window for A, or A's connection ID
  (conn-id) sent with the message and the conn-id of the ReceiverEntry don't match,
  B sends a SEND-FIRST-SEQNO (unless another one was already sent within the last N msecs) to A
- On reception of SEND-FIRST-SEQNO, A:
  - Resends the lowest message and highest sent message to B
    --> This will trigger retransmission requests from B to A

Implementation
--------------
- New property sync-min-interval (ms): the min time to wait after sending a SEND-FIRST-SEQNO message, before
  sending another one. Reduces traffic.
- Additional table last-sync-sent (long) in UNICAST3
  - Maintains mappings between members and timestamps (possibly ExpiryCache)
  - Needs to be a separate table because recv-entry might be null and can thus not store the timestamp
  - To prevent sending more than 1 SEND-FIRST-SEQNO within get-first-seqno-interval ms
- Additional field last-timestamp-received (int) in SenderEntry: stores the timestamp of the last ACK or SEND-FIRST-SEQNO received
  - ACKs or SEND-FIRST-SEQNO messages with lower timestamps will be dropped


Receiver (B)
------------
- When a message from A is received:
  - If the ReceiverEntry for A is null or (the message is not marked as first and the conn-ids don't match):
    --> If recv-entry is null or last-sync-sent for A is null or last-sync-sent timestamp for A is
        older than sync-min-interval:
        --> If last-sync-sent was updated to the current time successfully (atomic operation):
            --> Send SEND-FIRST-SEQNO to A (with a new timestamp)


Sender (A)
----------
- On reception of a SEND-FIRST-SEQNO(timestamp) from B:
  - If timestamp is out-of-order (timestamp <= send-entry.last-timestamp-received)
    --> Drop SEND-FIRST-SEQNO message
  - Else:
    --> Set send-entry.last-timestamp-received to timestamp
    --> Send the lowest message to B (the highest is sent periodically by retransmission task of sender A)

- On reception of an ACK(seqno,timestamp) from B:
  - If timestamp is out-of-order (timestamp <= send-entry.last-timestamp-received)
    --> Drop ACK
  - Else:
    --> Set send-entry.last-timestamp-received to timestamp
    --> Regular processing


Misc:
-----

- Timestamps are generated by the receiver with an AtomicInteger (field in UNICAST3), which is incremented on every
  getTime(). This ensures that timestamps are always increasing and can be compared to each other, even on numeric
  overflow (see System.nanoTime() on overflow discussion). Using an int saves memory and starting at 1 saves
  network bandwidth as timestamps can be compressed (see Bits.writeInt()).



