Minor fixes and optimizations
This minor update contains several improvements.
Note : jar version is for Java 6, jre8-jar for Java 8, jre11-jar for Java 11.
Improve network closing when necessary and using native ByteBuf
Reason:
While network connections were closed normally, sometimes they were closed too early
in the sense that another "local" connection (local channel) could still used it.
Moreover some ByteBuf were wrapped of byte arrays, which is not optimized for Netty.
Changes:
Improve testing on reusability of network connection and centralization of this
process, such that all local closings use the same algorithm and codes.
This change allows to keep as much as possible existing connections if they can be
reused soon. If not, within a certain delay (Timeout), the connection is closed.
Byte Arrays are now written directly within a real ByteBuf from pool.
Result:
Now loop based IT tests or direct multi transfers give good performances and no more issues.
- Recv Direct using 3000 files of 360 KB gives an average of 40 transfers per second,
while not optimal since even if multi-threaded, there is a latence on thread creation
and execution - Loop transfers (from one server to another and repeating) of 700 concurrent files in average
of 360 KB gives an average of 77 transfers per second.
This last result is a good start for a real benchmark (211 Mbps, roughly the speed of disk).
Conditions of benchmarks:
- 2 and 4 cores, 8 GB RAM, 200 Mbps disks capacity
- 2 Waarp servers started, 1 PostgreSQL database started (docker) on the same server
Comparisons on loop transfers:
- v3.5.1 : 4 cores gives about 30 transfers/second
- v3.5.2 : 2 cores gives about 50 transfers/second
- v3.5.2 : 4 cores gives about 76 transfers/second (roughly the disk speed), 4500/minute
One can expect a non linear scalability extending vertically the server. However note that
these benchmarks are on the same server so:
- almost no latency between services (the 2 Waarp servers and the database)
- but still 3 high consuming services on the same 4 cores (2 Waarp servers, each about 45%
of CPU, and 1 PostgreSQL database about 10% CPU with about 2 Mbps SQL traffic network
bandwidth)
Probably 4 cores with 8 GB for the JVM and only one Waarp server, having in a separate
host the PostgreSQL database, and of course the second Waarp server with its database
somewhere else, shall give better performances, about 150 transfers/second
7800 transfers/minute).
Scalability can be both vertical (increasing CPU mainly, but also Memory) and
horizontal (using a loadbalancer, a shared database and a shared efficient
filesystem).
Improve file canRead method
Since filesystem is sometime unsynchronized with Java, this method tends
to try to leverage this issue by trying multiple times if not OK the first
time (at most 30 ms).
Allow Windows FTP client to connect to FTP Gateway
Windows FTP client seems to send immediately after the connexion an OPTS command
while it shall be done after the authentication step. We allow this special command to occur
just before the connexion, such that Windows FTP client can access to the FTP Gateway.