Skip to content

New major version
Compare
Choose a tag to compare
@fredericBregier fredericBregier released this 18 Aug 15:23
· 24 commits to v3.6 since this release
v3.6.0
cc65fc4

Add S3 Support for Read or Write File to R66

Only available from JRE 8 (so not with JRE 6) due to unerlying S3 client used (Minio-Java),
it allows to extend the Waarp R66 tasks with new operations:

  • S3 GET (and eventually S3 DELETE) from remote S3 file and implicit R66 RENAME
  • S3 PUT (and eventually R66 DELETE) to remote S3 file
  • S3 DELETE remote S3 file

Note that for GET task types as pre-task, as it may be the source file for a transfer,
either in SEND mode or RECV mode (push or pull),
the mode for the Rule shall be SENDTHROUGHMODE or SENDMD5THROUGHMODE,
respectively 5 or 7, or RECVTHROUGHMODE or RECVMD5THROUGHMODE,
respectively 6 or 8, since the file does not exist at startup.

This change allows also to add new tasks to the existing tasks by adding
one or many TaskFactory to the TaskType object within WaarpR66 using an
extra Jar.

In order to have a full version for Waarp for JRE 8 and JRE 11, 2 additionnal
WaarpR66 with full dependencies are created, named WaarpR66-S3.

Add Remote monitoring in Push mode

Currently, only Pull mode (request on REST API) is available to integrate
status of transfers from Waarp R66. It is also available to connect directly
to a remote Elasticsearch but using JRE 8 or 11 versions only.

This add the possibility to configure each Waarp R66 servers to send
through HTTP(S) requests to a remote API REST (configurable)
every x delay (configurable) with the last R66 transfers informations
based on Stop Date (last updated information date) as a Json array
as

 {
   "results" : [
    { transfer information as Json },
    ...
  ],
 }

Each transfer informations as:

 {
   "hostId":"server1",
   "waarpMonitor":{
     "to":"2021-04-08T17:29:42.690Z",
     "from":"2021-04-08T17:29:41.676Z",
     "index":"server1"
   },
   "blockSize":1048576,
   "stop":"2021-04-08T17:29:41.965Z",
   "errorMessage":"Unknown status",
   "filename":"hello12560",
   "type":"r66json",
   "stepStatus":"Unknown",
   "transferInfo": "{\"follow\":-8739926495694136052, \"ORIGINALSIZE\":12560},
   "originalFilename":"hello12560",
   "originalSize": 12560,
   "specialId":-9223372036854775807,
   "followId":-8739926495694136052,
   "requester":"server1",
   "globalStep":"NOTASK",
   "ruleName":"loop",
   "start":"2021-04-08T17:29:41.954Z",
   "requested":"server2",
   "fileInfo":"Test Loop Send 12560 {\"follow\":-8739926495694136052}",
   "status":"TOSUBMIT",
   "errorCode":45,
   "retrieve":true,
   "globalLastStep":"NOTASK",
   "step":-1,
   "uniqueId":"server1.server1.server2.-9223372036854775807"
 }

And in the Headers of Http POST request:

 X-WAARP-ID : hostId
 X-WAARP-START : Date Time From Interval
 X-WAARP-STOP : Date Time To Interval

This can be used for instance with a Logstash service that sends back to an
Elastic index, using for instance waarpr66.waarpMonitor.index.date as final index.

Add configuration:

  • Within server set
  • URL for remote REST Service
    • pushmonitorendpoint /log
  • Remote End point for REST Service
    • pushmonitordelay 1000
  • Period of check to push information on changed transfers statuses
    • pushmonitorkeepconnection True
      • If True, the HTTP(S) connection will using Keep connection, while False means
        closing connection after each POST (default True)
    • pushmonitorintervalincluded True
  • If True, the Json will contain interval information (from, to, suggested partial index name) (default True)
    • pushmonitortransformlongasstring True
      • If True, the Json will transform "Long" as "String" (for ELK) (default False)

An example of Logstash configuration is given in WaarpR66/src/main/config/logstash-r66.conf

Benchmark shows only 1% of degradation using 4 vcore.

Other new functionalities

  • If a certificate is no longer valid, Waarp will raised a warning but will not stop the execution
  • Waarp R66: the Web administration allows to modify or create an transfert for the current server
  • Waarp R66: the Web administration allows to search transfers according to "follow id"
  • Enhanced facility to add new tasks to Waarp R66
  • Add compression task based on ZSTD algorithm (fast and efficient)
  • Add compression possibility (inactive by default) on network level using again ZSTD algorithm
  • JRE up to version 16 compatibility ensured (using JRE 11 package)
  • Add automatic packaging (not official yet) of RPM, DEB, TGZ and ZIP, including documentation in HTML and PDF
  • Gateway FTP: lets the possibility to specify if PASSIVE and ACTIVE modes are accepted:
    • in limit, activepassive n where n could be -1 = passive, 1 = active, 0 = both

Fix Doc and improve performances

  • LocalExec increase thread possibility
  • Change minimal / maximal threads for retrieve
  • Increase runlimit to 50 000
  • Improve Client Thread / Server Thread
  • Improve Buffer using ioBuffer for sent parts
  • Add Benchmark tests on Multiple scenarios (multiple servers, Monitoring, S3, FTP...) and also ascendant compatibility using
    v3.5.2 as reference
  • Fix issue on Oracle JDBC driver from Maven Central inspired from Pull Request #92
  • Fix issue #91 on profile option description in README.md, inspired from Pull Request #93
  • Fix Json and XML issues
  • Change to Saxon when possible
  • Fix various FTP issues (both server and clients sides)
  • Improvements on SQL requests and Indexes (mainly for Waarp R66)
  • Improvements on memory usages
  • Improvements on documentation
  • Creating a full build (jar, RPM, DEB, ZIP, TGZ) from Maven
  • Upgrade dependencies

Some Benchmark Results

Waarp R66 Benchmarks

  • Loop Benchmarks : 2 Waarp Servers, 1 with PostgreSQL as Container, 1 with H2 Database on the same server with 2000 files of size average of 250 KB
    • TLS efficiency: up to 127 trasfers/second
    • No TLS efficiency: up to 133 transfers/second
  • Dual Waarp servers on the same host with the same PostgreSQL Database as Container and Filesystem with one Client on the same server with 4000 concurrent transferts of size average of 250 KB
    • 1 server: 89 transfers/second with 30% CPU usage
    • 2 servers: 192 transfers/second, so 100% more than 1 server in cluster, 40% more compare to 1 server only, with 60% CPU usage
  • Big file transfers: up to 300 MB/s (2 400 Mbps)
  • Improvements: using a 2 CPU (4 vcore) and 16 GB RAM laptop with 2 instances of Waarp R66 on it using TLS
    • v3.0 ~ 30 transfers/s 100% CPU,
    • V3.2 ~60 t/s 100% CPU,
    • v3.5.2 ~71 t/s 100% CPU,
    • v3.6.0 ~ 100 t/s 90% CPU
    • on a 4 CPU (8 vcore), 2 instances: v3.6.0 ~ 127 t/s 30% CPU

Gateway FTP Server benchmark

  • 8 threads (4 core), 16 GB Memory
    • Single FTP client
    • Using Waarp FTP client (based on FTP4J): with 1KB file
      • Passive: 77 transfers/s
      • Active: 118 transfers/s
    • Same with 100 concurrent clients
      • Mixte Passive/active: 4200 transfers/s

IMPORTANT: DEB, RPM and other distribution (except JAR) are not yet official.

Assets 18
Loading