Advanced Replication White Paper
This paper is meant to address installations of FRP where there is a need to increase maximum throughput or if the installation will match or exceed any of the following situations.
- Files >100k
- Data >1TB
- Servers >10
- Replication Jobs >10
- Use of Encryption on multiple jobs
- Use of Compression on multiple jobs
- Adjustments to reduce the Dsync Bit level to < 10mb files
- Real Time Replication on Multiple Jobs
- Very Long File/Path names
- A very deep folder tree
The adjustments to FRP for these conditions cover two primary areas which will affect the throughput limits set by default in FRP as well as the amount of memory assigned to use by FRP.
All of these considerations must be balanced with the power of your hardware’s CPU, Throughput IO operations, and installed Memory. Additionally you must consider what other (if any) applications and uses are being run on the server that may impact FRP or that FRP may impact.
Tuning FRP for Maximum Throughput
Resource Manager can either limit or increase the number of CPU/memory intensive tasks running concurrently in FRP.
Please note in most cases you will also need to allocate more memory to FRP for this option as well. For details on increasing the memory allocation to FRP see the section below on Memory Allocated to FRP.
Overview:
The parameters can be set in the FRP s2s.properties file. The file is located in the /FileReplicationPro/lib directory. By default, the value for each of the parameters is set to 20. Please note the 2s.properties file does not include the parameters listed below. They should be added to the file for resource management customization for your environment. It is our recommendation to first test FRP with the default settings prior to changing the default settings.
The parameters that can be set are as follows:
s2s.max_concurrent_jobs: Defines the number of jobs allowed to run concurrently. Our recommended starting point when testing new parameters in limited environments is 40 for each value below.
s2s.max_concurrent_compressions: Sets the number of simultaneous compression/decompression threads.
s2s.max_concurrent_transfers: Defines the number of concurrent files that will be replicated. This parameter does not limit the number of replications within a specific job. It does limit the number of concurrent replications occurring at one time in different jobs.
s2s.max_concurrent_dsync: Sets the max number of concurrent dsync computations (for byte level differential replication).
Regarding bandwidth under-use issues - it is possible that FRP shares bandwidth with other network applications that consume a large amount of bandwidth. Another limiting factor may be due to system load and/or Java I/O operations.
Below is a sample you can cut and paste to your s2s.properties file to use as a starting point, you many need to adjust these up or down to get best performance. To restore default settings, remove these settings from your s2s.properties files.
These properties should be placed into any machine that is involved in a particularly intensive workload. The FRPrep service will need to be restarted after each change.
#--Try these settings and if replication is too slow make small changes to the items and retest until a balance of performance and stability is found.
s2s.max_concurrent_jobs=40
s2s.max_concurrent_compressions=40
s2s.max_concurrent_transfers=40
s2s.max_concurrent_dsync=40
Performance Tweaks
Two adjustments to your s2s.properties file on all of your machines should be considered. If you do not need real time replicaiton turning off this feature can increase throughput.
On ALL FRP servers open the file \FileReplicationPro\lib\s2s.properties with a plain text editor.
Change the following settings
s2s.buffer.size=64000 (change to 2x or 3x value)
s2s.realtime=true (change to false)
Save changes
Stop and restart the FRPrep service on each server.
Increased Memory Allocation
Users who need to replicate a very large number of files per directory or have a setup of multiple and frequent jobs should consider changing the default memory allocation for FRP. Symptoms that can result from too little memory assigned to FRP processes are: Excessive time to complete jobs causing FRP to run beyond the scheduled window for the job, unexplained job aborts, or FRP services that stop responding to the job schedule.
FRP memory consumption is highly effected by several factors. Some those factors are:
- The number of files that exist in a directory subject to replication IE: 10,000 or more.
- A large number of path lengths in excess of 128 char.
- The number of jobs being run in total and in parallel.
- Compression
- Encryption
- CRC checking
- Real Time Replication
It should also be obvious that if you are restricted in how much memory you can assign to FRP, reducing the use of the items above will reduce FRP's need for memory.
Any of these issues or a combination of them can cause FRP to suffer from insufficient memory, therefore it is highly recommended to change the default memory usage. This is necessary only for the FRP Replication Service and not for the other two services: FRP Management and FRP Helper.
During installation, the FRP replication service is configured to use a maximum of 256MB for versions thru 5.1.13 all versions after have 512MB set as default. FRP is very efficient and does not require massive amounts of memory to accomplish a great deal of work. In most cases with only a few jobs and not too many other memory consuming demands, FRP does not need additional memory. The threshold for increasing memory to 512MB or 1GB (or something else) is very individual to your environment and the demands you place on FRP. As you can see from this process it is not difficult to make this change and you can reverse it just as easily.
Our examples below for each OS assume a modern machine with plenty of ram 4 to 8 GB or more. Examples are provided for increasing FRP usage to 512MB or 1GB. To give you some idea of the impact of this change, we have seen Networks of 50 or more servers replicating very nicely on scheduled jobs with 1GB of ram assigned to FRP.
Be aware that you must consider making these changes on all servers doing replication. It takes just a much in resources to prepare the files for replication as it does to receive them and process them to the hard drive on the destination server.
For the complete memory article with step by step instructions see the article Increase Memory Allocation
--------------------------------------------------