Actions

TMG:Automatic Text CDR Retrieval: Difference between revisions

No edit summary
(Removed install section from this page - it is always included now)
 
(11 intermediate revisions by 5 users not shown)
Line 1: Line 1:
This procedure shows how to configure a Tmedia to send its CDR files to a remote server automatically and perform daily (or monthly) file archives.
{{DISPLAYTITLE:Automatic Text CDR Retrieval}}
 
This procedure describes how to configure a Tmedia and [[ProSBC]] to send its Call Detail Records (CDR) files to a remote server automatically and perform daily (or monthly) file archives.


= Overview =
= Overview =
The ''tbcopycdr'' script support the following features:
The ''tbcopycdr'' script supports the following features:
*Move cdr files from TMG_CDR_DIR to TMG_TEMP_DIR
*Move cdr files from TMG_CDR_DIR to TMG_TEMP_DIR
*Send files to remote server
*Send files to a remote server
*Move sucessfully sent files (rename to .sent.gz) from TMG_TEMP_DIR to TMG_ARCHIVE_DIR (ARCHIVE=true)
*Move sucessfully sent files (rename to .sent.gz) from TMG_TEMP_DIR to TMG_ARCHIVE_DIR (ARCHIVE=true)
*Rename sucessfully sent files (.sent.gz) in TMG_TEMP_DIR (ARCHIVE=false; ERASE_AFTER_SEND=false)
*Rename sucessfully sent files (.sent.gz) in TMG_TEMP_DIR (ARCHIVE=false; ERASE_AFTER_SEND=false)
*Delete sucessfully sent files from TMG_TEMP_DIR (ARCHIVE=false; ERASE_AFTER_SEND=true)
*Delete sucessfully sent files from TMG_TEMP_DIR (ARCHIVE=false; ERASE_AFTER_SEND=true)
*If a file failed to be send to remote server, it is kept in TMG_TEMP_DIR until it is successfully sent
*If a file failed to be sent to remote server, it is kept in TMG_TEMP_DIR until it is successfully sent
<br>
<br>
The ''tbarchivecdr'' script create a gzip tar archive with all gz files copied by ''tbcopycdr'' in TMG_ARCHIVE_DIR
The ''tbarchivecdr'' script creates a gzip tar archive with all gz files copied by ''tbcopycdr'' in TMG_ARCHIVE_DIR
*Change directory to TMG_ARCHIVE_DIR
*Change directory to TMG_ARCHIVE_DIR
*Create a temporary folder with the current date (ie. 2016-01-30)
*Create a temporary folder with the current date (for example: 2016-01-30)
*Sleep for 90 second
*Sleep for 90 second
*Move all *.gz files and $TMG_TEMP_DIR/$LOG_FILE to this folder
*Move all *.gz files and $TMG_TEMP_DIR/$LOG_FILE to this folder
*Archive the folder in a gzipped tar file (cdr_2016_01_30_$hostname.tgz)
*Archive the folder in a gzipped tar file (cdr_2016_01_30_$hostname.tgz)
= Install =
== From repository ==
The ''tbcopycdr' and ''tbarchivecdr'' scripst is included in the ''tb_tools'' packages 1.2-32.el7 (CentOS7) and 1.2-32.el5 (CentOS5).
#yum install tb_tools
#yum list tb_tools
tb_tools.noarch        '''1.2-32'''.el7.tb
== Manually ==
If you don't have access to the repository from the Tmedia or the tb_tools package not yet updated on TB repository, you can manual install the script.<br>
1. Get the script here: [[Media:tbcopycdr.gz|tbcopycdr.gz [click to download]]]
2. Copy the scrips to Tmedia (using a sftp client)
gunzip tbcopycdr.gz
tar xf tbcopycdr.tar
cp tbcopycdr/tbcopycdr /usr/bin/.
cp tbcopycdr/tbarchivecdr /usr/bin/.
chmod +x /usr/bin/tbcopycdr /usr/bin/tbarchivecdr
rm -fr tbcopycdr


= Configuration =
= Configuration =
Line 70: Line 53:
  CONNECT_TIMEOUT=10
  CONNECT_TIMEOUT=10


*''ARCHIVE'': true or false. If set to true, move sucessfully sent files (rename to .sent.gz) from TMG_TEMP_DIR to TMG_ARCHIVE_DIR. Default is false.
*''ARCHIVE'': true or false. If set to true, move successfully sent files (rename to .sent.gz) from TMG_TEMP_DIR to TMG_ARCHIVE_DIR. Default is false.
  ARCHIVE=false
  ARCHIVE=false
*''TMG_ARCHIVE_DIR'': Directory where the cdrs are moved if ARCHIVE=true
*''TMG_ARCHIVE_DIR'': Directory where the cdrs are moved if ARCHIVE=true
  TMG_ARCHIVE_DIR=/lib/tb/toolpack/setup/12358/cdr_archive
  TMG_ARCHIVE_DIR=/lib/tb/toolpack/setup/12358/cdr_archive
*''ARCHIVE_MAX_DAYS'': Define how many days archive file (.tgz) should be kept in TMG_ARCHIVE_DIR. This user variable is used by ''tbarchivecdr'' only.
*''ARCHIVE_MAX_DAYS'': Define how many days the archive file (.tgz) should be kept in TMG_ARCHIVE_DIR. This user variable is used by ''tbarchivecdr'' only.
  # Assign no value to disable
  # Assign no value to disable
  ARCHIVE_MAX_DAYS=
  ARCHIVE_MAX_DAYS=
Line 81: Line 64:
  ERASE_AFTER_SEND=false
  ERASE_AFTER_SEND=false


Note: if you set ERASE_AFTER_SEND to false the CDR text file will be available in the /home/telcobridges/cdr/ directory and rename *.gz.already_send.
Note: If you set ERASE_AFTER_SEND to false, the CDR text file will be available in the /home/telcobridges/cdr/ directory and renamed *.gz.already_send.


= Remote Server Copy =
= Remote Server Copy =
You must create an SSH key pair and copy it to the remote server, if the remote server copy is configured. This will enable the Tmedia to copy the backup files to the remote host without asking for a password:
You must create an SSH key pair and copy it to the remote server, if the remote server copy is configured. This will enable copy the backup files to the remote host without asking for a password. Follow the instructions here:  
 
* [[Password_less_ssh|Password-less SSH]]
*Connect to the Tmedia using SSH as root
 
*Create Tmedia public/private dsa key pair
ssh-keygen -t rsa
Note: Do not add a pass phrase, simply press enter until the key pair is generate.
*Append the content of the public key to the remote host authorized keys
 
Tmedia public key location
/root/.ssh/id_rsa.pub
 
Remote Host authorized keys location
~/.ssh/authorized_keys
 
More details here: [[Password_less_ssh|Password-less SSH]]


=Troubleshooting =
=Troubleshooting =
Line 107: Line 76:
  tbcopycdr -v
  tbcopycdr -v


If no error were seen on the shell, your configuration is correct. Also, you should have a hello_world.log file on the remote host.
If no errors are seen in the shell, your configuration is correct. In addition, you should have a hello_world.log file on the remote host.


== Log File ==
== Log File ==
Line 116: Line 85:
*Create a new crontab on the [[Tmedia]]
*Create a new crontab on the [[Tmedia]]
  crontab -e
  crontab -e
*Add new line with the cron configuration
*Add new lines with the cron configuration


Example:
'''''tbcopycdr''''':
{| cellpadding="5" border="1" class="wikitable"
{| cellpadding="5" border="1" class="wikitable"
|-
|-
Line 132: Line 101:
| width="210" style="background: none repeat scroll 0% 0% rgb(255, 255, 255); text-align:center;" | daily (midnight)
| width="210" style="background: none repeat scroll 0% 0% rgb(255, 255, 255); text-align:center;" | daily (midnight)
| width="400" style="background: none repeat scroll 0% 0% rgb(255, 255, 255);" | <code>0 0 * * * /usr/bin/tbcopycdr > /dev/null 2>&1</code>
| width="400" style="background: none repeat scroll 0% 0% rgb(255, 255, 255);" | <code>0 0 * * * /usr/bin/tbcopycdr > /dev/null 2>&1</code>
|}
'''''tbarchivecdr''''':
{| cellpadding="5" border="1" class="wikitable"
|-
! width="210" style="background: none repeat scroll 0% 0% rgb(255, 255, 255);" | Frequency
! width="400" style="background: none repeat scroll 0% 0% rgb(255, 255, 255);" | Cron Configuration
|-
| width="210" style="background: none repeat scroll 0% 0% rgb(255, 255, 255); text-align:center;" | daily@23h59
| width="400" style="background: none repeat scroll 0% 0% rgb(255, 255, 255);" | <code>59 23 * * * /usr/bin/tbarchivecdr > /dev/null 2>&1</code>
| width="400" style="background: none repeat scroll 0% 0% rgb(255, 255, 255);" | <code>59 23 * * * /usr/bin/tbarchivecdr > /dev/null 2>&1</code>
|-
| width="210" style="background: none repeat scroll 0% 0% rgb(255, 255, 255); text-align:center;" | weekly (sunday@23h59)
| width="400" style="background: none repeat scroll 0% 0% rgb(255, 255, 255);" | <code>59 23 * 7 * /usr/bin/tbarchivecdr > /dev/null 2>&1</code>
|}
|}
Cron table example for a cdr copy to an external server every 10 minutes with a daily archive of all the transmitted files:
*/10 * * * * /usr/bin/tbcopycdr > /dev/null 2>&1
59 23 * * * /usr/bin/tbarchivecdr > /dev/null 2>&1

Latest revision as of 15:16, 13 June 2024


This procedure describes how to configure a Tmedia and ProSBC to send its Call Detail Records (CDR) files to a remote server automatically and perform daily (or monthly) file archives.

Overview

The tbcopycdr script supports the following features:

  • Move cdr files from TMG_CDR_DIR to TMG_TEMP_DIR
  • Send files to a remote server
  • Move sucessfully sent files (rename to .sent.gz) from TMG_TEMP_DIR to TMG_ARCHIVE_DIR (ARCHIVE=true)
  • Rename sucessfully sent files (.sent.gz) in TMG_TEMP_DIR (ARCHIVE=false; ERASE_AFTER_SEND=false)
  • Delete sucessfully sent files from TMG_TEMP_DIR (ARCHIVE=false; ERASE_AFTER_SEND=true)
  • If a file failed to be sent to remote server, it is kept in TMG_TEMP_DIR until it is successfully sent


The tbarchivecdr script creates a gzip tar archive with all gz files copied by tbcopycdr in TMG_ARCHIVE_DIR

  • Change directory to TMG_ARCHIVE_DIR
  • Create a temporary folder with the current date (for example: 2016-01-30)
  • Sleep for 90 second
  • Move all *.gz files and $TMG_TEMP_DIR/$LOG_FILE to this folder
  • Archive the folder in a gzipped tar file (cdr_2016_01_30_$hostname.tgz)

Configuration

The script is configured with user variables.

  • CONFIG_FILE: Configuration file name and location. By default, it is set to /root/tbcopycdr.cfg
CONFIG_FILE=/root/tbcopycdr.cfg

tbcopycdr.cfg file example: 

TMG_TEMP_DIR=/home/tbcopycdr
TMG_CDR_DIR=/lib/tb/toolpack/setup/12358/cdr_logs
ARCHIVE=true
TMG_ARCHIVE_DIR=/lib/tb/toolpack/setup/12358/cdr_archive
ARCHIVE_MAX_DAYS=180
REMOTE_HOST=192.168.101.170
REMOTE_PORT=22
REMOTE_DIR=/root/cdr_backups
REMOTE_USER=root

The configuration file can redefine any of the following user variables. To see the default variables: 

head -n 50 /usr/bin/tbcopycdr
  • TMG_CDR_DIR: Tmedia directory where the CDR are located. Default is /lib/tb/toolpack/setup/12358/cdr_logs
TMG_CDR_DIR=/lib/tb/toolpack/setup/12358/cdr_logs
  • REMOTE_HOST : Remote hostname or Ip address. Remote host copy is disabled by default.
REMOTE_HOST=10.10.10.10
  • REMOTE_PORT : Remote host ssh port.
REMOTE_PORT=22
  • REMOTE_DIR : Remote directory where to copy the backup files (/root/backup by default)
REMOTE_DIR=/root/backup
  • REMOTE_USER : Remote host login user.
REMOTE_USER=root
  • CONNECT_TIMEOUT : scp connection timeout
CONNECT_TIMEOUT=10
  • ARCHIVE: true or false. If set to true, move successfully sent files (rename to .sent.gz) from TMG_TEMP_DIR to TMG_ARCHIVE_DIR. Default is false.
ARCHIVE=false
  • TMG_ARCHIVE_DIR: Directory where the cdrs are moved if ARCHIVE=true
TMG_ARCHIVE_DIR=/lib/tb/toolpack/setup/12358/cdr_archive
  • ARCHIVE_MAX_DAYS: Define how many days the archive file (.tgz) should be kept in TMG_ARCHIVE_DIR. This user variable is used by tbarchivecdr only.
# Assign no value to disable
ARCHIVE_MAX_DAYS=
  • ERASE_AFTER_SEND: Take effect only if ARCHIVE is false. If ERASE_AFTER_SEND is true, erase file from TMG_TEMP_DIR, else, rename and keep files in working directory.
ERASE_AFTER_SEND=false

Note: If you set ERASE_AFTER_SEND to false, the CDR text file will be available in the /home/telcobridges/cdr/ directory and renamed *.gz.already_send.

Remote Server Copy

You must create an SSH key pair and copy it to the remote server, if the remote server copy is configured. This will enable copy the backup files to the remote host without asking for a password. Follow the instructions here:

Troubleshooting

Test

  • Run the tbcopycdr script with the -v option
tbcopycdr -v

If no errors are seen in the shell, your configuration is correct. In addition, you should have a hello_world.log file on the remote host.

Log File

Transfer results are written in $TMG_TEMP_DIR/$LOG_FILE

Schedule the Scripts

  • Create a new crontab on the Tmedia
crontab -e
  • Add new lines with the cron configuration

tbcopycdr:

Frequency Cron Configuration
Every 10 minutes */10 * * * * /usr/bin/tbcopycdr > /dev/null 2>&1
hourly (top of the hour) 0 * * * * /usr/bin/tbcopycdr > /dev/null 2>&1
daily (midnight) 0 0 * * * /usr/bin/tbcopycdr > /dev/null 2>&1

tbarchivecdr:

Frequency Cron Configuration
daily@23h59 59 23 * * * /usr/bin/tbarchivecdr > /dev/null 2>&1
weekly (sunday@23h59) 59 23 * 7 * /usr/bin/tbarchivecdr > /dev/null 2>&1

Cron table example for a cdr copy to an external server every 10 minutes with a daily archive of all the transmitted files: 

*/10 * * * * /usr/bin/tbcopycdr > /dev/null 2>&1
59 23 * * * /usr/bin/tbarchivecdr > /dev/null 2>&1
Retrieved from "https://docs.telcobridges.com/w/index.php?title=TMG:Automatic_Text_CDR_Retrieval&oldid=33785"