Hi,
Im trying to find best solution for parallel file transfer, which is
event driven after the file written to a directory with extension
like .done, means the file is ready for file transfer.
1. Monitoring the directory for .done is cpu cycle time waster
2. running a cron on particular interval to transfer files will
not help fasteness to the data.
what are other options existing for file transfer.
Im trying to find best solution for parallel file transfer, which is
event driven after the file written to a directory with extension
like .done, means the file is ready for file transfer.
1. Monitoring the directory for .done is cpu cycle time waster
2. running a cron on particular interval to transfer files will not
help fasteness to the data.
On 2020-06-14, t.sathyanarayanan@gmail.com <t.sathyanarayanan@gmail.com> wrote:
Im trying to find best solution for parallel file transfer, which is
event driven after the file written to a directory with extension
like .done, means the file is ready for file transfer.
1. Monitoring the directory for .done is cpu cycle time waster
2. running a cron on particular interval to transfer files will not
help fasteness to the data.
https://man7.org/linux/man-pages/man7/inotify.7.html https://en.wikipedia.org/wiki/Inotify
On Sun, 2020-06-14, t.sathyanarayanan@gmail.com wrote:
Hi,
Im trying to find best solution for parallel file transfer, which is
event driven after the file written to a directory with extension
like .done, means the file is ready for file transfer.
1. Monitoring the directory for .done is cpu cycle time waster
2. running a cron on particular interval to transfer files will
not help fasteness to the data.
I cannot understand what you want to do, or how you want to do it;
you would have to explain better.
what are other options existing for file transfer.
Personally I always use rsync over ssh. Unless I download files over
http, then I use firefox or curl. Bittorrent is another option.
/Jorgen
--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
On Sunday, 14 June 2020 20:09:24 UTC+5:30, Jorgen Grahn wrote:...
On Sun, 2020-06-14, t.sathyanarayanan@gmail.com wrote:
Hi,
Im trying to find best solution for parallel file transfer, which is
event driven after the file written to a directory with extension
like .done, means the file is ready for file transfer.
1. Monitoring the directory for .done is cpu cycle time waster
2. running a cron on particular interval to transfer files will
not help fasteness to the data.
I cannot understand what you want to do, or how you want to do it;
you would have to explain better.
what are other options existing for file transfer.
Personally I always use rsync over ssh. Unless I download files over
http, then I use firefox or curl. Bittorrent is another option.
we have a distrubuted systems which collects peta bytes of data and
deliver the data to netstorage, the problem we have is transmission
of files at faster rate, the one problem we sees as soon as the file
comes in, we are not delivering the data, we have to wait for cron
to run and pick those files and to deliver netstorage, the problem
in cron is we schedule a time to run but practically this will lead
to big data backlog, my question here is there any event based for
monitoring file system to initiate the job as soons as files that
comes into the directory, like (epoll mechanism)
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 296 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 58:32:24 |
| Calls: | 6,652 |
| Calls today: | 4 |
| Files: | 12,200 |
| Messages: | 5,331,145 |