Difference between revisions of "Remotemanager-call"

From GreenVulcano Wiki
Jump to: navigation, search
(→‎{{VULCON}} / {{GVCONSOLE}} Configuration)
 
(10 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
==Description==
 
==Description==
  
−
This {{GVESB}} plug-in allows you to perform a sequence of file operations in a remote file system accessible via an (S)FTP account.
+
This {{GVESB}} plug-in allows you to perform a sequence of file operations in a remote file system accessible via various protocols.
  
 
=={{VULCON}} / {{GVCONSOLE}} Configuration==
 
=={{VULCON}} / {{GVCONSOLE}} Configuration==
Line 27: Line 27:
  
 
* Description
 
* Description
−
* [[FTPManager]] or [[SFTPManager]]
+
* [[FTPManager]] | [[FTPSManager]] | [[SFTPManager]]
 +
<div class="version_ge3.4.0.10">
 +
| [[HDFSRemoteManager]]
 +
</div>
 
* RemoteCommands
 
* RemoteCommands
  
Line 35: Line 38:
 
sequence.
 
sequence.
  
−
The RemoteCommands Element is used by [[remotemanager-call]] and internally, it is possible to configure the following operations:
+
The RemoteCommands Element is used by [[remotemanager-call]] and, internally, it is possible to configure the following operations:
−
* RemoteSearchCommand  
+
{|class="gvtable"
−
* RemoteMoveCommand  
+
! Remote Command !! Use this for...
−
* RemoteDeleteCommand  
+
|-
−
* RemoteDownloadCommand
+
| [[Remotemanager-call#RemoteSearchCommand | RemoteSearchCommand]] || Controlling one or more files (whose name corresponds to a regular expressions) in a remote directory
−
* RemoteUploadCommand
+
|-
 +
| [[Remotemanager-call#RemoteMoveCommand |RemoteMoveCommand]] || Renaming/moving a file/directory in a remote directory
 +
|-
 +
| [[Remotemanager-call#RemoteDeleteCommand |RemoteDeleteCommand]] || Deleting a file/directory from a remote directory
 +
|-
 +
| [[Remotemanager-call#RemoteDownloadCommand |RemoteDownloadCommand]] || Downloading a file/directory from a remote directory to a local directory
 +
|-
 +
| [[Remotemanager-call#RemoteUploadCommand|RemoteUploadCommand]] || Uploading a file/directory from a local directory to a remote directory
 +
|}
 +
 
 +
The remote commands can have the following subelements:
 +
* Description
 +
<div class="version_ge3.4.0.10">
 +
* [[PropertyDef]]
 +
</div>
 +
 
 +
====RemoteSearchCommand====
 +
 
 +
The {{GVESB}} operation '''RemoteSearchCommand''' checks if one or more files, whose name conforms to a given regular expression, exists within the specified directory.
 +
 
 +
On completion and in the absence of errors the following properties of output GVBuffer are setted:
 +
# GVRM_FIND_FILES_NUM Number of files found (>= 0)
 +
# GVRM_FIND_FILES_LIST List of names of found files, separated by ','. Set only if the value of field 'GVRM_FIND_FILES_NUM' is greater than 0. <br/> Both the pattern of the file and that the directory can be static or contain [[placeholders]] that are replaced at runtime.
  
−
The following table shows its details:
+
The following table shows its attributes:
  
 
{|class="gvtable"
 
{|class="gvtable"
−
! Operation !! Use this for... !! Attributes
+
! Attribute !! Type !! Description
 +
|-
 +
| type || fixed || This attribute must assume the value '''remote-command'''.
 +
|-
 +
| class || fixed || This attribute must assume the value '''it.greenvulcano.gvesb.virtual.file.remote.command.GVSearch'''.
 +
|-
 +
| sourcePath || required || Absolute path of the directory to search in.
 +
|-
 +
| filePattern || required || File name to search for. Can be a regular expression.
 
|-
 
|-
−
| RemoteSearchCommand || Checking if one or more files, whose name conforms to a given regular expression, exists within the specified directory. <br/> On completion and in the absence of errors the following properties of output GVBuffer are setted:
+
| isCritical || optional || If true an internal error interrupt the command sequence. Default true.
−
# GVRM_FIND_FILES_NUM Number of files found (>= 0)
+
|}
−
# GVRM_FIND_FILES_LIST List of names of found files, separated by ','.
+
 
−
Set only if the value of field 'GVRM_FIND_FILES_NUM' is greater than 0. <br/> Both the pattern of the file and that the directory can be static or contain [[placeholders]] that are replaced at runtime.
+
====RemoteMoveCommand====
−
|| ''type'': remote-command.  <br/> ''class'': it.greenvulcano.gvesb.virtual.file.remote.command.GVSearch.  <br/> ''sourcePath'': Absolute path of the directory to search in. <br/> ''filePattern'': File name to search for. Can be a regular espression.  <br/> ''isCritical'': If true an internal error interrupt the command sequence. Default true.
+
 
 +
The {{GVESB}} operation '''RemoteMoveCommand''' moves/renames a file/directory.
 +
 
 +
The file name, the source and destination directory names can be static or contain [[placeholders]] that are replaced at runtime. After the replacement of the [[placeholders|placeholder]], the value of the ''filename'' may contain a regular expression, in which case, all files of the source directory, whose name is in accordance with the corresponding pattern, will be moved to the destination directory. If the destination directory already contains files with the same name, they will be overwritten.
 +
 
 +
The following table shows its attributes:
 +
 
 +
{|class="gvtable"
 +
! Attribute !! Type !! Description
 +
|-
 +
| type || fixed || This attribute must assume the value '''remote-command'''.
 +
|-
 +
| class || fixed || This attribute must assume the value '''it.greenvulcano.gvesb.virtual.file.remote.command.GVMove'''.
 +
|-
 +
| targetPath || required || Absolute path of the file/directory where is the file to move.
 +
|-
 +
| oldName || required || Name of the file to be moved.
 +
|-
 +
| newName || required || Name of the file moved.
 +
|-
 +
| isCritical || optional || If true an internal error interrupt the command sequence. Default true.
 +
|}
 +
 
 +
<div class="version_ge3.4.0.10">
 +
The following table shows the PropertyDef possible elements that can be set when the command is executed on the [[HDFSRemoteManager|HDFS]] file system:
 +
{|class="gvtable"
 +
! Property !! Possible Values
 +
|-
 +
| dfs.file.overwrite || true/false; default=false
 +
|}
 +
This parameter can be used to owerwrite the target destination with the source one when a move operation is used on the HDFS file system.
 +
 
 +
====RemoteDeleteCommand====
 +
 
 +
The {{GVESB}} operation '''RemoteDeleteCommand''' deletes a file or directory.
 +
 
 +
Both the file name and the directory can be static or contain [[placeholders]] that are replaced at runtime. After the replacement of the [[placeholders|placeholder]], the value of the ''filename'' may contain a regular expression, in which case, will erase all files in the specified directory whose name is consistent with the corresponding pattern.
 +
 
 +
The following table shows its attributes:
 +
 
 +
{|class="gvtable"
 +
! Attribute !! Type !! Description
 +
|-
 +
| type || fixed || This attribute must assume the value '''remote-command'''.
 +
|-
 +
| class || fixed || This attribute must assume the value '''it.greenvulcano.gvesb.virtual.file.remote.command.GVDelete'''
 +
|-
 +
| targetPath || required || Absolute path of the file/directory to remove.
 +
|-
 +
| filePattern || optional || File name to remove. Might contain a regular expression. It is used only if ''targetPath'' is a directory, and if empty the filter is ignored.
 +
|-
 +
| isCritical || optional || If true an internal error interrupt the command sequence. Default true.
 +
|}
 +
 
 +
====RemoteDownloadCommand====
 +
 
 +
The {{GVESB}} operation '''RemoteDownloadCommand''' downloads a file or directory from a remote directory and saved in the local directory you specified (or in GVBuffer).
 +
 
 +
File names and directories can be static or contain [[placeholders]] that are replaced at runtime.
 +
 
 +
The following table shows its attributes:
 +
 
 +
{|class="gvtable"
 +
! Attribute !! Type !! Description
 +
|-
 +
| type || fixed || This attribute must assume the value '''remote-command'''.
 +
|-
 +
| class || fixed || This attribute must assume the value '''it.greenvulcano.gvesb.virtual.file.remote.command.GVDownload'''.
 +
|-
 +
| remotePath || required || Pathname of remote directory containing the file to download. It can be an absolute pathname or relative, in the second case it is relative to home directory of ftp user.
 +
|-
 +
| remoteFilePattern || required || The name of the file to download.
 +
|-
 +
| toGVBufferExpression || optional || GVBuffer OGNL expression used to specify how to handle the current downloaded file. In the expression, the downloaded file will be referenced by the variable #input, and will be a byte [].
 +
|-
 +
| targetPath || optional || Pathname of the local directory to save the downloaded file. Must be an absolute pathname.
 +
|-
 +
| isCritical || optional || If true an internal error interrupt the command sequence. Default true.
 +
|}
 +
 
 +
<div class="version_ge3.4.0.10">
 +
The following table shows the PropertyDef possible elements that can be set when the command is executed on the [[HDFSRemoteManager|HDFS]] file system:
 +
{|class="gvtable"
 +
! Property !! Possible Values
 +
|-
 +
| dfs.file.delete-source || fixed=false
 +
|-
 +
| dfs.file.raw-local-filesystem || fixed=true
 +
|-
 +
| dfs.stream-buffer-size || default: 4096
 +
|}
 +
These parameters can be used to specify the related action when a download operation is used on the HDFS file system.
 +
 
 +
====RemoteUploadCommand====
 +
 
 +
The {{GVESB}} operation '''RemoteUploadCommand''' uploads a file from a local directory (or [[GVBuffer]]) to the remote directory specified.
 +
 
 +
File and/or directory names can be static or contain [[placeholders]] that are replaced at runtime.
 +
 
 +
The following table shows its attributes:
 +
 
 +
{|class="gvtable"
 +
! Attribute !! Type !! Description
 +
|-
 +
| type || fixed || This attribute must assume the value '''remote-command'''.
 +
|-
 +
| class || fixed || This attribute must assume the value '''it.greenvulcano.gvesb.virtual.file.remote.command.GVUpload'''.
 +
|-
 +
| sourcePath || optional || Pathname of the local directory where the file is transferred. Must be an absolute pathname.
 +
|-
 +
| sourceFilePattern || optional || The name of the file to transfer.
 +
|-
 +
| fromGVBufferExpression || optional || OGNL Expression  applied to the current [[GVBuffer]] that specifies how to build the file content. In the expression, the [[GVBuffer]] is associated with #object, and must return a byte[].
 +
|-
 +
| remotePath || required || Pathname of remote directory where upload the file. It can be an absolute pathname or relative, in the second case it is relative to home directory of ftp user.
 +
|-
 +
| isCritical || optional || If true an internal error interrupt the command sequence. Default true.
 +
|}
 +
 
 +
<div class="version_ge3.4.0.10">
 +
The following table shows the PropertyDef possible elements that can be set when the command is executed on the [[HDFSRemoteManager|HDFS]] file system:
 +
{|class="gvtable"
 +
! Property !! Possible Values
 +
|-
 +
| dfs.file.delete-source || true/false; default=false
 +
|-
 +
| dfs.file.overwrite || true/false; default=false
 +
|-
 +
| dfs.permissions.path || Unix-like permission Example: 777; default 700
 +
|-
 +
| dfs.stream-buffer-size || default: 8192
 
|-
 
|-
−
| RemoteMoveCommand || Moving/renaming a file/directory. <br/> The file name, the source and destination directory names can be static or contain [[placeholders]] that are replaced at runtime. <br/> After the replacement of the [[placeholders|placeholder]], the value of the ''filename'' may contain a regular expression, in which case, all files of the source directory, whose name is in accordance with the corresponding pattern will be moved to the destination directory. <br/> If the destination directory already contains files with the same name, they will be overwritten.
+
| dfs.replication || default: 1
−
|| ''type'': remote-command. <br/> ''class'': it.greenvulcano.gvesb.virtual.file.remote.command.GVMove. <br/> ''targetPath'': Absolute path of the file/directory where is the file to move. <br/> ''oldName'': File name to move. <br/> ''newName'': File name moved. <br/> ''isCritical'': If true an internal error interrupt the command sequence. Default true.
 
 
|-
 
|-
−
| RemoteDeleteCommand || Deleting a file/directory. <br/> Both the file name and the directory can be static or contain [[placeholders]] that are replaced at runtime. <br/> After the replacement of the [[placeholders|placeholder]], the value of the ''filename'' may contain a regular expression, in which case, will erase all files in the specified directory whose name is consistent with the corresponding pattern. || ''type'': remote-command. <br/> ''class'': it.greenvulcano.gvesb.virtual.file.remote.command.GVDelete.<br/> ''targetPath'': Absolute path of the file/directory to remove.<br/> ''filePattern'': File name to remove. <br/> Can be a regular espression. It is used only if ''targetPath'' is a directory, and if empty the filter is ignored.<br/> ''isCritical'': If true an internal error interrupt the command sequence. Default true.
+
| dfs.blocksize || default: 4194304
 
|-
 
|-
−
| RemoteDownloadCommand || Downloading file/directory from remote directory and saved in the local directory you specified (or in GVBuffer). <br/> File names and directories can be static or contain [[placeholders]] that are replaced at runtime.
+
| dfs.permissions.user || hdfs user owner
−
|| ''type'': remote-command.<br/> ''class'': it.greenvulcano.gvesb.virtual.file.remote.command.GVDownload. <br/> ''remotePath'': Pathname of remote directory containing the file to download. It can be an absolute pathname or relative, in the second case it is relative to home directory of ftp user.<br/> ''remoteFilePattern'': The name of the file to download.<br/> ''toGVBufferExpression'': GVBuffer OGNL expression used to specify how to handle the current downloaded file. In the expression, the downloaded file will be referenced by the variable #input, and will be a byte [].<br/>  ''targetPath'': Pathname of the local directory to save the downloaded file. Must be an absolute pathname. <br/> ''isCritical'': If true an internal error interrupt the command sequence. Default true.
 
 
|-
 
|-
−
| RemoteUploadCommand || Upload a file from local directory (or GVBuffer) to the remote directory specified. <br/> File names and directories can be static or contain [[placeholders]] that are replaced at runtime.
+
| dfs.permissions.group || hdfs user group
−
|| ''type'': remote-command. <br/> ''class'': it.greenvulcano.gvesb.virtual.file.remote.command.GVUpload. <br/> ''sourcePath'': Pathname of the local directory where the file is transferred. Must be an absolute pathname. <br/> ''sourceFilePattern'': The name of the file to transfer. <br/> ''fromGVBufferExpression'': Expression OGNL apply to current GVBuffer that specifies how to building the file content. In the expression, the GVBuffer is associated with #object, and must return a byte[]. <br/> ''remotePath'': Pathname of remote directory where upload the file. It can be an absolute pathname or relative, in the second case it is relative to home directory of ftp user.<br/> ''isCritical'': If true an internal error interrupt the command sequence. Default true.
 
 
|}
 
|}
 +
These parameters can be used to owerwrite the target destination with the source one, set the replication, permissions and the other properties on destination target when an upload operation is used on the HDFS file system.

Latest revision as of 18:57, 20 November 2015

Description

This GreenVulcano® ESB plug-in allows you to perform a sequence of file operations in a remote file system accessible via various protocols.

VulCon / GV Console Configuration

The operations allowed are:

  • Download a file/directory from a remote directory to a local directory
  • Upload a file/directory from a local directory to a remote directory
  • Control of one or more files (whose name corresponds to a date regular expressions) in a remote directory
  • Deleting a file/directory from a remote directory
  • Rename/move a file/directory in a remote directory


The remotemanager-call Element is used by Channel and routed-call. It has the attributes:

Attribute Type Description
type fixed This attribute must assume the value call.
class fixed This attribute must assume the value it.greenvulcano.gvesb.virtual.file.remote.RemoteManagerCall.
name required Operation name. Used in the Flow section to associate workflow nodes to VCL operations.

and the subelements:

| HDFSRemoteManager

  • RemoteCommands

RemoteCommands

The RemoteCommands element contains the sequence of FTP operations to be carried out. These operations will be performed in the order they were configured. An error in one of the operation marked as critical will terminate the execution of the entire sequence.

The RemoteCommands Element is used by remotemanager-call and, internally, it is possible to configure the following operations:

Remote Command Use this for...
RemoteSearchCommand Controlling one or more files (whose name corresponds to a regular expressions) in a remote directory
RemoteMoveCommand Renaming/moving a file/directory in a remote directory
RemoteDeleteCommand Deleting a file/directory from a remote directory
RemoteDownloadCommand Downloading a file/directory from a remote directory to a local directory
RemoteUploadCommand Uploading a file/directory from a local directory to a remote directory

The remote commands can have the following subelements:

  • Description

RemoteSearchCommand

The GreenVulcano® ESB operation RemoteSearchCommand checks if one or more files, whose name conforms to a given regular expression, exists within the specified directory.

On completion and in the absence of errors the following properties of output GVBuffer are setted:

  1. GVRM_FIND_FILES_NUM Number of files found (>= 0)
  2. GVRM_FIND_FILES_LIST List of names of found files, separated by ','. Set only if the value of field 'GVRM_FIND_FILES_NUM' is greater than 0.
    Both the pattern of the file and that the directory can be static or contain placeholders that are replaced at runtime.

The following table shows its attributes:

Attribute Type Description
type fixed This attribute must assume the value remote-command.
class fixed This attribute must assume the value it.greenvulcano.gvesb.virtual.file.remote.command.GVSearch.
sourcePath required Absolute path of the directory to search in.
filePattern required File name to search for. Can be a regular expression.
isCritical optional If true an internal error interrupt the command sequence. Default true.

RemoteMoveCommand

The GreenVulcano® ESB operation RemoteMoveCommand moves/renames a file/directory.

The file name, the source and destination directory names can be static or contain placeholders that are replaced at runtime. After the replacement of the placeholder, the value of the filename may contain a regular expression, in which case, all files of the source directory, whose name is in accordance with the corresponding pattern, will be moved to the destination directory. If the destination directory already contains files with the same name, they will be overwritten.

The following table shows its attributes:

Attribute Type Description
type fixed This attribute must assume the value remote-command.
class fixed This attribute must assume the value it.greenvulcano.gvesb.virtual.file.remote.command.GVMove.
targetPath required Absolute path of the file/directory where is the file to move.
oldName required Name of the file to be moved.
newName required Name of the file moved.
isCritical optional If true an internal error interrupt the command sequence. Default true.

The following table shows the PropertyDef possible elements that can be set when the command is executed on the HDFS file system:

Property Possible Values
dfs.file.overwrite true/false; default=false

This parameter can be used to owerwrite the target destination with the source one when a move operation is used on the HDFS file system.

RemoteDeleteCommand

The GreenVulcano® ESB operation RemoteDeleteCommand deletes a file or directory.

Both the file name and the directory can be static or contain placeholders that are replaced at runtime. After the replacement of the placeholder, the value of the filename may contain a regular expression, in which case, will erase all files in the specified directory whose name is consistent with the corresponding pattern.

The following table shows its attributes:

Attribute Type Description
type fixed This attribute must assume the value remote-command.
class fixed This attribute must assume the value it.greenvulcano.gvesb.virtual.file.remote.command.GVDelete
targetPath required Absolute path of the file/directory to remove.
filePattern optional File name to remove. Might contain a regular expression. It is used only if targetPath is a directory, and if empty the filter is ignored.
isCritical optional If true an internal error interrupt the command sequence. Default true.

RemoteDownloadCommand

The GreenVulcano® ESB operation RemoteDownloadCommand downloads a file or directory from a remote directory and saved in the local directory you specified (or in GVBuffer).

File names and directories can be static or contain placeholders that are replaced at runtime.

The following table shows its attributes:

Attribute Type Description
type fixed This attribute must assume the value remote-command.
class fixed This attribute must assume the value it.greenvulcano.gvesb.virtual.file.remote.command.GVDownload.
remotePath required Pathname of remote directory containing the file to download. It can be an absolute pathname or relative, in the second case it is relative to home directory of ftp user.
remoteFilePattern required The name of the file to download.
toGVBufferExpression optional GVBuffer OGNL expression used to specify how to handle the current downloaded file. In the expression, the downloaded file will be referenced by the variable #input, and will be a byte [].
targetPath optional Pathname of the local directory to save the downloaded file. Must be an absolute pathname.
isCritical optional If true an internal error interrupt the command sequence. Default true.

The following table shows the PropertyDef possible elements that can be set when the command is executed on the HDFS file system:

Property Possible Values
dfs.file.delete-source fixed=false
dfs.file.raw-local-filesystem fixed=true
dfs.stream-buffer-size default: 4096

These parameters can be used to specify the related action when a download operation is used on the HDFS file system.

RemoteUploadCommand

The GreenVulcano® ESB operation RemoteUploadCommand uploads a file from a local directory (or GVBuffer) to the remote directory specified.

File and/or directory names can be static or contain placeholders that are replaced at runtime.

The following table shows its attributes:

Attribute Type Description
type fixed This attribute must assume the value remote-command.
class fixed This attribute must assume the value it.greenvulcano.gvesb.virtual.file.remote.command.GVUpload.
sourcePath optional Pathname of the local directory where the file is transferred. Must be an absolute pathname.
sourceFilePattern optional The name of the file to transfer.
fromGVBufferExpression optional OGNL Expression applied to the current GVBuffer that specifies how to build the file content. In the expression, the GVBuffer is associated with #object, and must return a byte[].
remotePath required Pathname of remote directory where upload the file. It can be an absolute pathname or relative, in the second case it is relative to home directory of ftp user.
isCritical optional If true an internal error interrupt the command sequence. Default true.

The following table shows the PropertyDef possible elements that can be set when the command is executed on the HDFS file system:

Property Possible Values
dfs.file.delete-source true/false; default=false
dfs.file.overwrite true/false; default=false
dfs.permissions.path Unix-like permission Example: 777; default 700
dfs.stream-buffer-size default: 8192
dfs.replication default: 1
dfs.blocksize default: 4194304
dfs.permissions.user hdfs user owner
dfs.permissions.group hdfs user group

These parameters can be used to owerwrite the target destination with the source one, set the replication, permissions and the other properties on destination target when an upload operation is used on the HDFS file system.

Retrieved from "https://gvwiki.greenvulcano.com/gvwiki/index.php?title=Remotemanager-call&oldid=4455"