Skip to Content
Last modified by publicadmin on 2025/12/16 13:04
Show last authors
1 VRE users are able to access to own projects' data in VRE workbench tool for further analysis, and push the outputs back to VRE projects by using VRE Command Line Tool.
2
3 === 1. Access to Command Line Tool within Guacamole VM ===
4
5 Please see [[VRE User Guide/Analyzing Data/Gucamole>>doc:Main.user_guide.Analyzing Data.Guacamole.WebHome]] for additional details regarding:
6
7 * What is the Guacamole VM workbench tool?
8 * How can I request access to my Project's Guacamole VM workbench tool?
9 * How can I access the Guacamole VM and launch terminal?
10
11 To decide which VM you wish to enter, please be advised:
12
13 {{info}}
14 The Project Zone (Green Room or Core) that the VM is being deployed within will decide which version the VRE CLI is configured within this VM.
15
16 * If users enter Green Room Zone Workbench VM, the VRE CLI deployed within this VM will support users download Project data from Green Room. Only very limited Project members can access to the Green Room Zone Workbench VM.
17 * If users enter Core Zone Workbench VM, the VRE CLI deployed within this VM will **NOT** support users download Project data from Green Room, users can only download Project data from Core zone.
18 {{/info}}
19
20 Within Guacamole VM, users can then launch VRE Command Line Tool to access to Project data for further analysis. The VRE Command Line Tool (CLI) is a binary executable program that provides a convenient way for users to perform file actions and platform-related tasks programmatically.
21
22 By typing {{code language="none"}}vrecli{{/code}} in Guacamole VM terminal, users can launch the latest version of Command Line Tool:
23
24 {{code language="none"}}
25 uname@s-hdp-vre-h009:/home/uname@charite.de$ vrecli
26 Usage: vrecli [OPTIONS] COMMAND [ARGS]...
27
28 What's new (Version 1.4.0):
29
30 1. Support file annotation when uploading folder
31
32 2. Support PACS data integration with VRE
33
34
35
36 Options:
37 --help Show this message and exit.
38
39 Commands:
40 file File Actions
41 project Project Actions
42 user User Actions
43
44 {{/code}}
45
46 === 2. User login through Command Line Tool ===
47
48 The commands users can input are categorized into groups of File, Project, and Users as shown in the above screenshot.
49
50 Users need to login to the VRE by their Charite AD credentials before performing any tasks through Command Line Tool, by typing {{code language="none"}}vrecli user login{{/code}}, users are asked to provide user name and password:
51
52 {{code language="none"}}
53 uname@s-hdp-vre-h009:/home/uname@charite.de$ vrecli user login
54 Username: uname
55 Password:
56 Welcome to the VRE Command Line Tool!
57 {{/code}}
58
59 === 3. Download Project data through Command Line Tool ===
60
61 a. The commands supporting the file related tasks are grouped within {{code language="none"}}file{{/code}} , users can print help screen to preview which file related commands they are input by typing {{code language="none"}}vrecli file --help{{/code}}:
62
63 {{code language="none"}}
64 uname@s-hdp-vre-h009:/home/uname@charite.de$ vrecli file
65 Usage: vrecli file [OPTIONS] COMMAND [ARGS]...
66
67 Options:
68 --help Show this message and exit.
69
70 Commands:
71 attribute-export export attribute template from a given project
72 attribute-list check attributes of a given project
73 list List files and folders inside a given project/folder
74 sync Download file from a given project/folder/file
75 upload upload file to a given project file upload <file-
76 path>...
77 {{/code}}
78
79 b. From the above screenshot, the command users can input to download Project data is {{code language="none"}}file sync{{/code}}, let's take a look what options users need to provide in order to download a specific Project file/folder by typing {{code language="none"}}vrecli file sync --help{{/code}} :
80
81 {{code language="none"}}
82 uname@s-hdp-vre-h009:/home/uname@charite.de$ vrecli file sync --help
83 Usage: vrecli file sync [OPTIONS] [PATHS]... OUTPUT_PATH
84
85 Download file from a given project/folder/file
86
87 Options:
88 -z, --zone TEXT Target Zone, core/greenroom [default: greenroom]
89 --help Show this message and exit.
90 {{/code}}
91
92 As part of the file downloading command, users are required to provide
93
94 * the target file/folder's path within the project, and
95 * the local path you want to save the downloaded file/folder
96
97 c.** Example**: a file of {{code language="none"}}TEST_csv.csv{{/code}} can be downloaded from {{code language="none"}}Indoc Test Project{{/code}} Core zone within the folder {{code language="none"}}uname{{/code}} to user's Home Directory by typing {{code language="none"}}vrecli file sync indoctestproject/uname/TEST_csv.csv . -z core{{/code}}
98
99 {{code language="none"}}
100 uname@s-hdp-vre-h009:/home/uname@charite.de$ vrecli file sync indoctestproject/uname/Test_csv.csv . -z core
101 Preparing status: READY_FOR_DOWNLOADING
102 start downloading...
103 Downloading TEST_csv.csv |██████████████████████████████ 100% 00:00
104 File has been downloaded successfully and saved to: ./TEST_csv.csv
105 {{/code}}
106
107 d. {{code language="none"}}TEST_csv.csv{{/code}} is successfully downloaded to the Home Directory:
108
109 {{code language="none"}}
110 uname@s-hdp-vre-h009:/home/uname@charite.de$ ls
111
112 Desktop indoctestproject_1627578185.2418823.zip indoctestproject_zipped_1622255309.zip lder Public TEST_csv.csv test_folder.zip testing_data_0104.png
113 Documents indoctestproject_1628537735.1548965.zip indoctestproject_zipped_1622257377.zip Music uname test_folder_0909 testindoc.png thinclient_drives
114 Downloads indoctestproject_1631202299.1036603.zip indoctestproject_zipped_1622316277.zip Pictures Templates test_folder_test testing0529.png Videos
115
116 {{/code}}
117
118 e. Trouble shooting. If downloading failed, please ensure:
119
120 * Whether your session expired for being inactive for an hour.
121 * Whether the target file/folder exists under correct path you provided from the portal.
122 * Whether you have access to the target file/folder:
123 ** you need to be a member of the Project where your target file/folder is within, and
124 ** your Project role is Project Administrator or Project Collaborator, who have the access to all files/folders within the Project Core zone.
125
126 === 4. Upload Project data through Command Line Tool ===
127
128 After grabbing Project data, users are then able to run containerized pipelines in Workbench Tools to analyze the data. The generated outputs can be uploaded back into the Project by the Command Line Tool.
129
130 a. Let's take a look what options users need to provide in order to upload a file or folder by typing {{code language="none"}}vrecli file upload --help{{/code}} :
131
132 {{code language="none"}}
133 uname@s-hdp-vre-h009:/home/uname@charite.de$ vrecli file upload --help
134 Usage: vrecli file upload [OPTIONS] [PATHS]...
135
136 upload file to a given project
137
138 file upload <file-path> -g <generate-id> -a <attribute> -p <project-code/folder> -t <tag>
139 -z <zone> -m <upload-message> -s <folder/source-file> --pipeline <pipeline> --zip --pacs
140
141 Options:
142 -p, --project-code TEXT Project Code
143 -g, --generate-id TEXT Generate ID (Only for project Generate)
144 -a, --attribute TEXT File Attribute Template
145 -t, --tag TEXT Add tag to the uploaded file
146 -z, --zone TEXT Target Zone, core/greenroom [default:
147 greenroom]
148
149 -m, --upload-message TEXT When upload processed files, must commit a
150 message to explain the purpose.
151
152 -s, --source-file TEXT The name of the source file of your processed
153 files.
154
155 --pipeline TEXT The processed pipeline of your processed files.
156 optional for source.
157
158 --zip Upload folder as a compressed zip file
159 --pacs Identify uploading data as pacs
160 --help Show this message and exit.
161
162 {{/code}}
163
164 As part of the file uploading command, users are required to provide the options of:
165
166 * the target file/folder's path on your local, and
167 * the Project directory you want to upload the file/folder to
168 * the Project zone. By default, data should be uploaded to Green Room for pseudonymisation. However, data can also be uploaded directly into Core zone if the data is output of source data from Core zone, if this's the case, you need to confirm on the VRE Terms of Use to proceed the uploading.
169 * the upload message if upload to Core zone. Users are required to provide a message to explain why data is uploaded directly to the Core zone.
170
171 b. (% id="cke_bm_1895S" style="display:none" %)** **(%%)**Example**: a file of {{code language="none"}}TEST_csv.csv{{/code}} can be uploaded from user's Home Directory into a (% id="cke_bm_1908S" style="display:none" %) (%%)folder of {{code language="none"}}uname{{/code}}(% id="cke_bm_1908E" style="display:none" %) (%%) within Project of {{code language="none"}}Indoc Test Project{{/code}} Core zone by typing {{code language="none"}}vrecli file upload ./TEST_csv.csv -p indoctestproject/uname -z core -m "workbench output"{{/code}}
172
173 {{code language="none"}}
174 uname@s-hdp-vre-h009:/home/uname@charite.de$ vrecli file upload ./TEST_csv.csv -p indoctestproject/uname -z core -m "workbench output"
175 You are about to transfer data directly to the VRE Core! In accordance with the VRE Terms of Use, please confirm that you have made your best efforts to pseudonymize or anonymize the data and that you have
176 the legal authority to transfer and make this data available for dissemination and use within the VRE .To review the Terms of Use, please visit https://vre.charite.de .If you need to process the data to remove
177 sensitive identifiers, please cancel this transfer and upload the data to the Green Room to perform these actions.
178 To cancel this transfer, enter [n/No]
179 To confirm and proceed with the data transfer, enter [y/Yes]
180 [y/N]: y
181 Starting upload of: ./TEST_csv.csv
182 Pre-upload complete.
183 uploader: uname container_id: 37 total_size: 303684 total_chunks: 1 uniq_identifier: f45fbb58-c135-4b94-a224-9bcd469a08ee-1631558977 resumable_relative_path uname/
184 Upload job is finalizing, please wait...
185 Upload job complete.
186 All files uploaded successfully.
187
188 {{/code}}
189
190 c. **Example 02**:  a file of {{code language="none"}}TEST_csv.csv{{/code}} can be uploaded from user's Home Directory into a  folder of {{code language="none"}}uname{{/code}}  within Project of {{code language="none"}}Indoc Test Project{{/code}} Core zone, and build a data lineage with a source file {{code language="none"}}testing_data.png{{/code}} in  a  folder of {{code language="none"}}uname{{/code}}  within same Project Core zone and with a pipeline name of {{code language="none"}}testing_pipeline{{/code}} by typing {{code language="none"}}vrecli file upload ./TEST_csv.csv -p indoctestproject/uname -z core -m "testing" --source-file uname/testing_data.png  --pipeline "test_pipeline"  {{/code}}
191
192 {{code language="none"}}
193 uname@s-hdp-vre-h009:/home/uname@charite.de$ vrecli file upload ./TEST_csv.csv -p indoctestproject/admin -z core -m "testing" --source-file admin/testing_data.png  --pipeline "test_pipeline"  
194 You are about to transfer data directly to the VRE Core! In accordance with the VRE Terms of Use, please confirm that you have made your best efforts to pseudonymize or anonymize the data and that you have 
195 the legal authority to transfer and make this data available for dissemination and use within the VRE .To review the Terms of Use, please visit https://vre.charite.de .If you need to process the data to remove 
196 sensitive identifiers, please cancel this transfer and upload the data to the Green Room to perform these actions. 
197 To cancel this transfer, enter [n/No] 
198 To confirm and proceed with the data transfer, enter [y/Yes] 
199   [y/N]: y
200 Starting upload of: ./TEST_csv.csv
201 Pre-upload complete.
202 uploader: swang project_code: indoctestproject total_size: 303684 total_chunks: 1 resumable_relative_path admin
203 Uploading TEST_csv.csv:  |██████████████████████████████ 100% 00:00
204 Upload job is finalizing, please wait...
205 Upload job complete.
206 All files uploaded successfully.
207 {{/code}}
208
209 The Data Lineage Graph in Properties of file {{code language="none"}}TEST_csv.csv{{/code}} or {{code language="none"}}testing_data.png{{/code}} will display the lineage between {{code language="none"}}testing_data.png{{/code}} as a Upstream file, {{code language="none"}}testing_pipeline{{/code}} as a Process, and {{code language="none"}}TEST_csv.csv{{/code}} as a Downstream file linked within the lineage graph:
210
211 [[image:lineage.png||data-xwiki-image-style-alignment="center"]]
212
213 ----
214
215 {{box}}
216 [[image:https://i.creativecommons.org/l/by-sa/4.0/88x31.png||data-xwiki-image-style-alignment="center"]]
217
218 (% style="text-align: center;" %)
219 Copyright © 2022, Indoc Research. This work is licensed under a [[Creative Commons Attribution-ShareAlike 4.0>>http://creativecommons.org/licenses/by-sa/4.0/ ||rel="noopener noreferrer" target="blank"]] International License.
220 {{/box}}