Accessing Project data in Jupyterhub using the VRE Command Line Tool

VRE users are able to access to own projects' data in VRE workbench tool of Jupyterhub for further analysis, and push the outputs back to VRE projects by using VRE Command Line Tool. 

1. Access to Jupyterhub Terminal

Jupyterhub is integrated with VRE user account with Single Sign-On, so users can launch JupyterHub from the VRE Portal within the Project canvas. 

Please visit this page: VRE User Guide/Analyzing Data/Jupyterhub for details on

• What is Jupyterhub workbench tool?
• How to enter Jupyterhub and launch Terminal?
• How to install Python packages, create a Python Virtual Environment and register a Kernel?

2. Access to Command Line Tool within Jupyterhub Terminal

Within Jupyterhub Terminal, users can then launch VRE Command Line Tool to access to Project data for further analysis. The VRE Command Line Tool (CLI) is a binary executable program that provides a convenient way for users to perform file actions and platform-related tasks programmatically. For more technical details, please find: Developer Guide/Command Line Tool

By typing vrecli in Jupyterhub Terminal, users can launch the latest version of Command Line Tool:

3. User login through Command Line Tool

The commands users can input are categorized into groups of File, Project, and Users as shown in the above screenshot.

Users need to login to the VRE by their Charite AD credentials before performing any tasks through Command Line Tool, by typing vrecli user login, users are asked to provide user name and password:

4. Download Project data through Command Line Tool

a. The commands supporting the file related tasks are grouped within file , users can print help screen to preview which file related commands they are input by typing vrecli file --help:

b. From the above file related commands, the one users can input to download Project data is file sync, let's take a look at what options users need to provide in order to download a specific Project file/folder by typing vrecli file sync --help :

As part of the file downloading command, users are required to provide

• the target file/folder's path within the project, and
• the local path you want to save the downloaded file/folder
• the Project zone (Green Room or Core) where contains the target file/folder

c. Example: a file of TEST_csv.csv can be downloaded from Indoc Test Project Core zone within the folder swang to user's Home Directory by typing vrecli file sync indoctestproject/swang/TEST_csv.csv . -z core 

d. TEST_csv.csv is successfully downloaded to the Home Directory:

e. Trouble shooting. If downloading failed, please ensure:

• Whether your session expired for being inactive for an hour.
• Whether the target file/folder exists under correct path from the portal.
• Whether you have access to the target file/folder:
  • you need to be a member of the Project where your target file/folder is within, and
  • your Project role is Project Administrator or Project Collaborator if you are trying to download files/folders from the Project Core zone. 

4. Upload Project data through Command Line Tool

After grabbing Project data, users are then able to run containerized pipelines in Workbench Tools to analyze the data. The generated outputs can be uploaded back into the Project by the Command Line Tool. 

a. Let's take a look what options users need to provide in order to upload a file or folder by typing vrecli file upload --help :

As part of the file uploading command, users are required to provide the options of:

• the target file/folder's path on your local, and
• the Project directory you want to upload the file/folder to
• the Project zone. By default, data should be uploaded to Green Room for pseudonymisation. However, data can also be uploaded directly into Core zone if the data is output of source data from Core zone, if this's the case, you need to confirm on the VRE Terms of Use to proceed the uploading. 
• the upload message if upload to Core zone. Users are required to provide a message to explain why data is uploaded directly to the Core zone. 

b.  Example: a file of TEST_csv.csv can be uploaded from user's Home Directory into a folder of swang  within Project of Indoc Test Project Core zone by typing vrecli file upload ./TEST_csv.csv -p indoctestproject/swang -z core -m "workbench output"