Managing Data

This section describes how to manage data in the IMQCAM including:

• Creating and managing access to collections

• Creating and managing folders

• Uploading folders and files

• Uploading folders and files using the girder-client

• Ingesting data

Collections

Data in the IMQCAM portal is organized into collections with the three main collections relating to Processing (Enclave 1), Multiscale Modeling (Enclave 2), and Q&C (Enclave 3). Collections are open for all IMQCAM members but Enclave PIs may request that write access be limited to specific team members.

After signing in to your account, select the “Collections” link to view any collections that you have access to. Select the collection that you want to access.

Creating Folders

To create a new folder, select the dropdown on the right and “Create folder here”:

Uploading folders and files

To upload a file or folder, select the green upload icon:

Using the “Upload files” window, browse or drag-and-drop the files you want to upload. Select “Start Upload” to initiate the upload process:

When uploading folders, you may be prompted to allow your browser to upload multiple files:

Upload files and folder using girder-client

It is also possible to upload files and folders from the command line using the Python-based girder-client application. This requires that you configure an API Key in the IMQCAM DMS system.

Install girder-client:

pip install girder-client

Note

The girder-client version shoulb be >3.x.x Check using this command: pip show girder-client

To configure an API key for use with the girder-client, select the “My account” option from the user menu:

Select the “API keys” tab and then “Create new key”:

Name your API key and keep the default options then select “Create”:

Click the “show” link to view and copy your key:

Now use the girder-client command line tool to upload:

girder-client --api-url https://data.imqcam.org/api/v1 --api-key API_KEY upload SERVER_REMOTE_PATH OR ID "YOUR_LOCAL_PATH"

For example, to upload the local folder “test” to the remote “Example” in the “Demo”:

Upload with help of SERVER_REMOTE_PATH:

How to get the path:

Go inside the Collections, Select the right collection. Eg: Demo

select the right folder eg: Example

FOR THE GIVEN example the SERVER_REMOTE_PATH is “Demo/Example”

girder-client --api-url https://data.imqcam.org/api/v1 --api-key API_KEY upload "/collection/Demo/Example" "./test"

Creating Folder from test
[####################################]  9.00/9.00  100%  file.txt
Creating Folder from test/subfolder
[####################################]  5.00/5.00  100%  file.txt

Upload with help of ID:

How to get the ID:

Go inside the Collections, Select the right collection. Eg: Demo

Copy the last ID from the URL like shown below

girder-client --api-url https://data.imqcam.org/api/v1 --api-key API_KEY upload 6839b47028b402e4f5r4 "./test"
Creating Folder from test
[####################################]  9.00/9.00  100%  file.txt
Creating Folder from test/subfolder
[####################################]  5.00/5.00  100%  file.txt

Transferring data

When data needs to be transferred automatically or continuously, the DMS supports several options:

• Direct transfer using the manage-girder-client

• Direct transfer using common protocols including SFTP, rsync, or Globus.

• Automated transfer using OpenMSIStream