Upload large datasets and bulk upload using the FTP uploader, desktop uploader or API

07/09/2020 - Please note, ongoing support for the desktop uploader has shifted to the FTP client outlined above. Existing installations may still work, but no new downloads or support for existing installations will be offered

 

There are several ways to upload your data, depending on the size:

 

1. Through My data where you can drag and drop files of up to 5GB (default limit), if you need to upload a single file that is bigger than 5GB please submit a support ticket here and we will lift the limit to allow you to do so

 

2. Using the FTPS solution (scroll down for more details) which can work with either small or large files. 

 

3. Using our desktop uploader (scroll down for more details) or the Figshare API, especially when working with large file/s - most web browsers can cope with ~5GB single file uploads only  

 

If these options do not meet your data needs please create a support ticket here or get in touch with us. Figshare can support single file uploads of up to 5TB in size.

All Figshare.com accounts are provided with 20GB of free private space and unlimited public space! For institutional clients your private space limit is dependent on your institution, please contact your institutional admins if you require more space.   

Individual account limits - such as the number of files per item - are listed here

 

How to use the FTP solution

Regardless of whether you have an account with Figshare authentication (either an individual account or institutional) or you use an institutional account with SSO, you can upload files in the account by using the new FTP uploader.

The method allows you to easily and securely upload files in your account directly from your computer by using a secure FTP connection. To use this method, you will need to install an FTP client like Filezilla (but any FTP client will work).

There are several details you need in order to connect to the Figshare FTP server to be able to upload data to your account.

Host: ftps.figshare.com

Stage host: ftps.figsh.com

Transfer mode: This should be set to passive. This is the default mode for some FTP clients, however please make sure you are using this mode.

Username: You can find the username in the Application page of your account, which is accessible via the user menu.

Password: Depending on the authentication method for your account, you will either be using the same password as the one used to log in to Figshare or you will be able to generate one if your account has SSO.

If you generate a password, you need to store it safely as it will not be displayed in the interface. However, if you forget it you’ll always be able to generate a new one. In this case, the old password becomes unusable.

 

 

 

 
After you have all the above information, connect to the FTP and by default we will create a folder called “data” upon your first log in. The data folder is your home directory.

 

To upload your file(s) you need to create a folder first within the data directory. Each item needs its own folder. The folder name will become the item title. For example, if you have several files and you want to upload them into separate items — and get a DOI for each of them — you will need to make multiple folders. If you want to group the files together, create a single folder and upload them all inside.

 

Alternatively, if you have the file/files already organised in a single folder in your file system, just drag and drop the folder in the FTP root directory.

 

After you create the folder, navigate inside it and upload the file(s). The files will be processed immediately. After successfully processing the file(s), they are deleted from the folder and a log event is inserted into processed_files_log.txt, which is created on the fly. The log file is located in the item folder. Another log file is created debug_log.txt containing all actions for items/files.

 

There are some simple rules you need to follow when creating a folder:

      • The folder name has a maximum of 200 characters. After you finish uploading, you can modify the item title from My data in the Figshare interface. After you rename the item title, you will still be able to upload files in the folder, even if it has a different name.
      • There are some characters that you cannot use, as they are not allowed by operating systems: \ / : * ? " < > | ' { }
      • You cannot upload a file called debug_log.txt or processed_files_log.txt.
      • The FTP solution allows you only to upload files. Deleting them is an action that must be performed from My data in the Figshare interface. 

 

What can be added to the folder?

After creating the item folder, you will be able to upload any number of files inside it.

If you have multiple files grouped in sub-folders, you would need to remove the folder structure first, as Figshare won’t allow you to add folders in the item folder you’ve just created in the root directory.

If you add a sub-folder inside the folder, we will simply ignore it and its contains won’t be uploaded into your Figshare account.

In case you need to maintain the structure, you need to make an archive and upload it in the original folder. You can upload any type of archive and we will be able to visualise it on our website like in the example below:

https://figshare.com/articles/Yahoo_Password_Frequency_Corpus/2057937

 

What is the maximum file size you can upload?

It is important to know that Amazon S3 has a limit of 5TB per file. This is a hard limit that we cannot surpass.

If you have an institutional (publisher/preprint/lab/conference) account, the file limit is based on your quota (storage), that you can see in your My data tab. You will be able to upload files that are smaller than the 5TB limit, if your remaining quota allows it. Institutional accounts do not have a file size limit at the moment.

If you have an individual account, you will be able to upload files up to 5GB, since this is the file limit for individual accounts, even if your overall storage is of maximum 20GB, provided you haven’t uploaded other files. However, the FTP process allows you to upload files greater than the limits we have for the two account types. The file will be uploaded, but it won’t get processed and you will see an error in the debug_log.txt. The file will remain in the folder until you choose to remove it.

 

How to use the desktop uploader

07/09/2020 - Please note, ongoing support for the desktop uploader has shifted to the FTP client outlined above. Existing installations may still work, but no new downloads or support for existing installations will be offered

 

 

The desktop uploader is a lightweight app that sits on your desktop and allows you to drag in your research outputs, where you can then track their progress as they are uploaded securely to the cloud. 

You can upload many files at once and resumable uploads are also supported, which means if your internet connection drops you don’t need to start the uploads again. 

You can download the uploader here. Once installed, login using your figshare credentials:

 

 

To generate a personal token, go to the application page - https://figshare.com/account/applications - and scroll down to: 

 

 

Click 'Create Personal Token' and use the code to sign into the Desktop uploader. 

 

How to use the API

Documentation on how to use Figshare's API can be found at https://docs.figshare.com/ and some examples have been highlighted at https://knowledge.figshare.com/articles/item/how-to-use-the-figshare-api