Requesting Storage
Our main storage system is Bucket. It is the main repository for all active research data at OIST. Each unit gets an initial allocation. If the unit runs out of space on Bucket, the unit leader can request an increase.
- You need to be the unit leader to request storage.
- You can only request storage increases on Bucket and other long-term systems. The in-cluster storage systems Flash and Work are not expandable.
The link to the form is here:
Storage Allocation Request Form
The Request Process
Here is a summary of the process. See further below for all the details.
- First, we ask you to clean up, delete and archive existing data.
- Second, you fill in the request form, where you answer a few questions that will help us determine the storage you need, and to help us maintain and expand the systems into the future.
- If your request is simple and not very large, we can typically accommodate you within a few days to a week or so (staffing, technical issues and holidays permitting).
- For large or complex requests we may not be able to accommodate you with the existing resources. We may need to apply for funding and submit a tender to acquire new hardware, which will take time (several months to a year). For large-scale storage requests, please contact us as soon as possible.
Clean up the existing storage
Our storage space is large but it’s not unlimited. Before you ask for an increase, please make sure you ask your unit members to do the following:
Delete all unneeded files
It’s important to keep the storage reasonably clean. Delete any temporary files, failed data, duplicate files and so on. They all take up space for no good reason.
Many tools generate a lot of intermediate data that is not useful afterwards. And you generally don’t want to keep all the data from failed experiments or failed simulations, just like you would not keep all the intermediate byproducts from a chemical process, or permanently store the end result of a failed reaction.
Archive any finished research projects to free up space.
Once a research project is done, the papers are published and the post-doc or student has left, there is normally little reason to keep the data in live storage any longer.
If you don’t need the data for any current or future project, you can ask us to archive it to tape, where it will no longer take up space on your unit storage. Read more about it here: Archive your data
Remove any files that are not research-related
Bucket is not a general office computer backup or storage system. It is specifically for research data and nothing else. Office documents, PC software, games, movies, music files and so on are not allowed on Bucket.
Unit personnel information and other confidential internal data is absolutely off-limits; Bucket is not designed or certified for such use. It’s all too easy to accidentally give unit members access without realizing it.
Fill in your storage request
We ask you three questions regarding your storage request. We are not evaluating the scientific value of your work. Your answers will help us determine the type and amount of storage you need, and help us get the resources to provide you with it.
1. Storage Amount
- What amount of storage do you need?
- When you need it?
- Are there any other conditions or special considerations?
why do we ask this? This helps us determine what storage system is appropriate, and how we best can go about giving you the storage.
2. Your data plan
- What kind of data do you want to store, and how much?
- From where is it transfered in; and how will you anayze it or otherwise use it?
- How long do you need the data to be stored?
- What will you do with the data afterwards?
why do we ask this? This tells us what the flow of data looks like and what connectivity the storage needs; and for how long we need to provide the storage.
“forever” is not a valid response; nothing is stored forever. Once you no longer actively need a data set, it should probably be archived and the space freed up for other uses.
3. Your research project
- A title or simple description is sufficient
- For small or recurring requests you can disregard this
- If you do have a more detailed project summary, you can paste it here
why do we ask this? We have no generic spare storage. All our storage resources come through budget requests that specifies the use case and justify the scientific value. This information from you help us make effective resource requests and get the resources we need to support you.
Request timeline and resources
With a normal, simple storage increase, with no additional constraints, we can usually provide it within a few days to a week.
In some cases there are technical issues and we may need to migrate your unit storage from one subsystem to anther; in such cases it can take a bit longer, and you may have a brief downtime during the migration.
For large and specific requests, that ask for lots of storage or specific features or constraints, we may not have a system that can fulfill your request on hand.
In such cases, we may need to repurpose other hardware, or purchase new hardware that can meet your needs. The budget for this is requested through the regular OIST budget process. We need a more detailed scientific background and justification, and we need to conform to the OIST budget timelines.
OIST has a supplementary budget request that can accommodate smaller requests around September, and the main budget is submitted in December. If approved, actual work can commence from the next budget year, from April onwards.
This means that large-capacity requests may take from several months up to a year to fulfill. If you need large-scale storage for a project it is imperative that you contact us as soon as possible, or we may be unable to provide you with the resources you need.
We will contact you with any further questions or clarifications. Once you receive the new storage we will let you know.
If you have any questions on this process, let us know at ask-scda@oist.jp.