Using HPCI Resources
OIST has access to the Japanese High-Performance Computing Infrastructure, or HPCI. The organization connects Japanese supercomputers — the K computer system especially — to a nationwide high-speed network and a single identity management system. You can get an ID from HPCI that lets you apply for access to high-performace systems throughout Japan.
Available Resources
The list of resources is available on this webpage. Resources range from the Fugaku supercomputer and specialized accelerator systems, down to smaller university-level clusters similar in size to Deigo. Click through to each system's page to see details on specifications and availability.
Large HPC systems are a valuable resource and they have serious security concerns (they are used for commercial and classified computing along with academic projects), so getting access to a system can be quite involved.
NOTE: The below is not yet updated for Fugaku. The overall workflow is still br9oadly correct.
The K Computer
The flagship system of the HPCI network is the K computer system. At over 80000 nodes and 800K cores it is currently (spring 2017) the largest system in Japan and one of the largest in the world. For OIST members the K computer is probably the primary reason for wanting to use HPCI resources.
Using the K system is a little different from smaller cluster computers such as Sango. The per-core performance of K is lower than that of Sango; it attains its speed from spreading the work over many thousands or tens of thousands of cores. It uses purpose-built hardware and custom compilers and system software in order to gain as much speed as possible.
A lot of common commercial software such as Java, Matlab or Comsol are not available on K, and the K system does not provide preinstalled versions of most scientific software. The software you want to run will most likely need to be adapted and rebuilt, and you may need to partially rewrite it in order to attain good performance.
Which is to say that using K is not a quick fix for improving your calculation times, but represents a substantial long-term commitment in time and resources. The system makes most sense for very large modelling or analysis jobs that are simply not possible or feasible to run on smaller systems.
Get Access
We're going to assume that you are mainly interested in using the K system. Other systems each have somewhat different application and access procedures, so please refer to their respective web pages for more information.
You need three things to get computing time on an HPC resource in the HPCI network:
- Your own HPCI-ID
- Membership in an accepted HPCI project
- A browser certificate or equivalent for access to the resource you want to use.
The entire process is outlined in English in this pdf file. We will explain the main points below.
Obtain a HPCI-ID
Obtaining the HPCI-ID is straightforward, and you can apply for it on this page. The English-language explanation is here (pdf). Most form fields should be quite straightforward.
The "Primary Center" field refers to a "resource center". You need to belong to one of the available resource centers, and each resource center has their own procedures for paperwork and supplying password credentials. Some HPCI system providers ask that you belong to their resource center to get access to their resources. In practice it's usually most convenient to belong to the resource center corresponding to the system you want to use, and for the K computer it is RIKEN AICS in Kobe.
The HPCI-ID merely adds you as a potential user in the HPCI system. To get an HPCI account and access to the resource, you need to belong to an accepted HPCI project. The difficulty and time taken to get a project accepted depends in large part on the resource you wish to access.
Form a Project
In order to use a resource through the HPCI framework, you need to have or be part of an accepted research project.
Regular projects on K have a call for proposals twice a year, and typically involve projects that can show they need such large-scale resources. The application process takes several months from to complete, and is intended for larger multi-year research projects.
For new HPC users, there is also a General Trial Use project category, where research groups without direct experience can try out K for six months to see if the system is a good fit for their needs. You can apply at any time, and projects are awarded four times a year.
The project application process is explained here. Briefly, all participants register a HPCI-ID, the project representative — typically the PI — fills in the application, which is submitted both online and in post by the end of the submission period, and the award result is sent to the project representative once the decisions have been made.
If the application is successful, the representative has to visit one of the "neighbourhood centers" where HPCI staff is available, and identify themselves in person to get an HPCI account. They can then in turn identify the project members on the centers' behalf, so only the representative needs to visit in person.
Once the HPCI accounts are created, the members can apply for and get user account on the resource they applied to use.
Note that many of the smaller resources are also available through other means, typically through research collaboration with a member of the hosting organization. If you are already in contact with or working with such members, that may be a more expedient way to gain access.
User accounts
Once access is granted for the K computer, each member receives two things: an email with a browser certificate (a file named after your K account name and ending in "k12"), and a postcard with the project ID, your K account name and a passphrase.
You need to install the certificate into Firefox. There is a guide here (pdf) but briefly, you go through these steps:
Set up Firefox
- Save the browser certificate from the email.
- In Firefox, go to settings->advanced, choose "certificates" in the tab and
click "show certificates". - Select "my certificates, and "import" and select the certificate file.
- Enter the passphrase you got with the postcard. It may help to first type
it out in a text editor, then copy and paste it to avoid typing errors. - Go to settings->advanced->certificates again and choose "security devices".
Choose "software security device" and "set password. Set a password.
Set up Java.
Make sure you have Oracle Java installed. We need to set up the security certificate for Java as well.
- run the Java Control Panel application or run "jcontrol" from the command line.
- Click "security" tab, then "manage certificates".
- Click "import", select the certificate file and enter the passphrase.
Log in to the portal
Now you should be able to go to https://k.aics.riken.jp (note the "https") and get to the portal. It should ask you for the password you set earlier before connecting.
Choose English or Japanese in the very top left, click "Start" and you're logged in. At the top right tab "Documents" you find a lot of instruction manuals and other information that are essential to using K.
Set up your SSH keys
You will need an SSH terminal connection for most uses. To set that up, you need to create and copy an SSH key to your K account.
First make sure you've created a key (we cover that in the HPC introduction). In a terminal, do:
ssh-keygenNote that we do not add the "-b 2048" parameter as suggested by the manual. At the time of writing (March 2017) this generates a key that is ultimately rejected by the system.
When asked, do not use the default key name (".ssh/id_rsa") if you have already set it up for Sango or other computers. This will overwrite that key. Instead give it a name such as ".ssh/id_rsa_K" so you know what the key is for. Set a passphrase and just press return for any other prompts.
On the portal website, click the "Status/Configuration" tab, then "SshKeyAdd" in the lefhand-side list.
On Linux and OSX, go to the directory .ssh/ in your home directory. Copy the contents of "id_rsa_K.pub" (that is your public key), and paste the entire contents, without linefeeds, into the text entry window. Press "register" and the key will be added to your list of authorized keys on K.
Now you should be able to log in, using ssh and your account name:
ssh <k account name>@k.aics.riken.jpIf that succeeds then you are good to go. For actually using the K system (or other HPCI systems), we refer you to the online manuals and other documentation on the K portal website.