Accounts on High Performance Systems

The Big Red Cluster

Big Red is the most powerful university-owned computer in the US, and one of the 50 fastest supercomputers in the world. Part of a comprehensive strategy to build an advanced cyberinfrastructure to support research at Indiana University, Big Red has a theoretical peak performance of more than 20 teraflops, and has achieved more than 15 teraflops on numerical computations.

Big Red is a distributed shared-memory cluster, consisting of 512 IBM JS21 Blades, each with two dual-core PowerPC 970 MP processors, 8GB of memory, and a PCI-X Myrinet 2000 adapter for high-bandwidth, low-latency MPI applications. In addition to local scratch disks, the Big Red compute nodes are connected via gigabit Ethernet to a 200TB GPFS file system, hosted on 16 IBM p505 Power5 systems.

Access to Big Red is provided to all IU faculty and graduate students, and faculty-sponsored undergraduates and staff. Instructional use is limited to courses that have been approved by the Director for Research Computing.

The Big Red cluster runs the SuSE Linux Enterprise Server operating system. Batch jobs are managed with IBM's LoadLeveler and the Moab Workload Manager.

Big Red is configured for massively parallel computing. User support, including migrating code to Big Red and parallelizing it, is available from the High Performance Applications team, hpa-admin@iu.edu. To apply for an account, visit the Account Management Service at:

https://itaccounts.iu.edu/

The Libra Cluster

The Libra Cluster accommodates interactive, serial and OpenMP programs within an AIX environment. Statistics, mathematics, scientific/numerical libraries, bioinformatics, database and many other software packages are available on Libra. It is comprised of three IBM BladeCenters and seven IBM p575s all running AIX 5.3. The Power chips in the p575s and the Blades can attain up to four floating point operations per cycle, providing extremely high throughput for scientific workloads.

The p575 nodes libra43, libra47, libra48, libra49, and libra50 are dedicated to large-memory serial applications and to shared-memory multiprocessor (SMP) applications requiring up to eight CPUs. Serial applications and two-CPU SMP applications requiring less than 4GB of memory run on the Blade nodes, libra01-libra42. The p575 nodes libra45 and libra46 are dedicated to research databases (the Research Database Complex, RDC).

Access to Libra is provided to all IU faculty and graduate students, and faculty-sponsored undergraduates and staff. To apply for an account, visit the Account Management Service at:

https://itaccounts.iu.edu/

The Quarry Cluster

Indiana University's newest supercomputer, Quarry, went into production service on August 20. The 7 Teraflop Quarry system replaces the 2 Teraflop AVIDD system, which is being retired.

Like AVIDD, Quarry is built from Intel processors and runs Red Hat Linux as its operating system. When the retirement of AVIDD was discussed at the Research Technologies Round Table in January, it became clear that many researchers preferred to continue working in that familiar environment, rather than migrating to the more powerful Big Red cluster. To meet the needs of these ongoing research programs, the Quarry cluster was commissioned.

Quarry consists of 112 IBM HS21 Blade servers, each containing two Intel Xeon 5335 quad-core processors, 8GB of memory, a 36GB locally- attached SCSI disk for local scratch space space, and gigabit ethernet for system interconnects. The Blade configuration provides electrical power and cooling advantages which makes Quarry much more cost-effective than AVIDD, while occupying much less space.

Quarry uses the Torque resource manager for batch jobs (an open source project based on PBS) and the Moab Workload Manager to implement scheduling optimization and fairness policies. There are four queues: defaultq includes 64 nodes, fastq includes 4 nodes, osg (Open Science Grid queue) includes 16 nodes, and lead (the LEAD queue) includes 16 nodes. Jobs submitted from Quarry login nodes are automatically sent to defaultq. The fastq is for jobs requiring 30 minutes of walltime or less.

The user environment is generally similar to AVIDD's. The Softenv environment management system is used to simplify application and environment configuration. When you login to Quarry for the first time, a file named .soft will be created, defining system defaults, such as the Intel compilers. Additional packages may be added, using the Nano editor; to get a list of the possibilities, enter the command softenv.

The High Performance Applications team can help local users with migration issues and code optimization. Send email to hpa-admin@iu.edu.

Quarry accounts are available to all IU faculty, staff and students. To create one, visit the Account Management Service at:

https://itaccounts.iu.edu/

After you log in (using your Network ID username and passphrase), click create more accounts. Follow the instructions to create your Quarry account. It may take up to six hours for your request to be processed.

If you have questions about the account request process, please contact your campus Support Center.

The Research Database Complex (RDC)

The Research Database Complex provides Oracle database services for research. Access to the RDC is provided to all IU faculty and graduate students, and faculty-sponsored undergraduates and staff. To apply for an account, visit the Account Management Service at:

https://itaccounts.iu.edu/

Account Duration

Accounts on all the above systems will remain valid while the account holder is a registered student, faculty or staff member at IU. Steel accounts will be disabled during the semester following the account holder's departure from IU, and will be purged within six months. On the research systems (Big Red, Libra, Quarry, Research Database Complex) accounts will be similarly disabled and purged. However, research systems accounts will be exempted from disabling if the user sends a request to ITPO Academic Accounts, and in this case the research account will remain activated for one calendar year beyond the account holder's departure from IU. At the end of the year the account will be purged. Extensions beyond one year for research accounts will be granted only for those accounts involved in funded research and having an IU faculty sponsor, or with approval of the Dean or Director of Research and Academic Computing.

Affiliate Accounts

Affiliate accounts on the High Performance Systems are available for non-IU researchers. Please send any questions to hps-admin@iu.edu.

The activities of the High Performance Systems group support and have been funded in part by the National Science Foundation [NSF] under Grant No. 0116050 and Grant CDA-9601632; Shared University Research grants from International Business Machines, Inc. [IBM]; and the Indiana Genomics Initiative [INGEN]. The Indiana Genomics Initiative of Indiana University is supported in part by Lilly Endowment, Inc. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessaritly reflect the views of NSF, IBM, or Lilly Endowment, Inc.