Author Archives: Blair Bethwaite

R@CMon IDC Spotlight – AMD & DELL

Late in 2013 Monash was approached by AMD and Dell to provide information for an IDC report on the Monash Node of the NeCTAR Research Cloud . We were more than happy to contribute and highlight the success of our Node and NeCTAR as a whole, plus share lessons learned along the way.

Download (PDF, Unknown)

Whilst it contains the requisite commercial “fluff” (the R@CMon tech team speaking here), we were keen to share our experience with enterprise customers who might take note of such reports and thus further assist OpenStack’s penetration in the enterprise sector.

Note that the report contains some outdated information about the full capability of the Monash Node, notably that we’ll have over 4500 cores capacity, which we’ll correct next week when we post about the progress of phase 2!

GPU Flavors on R@CMon

We’re pleased to announce we now have GPGPU accelerated cloud instances working, a first for the NeCTAR Research Cloud!

If you’re not as excited by SSH session logs as I am you might like the following screenshots captured by Juptier Hu from the CVL team, which illustrate the CVL running on both a new GPU flavor and a standard non-GPU flavor. The CVL uses TurboVNC to achieve remote hardware accelerated rendering (care of VirtualGL), so the framerates here show CPU versus GPU rendering speeds.

CPU Render - 9 fps

CVL Desktop + CPU Rendering

GPU Render - 380 fps

CVL Desktop + GPU Rendering

And to really prove it, here’s the obligatory mono-spaced console dump you’d expect from a technical blog:

01:47:51 blair@bethwaite:~$ nova show f44cad73-40e1-4e83-8699-dd3f7f2e9ead | grep flavor
| flavor | cg1.medium (19) |
01:47:54 blair@bethwaite:~$ nova show f44cad73-40e1-4e83-8699-dd3f7f2e9ead | grep network
| monash-test network | 11x.xxx.255.20 |

01:48:39 blair@bethwaite:~$ sshrc root@11x.xxx.255.20
root@ubuntu:~# lshw | grep -C 2 NVIDIA
description: 3D controller
product: GF100GL [Tesla M2070-Q]
vendor: NVIDIA Corporation
physical id: 6
bus info: pci@0000:00:06.0
root@ubuntu:~# nvidia-smi
Mon Jan 13 01:52:22 2014
+------------------------------------------------------+
| NVIDIA-SMI 5.319.76   Driver Version: 319.76         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla M2070-Q       Off  | 0000:00:06.0     Off |                    0 |
| N/A   N/A    P0    N/A /  N/A |       10MB /  5375MB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|  No running compute processes found                                         |
+-----------------------------------------------------------------------------+

root@ubuntu:~# cd NVIDIA_CUDA-5.5_Samples/7_CUDALibraries/MersenneTwisterGP11213/
root@ubuntu:~/NVIDIA_CUDA-5.5_Samples/7_CUDALibraries/MersenneTwisterGP11213# ./MersenneTwisterGP11213 ./MersenneTwisterGP11213 Starting...

GPU Device 0: "Tesla M2070-Q" with compute capability 2.0
Allocating data for 2400000 samples...
Seeding with 777 ...

Generating random numbers on GPU...
Reading back the results...

Generating random numbers on CPU...
Comparing CPU/GPU random numbers...

Max absolute error: 0.000000E+00
L1 norm: 0.000000E+00

MersenneTwister, Throughput = 3.1591 GNumbers/s, Time = 0.00076 s, Size = 2400000 Numbers

Shutting down...

Currently these are available as cg1.* flavors in the monash-test cell, so not open for general consumption – and they won’t be for a while until we purchase and deploy more GPU nodes. The allocation process also needs to be tweaked to deal with this new capability. So currently GPU flavors are only accessible by special arrangement with the R@CMon team.

We’ll be deploying a considerable GPU capability as part of R@CMon in order to support, e.g., GPU accelerated rendering and GPGPU accelerated viz processing for the CVL. GPU capabilities can also be useful for on-demand development work, such as providing hardware rendering for the CAVE2 development environment.

At the moment we have a handful of NVIDIA Tesla M2070-Q’s, with Kepler K2’s coming as part of R@CMon phase2. If you’re keen to get access or try this out then drop us a line.

Computational Resource Framework

Over the past year the Monash NeCTAR porting programme has worked with Griffith University eResearch developers on their Computational Resource Framework (CRF) project. We’re well overdue to promote their excellent work (and a little of ours), and as there will be a presentation on this at eResearch Australasia (A RESTful Web Service for High Performance Computing based on Nimrod/G), now seems a good time to blog about it!

hpcportal_login_sm

The CRF aims to address one of the long-standing issues in HPC, that is uptake by non-technical users. HPC is a domain with a well-entrenched command-line ethos, unfortunately this does alienate a large cohort of potential users, and that has negative implications for research productivity and collaboration. At Monash, our HPC specialists go to a great deal of effort to ease new users into the CLI (command line interface) environment, however, this is a labour-intensive process that doesn’t catch everybody and often leaves users reliant on individual consultants or team-members.

For some time portals have been the go-to panacea for democratising HPC and scientific computing, and there are some great systems being deployed on the RC, but they still seem to require a large amount of development effort to build and typically cater to a particular domain. Another common issue with “job-management” style portals (including Nimrod’s own ancient CGI beauty) is that they expose and delegate too much information and responsibility to the end user – typically an end user who doesn’t know, or want to know, about the intricacies of the various computational resources. Such mundanities as what credentials they need, which ones are actually working today, etc.

The CRF is different in this respect as it is not a domain-specific interface, instead the Griffith team have taken the approach of concentrating on a minimum set of functionality for some popular HPC and scientific computing applications. The user just inputs info relevant to the application and the CRF configuration determines where and how the job is run. Currently there are UIs for creating, submitting and managing NAMD, R and MATLAB jobs and sets thereof; AutoDock, Octave and Gaussian are in the works. They’ve put a bunch of effort into building out a web-service in front of Nimrod/G, that layer means their UIs are fairly self-contained bits of php that present an interface and translate inputs into a template Nimrod experiment. The web-service has also enabled them to build some other cool bits and pieces, like experiment submission and results over email.

hpcportal_namd_sm

Using Nimrod/G as the back-end resource meta-scheduler means the CRF can intermingle jobs over the local cluster and burst or overflow into the NeCTAR Research Cloud, and that’s the focus of the Monash & Griffith collaboration for this project. We’re now looking to implement scheduling functionality that will make it possible to put simple policies on resources, e.g., use my local cluster but if jobs don’t start within 30 mins then head to the cloud. There should be some updates following in that area after the eResearch conference!

Volumes on R@CMon

The Monash node now has a block-storage volume service in pre-production. The volumes are delivered via OpenStack Cinder and Ceph. Cinder hooks up networked real or virtual block devices to your virtual machine instances from a huge variety of backend storage systems. This gives you persistent storage on-demand, in the mold of Amazon EBS.  Like object storage, volumes exist independent of your instances; but unlike object storage, volumes provide virtual direct-attached storage suitable for your favourite filesystem, database or application store. Some tools built on top of the RC require volumes to unleash their full abilities, e.g., Galaxy. There is more general information about the volume service on the NeCTAR wiki Research Cloud Storage page.

Originally, NeCTAR never specifically required the RC nodes to provide persistent block-storage, only to capable of interfacing with it at some stage in the future – clearly that was an intended meeting/union point (in budget line items) for NeCTAR and RDSI. At Monash we have seeded our volumes capability via NeCTAR and plan to expand it via RDSI. We plan to offer trial quota of about 50GB to those who ask, we’ll give some out via NeCTAR merit-allocation, and more via ReDS. Thanks to the “data-deluge” (please don’t hurt me for repeating that), block-storage will prove to be the scarcest resource across the RC, so we will probably have to implement some form of quota retirement – similar to how HPC resources put tight quotas on and have regular cleanups of their /short or /scratch filesystems.

We expect the Monash volume service to stay in “pre-production” until the end of the year. What we mean by that is that we are giving it best-effort support. It is a highly available and redundant system (Ceph, if you are interested – more specific details below), but we have not operationalised the system administration to give it 24/7 attention, or for that matter done much in the way of tuning. So in summary, we don’t expect or plan to cause any problems/outages, but we also have no fixed service levels or detailed DR plans as yet.

The current setup is running across eight Dell R720xd boxes with a total of 192TB raw storage, all very closely coupled with the monash-01 compute AZ (availability-zone). The “nova” pool has two replicas of each object, so in practice we have about 90TB of usable capacity at the moment. It seems to be working out quite well so we expect to expand this as part of R@CMon stage 2. A bunch of folks are already using it, as you can see from the status:

root@rcstor01:~# ceph -s
 cluster b72.....x
 health HEALTH_OK
 monmap e1: 5 mons at {0=w.x.y.z2:6789/0,1=w.x.y.z3:6789/0,2=w.x.y.z4:6789/0,3=w.x.y.z5:6789/0,4=w.x.y.z6:6789/0}, election epoch 184, quorum 0,1,2,3,4 0,1,2,3,4
 osdmap e696: 96 osds: 96 up, 96 in
 pgmap v2671880: 4800 pgs: 4800 active+clean; 8709 GB data, 18387 GB used, 156 TB / 174 TB avail; 17697B/s rd, 134KB/s wr, 21op/s
 mdsmap e1: 0/0/1 up

The awesome thing about Ceph is the bang for buck and fine-grained scalability. The value proposition is exactly the same as for object storage – you can expand a server at a time at commodity prices per TB (which seem to be about half to two-thirds that of vendor proprietary solutions), you don’t ever need to buy a huge amount more storage than you really need, you keep buying today’s spec of the hardware and reap the capacity and performance improvements, and every time you add a storage server you’re increasing the number of clients you can serve (no other components to scale).

If you think your project needs volumes please put a request in or edit an existing request through the NeCTAR Dashboard

R@CMon accelerates forensic data research

Campbell Wilson and Janis Dalins of Monash’s Faculty of IT are developing a forensic data ranking and analysis tool, focusing on accelerating existing multimedia forensics analytics. Their prototype will collect, index and analyse geo-tagged multimedia in a tool intended for use by international law enforcement agencies for multi-jurisdictional victim identification.

Till recently they were running their own Solr cluster on a set of spare desktop machines which Janis was spending a good deal of time maintaining. Then their NAS box died, badly. R@CMon to the rescue!

Janis and I met previously in the server room at Caulfield where I told him all about the research cloud and the imminent availability of the Monash node, so we were both excited to see what kind of Solr performance they could get on the cloud compared to their dedicated hardware. The sticking point previously had been storage performance (or the lack thereof) on the existing cloud node and the relatively small size of the ephemeral storage available in the standard NeCTAR flavors. Fortunately R@CMon came online just in time and with enough flexibility to help them recompute their lost data. We created a special large flavor with 32 cores, 128 GB RAM and 2.4TB of ephemeral storage – a mini cluster-in-a-box.

Janis kindly shared a recent progress update:

Since restarting our indexing almost a fortnight ago, we’ve successfully indexed over 1.2 billion (1,226,707,169 at last count) tweets. This is almost double the count we achieved on our existing cluster, I’d guesstimate the overall index construction workload/performance as being around 130% that of the cluster. Faceted querying now works, and in a timely fashion.

So performance is great, but Janis also nicely sums up the key empowering aspect of the NeCTAR research cloud:

Beyond simple performance measures, I wanted to emphasise probably the most important thing: Unlike the existing indexing cluster, the NeCTAR instance has been treated very much as a turn-key project. We used the instance “as is” (with the exception of adding user accounts and an nginx based reverse proxy for security reasons), and haven’t performed any maintenance or configuration work. I can’t emphasise enough how much time that’s freed up for actually doing research – in commercial lingo, I’d say that’s improved efficiency greatly.

Thanks a bunch to Janis for taking the time to share the story. If you like the sound of this and think we might be able to help with your research computing please contact merc@monash.edu.