tag:blogger.com,1999:blog-75062596881804337772024-02-07T01:28:47.298-08:00Inside OSG OpsI've set up this blog as a place to tell stories about OSG Operations. These stories may include interesting events, technical problems that have led to important lessons learned, or anything about OSG Operations that may tend itself toward interesting telling. Contributor will consist of all OSG Operations personnel at the various institutions were OSG houses Operational Services.Rob Qhttp://www.blogger.com/profile/06837197517253625772noreply@blogger.comBlogger12125tag:blogger.com,1999:blog-7506259688180433777.post-40541819826286524902012-10-16T07:55:00.001-07:002012-10-16T07:58:46.936-07:00OSG Operations at the EGI Technical Forum<br />
Reflecting on the <a href="http://tf2012.egi.eu/">European Grid Initiative Technical Forum</a> which took place in Prague September 17-21, I am reminded of the maturity of the relationship that OSG Operations has with our European collaborators. We've had interoperating ticketing, information systems, topology, and monitoring tools for most of the life of the OSG project. We've recently added accounting to this list, and are now working on collaborating on communication and information dissemination efforts. Due to the diligent Operational effort on the parts of OSG, EGI, and WLCG, interoperability of these tools has become second nature.<br />
<br />
I had the privilege of being accompanied by Bill Barnett, IUs Director of Science Community Tools, and introduce him to our EGI and WLCG Operational peers. Giving him a chance to understand some of the technical and logistical challenges facing OSG Operations when working with international collaborators.<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
I gave presentations on <a href="https://indico.egi.eu/indico/contributionDisplay.py?sessionId=39&contribId=192&confId=1019">current Operational status</a>, <a href="https://indico.egi.eu/indico/contributionDisplay.py?sessionId=45&contribId=170&confId=1019">future Operational plans</a>, and on <a href="https://indico.egi.eu/indico/contributionDisplay.py?sessionId=45&contribId=75&confId=1019">OSG's plans for GLUE 2.0</a>. While the current and future plans were in tune with EGI's and WLCG's direction, OSG is looking at new solutions for the information system and not immediately embracing the need for GLUE 2.0, unlike EGI Operations. Each presentation sparked welcome discussion and input from our EGI Collaborators. I was also asked to do an interview with Grid Talk on the OSG and EGI Interoperability relationship, available at <a href="http://gridtalk-project.blogspot.com/2012/09/rob-quick-from-open-science-grid-on.html">http://gridtalk-project.blogspot.com/2012/09/rob-quick-from-open-science-grid-on.html</a>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen='allowfullscreen' webkitallowfullscreen='webkitallowfullscreen' mozallowfullscreen='mozallowfullscreen' width='320' height='266' src='https://www.youtube.com/embed/Au0cAKH7kWw?feature=player_embedded' frameborder='0'></iframe></div>
<br />
<br />
Bill and I were also able to meet with the iSGTW European Editors and work out some of the initial plans for the new iSGTW US based position that will be hired shortly. This new collaboration will sponsor a US Reporting Desk for iSGTW and will function as a satellite project to OSG.<br />
<br />
Looking back at my presentations and several other presentations from our EGI and WLCG partners, the one idea that kept coming up in various fashions was how mature these relationships and tools have become over the several years I've been acting as the Operations Coordinator for OSG. This is due to continuous effort in the areas of evolution of services, change management, and constant communication. The effort of maintaining this ongoing relationship makes it possible to continue robust and reliable infrastructure services while expanding into new areas of collaboration. Rob Qhttp://www.blogger.com/profile/06837197517253625772noreply@blogger.com0tag:blogger.com,1999:blog-7506259688180433777.post-49030588762720110252012-08-08T05:40:00.000-07:002012-08-08T17:00:16.126-07:00Reflecting on the Ghana Grid SchoolAs I sit in the final day of the <a href="http://africanschoolofphysics.web.cern.ch/africanschoolofphysics/AboutUs.html">African School for Fundamental Physics and its Applications</a> (ASP) at <a href="http://www.knust.edu.gh/pages/">Kwame Nkrumah University of Science and Technology</a> in Kumasi, Ghana I can't help but reflect on the grid portion of course and my interactions with the students and my fellow instructors.<br />
<br />
On arrival in Accra, Ghana I met up with some long time OSG and DOSAR VO collaborators Horst Severini from Oklahoma University and Scot Kronenfeld from the University of Wisconsin Madison. I was also introduced to a few new colleagues that would be joining us during the Grid School, Julia Gray from CERN and Stony Brook University and Pat Skubic from Oklahoma University. After a short flight from Accra to Kumasi we spent several days configuring the machines in the classroom to act as a Condor cluster and working through the lessons one final time trying to find any final adjustments to the <a href="http://twiki.grid.iu.edu/bin/view/Education/AfricaGridSchool12Materials">course material</a>. We were joined by a few other well known OSG faces at the end of the weekend, Dick Greenwood from Louisiana Tech and Jae Yu from the University of Texas Arlington completing the instructional staff. <br />
<br />
We were able to make our first visit to the classroom on Friday evening and found a modern computer classroom with reliable, though not necessarily robust network. After a few hours of tweaking the machines in the classroom we were confident the school was ready to begin on Monday morning.<br />
<br />
The weekend consisted of a bit of free time, we did some shopping and we visited the woodworking village Ahwaii on the outskirts on Kumasi Saturday and held a session with ASP students on Sunday where each US school represented talked with students about graduate student opportunities. <br />
<br />
Monday brought the start of class and much to my surprise a long time friend, and former South Padre Grid School (2004) roommate, Addy Tettah who was in attendance to help with instruction. The lectures by Scot introducing Condor and HTC concepts were excellent, but the hands on exercises were a challenge, mostly due to the students being unfamiliar with UNIX navigation.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGsIJarOvN00-Vw-AEDjccRF8YqqxqkBZCJmvbvbzSSfQ-wsmlKBkdAC5ig5VoMJyCDABuYkvKAXGxk_AalOwhbsCfLqhCT1ncVkE41i-VdskNEYGv7REzEemr5OluIbvAcDJMFl_zqvop/s1600/IMG_1399.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGsIJarOvN00-Vw-AEDjccRF8YqqxqkBZCJmvbvbzSSfQ-wsmlKBkdAC5ig5VoMJyCDABuYkvKAXGxk_AalOwhbsCfLqhCT1ncVkE41i-VdskNEYGv7REzEemr5OluIbvAcDJMFl_zqvop/s320/IMG_1399.jpg" width="263" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Addy and Rob renewing a long time friendship. </td></tr>
</tbody></table>
<br />
<br />
It was my turn Tuesday, recapping day one and introducing the students to the Open Science Grid, BLAST, DHTC and Glide-Ins. Hands on exercises went much faster, mostly due to the students quickly grasping basic UNIX. Horst took the afternoon session to introduce storage and SRM. Real science concepts such as BLAST seemed to excite and energize the learning process. I left day two both elated and exhausted.<br />
<br />
This morning we are back to physics and examples based on the ROOT software. Time is flying and we have the afternoon scheduled to talk with students about their experience and answer any questions they have for us.<br />
<br />
It's been a whirlwind trip, but I'm honored to have been a part of ASP 2012 and am very proud of all the hard working students. Seeing the increase in knowledge and understanding in such a short time is extremely rewarding. I've greatly enjoyed my time in Ghana and working with collaborators and future grid users has been gratifying and the extra bonus of renewing an acquaintance after several long years on his home turf, this has turned into an unforgettable experience I will treasure for a very long time. I look forward to hearing from these students as they begin their scientific careers and to return for the next Grid School, no matter where it happens to be held. <br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhq1jv5ztHyjCIszFKhg70rkUtsZMCU5HUFtue5jHw_RjKZwDmnNM7hQA8NDE34UUYqyL5GI4bht7evA8iseM0lRO0_xxg3mkROEzbabRnGsNGd0TpzwXKq840oiRzajCIVPZQWbRqlZ9sY/s1600/Screen+Shot+2012-08-08+at+7.45.37+AM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="170" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhq1jv5ztHyjCIszFKhg70rkUtsZMCU5HUFtue5jHw_RjKZwDmnNM7hQA8NDE34UUYqyL5GI4bht7evA8iseM0lRO0_xxg3mkROEzbabRnGsNGd0TpzwXKq840oiRzajCIVPZQWbRqlZ9sY/s320/Screen+Shot+2012-08-08+at+7.45.37+AM.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Students and Instructors of the ASP2012 Grid School.</td></tr>
</tbody></table>
<br />
Rob Quick<br />
OSG Operations Area Coordinator<br />
Indiana UniversityRob Qhttp://www.blogger.com/profile/06837197517253625772noreply@blogger.com0tag:blogger.com,1999:blog-7506259688180433777.post-15662155604925770112012-05-07T09:32:00.002-07:002012-05-07T09:32:49.948-07:00CondorWeek 2012During the week of May 1-4, I attended <a href="http://research.cs.wisc.edu/condor/CondorWeek2012/">CondorWeek 2012</a> in Madison, WI. The event was hosted in the <a href="http://discovery.wisc.edu/">Wisconsin Institutes for Discovery</a> at the <a href="http://www.wisc.edu/">University of Wisconsin-Madison</a>.<br />
<br />
The first day consisted of tutorials explaining the basics of <a href="http://research.cs.wisc.edu/condor/">Condor</a> usage, workflows, administration, and security. Having spent more time doing general support with OSG and working with the support workflow, I haven't spent as much time as I'd like with Condor, but the tutorials did show me some things that I would like to try. I think that next year, these tutorials would be even more useful for me after trying some new things with Condor in the coming months.<br />
<br />
The final three days consisted of researchers, admins, and companies showing and discussing off the multitude of ways that they use Condor to enhance and extend their workflows. Of particular interest to me was the way that DreamWorks used Condor to send jobs to their rendering software to create their animated films. They showed the trailer for Madagascar 3, which is their first movie to be created using Condor from beginning to end in the rendering process.<br />
<br />
It was interesting to hear other outfits explain how they have had to work around some things Condor currently can't do and see how the Condor team was curious to see if these things could be implemented into coming versions of Condor. The team was very interested to expand and enhance Condor, which is already a very capable product.<br />
<br />
Overall it was a good experience that made me aware of all the things that can be done with Condor. I look forward to trying some of these things for my own, and hopefully returning next year with a much better understanding of Condor as a whole so that I can gain even more information from the talks and tutorials.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-7506259688180433777.post-72143111895209552812012-03-29T08:04:00.003-07:002012-05-07T07:53:54.925-07:00RabbitMQ & CometDWe've been experimenting with following new services in the last few month here at GOC.<br />
<ul>
<li>event.grid.iu.edu (RabbitMQ/AMQP Server)</li>
<li>comet.grid.iu.edu (CometD Server) </li>
</ul>
<div>
RabbitMQ (event.grid.iu.edu) allows GOC, and OSG users to publish and/or subscribe to various messages generated by our services, and OSG in general. Currently, we receive following messages.</div>
<div>
<ul>
<li>RSV status changes</li>
<li>OIM updates</li>
<li>GOC Ticket updates</li>
<li>GIP information changes (prototype)</li>
</ul>
<div>
Anyone can subscribe to these messages in XML format, and be notified in real time using AMQP messaging client. APIs are available in many languages including, Java, PHP, Python, etc.. </div>
</div>
<div>
<br /></div>
<div>
CometD (comet.grid.iu.edu) allows us to push messages to our various web applications. For example, GOC Ticket uses it to display users who are currently viewing a ticket. If someone updates a ticket while someone else is viewing, it will send page refresh request to all viewers. CometD can also be used to implement features such as chat, shared editing, and other functionalists.</div>
<div>
<br /></div>
<div>
CometD itself is a Java application framework where we can implement various services that client (web browsers) can make requests to. CometD acts as a glue between RabbitMQ and the web browsers. For example, "GOC event service" in comet.grid.iu.edu subscribes to RSV, OIM, and GOC tickets events, and pools all recent events. A web browsers can then make a request to download these events during the initial loading of a page, and it will subscribe to "new event" queue on comet in order to receive new events in real time until user closes the page.</div>
<div>
<br /></div>
<div>
By using Event & Comet services, we can implement interesting features such as <a href="http://myosg.grid.iu.edu/miscevent/index?datasource=event&count_sg_1=on&count_active=on&count_enabled=on">Realtime GOC event</a> (prototype) in MyOSG. My current goal is to continue experimenting with RabbitMQ/CometD and see what I can (and can not) accomplishing using these tools. </div>
<div>
<br /></div>
<div>
If anyone has an idea about what we can do with these tools, please feel free to send me a message.</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>Anonymoushttp://www.blogger.com/profile/06484174015994780168noreply@blogger.com0tag:blogger.com,1999:blog-7506259688180433777.post-79157601304682346032012-02-15T07:46:00.000-08:002012-02-15T07:47:21.127-08:00New Home for OSG Web Site<!--[if gte mso 9]><xml> <o:documentproperties> <o:revision>0</o:Revision> <o:totaltime>0</o:TotalTime> <o:pages>1</o:Pages> <o:words>154</o:Words> <o:characters>884</o:Characters> <o:company>Indiana University</o:Company> <o:lines>7</o:Lines> <o:paragraphs>2</o:Paragraphs> <o:characterswithspaces>1036</o:CharactersWithSpaces> <o:version>14.0</o:Version> </o:DocumentProperties> <o:officedocumentsettings> <o:allowpng/> </o:OfficeDocumentSettings> </xml><![endif]--> <!--[if gte mso 9]><xml> <w:worddocument> <w:view>Normal</w:View> <w:zoom>0</w:Zoom> <w:trackmoves/> <w:trackformatting/> <w:punctuationkerning/> <w:validateagainstschemas/> <w:saveifxmlinvalid>false</w:SaveIfXMLInvalid> <w:ignoremixedcontent>false</w:IgnoreMixedContent> <w:alwaysshowplaceholdertext>false</w:AlwaysShowPlaceholderText> <w:donotpromoteqf/> <w:lidthemeother>EN-US</w:LidThemeOther> <w:lidthemeasian>JA</w:LidThemeAsian> <w:lidthemecomplexscript>X-NONE</w:LidThemeComplexScript> <w:compatibility> <w:breakwrappedtables/> <w:snaptogridincell/> <w:wraptextwithpunct/> <w:useasianbreakrules/> <w:dontgrowautofit/> <w:splitpgbreakandparamark/> <w:enableopentypekerning/> <w:dontflipmirrorindents/> <w:overridetablestylehps/> <w:usefelayout/> </w:Compatibility> <m:mathpr> <m:mathfont val="Cambria Math"> <m:brkbin val="before"> <m:brkbinsub val="--"> <m:smallfrac val="off"> <m:dispdef/> <m:lmargin val="0"> <m:rmargin val="0"> <m:defjc val="centerGroup"> <m:wrapindent val="1440"> <m:intlim val="subSup"> <m:narylim val="undOvr"> </m:mathPr></w:WordDocument> </xml><![endif]--><!--[if gte mso 9]><xml> <w:latentstyles deflockedstate="false" defunhidewhenused="true" defsemihidden="true" defqformat="false" defpriority="99" latentstylecount="276"> <w:lsdexception locked="false" priority="0" semihidden="false" unhidewhenused="false" qformat="true" name="Normal"> <w:lsdexception locked="false" priority="9" semihidden="false" unhidewhenused="false" qformat="true" name="heading 1"> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 2"> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 3"> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 4"> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 5"> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 6"> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 7"> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 8"> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 9"> <w:lsdexception locked="false" priority="39" name="toc 1"> <w:lsdexception locked="false" priority="39" name="toc 2"> <w:lsdexception locked="false" priority="39" name="toc 3"> <w:lsdexception locked="false" priority="39" name="toc 4"> <w:lsdexception locked="false" priority="39" name="toc 5"> <w:lsdexception locked="false" priority="39" name="toc 6"> <w:lsdexception locked="false" priority="39" name="toc 7"> <w:lsdexception locked="false" priority="39" name="toc 8"> <w:lsdexception locked="false" priority="39" name="toc 9"> <w:lsdexception locked="false" priority="35" qformat="true" name="caption"> <w:lsdexception locked="false" priority="10" semihidden="false" unhidewhenused="false" qformat="true" name="Title"> <w:lsdexception locked="false" priority="1" name="Default Paragraph Font"> <w:lsdexception locked="false" priority="11" semihidden="false" unhidewhenused="false" qformat="true" name="Subtitle"> <w:lsdexception locked="false" priority="22" semihidden="false" unhidewhenused="false" qformat="true" name="Strong"> <w:lsdexception locked="false" priority="20" semihidden="false" unhidewhenused="false" qformat="true" name="Emphasis"> <w:lsdexception locked="false" priority="59" semihidden="false" unhidewhenused="false" name="Table Grid"> <w:lsdexception locked="false" unhidewhenused="false" name="Placeholder Text"> <w:lsdexception locked="false" priority="1" semihidden="false" unhidewhenused="false" qformat="true" name="No Spacing"> <w:lsdexception locked="false" priority="60" semihidden="false" unhidewhenused="false" name="Light Shading"> <w:lsdexception locked="false" priority="61" semihidden="false" unhidewhenused="false" name="Light List"> <w:lsdexception locked="false" priority="62" semihidden="false" unhidewhenused="false" name="Light Grid"> <w:lsdexception locked="false" priority="63" semihidden="false" unhidewhenused="false" name="Medium Shading 1"> <w:lsdexception locked="false" priority="64" semihidden="false" unhidewhenused="false" name="Medium Shading 2"> <w:lsdexception locked="false" priority="65" semihidden="false" unhidewhenused="false" name="Medium List 1"> <w:lsdexception locked="false" priority="66" semihidden="false" unhidewhenused="false" name="Medium List 2"> <w:lsdexception locked="false" priority="67" semihidden="false" unhidewhenused="false" name="Medium Grid 1"> <w:lsdexception locked="false" priority="68" semihidden="false" unhidewhenused="false" name="Medium Grid 2"> <w:lsdexception locked="false" priority="69" semihidden="false" unhidewhenused="false" name="Medium Grid 3"> <w:lsdexception locked="false" priority="70" semihidden="false" unhidewhenused="false" name="Dark List"> <w:lsdexception locked="false" priority="71" semihidden="false" unhidewhenused="false" name="Colorful Shading"> <w:lsdexception locked="false" priority="72" semihidden="false" unhidewhenused="false" name="Colorful List"> <w:lsdexception locked="false" priority="73" semihidden="false" unhidewhenused="false" name="Colorful Grid"> <w:lsdexception locked="false" priority="60" semihidden="false" unhidewhenused="false" name="Light Shading Accent 1"> <w:lsdexception locked="false" priority="61" semihidden="false" unhidewhenused="false" name="Light List Accent 1"> <w:lsdexception locked="false" priority="62" semihidden="false" unhidewhenused="false" name="Light Grid Accent 1"> <w:lsdexception locked="false" priority="63" semihidden="false" unhidewhenused="false" name="Medium Shading 1 Accent 1"> <w:lsdexception locked="false" priority="64" semihidden="false" unhidewhenused="false" name="Medium Shading 2 Accent 1"> <w:lsdexception locked="false" priority="65" semihidden="false" unhidewhenused="false" name="Medium List 1 Accent 1"> <w:lsdexception locked="false" unhidewhenused="false" name="Revision"> <w:lsdexception locked="false" priority="34" semihidden="false" unhidewhenused="false" qformat="true" name="List Paragraph"> <w:lsdexception locked="false" priority="29" semihidden="false" unhidewhenused="false" qformat="true" name="Quote"> <w:lsdexception locked="false" priority="30" semihidden="false" unhidewhenused="false" qformat="true" name="Intense Quote"> <w:lsdexception locked="false" priority="66" semihidden="false" unhidewhenused="false" name="Medium List 2 Accent 1"> <w:lsdexception locked="false" priority="67" semihidden="false" unhidewhenused="false" name="Medium Grid 1 Accent 1"> <w:lsdexception locked="false" priority="68" semihidden="false" unhidewhenused="false" name="Medium Grid 2 Accent 1"> <w:lsdexception locked="false" priority="69" semihidden="false" unhidewhenused="false" name="Medium Grid 3 Accent 1"> <w:lsdexception locked="false" priority="70" semihidden="false" unhidewhenused="false" name="Dark List Accent 1"> <w:lsdexception locked="false" priority="71" semihidden="false" unhidewhenused="false" name="Colorful Shading Accent 1"> <w:lsdexception locked="false" priority="72" semihidden="false" unhidewhenused="false" name="Colorful List Accent 1"> <w:lsdexception locked="false" priority="73" semihidden="false" unhidewhenused="false" name="Colorful Grid Accent 1"> <w:lsdexception locked="false" priority="60" semihidden="false" unhidewhenused="false" name="Light Shading Accent 2"> <w:lsdexception locked="false" priority="61" semihidden="false" unhidewhenused="false" name="Light List Accent 2"> <w:lsdexception locked="false" priority="62" semihidden="false" unhidewhenused="false" name="Light Grid Accent 2"> <w:lsdexception locked="false" priority="63" semihidden="false" unhidewhenused="false" name="Medium Shading 1 Accent 2"> <w:lsdexception locked="false" priority="64" semihidden="false" unhidewhenused="false" name="Medium Shading 2 Accent 2"> <w:lsdexception locked="false" priority="65" semihidden="false" unhidewhenused="false" name="Medium List 1 Accent 2"> <w:lsdexception locked="false" priority="66" semihidden="false" unhidewhenused="false" name="Medium List 2 Accent 2"> <w:lsdexception locked="false" priority="67" semihidden="false" unhidewhenused="false" name="Medium Grid 1 Accent 2"> <w:lsdexception locked="false" priority="68" semihidden="false" unhidewhenused="false" name="Medium Grid 2 Accent 2"> <w:lsdexception locked="false" priority="69" semihidden="false" unhidewhenused="false" name="Medium Grid 3 Accent 2"> <w:lsdexception locked="false" priority="70" semihidden="false" unhidewhenused="false" name="Dark List Accent 2"> <w:lsdexception locked="false" priority="71" semihidden="false" unhidewhenused="false" name="Colorful Shading Accent 2"> <w:lsdexception locked="false" priority="72" semihidden="false" unhidewhenused="false" name="Colorful List Accent 2"> <w:lsdexception locked="false" priority="73" semihidden="false" unhidewhenused="false" name="Colorful Grid Accent 2"> <w:lsdexception locked="false" priority="60" semihidden="false" unhidewhenused="false" name="Light Shading Accent 3"> <w:lsdexception locked="false" priority="61" semihidden="false" unhidewhenused="false" name="Light List Accent 3"> <w:lsdexception locked="false" priority="62" semihidden="false" unhidewhenused="false" name="Light Grid Accent 3"> <w:lsdexception locked="false" priority="63" semihidden="false" unhidewhenused="false" name="Medium Shading 1 Accent 3"> <w:lsdexception locked="false" priority="64" semihidden="false" unhidewhenused="false" name="Medium Shading 2 Accent 3"> <w:lsdexception locked="false" priority="65" semihidden="false" unhidewhenused="false" name="Medium List 1 Accent 3"> <w:lsdexception locked="false" priority="66" semihidden="false" unhidewhenused="false" name="Medium List 2 Accent 3"> <w:lsdexception locked="false" priority="67" semihidden="false" unhidewhenused="false" name="Medium Grid 1 Accent 3"> <w:lsdexception locked="false" priority="68" semihidden="false" unhidewhenused="false" name="Medium Grid 2 Accent 3"> <w:lsdexception locked="false" priority="69" semihidden="false" unhidewhenused="false" name="Medium Grid 3 Accent 3"> <w:lsdexception locked="false" priority="70" semihidden="false" unhidewhenused="false" name="Dark List Accent 3"> <w:lsdexception locked="false" priority="71" semihidden="false" unhidewhenused="false" name="Colorful Shading Accent 3"> <w:lsdexception locked="false" priority="72" semihidden="false" unhidewhenused="false" name="Colorful List Accent 3"> <w:lsdexception locked="false" priority="73" semihidden="false" unhidewhenused="false" name="Colorful Grid Accent 3"> <w:lsdexception locked="false" priority="60" semihidden="false" unhidewhenused="false" name="Light Shading Accent 4"> <w:lsdexception locked="false" priority="61" semihidden="false" unhidewhenused="false" name="Light List Accent 4"> <w:lsdexception locked="false" priority="62" semihidden="false" unhidewhenused="false" name="Light Grid Accent 4"> <w:lsdexception locked="false" priority="63" semihidden="false" unhidewhenused="false" name="Medium Shading 1 Accent 4"> <w:lsdexception locked="false" priority="64" semihidden="false" unhidewhenused="false" name="Medium Shading 2 Accent 4"> <w:lsdexception locked="false" priority="65" semihidden="false" unhidewhenused="false" name="Medium List 1 Accent 4"> <w:lsdexception locked="false" priority="66" semihidden="false" unhidewhenused="false" name="Medium List 2 Accent 4"> <w:lsdexception locked="false" priority="67" semihidden="false" unhidewhenused="false" name="Medium Grid 1 Accent 4"> <w:lsdexception locked="false" priority="68" semihidden="false" unhidewhenused="false" name="Medium Grid 2 Accent 4"> <w:lsdexception locked="false" priority="69" semihidden="false" unhidewhenused="false" name="Medium Grid 3 Accent 4"> <w:lsdexception locked="false" priority="70" semihidden="false" unhidewhenused="false" name="Dark List Accent 4"> <w:lsdexception locked="false" priority="71" semihidden="false" unhidewhenused="false" name="Colorful Shading Accent 4"> <w:lsdexception locked="false" priority="72" semihidden="false" unhidewhenused="false" name="Colorful List Accent 4"> <w:lsdexception locked="false" priority="73" semihidden="false" unhidewhenused="false" name="Colorful Grid Accent 4"> <w:lsdexception locked="false" priority="60" semihidden="false" unhidewhenused="false" name="Light Shading Accent 5"> <w:lsdexception locked="false" priority="61" semihidden="false" unhidewhenused="false" name="Light List Accent 5"> <w:lsdexception locked="false" priority="62" semihidden="false" unhidewhenused="false" name="Light Grid Accent 5"> <w:lsdexception locked="false" priority="63" semihidden="false" unhidewhenused="false" name="Medium Shading 1 Accent 5"> <w:lsdexception locked="false" priority="64" semihidden="false" unhidewhenused="false" name="Medium Shading 2 Accent 5"> <w:lsdexception locked="false" priority="65" semihidden="false" unhidewhenused="false" name="Medium List 1 Accent 5"> <w:lsdexception locked="false" priority="66" semihidden="false" unhidewhenused="false" name="Medium List 2 Accent 5"> <w:lsdexception locked="false" priority="67" semihidden="false" unhidewhenused="false" name="Medium Grid 1 Accent 5"> <w:lsdexception locked="false" priority="68" semihidden="false" unhidewhenused="false" name="Medium Grid 2 Accent 5"> <w:lsdexception locked="false" priority="69" semihidden="false" unhidewhenused="false" name="Medium Grid 3 Accent 5"> <w:lsdexception locked="false" priority="70" semihidden="false" unhidewhenused="false" name="Dark List Accent 5"> <w:lsdexception locked="false" priority="71" semihidden="false" unhidewhenused="false" name="Colorful Shading Accent 5"> <w:lsdexception locked="false" priority="72" semihidden="false" unhidewhenused="false" name="Colorful List Accent 5"> <w:lsdexception locked="false" priority="73" semihidden="false" unhidewhenused="false" name="Colorful Grid Accent 5"> <w:lsdexception locked="false" priority="60" semihidden="false" unhidewhenused="false" name="Light Shading Accent 6"> <w:lsdexception locked="false" priority="61" semihidden="false" unhidewhenused="false" name="Light List Accent 6"> <w:lsdexception locked="false" priority="62" semihidden="false" unhidewhenused="false" name="Light Grid Accent 6"> <w:lsdexception locked="false" priority="63" semihidden="false" unhidewhenused="false" name="Medium Shading 1 Accent 6"> <w:lsdexception locked="false" priority="64" semihidden="false" unhidewhenused="false" name="Medium Shading 2 Accent 6"> <w:lsdexception locked="false" priority="65" semihidden="false" unhidewhenused="false" name="Medium List 1 Accent 6"> <w:lsdexception locked="false" priority="66" semihidden="false" unhidewhenused="false" name="Medium List 2 Accent 6"> <w:lsdexception locked="false" priority="67" semihidden="false" unhidewhenused="false" name="Medium Grid 1 Accent 6"> <w:lsdexception locked="false" priority="68" semihidden="false" unhidewhenused="false" name="Medium Grid 2 Accent 6"> <w:lsdexception locked="false" priority="69" semihidden="false" unhidewhenused="false" name="Medium Grid 3 Accent 6"> <w:lsdexception locked="false" priority="70" semihidden="false" unhidewhenused="false" name="Dark List Accent 6"> <w:lsdexception locked="false" priority="71" semihidden="false" unhidewhenused="false" name="Colorful Shading Accent 6"> <w:lsdexception locked="false" priority="72" semihidden="false" unhidewhenused="false" name="Colorful List Accent 6"> <w:lsdexception locked="false" priority="73" semihidden="false" unhidewhenused="false" name="Colorful Grid Accent 6"> <w:lsdexception locked="false" priority="19" semihidden="false" unhidewhenused="false" qformat="true" name="Subtle Emphasis"> <w:lsdexception locked="false" priority="21" semihidden="false" unhidewhenused="false" qformat="true" name="Intense Emphasis"> <w:lsdexception locked="false" priority="31" semihidden="false" unhidewhenused="false" qformat="true" name="Subtle Reference"> <w:lsdexception locked="false" priority="32" semihidden="false" unhidewhenused="false" qformat="true" name="Intense Reference"> <w:lsdexception locked="false" priority="33" semihidden="false" unhidewhenused="false" qformat="true" name="Book Title"> <w:lsdexception locked="false" priority="37" name="Bibliography"> <w:lsdexception locked="false" priority="39" qformat="true" name="TOC Heading"> </w:LatentStyles> </xml><![endif]--> <!--[if gte mso 10]> <style> /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:12.0pt; font-family:Cambria; mso-ascii-font-family:Cambria; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:Cambria; mso-hansi-theme-font:minor-latin;} </style> <![endif]--> <!--StartFragment--> <p class="MsoNormal">On Frebruary 28<sup>th</sup> the OSG Webpages located at <a href="http://www.opensciencegrid.org">www.opensciencegrid.org</a> will move to the OSG Twiki. This move corresponds with the upcoming conclusion of a contract with the Chicago based web hosting service Tilted Planet.</p> <p class="MsoNormal">During the scheduled production service update on the 28<sup>th</sup>, browsers will be redirected from the current location to the new location on the main OSG Twiki page. A mockup of the new page can be seen at twiki-itb.grid.iu.edu.</p> <p class="MsoNormal">This is the first step, and likely an interim web page home, in a project that will affect the OSG Public Web Pages, the OSG Twiki, the DocDB, and possibly other OSG services with web-UI’s. Evaluation of content management systems, wikis, and documentation file database solutions has already begun and will continue over the next several months. Please contact the GOC (<a href="mailto:goc@opensciencegrid.org">goc@opensciencegrid.org</a>) if you have a suggestion for packages you think should be evaluated. <o:p></o:p></p> <!--EndFragment-->Rob Qhttp://www.blogger.com/profile/06837197517253625772noreply@blogger.com0tag:blogger.com,1999:blog-7506259688180433777.post-50190845695213718032011-11-21T06:05:00.000-08:002011-11-21T06:06:21.662-08:00GOC holiday scheduleFrom 24/Nov through 27/Nov the GOC will be operating on a Holiday<br />schedule. Staff will be available to respond to emergencies but<br />routine operations will resume at start of business Monday 28/Nov.<br /><br />The GOC wishes its users and OSG staff a happy and satisfying<br />Thanksgiving Holiday.scotthttp://www.blogger.com/profile/16325987765411148883noreply@blogger.com0tag:blogger.com,1999:blog-7506259688180433777.post-61242491121991042322011-11-03T08:07:00.000-07:002011-11-03T08:08:33.356-07:00Moving Services to BloomingtonAs you know, the GOC updates services on the second and fourth Tuesday of each month.<br />The update scheduled for November 8th marks a milestone for the infrastructure team.<br />After this date all GOC services (with one exception) be be hosted exclusively in<br />the Bloomington, Indiana data center.<br /><br />Previously, most services had two instances, one physically hosted in Indianapolis<br />the other in Bloomington. These instances are in DNS round robin allowing users<br />of these services transparent use of either instance. The GOC will continue to<br />operate (at least) two instances and keep them in round robin, but both instances<br />will be in Bloomington.<br /><br />So why the change? Originally, the Bloomington machine room was extremely unreliable.<br />Problems included a leaky roof, insufficient cooling and power and a lack<br />of space. In short, the systems hosted there had outgrown the facility. The machine<br />room in Indianapolis was larger, newer and considered more reliable. The old Bloomington<br />machine room went down during a thunderstorm when it was discovered that both electrical<br />feeds were, at one point, hung from the same utility pole. (Care to guess where the<br />lightning struck?) Two weeks were required to restore power during which many of the<br />university enterprise services were unavailable. This situation was clearly unacceptable<br />so the university decided to invest $37.2M in a new, state-of-the-art data center.<br /><br />The 92,000 sq. ft. Bloomington data center is designed to withstand category 5 tornadoes.<br />The facility is secured with card-key access and 7 x 24 x 365 video surveillance.<br />Only staff with systems or network administration privileges have access to the machine room<br />requiring biometric identity verification. Fire suppression is provided by a double interlock<br />system accompanied by a Very Early Smoke Detection Apparatus (VESDA). Three circuits feed<br />the Data Center, traveling redundant physical paths from two different substations.<br />Any two circuits can fully power the building. A flywheel motor/generator set conditions<br />the power and provides protection against transient events and uninterruptible power<br />supplies protect against failures of moderate (~1 hour) duration. Dual diesel generators<br />can provide power for 24 hours in the event of a longer term power failure. In house<br />chillers provide cooling. Externally supplied chilled water plus city water can be used<br />in the event of a failure of this system.<br /><br />Several advantages are realized by hosting all instances in one location. Service failures<br />associated with the network between Indianapolis and Bloomington are avoided. By using the<br />same LAN, DNS round robin can be replaced with Linux Virtual Server (LVS) giving control<br />of round robin to the GOC rather than the DNS administrators at Indiana University. Also<br />avoided are failures associated with the loss of one of two data centers. It is trivial to<br />move virtual machines from host to host since the IP address of the VM does not change, a<br />property allowing detailed load balancing on all VM hosts.<br /><br />The GOC looks forward to continuing providing services with the availability OSG users<br />have come to expect.scotthttp://www.blogger.com/profile/16325987765411148883noreply@blogger.com58tag:blogger.com,1999:blog-7506259688180433777.post-37625379742487895342011-11-01T10:14:00.000-07:002011-11-01T10:16:36.102-07:00EGI Technical ForumWrote this for the OSG Newsletter, but thought it would be good to drop here also.<br /><br />Henry Kissinger famously asked, “Who do I call if I want to call Europe?” I was reminded of this quote when I attended the EGI Technical forum in Lyon, France and then visited CERN in September. While I don’t need to call Europe as a whole, as Operations Coordinator for OSG I may need to contact any one of the 30+ National Grid Infrastructures (NGI) that make up the European Grid Infrastructure (EGI) in times of an operational crisis.<br /><br />Accompanied by Scott Teige the OSG Operations Technical Lead, we presented material on both technical and personal communications between OSG, WLCG, and EGI including Global Grid User Support System (GGUS) ticket synchronization, availability and reliability reporting, Berkeley Database Information Index (BDII) information exchange, and various other ways to keep communication channels open between OSG and our European counterparts. We came away with several technical action items from the SAM team. In addition, OSG Operations took a seat as a non-voting member of the EGI Operations Management Board. We are also participating in the WLCG Operations Technical Evaluation Group which will provide input to chart the future activities within the WLCG. We look forward to continued collaboration with WLCG and EGI on an operational level.<br /><br />~ Rob QuickRob Qhttp://www.blogger.com/profile/06837197517253625772noreply@blogger.com0tag:blogger.com,1999:blog-7506259688180433777.post-22337494658347087152011-10-02T11:10:00.000-07:002011-10-02T12:03:39.254-07:00Rob and Scott go to CERN and EGI Technical ForumEarly in the afternoon of 15th of September Rob Quick and I set off for Geneva to visit our collaborators at CERN and attend the European Grid Initiative (EGI) technical forum. We arrived without difficulty at 7:55 AM on the 16th and decided to get the paperwork needed to become CERN users completed. Paperwork being what it is, this turned out to consume the entire day. Many thanks are due to David Collados who picked us up and at days end delivered us to our hotel.<br /><br />We used Saturday to adjust our clocks by six timezones, working the entire day before actually helped. We hopped a bus to Ferney just across the border in France. There was an open air market where I bought some olives, bread, beer and goat cheese. Turns out David lives near the market and he and his wife gave us a ride back to Geneva. We visited the UN and gorged ourselves of fondue at a small local restaurant. Later in the evening Rob and I made dinner of my Ferney purchase.<br /><br />On Sunday we hopped a train to Lyon, France for the EGI conference. Quite an enjoyable trip through the mountains. Our talks were on Monday and we learned some interesting things and met many people we knew only by e-mail or con-call. Wednesday was a dinner at a Paul Bocuse restaurant featuring a dessert that made quite the impression on both of us. Raspberries, Chocolate Mousse and Ice Cream. On Friday we knocked off early and did tourist things by visiting the Roman ruins near the confluence of the Saone and Rhone rivers. Those folks built to last. Saturday was back on the train to Geneva.<br /><br />On Sunday we visited the old part of Geneva. As far as I could tell, old was defined as being contained within the original city walls. I ate a huge bucket of Mussels cooked in white wine, garlic and saffron. Rob had perch and we both had fries, bread and beer. Three days of very productive meetings with our CERN counterparts then back home Thursday at noon. We had a bit of a delay between Washington and Indianapolis but nothing serious.<br /><br />We both needed Friday to recover, Saturday I started catching up on chores neglected in my absence. Monday I will do the same at work. All in all, a productive if tiring trip.<br /><br />Sscotthttp://www.blogger.com/profile/16325987765411148883noreply@blogger.com0tag:blogger.com,1999:blog-7506259688180433777.post-73470356402343406772011-07-21T07:11:00.000-07:002011-08-02T11:35:57.654-07:002011 OSG Summer SchoolLast month I had the privilege to attend 2011 OSG Summer School in Madison Wisconsin with a group of select professionals from various countries around the world. My expectation of the class was at the very least, to gain a better understanding of the users perspective on the OSG. I thought that if I could fully experience the user side of OSG, I'd be able to offer even better assistance in resolving issues that are presented to the members of the GOC on a daily basis.<br /><br />One of the many benefits of attending 2011 OSG Summer School was the opportunity to personally meet all those associated with the OSG face-to-face in Madison. Miron Livny, Alain Roy, Tim Cartwright and Sarah Cushing were all very cordial and highly attentive to the needs of the students and visitors. I really liked the structure of the classes with equal emphasis on lecture and hands on sessions. It divided the day nicely allowing for interaction with my fellow students. For me, the 2 most enjoyable parts of 2011 OSG Summer School were the presentation by Miron Livny and the High-Throughput Computing Showcase. Miron's lecture on HTC which included a little history lesson helped paint a clearer picture in my mind of the OSG. The High-Throughput Computing Showcase invited four presenters (two faculty, two grad students) from different University of Wisconsin labs that make extensive use of local HTC resources and/or OSG. They shared a brief background of their work and how much of a role OSG has aided them in obtaining their goals as scientists. I found the presentations to be very exciting and informative. I've gained a deeper appreciation for the OSG and the collaborators that have worked together in making such an impact on the world we live in.<br /><br />With a fuller knowledge of each step in submitting a job to the grid, I will be able to efficiently assess the the many issues a user may encounter and provide a solution or direction with greater accuracy. I'll also be able to provide the users assistance with a multitude of tools and technologies, and assist them in efficiently and properly utilizing the OSG.<br /><br />2011 OSG Summer School was educational and fun. I highly recommend 2011 OSG Summer School for everyone interested in gaining a better understanding of the OSG and making connections with fellow students from around the world. Whether you're part of operations to support the OSG or a scientist in search of better ways to handle your data, there's a lot to be gained from attending the 2011 OSG Summer School.<br /><br />-Alain DeximoAlain Dhttp://www.blogger.com/profile/08261201344258504963noreply@blogger.com0tag:blogger.com,1999:blog-7506259688180433777.post-22362626180905715412011-06-16T20:38:00.000-07:002011-06-16T20:38:36.403-07:00GOC Alert & Service Monitor ScriptWe maintain many different services at OSG Operations Center; OIM, MyOSG, GOC Ticket, Software, BDII, TWiki, Jira, OSG Display, to name a few. We also have many other auxiliary services that works somewhat behind the scene; RSVProcess, Data Cluster, Monitor, Internal, Jump, RSV Clients, GOC TX, etc, etc.. Many of these services have multiple instances, and we also have ITB instances of most of these services, and some even have dedicated development instances. To make it worse, each services usually have more than 1 applications on them (Myosg has half a dozen consolidator applications) and most of them were developed & maintained by us. <br />
<br />
Sometime I wonder how on earth we are keeping up with all of these, but one of the infrastructures that has helped us in recent years is a system we call "GOC Alert & Service Monitoring" system.<br />
<br />
Since there is no way to monitor every single services and instances manually, we have a set of scripts for each service that monitor themselves. Also, our services are very dynamic; meaning they are constantly updated and environment that surrounds them also changes frequently which creates new set of issues and possible scenarios that something could go wrong. Unlike Nagios or RSV where similar tests are executed across all services, our services have their unique set of scripts which are constantly updated and fine tuned during a course of our normal operations. If anything goes wrong, it is usually considered a failure of our service monitoring script and they will be updated to prevent any future occurrence of a similar issue. (We also have meta-service monitor which remotely checks our service monitors that they are executed on each services.)<br />
<br />
Service monitor scripts are usually simple shell scripts that sends out messages if something is wrong or about to go wrong. The messages are sent directly to our off-site messaging bus which we call "GOC Alert". Currently, GOC Alert is implemented using Google Group's mailing list. The idea here is that, if something goes wrong at Operations Center, we want to ship that information out of our environment as soon as possible, and as far away as possible in order to prevent such messages to be destroyed or become unavailable during the outage. Have you wondered why airplanes store their flight data recorder inside each airplane, instead of sending out the information via satellite to a off-site location so that people can analyze them as soon as any incident occurs? A similar idea here.<br />
<br />
Every single services at Operations Center posts messages with its own tags to GOC Alert. Once the messages are published there, we can then subscribe to those messages with specific tags or keywords in various kinds of ways. We can setup an email filter which will only pull certain tags, or let another monitoring service to handle messages posted. One of such service is what we call "Central Monitor". It has a component called "GOC-Alert forwarder" which receives all GOC Alerts, and analyze its content, and based on information such as which service is having what type of problems, and when, it will send a message or simply forward the message to some destination. For example, I have setup our forwarder so that if one of our high priority service's RSV status becomes critical during my off-hours (after 5, weekened, etc.) it will send a SMS message to my cell phone with basic information about the error. Central Monitor service also tracks incoming critical priority GOC tickets and sends alerts to our group chat room via XMPP, among many other things that it tracks.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWOYix74fcu2tDP_M8bI-u_NyHvXzMmExQt_wilFYc7hQqL-YSyo3NQ801QjZ3CAC_IkeT8H20rvRODXk4lnswe2D_HFSOnNL5BTyxANYad8TXiscWHU_R_dIS8TOfwlLHYS6vOs4Qy4vG/s1600/GOCMonitoringLayers.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWOYix74fcu2tDP_M8bI-u_NyHvXzMmExQt_wilFYc7hQqL-YSyo3NQ801QjZ3CAC_IkeT8H20rvRODXk4lnswe2D_HFSOnNL5BTyxANYad8TXiscWHU_R_dIS8TOfwlLHYS6vOs4Qy4vG/s640/GOCMonitoringLayers.png" width="640" /></a></div><br />
Above diagram shows a simplified interaction between our various monitoring components which includes Service Monitor, GOC-Alert, and Central Monitor. There are several other components that works together that allows us to keep up with all of our services.Anonymoushttp://www.blogger.com/profile/06484174015994780168noreply@blogger.com1tag:blogger.com,1999:blog-7506259688180433777.post-45510740738961547222011-06-06T12:39:00.000-07:002011-06-06T20:20:39.671-07:00Jumping off point...Welcome to Inside OSG Ops, a place for stories revolving around the Operations services, events, and people that make up OSG Operations.<br />
<br />
I'm going to start out the new blog talking about the recent upgrade to the CERN Top-Level BDII, how it affected OSG Ops, and it's status as of today. A few weeks ago OSG Ops got wind of a WLCG Top-Level BDII Upgrade via the daily WLCG Ops call. At this time of reporting this upgrade (May, 12th) the change date was already quite near (May, 17th). This left a Friday and Monday to test. This led to a quite extensive debugging <a href="https://ticket.grid.iu.edu/goc/viewer?id=10411">ticket</a> opened on May, 13th when FNAL did not appear in the test systems.<br />
<br />
Due to this testing the release was delayed May, 26th. Though the initial discrepancies were never identified it was determined the upgrade was not the cause of the issue reported. Due to the fact we've never identified was caused the issue, or what the solution was the GOC implemented ongoing testing to determine if the WLCG Top-Level BDII or the OSG BDIIs drop any resources from reporting. Here is a recent visualization that shows the BDII stability over the past several days.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWU3xFldTVDh_ItGcwWHK-iD1CmoEY7Q_D4doYiI9wGvqm0o0nSCIejrGvTnrXFfCWaECe32M9BB1lxQPaIzzLsDLoiWM7uY7hFW9psf2pghkdhBCYhHc6C4QY2OqvH5wqu-oiDpcbNbcl/s1600/Picture+12.png" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5615199615751331314" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWU3xFldTVDh_ItGcwWHK-iD1CmoEY7Q_D4doYiI9wGvqm0o0nSCIejrGvTnrXFfCWaECe32M9BB1lxQPaIzzLsDLoiWM7uY7hFW9psf2pghkdhBCYhHc6C4QY2OqvH5wqu-oiDpcbNbcl/s320/Picture+12.png" style="display: block; height: 229px; margin: 0px auto 10px; text-align: center; width: 320px;" /></a><br />
As always, we'll keep watch for any unusual activity.<br />
<br />
Rob Quick<br />
OSG Operations CoordinatorRob Qhttp://www.blogger.com/profile/06837197517253625772noreply@blogger.com0