A "Birds of a Feather" Session held Monday, March 23, 1998 at the SPIE Astronomy Conference in Kona, Hawaii.
Co-chairs:
6:30 Kibrick: Introduction and Welcome
6:35 Kibrick: Summary of issues
6:50 Almes: Internet-2: current status and issues (deferred until later in session)
6:55 Wasley: CalREN-2: current status and issues
7:00 Griffin/Spence: NASA, NGI, and NREN
7:05 Cohen/Shopbell: Caltech tests with NASA ACTS satellite
7:10 Lewis: Keck remote observing from Waimea
7:15 Kibrick: Keck remote observing tests from mainland via Internet
7:20 Rhoads/Whinery: Options for improving Hawaii access
7:30 Kibrick: ISDN options for alternative access
7:35 Group: Technical and economic challenges of mainland link
7:45 Ogasawara: Network plans for Subara Telescope project
7:50 All: open forum
Opening Summary
A transcript of the opening "summary of issues" is being prepared and will be attached to the final draft of these minutes.
Minutes
The following minutes of the meeting were recorded by Will Deich of UCO/Lick Observatory, and they represent only a very partial transcript of the meeting. We have attempted to correctly attribute remarks to the individuals that made them, but there may be some errors, especially in terms of correctly identifying individuals who participated via teleconference.
Abbreviations:
Whinery: Mon-Fri, there is significant congestion at the Hawaii side of the 6Mbit/sec link between the mainland and Oahu. But more often congestion is in Stockton, California, which is where this link hits the mainland U.S., and beyond.
Q: Who pays for Hawaii-Oahu link?
Whinery: NASA (elaboration follows). Apparently this arrangement was negotiated some years ago (along with many other network links across the Pacific) by Torben Nielsen of the University of Hawaii. [Note: Nielsen was formerly on the faculty of the Department of Information and Computer Sciences at the University of Hawaii, Manoa campus, and is now with the Hawaii Institute of Geophysics and Planetology within the School Ocean and Earth Science and Technology (SOEST), University of Hawaii. The Oahu to mainland link was also originally arranged by Nielsen and funded by NASA many years ago, but is now funded by the University of Hawaii and managed by its Department of Information Technology Services.]
Griffin: NASA's management of the Hawaii-Oahu link is now administered by Marshall SFC in collaboration with Ames xxx. The Network Operations Center (NOC) at Ames handles these connections.
Griffin(?): What bandwidth is needed to support remote observing from the mainland?
Kibrick: At Keck, T1 is needed for readout of current generation of instruments; less when not reading out the detector (typically about 192 Kbits/sec). Next generation instruments will need peak higher rates, up to about 45 Mbit/sec peak rate for real-time instrument readout.
Wasley: CalREN-2
Oahu-Caltech goes through the Mae West connection and loses a great deal of data.
Suggest UH folks observe the paths taken today, and see where losses occur. Investigate alternate routes, or work with existing providers to avoid the congested exchange points.
Whinery: UH is considering changing Internet providers.
Wasley: One strategy is to get a good path to CalREN2 network. Peer w/ CalREN2 wherever it's convenient to do so. Also vBNS (in theory) would let you go cross-country; NSF permission is required for this but should be straightforward. PROBLEM: cost of connection from Hawaii to mainland. Our experience in California is that telecommunications carriers make money from dialtones. They price service based on the number phone calls they can support over a given bandwidth link. By contrast, other providers (e.g., SONET) which specialize in wholesale bandwidth may offer it at much lower price.
Cohen: I've priced private T1 between Pasadena and Waimea. It costs about $12,000/month.
Kibrick: We have, too. A T1 between Santa Cruz and Waimea will cost about $150,000/yr.
Cohen: This isn't a lot of money, but somebody has to put money on the table.
C: DS-3 is cheaper than aggregate T1's.
Rhoads: The cost for a DS-3 link between Hawaii and the mainland is $2million/yr (quote from Oceanic Cables, the Time-Warner company which provides cable TV and communications services in Hawaii.)
Q: What are yearly travel costs for sending observers from the mainland to observe at Keck?
Cohen: At Caltech, the annual costs for travel+airfare+hotel+car rental: $140,000; UC spends at at least that much and probably more.
[Note: NASA spends an additional amount to send NASA-sponsored observers from other mainland sites not associated with either UC or Caltech. Thus, aggregrate annual travel costs are at least $300,000 per year, and probably higher.]
Griffin + Spence: NASA, NGI, NREN
This is pretty interesting. We are getting ready to work with the Army at Tripler Medical Center on Oahu. They are connected to the Defense Research Network (DREN).
I understand there is a high-performance networking agreement in Maui, which we can get to via some kind of peer agreements.
Kibrick: We are aware of a high-speed connection between the mainland and the Maui High Performance Computing Center (MHPCC), but we currently have no means to connect to it from Keck.
Lewis: My understanding is that the Maui supercomputer folks are keen to have others use their capacity, at least at lower rates of bandwidth such as T1. The local high school in Waimea has a T1 line to MHPCC which they use for Internet access. It would cost $28K/year to rent a T1 for Maui from Waimea.
C: I understand connection for MHPCC is DS-3...
Wright: At Gemini we recently hired Tod Fujioka, former network manager at MHPCC. He says MHPCC has three DS-3 connections to mainland.
Rhoads: In December, there was a meeting at MHPCC. UH talked about maybe joint proposal to get into vBNS. MHPCC's lines don't currently connect into VBNS. MHPCC is interested in cooperative proposal to get onto VBNS.
Griffin: Department of Defense (DOD) in Hawaii has 2 x DS-3 links to mainland. May be able to negotiate a connection. Perhaps try to get funding to support a link between the existing networks on Hawaii (either from Oahu or Hawaii) to the DOD link on Maui.
Griffin/Spence(?): Noted that some of these DS-3 links carry classified traffic, but at least one probably does not, and is part of the Defense Research Network (DREN). _If_ it is possible to get onto DREN, then it isn't a problem to get from there to VBNS via NASA exchange point.
[Note: the following clarification was provided by Larry Bergman of JPL the day following the session: I had heard (and I think Bill Santos confirmed) that the Maui Supercomputer Center has two DS-3 links to the mainland. One is the earlier secure link that they have had all along (which is encrypted) that goes to Kirtland AFB in Albuquerque. Classified satellite data passes on this link, ultimately ending up at Cheyenne Mt in Colorado, and the second DS-3 is part of the DREN (and came on line recently). DREN is non-secure (in parts) and supposed to be used for universities and other institutions that are doing DoD related R&D using the Maui Supercomputer. You are right. I understand that they have much excess capacity. Its too bad that there isn't some little R&D collaboration between NASA and DoD (like the killer asteroids, or?) that might justify having some Keck users piggyback on the Maui DS-3 link. That would probably be the easiest way to get the service but may not satisfy all the Keck users and certainly does not offer a clear future path either.]
[Added note: David Lassner, Director of Information Technology Services for the University of Hawaii, and Jonathan Chock of Keck Observatory both confirmed that Maui HPCC has three DS-3 connections to the mainland: 1) The classified link 2) The DREN link 3) A commercial link. However, Lassner indicated that during discussions held last December, MHPCC was not interested in providing bandwidth to outside users at any rates above T1.]
Max: DOE has its own highspeed thing, NASA has its own, NSF has its own... is this [lack of cooperation] counterproductive?
A: The next-generation internet initiative is to tie these together: DOD's DREN, NASA's NREN, NSF VBNS.
Max: There is a disconnect between this lack of cooperation and the House Science Committee's demand for more cooperation. HSC is writing a report on science in post-cold-war era, calling for cooperation.
A: Oh, but they [the different networks] _are_ talking via exchange points. (...more along these lines...)
Griffin: NASA is creating 3 next-generation exchange points. Has agreement to peer with VBNS; working on agreement to peer with DREN; intent is to peer with all the rest, so that a lab with any of these kinds of connectivity can all work together.
Max: Are we trying to locally solve a national problem? Should this be happening from top down, not bottom up?
Griffin(?): We're talking about a work in process; not all agreements have been ironed out. (Note: peering w/ DREN can be political issue because they can have secret stuff on the network.)
Griffin(?): Excluding the politics of agreement, right now it's physically possible to transit to vBNS from Maui.
Cohen/Shopbell: Caltech tests with ACTS.
Shopbell: We used ACTS (DS-3 speeds) for 1-yr operational, running at night. This was mainly a testbed; note the satellite is geostationary.
Conclusions:
Cohen: reliability is key.
Kibrick: Tests of Keck remote observing from the mainland via the Internet
Lewis: [No; but] a videoconferencing capability between mainland remote remote observer and instrument specialist at Keck HQ might address this need.
Q: How can the video from the remote observer in California connect with video from the telescope operator at the Keck summit AND the instrument specialist at Keck HQ in Waimea?
Chock: The current videoconferencing bridge for the interferometry project will allow you to connect remote operations rooms 1 & 2 into a single one. But 3-way simultaneous is very expensive [due to telephone charges]. The 3-way bridge hardware itself is perhaps $50,000.
Kibrick: The ESO remote observing facility in Garching, Germany [which provides remote observing capability with the ESO NTT and CAT Telescopes in Chile via a dedicated 2 Mbits/sec satellite link] maintains an instrument specialist at the remote facility, but this [like the satellite link] is expensive. It may prove more cost effective to provide the remote videoconferenc ing capability, so that the existing instrument specialist staff at Keck HQ could provide support to remote observers on the mainland. But this would definitely require the 3-way bridge hardware.
Kibrick: ISDN options
Kibrick: We haven't done it yet -- this is just a cost calculation based on the number of phone calls that would have been required for the Marcy/Butler run.
Shopbell: You need video interaction, but the quality of video isn't too important. Has anybody tried using an MBONE tool to do this?
Lewis: Our experience is that video quality isn't too important, but audio quality is critical.
C: Audio is crummy over internet.
C: Worth exploring internet video when VBNS or whatever can give reliable connection at high rate.
Shopbell: Internet audio might be quite usable if a reliable, low-latency link is obtained along with an Internet telephony tool which provides echo suppression and related features.
C: There are cost-affordable solutions. Mention of SGI tools.
Griffin: NASA did video experiment to Russia, using video via IP, audio via telephone.
Almes: Internet-2
Internet connections to Hawaii are not typical for several reasons:
Almes: "Fix-west" at Ames is ideal place to converge.
C: Fix-west is the Ames half of mae-west and the NREN exchange. (?)
Griffin: Ames has pushed high-speed low-loss networking since 1989.
Almes: If you begin to talk about very high bandwidth connections from observatory to mainland, you get classic case of delayed delivery. It's very important to ensure that it is engineered in a way that ensures low packet loss.
Kibrick: we are acutely aware of that, and the UDP-based image transmission protocol we use does very well over links with long propagation delays.
Almes: that's very important... in terms of adaptive applications that handle this situation...
Cohen: The Sam Southard UDP-based image transmission protocol works at Keck for long propagation delay paths, but it is currently used only by the optical instruments currently in operation at Keck, and is not currently used by the infrared instruments. Claiming victory because the optical instruments use UDP is premature.
Kibrick: Suggests using tools to wrap other applications in UDP.
Almes(?): is that done by requesting dropped packets? (yes)
C: You use UDP to roll your own [reliable] algorithm? (yes)
Shopbell: Mentioned that folks at Mitre Corp. are working on just such an implementation of TCP, that is based on UDP. They are doing it under contract with NASA, for possible use in deep-space spacecraft, where the time delay may be enormous. We actually tried it out, but all they had at the time were custom forms of several tools, including ftp (i.e., it wasn't a kernel modification that would halp all of our tools, such as those based on X11).
Almes(?): Must make sure the nature of the high-speed connectivity won't have high packet loss. For instance OC-3 SONET is ok. Skeptical about UBR-ATM (the UBR part, not the ATM part). The UBR part could have high packet loss. You have to consider this. The key issue is that we'll need to ensure that the remote observation/control applications that need crisp performance are Able to get it. I am very concerned that the engineering be such that we can achieve this goal. Support for QoS and achieving high bandwidth AND low packet loss for the key applications will be crucial.
Ogasawara: Subaru Telescope and network plans
[The Subaru 8-meter Telescope is currently under construction atop the Mauna Kea summit and is being built by the National Astronomy Observatory of Japan.]
Rhoads, Whinery: Options for improving Hawaii access
Whinery:
Whinery: According to UH statistics, connections to the University of Arizona are pretty available except in mid-day. But notably, Netscape (which has a presence into same POP) was lightning fast all day, while all other points experienced severe delays.
Q: Is the bandwidth between Hawaii and Oahu going from 1.5Mbps to OC3 any time soon?
Whinery: We have 2 T1's to UH Hilo, astronomy T1, etc. These T1's will to UH-Hilo will be replaced by OC3 with the new inter-island network.
Rhoads: DS3 to Hilo from summit is $3000/month. Easy to get to Mauna Kea summit from Hilo. Fiber is currently available. But what about the OC3 between islands...
Whinery: we have an RFP to do this summer ... estimate that the OC3 might be in place within next 18-months, depending on how long the RFP takes.
Rhoads: What will be the OC3 nodes on the big island?
Whinery: UH Hilo.
Rhoads: Have to take into account Hilo-Waimea connection, if this new OC3 link is to provide increased Hawaii-Oahu bandwidth for the observatories whose headquarters are located in Waimea (i.e., Keck, CFHT).
Whinery:
Wright: Gemini has been talking to GTE for summit to Hilo Gemini temporary office connection. We will wait for our base facility to be constructed before turning on the DS-3. Per the contract negotiated by UH, there will be no installation cost at either summit or base facility.
For many reasons, most significantly the security architecture of our network, having the Internet connection at the base facility is preferable to having it at the summit. This includes the link to UH Manoa.
Rhoads: Bandwidth between the mountain and HQ sites, and between UH and the mainland are probably not the most urgent issues short term. Rather, the connection point in Stockton is the source of much of the connectivity problems. Must find some other connection point into the mainland, either onto vBNS via DREN or whatever.
Whinery: The problem is that we have two different kinds of traffic: WWW and other commodity Internet traffic (which doesn't currently qualify for connection into the vBNS) and high-bandwidth research applications, such as remote observing (which do). Similar concerns in terms of direct connection into NREN or DREN or whatever.
Griffin: it's very common practice to get a bunch of universities to build a gigapop, and from the gigapop provide multiple services (VBNS, NREN, internet access providers).
Kibrick: Wrap-up
Alternatively, it might be worth trying to hold such a workshop in conjunction with this year's ADASS conference.
[NOTE: Several days after this BOF session, SPIE organizers indicated some interest in organizing a "Remote Observing" conference as part of the 2000 SPIE Astronomy Symposium in Munich and also one at the 2002 SPIE Astronomy Symposium in Kona.]