Transcript - IPv6 Transition Plenary
Due to the difficulties capturing a live speaker's words, it is possible this transcript may contain errors and mistranslations. APNIC accepts no liability for any event or action resulting from the transcripts.
APNIC 32 Conference
Tuesday, 30 August 2011
09:00 (UTC +9)
IPv6 Transition Plenary
Miwa Fujii: Good morning, I am Miwa Fujii from APNIC and I am the MC of this program.
This IPv6 Plenary is designed to share lessons and experiences that we learned from the World IPv6 Day in supporting fast and robust IPv6 deployment in this region and also globally.
Today we have three panel discussions and one lightning talk session. In the first panel discussion we will review World IPv6 Day from the content providers' and CDNs' point of view and discuss about IPv6 readiness and the way forward. The moderator for the session is Dean Pemberton from IPv6 Task Force, New Zealand.
In the second panel discussion, we will do the same from the ISPs' and equipment vendors' point of view.
The moderator for this session is Philip Smith of APNIC.
Then we will have a short session to have a few lightning talks. Thank you for all who submitted your abstract. We will contact the three speakers shortly.
Then we will have the last panel discussion, to learn about the statistical analysis of IPv6 deployment and readiness, particularly focusing on the learnings from the World IPv6 Day. The moderator for this session is Tomoya Yoshida from NTT Com.
I would like to thank all speakers and moderators for their support for the plenary. This plenary was prepared with support from the program committee members, who are Yong Wan Ju, KISA, Korea; Seung Jae Lee, NIDA, Korea; Tony Hill, AP IPv6 Task Force and ISOC-AU; Erik Kline, Google; George Michaelson and Philip Smith from APNIC. I would like to thank them for their support provided in the last few months while we were working on preparing this program.
Now, let's have the opening remarks from Dr Hyun K Kahng, Chair, IPv6 Forum Korea, and after that Paul Wilson, IPv6 Director General.
Hyun K Kahng: Good morning, APNIC Members and speakers and ladies and gentlemen. Welcome to Busan. Busan is one of the beautiful cities in Korea.
My name is Kahng Hyun-Kook, I am the chair of IPv6 Forum Korea. I am honored to have a chance to make a short address in front of all of you.
Today I want to start my remarks with some introduction about Korea. Nowadays, most young Koreans cannot live without the Internet, that's true: smartphone, tab, notebook, PC, pad, smartcar, whatever, you name it. Without it, they cannot breathe. The same for my kids.
At the same time, we have problems, that is we have to pay too much. The Internet applications and services are getting smarter and smarter every day, but it seems the "Internet" seems to stay as it is. That is one of our problems in Korea.
However, as Chair of the IPv6 Forum Korea, I believe it is fortunate that IPv6 is ready. Without IPv6, in the near future we might have to pay much -- not because we use the content service, not because we access the network but because we use only one IP address, and this is IPv6.
IPv6 can provide a toll-free information highway, IPv6 can be a stepping stone to fair information access and distribution to anyone, regardless of geographical, economical, cultural or language barriers.
Yes, we know that IPv6 is not enough. It may be true. However, we have to remember that IPv4 has evolved. I remember when I started on the Internet, about 30 years ago, it was very simple, just a couple of weeks, that was done, it was about this thick of papers.
Now, RFC, including Internet draft, I cannot count the numbers.
If we implement IPv6 now, then IPv6 will evolve.
During the last century, IPv4 served the world, humbly and successfully. I believe that it is time that IPv6 will play its role, hopefully more than a quarter of a century, but at least for a decade.
Yesterday we were asked why we turned off IPv6 after the World IPv6 Day. That's one of the problems.
However, on the other plus side, I believe that we showed a potential from what we did on v6 Day. We can turn on IPv6 again, whenever it is ready.
Today we have a whole day of sessions about the IPv6 transition. I hope that we can, from these meetings, get a valuable momentum to turn on IPv6 again.
In the last IPv6 summit in Korea, I quoted a famous fable, one of Aesop's Fables, The Boy Who Cried Wolf.
You know what it means. We cried for a long time, for more than a decade, that IPv6 is coming, but I believe now it is the time. Maybe IPv6 is knocking on the door, but maybe we cannot find the proper doorknob to open it.
This is time, and I am really happy that this very important meeting can be held in Busan, Korea.
This morning I want to quote one more thing, which is a song, "We shall overcome." We have to. Who else can do it but us? I believe this is our job. We need patience, but we cannot give up.
Finally, I want to express my thanks to APNIC, KISA, and our valuable sponsors. Without them we could not have this very precious meeting today.
I hope, while you stay here, you enjoy Busan culture and return home safe with good memories. Thank you.
Miwa Fujii: Thank you, Dr Kahng. Now I would like to invite Paul Wilson, APNIC Director General.
Paul Wilson: Thank you very much, Miwa, and thank you, Dr Kahng, for those remarks as well.
Welcome back, everyone, welcome back to APNIC 32 and to this plenary session. This session reflects a collective preoccupation, maybe an obsession, in this community with IPv6 -- for good reason, I think. At the last APNIC 31 Meeting in Hong Kong, we had one of these full-day sessions, a packed program, about the transition; and here we are again with a whole lot of new content.
In between times, the issue has actually become even more pointed with the exhaustion by APNIC of our normal supply of IPv4 addresses. So this really is where the rubber is hitting the road, where we need to focus.
As Dr Kahng said before, the little boy has been crying wolf for many years now. We have kind of known that the wolf wasn't at the door for all those years, but it kind of is here now, and I think we all know that.
The transition is really pressing, it is really important. Our destination is IPv6, but there is still a huge amount of work -- planning, preparation, implementation -- right across the whole Internet.
I am sometimes asked by journalists, by policy people, "What do we have to do, what does a business have to do to transition to IPv6? What will it cost?"
As though you might go and buy IPv6 from the local electronics shop. It's not as simple as that, and we all know it.
An analogy that I have found quite useful, which illustrates the different challenges that we all face, is an analogy with the transportation system.
There is a transition coming up that we are all well aware of, from crude oil, from petroleum, to electricity -- and I think we are all pretty familiar with the way vehicles work and the way the system works.
We don't ask, "What's it going to cost?" or "What is it going to cost an anonymous business out there?" Because what someone has to do depends completely on their relationship with vehicles and transportation.
If you are a car driver, you still know how to drive your car, you just need to know where to get your fuel.
If you are a fuel station, you have to transition in a more serious way because your gasoline business is going to disappear. Probably many fuel suppliers are thinking now about how they will transition to providing electricity or any of the new technology and, if they haven't been too distracted with turning their gas stations into convenience stores and gambling dens and so forth, they will be focused on their core business, which is providing energy, and focused on what they need to do. I think that's a lesson for all of us.
The only people who don't really need to know about the transition are the kids who sit in the back of the car, because they should be able to sit back and enjoy the ride. They don't need to know the details. That's luckily the case with many of our end users on the Internet, they don't need to know either.
I made a presentation of this analogy and I have found it has shed a fair bit of light. It has also shed some light on the compatibility issue. No one imagines that electricity and gasoline are compatible. You don't try to charge up your diesel-powered truck, but you do drive these vehicles on the same roads and use the same steering wheel and carry the same passengers.
Again, we are talking about a replacement of a layer within a layered system and within that layer we really have to know how it is going to work and how the compatibility or incompatibility exists and how it affects the process that we are all going through.
Anyway, without further ado, back to today. I think we have a lot of great content coming up today, which will shed light for all of us, I'm sure.
To today's speakers, thanks very much for coming to our APNIC 32 Meeting to share your experiences, they are going to be really valuable and I think everyone here will gain a lot from your generosity in being here.
Lastly, I would like to thank Miwa Fujii from APNIC in particular for putting together this session and for all her great work in recent years on our IPv6 program.
Thanks, Miwa, and thank you all.
Miwa Fujii: Thank you, Paul.
Now we will have a keynote speech from Dr Yanghee Choi.
Yanghee Choi: Thank you, Miwa, and good morning everyone.
It is my great pleasure to be invited to give a plenary talk in this prestigious APNIC Meeting.
Actually, I was invited to talk about the future of the Internet, not about IPv6, so please forgive me if I do not mention IPv6 at all in my talk.
I am here as the director of the Future Internet Forum of Korea. Maybe you don't know about the Future Internet Forum of Korea, so let me introduce it briefly.
FIF Korea was established in 2006 at a very successful international event, a kind of symposium on the Internet of the future. After the meeting, the Korean experts felt that we needed a kind of forum, an informal forum, to expand the participation for the R&D for the future Internet inside Korea, so the forum was established.
The forum itself is composed of several working groups. For example, we have a working group on architecture, a working group on mobile and wireless network, a working group on the services and applications area and a working group on the testbed, which is quite active at the moment.
Along with the working groups, which hold regular meetings five or six times a year, we hold an International Conference on Future Internet every year around June/July. We had this international conference this year in Seoul. It was the fifth International Conference on Future Internet.
In addition to the conference, we also hold an event for demonstrating prototypes that implement new ideas for the future Internet. It is also an annual event.
We hold an extra event on standards for future Internet.
It may be too early to talk about the standards for the future, but anyhow we have started this kind of activity since the beginning of the Future Internet Forum.
Today my talk will be divided into two parts.
First, I will talk about the megatrends happening in the IT area, IT world, then I will talk about the Internet in the remote future, for example, Internet in 20 years, to 2030.
How much different will the Internet be in 2030?
I think the main characteristics of the future Internet in that remote year will be summarized in three key words -- realtime, quality and knowledge. I will return to these topics later in my talk.
Megatrends: the Internet is used by more than 2 billion people right now, and in 10 or 20 years it will be 4 billion or 5 billion people connected to the Internet. In addition to those human users, we will have hundreds of billions of devices, sensors and RFI devices and small servers connected to the Internet, and they will use the Internet in a very different way. So the Internet in the future has to support all these new kinds of requirements, all these new kinds of applications, and all these new kinds of content that will be available to the Internet.
Consequently, the researchers in the world are working very hard to come up with new solutions, new architectures for the future Internet; new system architecture, new device architecture, new application architecture and new security architecture.
For me, the solutions or proposals presented so far concentrate mostly on rather narrow technical things -- content, security, mobility, context awareness, et cetera. But they lack the fundamental vision for the future Internet, which should be derived from the observation of the megatrends of IT around the globe.
What will be the big trends in the IT area?
Because the Internet is there to support IT applications, the Internet should be a big infrastructure for enhancing the growth of IT in general.
What will be the IT in the future? I see three big megatrends in IT. The first one is everything goes smart; the second thing is increased interaction; the third thing is explosion of data and traffic in the world.
Smart means intelligent, fast, personalized, et cetera. So smart device, you have smart network and smart applications, you even have a smart society, smart hospital, smart education, et cetera.
Interaction: telecom and Internet used to be very useful medium for permitting increased interaction between human users, but the Internet is now used for interaction between machines and devices. I see the emergence of the applications which use the Internet as a medium for the communications and collaborations among content and data. So we will see more new types of interactions emerging in the future. In the future Internet, it will be there to support and enhance all these kinds of new emerging interaction types.
Explosions: you have two different types of explosions. During the last two to three years, data traffic over the Internet has increased at an amazing rate. For example, in Korea, the data traffic over the Internet increased five times.
The Chairman of Korea Telecom, which is one of the major ISPs in Korea, said in his speech about one year ago that he would not be surprised to see 1,000 times data traffic increase over his network in five years.
1,000 times over five years means six to seven times traffic increase every year. So he has to prepare his network ready to accept the rate and accommodate all the traffic increases. It needs an enormous investment for the company. So without changing to new technology, without changing new protocols and the architecture, you will need endless investment to support the explosion of data traffic over the Internet.
Another explosion is the explosion of information in the globe. Data traffic is there to carry the information available in the globe. The information is exploding. Almost all the information created on the globe is now in digital form. Video is in digital form, text and messaging, music, voice -- everything is in digital form. The information created in the globe is expanding every year at an alarming rate. Without understanding the overwhelming expanding rate of this information, you cannot understand how to create, how to build the future Internet.
This overwhelming pool of information created in the globe, we call it information universe. This is IU.
This new term was proposed by one of my colleagues, Prof Yong Gai, and we were quite satisfied with this new word, it is quite an attractive and sexy word, to describe the overwhelming pool of new information available on the globe.
An example of the smartness emerging right now is smart work. Smart work usually means remote working.
You don't go to your office every morning. You would rather stay at home or go to a smart work office nearby your home, and you carry out your project or job sent by some company to you. By smart work, you can even work for several companies at a time, because you don't have to report your presence to your employer every morning.
You stay at home and you accept assignments from different companies and divide your time according to the volume of the assignments given to you and you can work for several companies at a time. So then you become a freelancer.
This is quite natural for software engineers because a lot of top quality software engineers do not work for one company, they stay at home or they rent a room in an office building and work for several companies on a contract basis. This is kind of smart working.
To support smart working, the network has to do a lot of things. For smart work, you need a smart work environment, which should not be very different from the real one, even though it is a virtual one. You need a very fast network with cloud-based document services, et cetera, you need very high-speed and interactive teleconferencing facilities. To support smart work, you need a realtime, high-speed, large capacity, large storage network and services. So this is quite a big challenge for the designers of the new Internet in the future. For mobile workers, the requirements mentioned should also be provided equally.
New interaction, as I mentioned earlier, man-to-man, man-to-machine and machine-to-machine and content-to-content interactions are emerging and expanding. To support this, new modes of communication, collaboration and computing should be devised. The detailed list of the mechanisms and techniques are listed here: redundancy handling, provenance verification -- trust, visualization, aggregation, et cetera.
As a good example of the information explosion, let me talk about the Winter Olympics. In Korea we will hold the Winter Olympics in 2018, in seven years, in Pyeongchang. Pyeongchang is a small mountain area city near the east coast of the Korean Peninsula.
In the Winter Olympics, a lot of video clips and video information will be created and consumed by the spectators and by the television viewers around the globe. The video traffic will be usually generated by video broadcasting companies.
I see in seven years, during the Pyeongchang Winter Olympics, the broadcasting stations will be mostly personal. Personal mobile broadcasting will be common at that time, supported by future Internet with high-speed uplink and downlink capacity and with storage-based routers, et cetera.
Each spectator, equipped with high resolution mobile video cameras which possess a high-speed uplink to the Internet backbone, will send out the camera-captured video in real time. With the number of personal mobile stations over, for example, 100,000, you can create enormous video traffic or an enormous amount of video information and add an enormous amount of video information to the IU, which is already expanding. So the Winter Olympics is a good example for the information explosion in the world.
According to the IDC report recently announced, the information universe, the IU, will be expanding to reach 35 zettabytes in the year 2020, which is 44 times the information that was available on the globe in 2009. So the information is expanding at a very high speed.
IU is the name of the famous Korean female singer.
I do not know if she is 18 or 19 years old. When we decided to call the pool of information on the globe IU, the students in my laboratory at Seoul National University welcomed the decision because they like IU very much. Even in my laboratory on the wall, they have posters of IU, they listen to IU music, and IU even visited the membership training last year and she sang three songs there. So the students were quite excited to meet IU. So you remember IU. OK?
IU creates lots of issues, lots of problems. How do you capture, store, move and process all this data?
On the right side is the big list of problems and issues that will be researched out for the successful implementation and for the exploitation of the IU: data collection, data pre-processing, national language processing, aggregation, visualization, et cetera.
What will be the Internet in 2030? As I told you, I summarized the key features of the Internet in the remote future in three words: realtime Internet, quality Internet and knowledge Internet.
The Internet at present does not support realtime features, it is poor quality, it doesn't provide a knowledge service to the users.
First, the realtime Internet. For example, to support smart work, you need to have a network that is very fast; not only fast, but with limited delay, less than 100 milliseconds end to end. Without that, the collaboration and communication between smart workers will be interrupted and they would not have the impression that they are working in the same office. So smart work will not be as efficient as if they are working in the same office, in the same physical location.
For example, the Internet has been used as a good tool for time shift, place shift and content shift.
Because of the time limits, I will not talk about this shifting in detail, but all these different kinds of shifting require a very powerful network, which should be the future Internet's main feature.
A very small delay will be required for the future Internet. How to realize a very small delay in the future Internet? There are several available technologies already, but you should come up with new solutions.
There are available techniques like the reduced compression and the compression delays for video data or prediction and prefixing and distributed cloud services, et cetera. With this kind of support from the device, network and cloud servers, the end-to-end delay can be reduced a large amount, and with this reduced delay you can achieve a realtime Internet I think very soon.
The most important thing is to accept the idea that realtime is needed in the Internet, because many people are opposed to this idea, who say Internet is used for best effort, it does not need reduce the delay parameters, et cetera. I do not think this is right, because the Internet is for the people who are using it; the Internet is not for the designers who design the architecture and manufacture the Internet equipment. It is for the users.
The users request a realtime service on the Internet so we have to be ready to accept the idea that realtime is also needed in Internet, even though it is limited in distance, even though it is limited in speed, et cetera.
The next one is quality. If you look at the many proposals for the future Internet available right now, my impression is that the quality is not usually the first priority for the designers. Quality includes security, performance, availability, host complexity -- anything that is related to the users' satisfaction level, so the quality of the service and quality of satisfaction for the users.
To enhance, to improve the quality, the Internet designers should not come up with a single quality level Internet. That means different applications, different devices, different networks, different servers need different levels of quality. The Internet should be designed so as to satisfy those different levels of quality.
The notion of adaptive quality is needed. The security and protection level adaptiveness is also needed and the user satisfaction level adaptiveness is also needed.
We know that people in the office usually spend 20 per cent or 25 per cent of their time searching for data. The time for searching for data in the office is increasing. I guess that without suitable and intelligent tools for searching, it will increase to 30 or 40 per cent and you will spend half your time in the office just searching for the right data.
Why do we search for data? It is only to extract knowledge from the data collected from the Internet.
The Internet is used to collect intelligence or the knowledge from the IU, the information pool available on the Internet. So the Internet should be restructured to provide good knowledge at the request of the users.
Usually, in the past, knowledge or searching for knowledge was put outside the Internet architecture.
I think this is wrong. Internet searching, data searching, information searching or knowledge searching should be included in the Internet architecture, so that the Internet can satisfy the users' requests at the highest level.
The Internet as a tap for knowledge, you should accept this idea. For that, the information universe and the future Internet cannot be separated, so it should be integrated as one single architecture.
Up to now, if you look at the Internet from a big distance, it is usually separated in three parts -- network, nodes attached to the network and content or information available outside the Internet.
For the network designers, they optimize the architectures, the common architectures, independently from the device and the information. The device architects, for example, when you design your smartphone, you assume a certain set of interface and a certain set of dialogues between your smartphone and network smartphone and the cloud servers. But you do not try to change the internals of the network, you do not try to change the cloud servers, you just optimize your smartphone architecture on your own.
Those three independently developed architectures, what happens? Large overlaps, increased complexity, high cost and a low level of satisfaction to users.
So this should change. I think that integrated design, integrated architecture for those three components should be in place to decrease the cost, to decrease the complexity and to improve the user satisfaction level.
I see that this happens already in some areas. For example, in Korea, usually the manufacturers and telecom ISPs are cooperating very closely, so they are now talking about this kind of integrated architecture. For example, moving large storage space from the device to networks, from networks to clouds or in the reverse as well.
This will create lots of problems for the business models, but they will surely create a lot of changes for the newcomers. So a power shift is coming.
Let me conclude my talk briefly. In the remote future there will be expanding the information universe, pool of information. This should be handled effectively by the new Internet; and Internet Infra should be ubiquitous; and the new future Internet should provide three key features, which are realtime, quality and knowledge.
Thank you for your attention.
Miwa Fujii: Thank you, Dr Choi, for your very insightful keynote speech.
The next keynote speech will be given by Dr Shin Miyakawa, Director of Network Systems and Technologies, Innovative IP Architecture Center of NTT Com, Japan.
Shin Miyakawa: Thank you for the introduction.
My name is Shin Miyakawa. Today I would like to talk about -- some of you already heard about that, especially Erik is saying that the dual stack Internet will be here in three years. But I believe that a lot of people in the room still have not seen this yet, so I would like to introduce what we are doing in Japan and what we have to do.
As everyone knows, the first problem we are now thinking about is the IPv4 address shortage. Of course, many people know about that. This is the structure of how the IP addresses are assigned to the people. IANA has the origin, then all the RIRs, like APNIC, have a pool of IP addresses, and then each ISP and enterprise asks APNIC to assign the IP address. This is the structure.
The fundamental problem is, especially in this region, APNIC has already run out of IPv4 addresses.
This year, this is a very fundamental problem we are now enforcing to be solved.
We have to do two things simultaneously. Even though we would like to shift the entire system to the IPv6, but still we have to keep our assets. Many operating systems, like Linux, BSD, or other Unix-based systems, Windows, have already implemented IPv6 protocols in 10 years maybe, and many softwares, like Apache and also lots of Daemons, are also compatible with IPv6. But many people do not turn on the features, and also lots of content landing on top of it doesn't care about the IPv6.
As an ISP, we have to keep still IPv4 service, with some modifications, and at the same time we have to think about the smooth introduction of IPv6.
We know that there are still some issues, especially in the technical and operational issues exist in IPv6, but we have to do something. Again, we have to do these two things simultaneously.
Recently, especially our group, IETF, are discussing about the carrier grade NAT. We used to call that large-scale NAT but at the last IETF we went back to the name of carrier grade NAT. We are thinking about carrier grade NAT or maybe similar technologies, like DS Lite, especially the IPv4 address sharing systems.
The idea is the carrier grade NAT is going to introduce some NAT function in the IP network core, maybe the access network. The left side is today's IP Internet ISP architecture and the host received private v4 addresses and the CPE, the so-called router, has the one interface with global IP address; and then some access concentrator, maybe L2TP concentrator, maybe a cable headset, that hooks to the global v4 network.
Because of the shortage of the IPv4 addresses, to keep customers' equipment as it is, we have to do something. That is the carrier grade NAT function. NAT function is going to be introduced to the access concentrator or something somewhere, in between from there to the eBPG router.
Actually our group company, called NTT DoCoMo, already installed this function in their smartphones network. If you come to Japan, to Tokyo, if you turn on your smartphone devices with the DoCoMo network, you receive a private IP address only; no global IP addresses are available for the smartphones for the DoCoMo network. This has already started.
But the carrier grade NAT is not enough -- not perfect. Really bad idea. Then sharing IPv4 addresses means the modification of the IP communication model itself.
For example, there are several, tonnes of problems around this carrier grade NAT, but one very famous bad example is we cannot use the access control list. Also, we have to ask all the content providers to log not just only the IP address of the access but also the source port as well, to hunt out who is accessing that service.
This is required by the legal enforcement sometimes so we have no escape from this. However, this requires huge storage.
Then I will explain what happens there. If the good guys are using the servers through the firewalls, but one of the guys is sending some packets to do something bad -- maybe just multiple packets going to be sent to the servers -- according to today's practice, usually the server operators who are using firewalls are writing some ACL to filter out that IP address is brokered, at least for a while.
But after introducing CGN, one IP address is shared by multigroup users. What's happening on that? If the operators find out that IP address, then that means the other normal, innocent users also are going to be banned out. Probably we got lots of telephone calls on that, "Why can I not use this service?"
Thinking about lots of solutions against this problem, frankly speaking, I never found the answer for this. If we still stick to the carrier grade NAT based or any IPv4 address sharing scheme based IPv4, we have no hope. So this is one of the strong reasons why we have to do something else, which means IPv6.
Also, the IPv4 address sharing has another problem around the HTML. Because of allowing the people to use the HTTPs, sometimes we have to have a limit to the number of TCP sessions per user, or maybe we have to guarantee some of the numbers of the TCP sessions per user. Because the TCP/IP has only two bytes port number, in worst case one single IP address can accommodate 64K TCP sessions as a maximum. If you think about that, 1,000 people using one single same IP address, which means one single user can use around 64 TCPs.
We have done some experiments about that. We installed some TCP session limiter machines. We can create this very easily, like hacking Linux machines, and through that we run the web browser running on top of Windows, then see the Google Maps. With help from Erik Kline, back in 2008 I think, I showed up at their headquarters and made this presentation in front of Dr Vint Cerf, and still I keep using this file.
With the limit of 30 connections with the Google Maps, this looks just fine. But if we put the limit at 20, there is some ramp here. If we put a 15 TCP session limit, almost nothing can be seen. Then 10 connections, 5 connections, nothing happens.
What is happening on this is because of AJAX.
That is the reason why recent applications, like on Google Maps or on any other popular web page, run very, very quick.
According to our surveys, even if we have no operation, just turn on the Windows operating system or the Macintosh operating system, that consumes 5 to 10 TCP sessions that you don't even think you are using, because, for example, Adobe PDF reader is seeking an update periodically, sometimes Firefox does the same thing, and Windows update is checking whether security patches are available or not. This consumes 5 to 10 background TCPs.
Then if you turn on iTunes and go into the iTunes market, lots of photographs of the CDs and the artists are popping up very instantly. Each picture consumes one TCP.
Of course, there is lots of technical discussion behind this. Mostly, the TCP time-out is very key to think about this, to be very precise; but, anyway, this is a fact.
Please read my file carefully. I do not want to read these types of things, I just write down lots of the reasons and analyze them in my file, so please download my files.
Anyway, as a result, we think probably we should allow only 10 people per user, or something like that, a really modest number, of how many users can share one single TCP. But at the same time, still, lots of problems I described, ACL, and many things remain.
Thinking about how we can utilize this scenario, maybe we should implement carrier grade NAT this year or maybe within one or two years before we are running out of our own storage in ISP to extend the life of IPv4 for another 10 years; after that, IPv6 only. We really think about this, this 10 years is very important duration.
We have to modify our Internet, changing little by little, minimize the risks and also the cons of that transition, then we should realize a smooth transition to the IPv6. This is our thought.
Then, to be honest, I already purchased -- not me, but my company -- several number of carrier grade NAT implementations. The truth is the truth. Carrier grade NAT devices are quite expensive. From the business point of view, we have no hope to recover the money for it. When I think about that, if ISP put the carrier grade NAT to your residential ISP connection, do you want to pay more? I don't want to do that. I want to pay less for it, because my carrier grade NAT access is quite limited.
Think about it as a manager, a board of directors of the ISP business; do you want to put some expensive devices which have no chance to recover the cost? Think about that.
This is really the convincing logic, and NTT's executives agree with me. That is the really true reason why we, the NTT Group -- not NTT Communications only but NTT East and West -- is seriously committing to the IPv6 introduction in Japan nationwide and also the global networks as well.
Please, think about it yourself. Probably you can agree with me. So please think about IPv6 from a business point of view right now.
Then, last of all, IPv6 interaction. This summer we proudly announced that NTT Communications, who has one of the biggest broadband ISP services in Japan, has started its own ISP connection with IPv6 without additional fee. If you are users of NTT Communications ISP network in Japan -- of course, the area is still limited, but we are now enhancing that -- iPv6 is just an option, free of charge. If you are an IPv4 service customer, you can ask for IPv6, just order it.
Also the IPv6 compatible CD devices have now started to sell to the people. This is really cheap and good devices, they have 2-channel ESS wireless channel. We are now ready to do it.
I still have 10 minutes. I went too quickly, maybe.
Anyway, as you see, we are now seriously thinking about that, to shift the entire Internet system from IPv4 only to dual stack. Then we really would like to escape from the CGN, but unfortunately probably we have to install CGN some day.
Now that we are thinking about that and how to limit the number of CGN sessions deployment as few as possible, then to prevent lots of bad things caused by CGN, then promote IPv6 more and more, and maybe ask lots of content providers and users to upgrade their machines and also the software and the content, to be compatible with IPv6 as soon as possible.
I think we are looking forward to the new future.
Of course, lots of other people say that machine-to-machine has applications which could be deployable over the IPv6 network, but not just only for that kind of thing. The existing network is going to convert to IPv6, I think.
Some of the other people have also pointed out about introducing IPv6 has still problems, we believe that still, especially around the firewalling and the load balancers caused by path MTU discovery or something like that could be reported by other people, so I would like to omit these kinds of things.
Right now we are still working on that. I am very optimistic we can do it.
Then a very short presentation, but I would like to emphasize again, IPv4 address is already running out.
One of the fundamental problems still we think is the number of people who understand IPv6 today. The education and also lots of training is very important.
Please do that in your countries, to enhance the engineers who know IPv6, especially with hands-on experience. Seeing is believing.
I believe that everyone in this room still remembers when you turned on your Internet the first time, not just reading books, but using the Internet is the most important experience to learn that.
IPv6 is the same. So please install IPv6 into your environment, into your machines, and if you write the software next time, please make your program IPv6 compatible. That is a mandate, I think.
Also we are learning still huge around the efforts in our apps and with lots of colleagues in and out of our companies, then we still continue R&D and we are happy to share our R&D results in this kind of conference and at other venues, and I really hope that working together in a standardization body like IETF or maybe any places on the earth.
I am very looking forward to working with you.
Also, APNIC is doing a very important role for these activities, I really appreciate that, and thank you for allowing me to be a keynote speaker. Thank you very much.
Miwa Fujii: Thank you, Miyakawa-san, for the insightful keynote speech.
Now we will have short remarks from three people wanting to share regional efforts in order to promote World IPv6 Day and also to support the IPv6 transition in their industry.
Our first speaker is Tony Hill, chair of AP IPv6 Task Force and board of directors of ISOC-AU.
Unfortunately, Tony is arriving at Busan Airport around now, so he cannot deliver his message in person.
However, he sent his message to us, so I would like to read it for you: "I regret that I could not attend this important discussion of IPv6 at APNIC 32. Unfortunately, my schedule in Australia prevented me from a departure in time to join this session from the start. However, I have been working to join the discussion during the day.
"I believe that discussion of IPv6 continues to be of the utmost importance for continued development of the Internet worldwide and for building access to the Internet for citizens, businesses and other organizations throughout the Asia Pacific region.
"The exhaustion of the global pool of IPv4 addresses in February 2011 was quickly followed by the end of normal allocation of IPv4 addresses in Asia Pacific.
This means that users and service providers in our region need to rely on their own supply of IPv4 addresses or quickly move to adopt IPv6 so that the Internet continues to expand and allow access for growing populations and economies.
"Obtaining full benefit from the Internet depends on any user being able to access all services and all parts of the Internet without complications. As we know, IPv6 and IPv4 do not directly interwork, apart from edge operating systems and servers. Therefore, it is essential that we maintain effective communication between economies within the Asia Pacific Region to maximize interoperability of the Internet.
"As is the normal culture of the Internet, everyone and every economy makes their own decisions on how and when to adopt Internet technology. The approach in one economy may be quite different from another economy.
Nevertheless, we can still learn much from the approaches that each of us is using to explore adoption of IPv6.
"For instance, in Australia we continue a national discussion of IPv6 through the annual Australian IPv6 Summit process, where individual organizations can present their approach to IPv6 and describe their experiences in discussion with others.
"The Australian Government continues with their IPv6 transition plan, which will have all services available via IPv6 by end of 2012. We need to continue the discussion through the Asia Pacific IPv6 Task Force to make sure that all our economies are fully informed.
Many of the plans already in place around Asia Pacific economies include a target date of end of 2012 for IPv6 adoption.
"For the first time, World IPv6 Day on 8 June provided the opportunity for a live demonstration of the operation of IPv6 on the Internet, and its impact on the usual experience of users. The good news is that nothing really bad happened. The Internet continued to function.
"But today we will have the opportunity to hear a much more detailed discussion of experience: what are the issues that came up; how were they managed on World IPv6 Day; what will organizations be doing in response to their experience?
"I wish everyone well with today's discussions and look forward to talking to you individually."
That is Tony's message.
The next speaker is Prof Ma Yan, the deputy director of NIC of Beijing University of Posts and Telecommunications.
Ma Yan: Good morning, everyone.
The topic today is setting the scene, so I've followed the theme to talk about what has been done and what has happened in the case of China. We are talking about IPv6 and what we need, so we are doing, as we said yesterday, we do what we need.
IPv6 is really one important factor to enable the technique to move forward to facilitate the economy and also to give a good education to the people in China.
The Chinese Government has put in a very big effort.
Starting from 2003, we have a national plan set up, a nationwide testbed, an IPv6 pure backbone for CERNET and CERNET2. At the moment over 100 universities have been connected by using IPv6. Something over 1 million students enjoy the connectivity of IPv6. I tried yesterday using the connectivity from Busan here to access the resources in my university and other universities by using IPv6. So this is what has been done.
According to the CNNIC status data, there are some 15 billion daily inquiries, among them some 700 million inquiries, and still getting an even larger number of IPv6 inquiries starting.
The Chinese Government will set up another big project connecting all the universities, primary schools and middle schools getting into the Internet.
According to the stat data, also CNNIC has released data, 34 per cent of the population is getting connected to the Internet. So we have still a lot of work to be done in the future.
More people, more students, will get connected to the Internet, so we need more IP addresses. No more IPv4 left, so we have to do IPv6, just for our sake, for our economy, and also could be one example to show you the necessity of promoting IPv6.
According to the stat data I saw yesterday, just in my POP, in my CERNET2 operation centre, our traffic still keeps growing. Yesterday, the traffic came to 9.6 giga, connecting something around 10 universities. We still have another 30 universities to be connected. So this is the real data that happens on my site.
I would like to say, we have really tried our efforts, not only just trying v6, but deploying IPv6 and it is getting a big welcome from our university researchers and students, and they are enjoying their life, enjoying their study, enjoying their network connectivity.
We also met troubles because of IPv4 operation. V4, the engineer, the operator was only facing one network, one routing table, one circuit thread, one troubleshooting, but when we introduced IPv6, they had two problems, both IPv4 and IPv6. We had to train our engineers and operations staff to know what would happen, for the troubleshooting, for the maintenance, for our customers, to care about our customers.
We also had to migrate a lot of applications from the original IPv4 only to support the dual stack or even pure IPv6 only, pure IPv6 only services; for example, some of our video services for our content, for our realtime TV streaming, we are moving to pure IPv6 servers.
In this case, the students really do not know, they do not care. They just click on the website, and they do not ask about the resolution, whether it will automatically convert their traffic from traditional v4 to v6, because at this moment v4 is relatively getting more crowded and more congested. V4 has more and more users, but v6 has still not so much, so we are encouraging people to migrate from the traditional v4 to v6, to have a larger bandwidth and better user experience, so in this case really facilitates the transition from v4 to v6.
Also, we have met problems, a dilemma, because some customers say, "No, I did not install v6 yet, I do not know how to install it, I do not like to install it, but I need the service." So we are also trying the method, the transition technology, including the popular 6rd, DS Lite and NAT 6to4. Also, Professor Xing Li has made several speeches in previous APNIC talks about the technique of IVI, so we are also implementing IVI technology, to let the traditional IPv4 users access the resources in the IPv6 network. So we are doing all these kinds of things.
We have also set up some of the pure v6 network to do the trial. At this moment, not every service, every application, can run on top of, by using the traditional, the first version, the first generation IVI. More improvements, more next generation technology of IVI will be doing even detailed investigation. But at this moment, major applications, including the website, web surfing, the video streaming, the file sharing, all major student applications really have a very smooth operation. So those kinds of things, we are trying our best to do things.
This is from our educational sector's story. Also, from the server provider side, we facilitate and help the server providers to know how things could be happening. We have set up an organization called China Internet Service Provider Association and we have training lectures, several months, two or three months at a time.
We let the service providers know what is the awareness that should be taken and also what kind of technical problems they may face, and let our experience be a real example, just as Miyakawa-san has just mentioned. What you see is what you believe, so let them know this is really working.
Also, for the manufacturers we have a joint industry, so they join Huawei ECP, Amada, even a lot of the foreign technical companies -- Juniper, Cisco, Microsoft, French Telecom, those people, foreign companies -- join in as one of the driving forces for the deployment of IPv6. So they try different kinds of technologies. When they are ready for these kinds of things, their product could be ready for the market. If the market is available and comes up, their product will be ready.
During World IPv6 Day just one or two months earlier, on 8 June, we tested some of the new services.
Major service providers in China, baidu.com and sohu.com, also set up their websites enabling IPv6 access. I think those are the kinds of things that are really happening in China.
I would like to share this kind of information with you and during the conference I would like to have more discussion with all of you and learn from all of you.
Thank you. This is my story. Thank you.
Miwa Fujii: Thank you, Prof Ma Yan.
The next speaker is Ms Ka Ping Wong, the Secretary General of ISOC Hong Kong. Ping and I have been working together for years, and if I ask Ping something, she always comes back to ask me to do two more things for her. I am very happy to have Ping here to share her knowledge about the regional efforts of ISOC Hong Kong.
Ka Ping Wong: Thank you, Miwa and APNIC for inviting me to be a speaker to update all of you about Hong Kong's Internet community's effort in supporting IPv6 transition.
The Internet Society of Hong Kong has been working on promoting IPv6 since 2007, and we have formed an IPv6 working group and we have a group of volunteers who are IPv6 experts, as well as those who are interested in IPv6. Our convenor is Che-Hoo Cheng, whom most of you know. He is a big guy and is the superhero behind all the IPv6 initiatives.
We have built up a thematic website on IPv6, which is ipv6world.asia, and we have built an IPv6 enabled website directory. You can go to ipv6world.asia and input your URL, your domain name, and the system will automatically check your IPv6 availability and check regularly, so we keep the most up to date Hong Kong IPv6 website there.
We have been doing a lot of events during the past few years. Overall, we have organized over 12 events in Hong Kong, including APNIC training, conferences. The biggest one and the latest one was APRICOT 2011. We joined with APNIC and other partners and had a well attended conference, over 600 people attended. We have an ipv6world.asia conference every year, which is also well attended -- last time it was over 300 people.
Recently, we organized a conference in Hong Kong on World IPv6 Day, and we organized another one this Friday, "After World IPv6 Day, What's Next?" conference in Hong Kong.
We did a lot of technical training for technical people. So far we have trained over 400 people in Hong Kong for technical training.
Overall, as I mentioned, in total, the accumulated attendees of our events and activities is about 5,000 people.
Of course, we will continue our IPv6 promotion in Hong Kong, not only to the IT community. We are trying to outreach to the general public as well as to SMEs, the enterprises.
A good demonstration on how we work with other partners for our IPv6 promotion in Hong Kong, I think we cannot do it alone. We need different partners to support us. We need a joint effort.
So one of the events that we did was the World IPv6 Day Conference, and on that day, before that, we called for action for different enterprises, ISPs, to test run their IPv6, and also we organized a conference on that day, and we had different partners to support us. You can see that we have APNIC, of course, supporting us, we have DotAsia and we have the Internet Society supporting us. Those are all regional organizations.
The key partners include Hong Kong Internet Exchange, DotHK, ISP Association, IPv6 Forum Hong Kong and Cyberport.
Besides those organizations we need sponsors and commercial partners and you can see the vendors and ISPs are supporting us, not only financially, I think that they also help us to promote the message too.
Of course, the most important is the Hong Kong Government. So far, the Hong Kong Government has been very supportive to us for our IPv6 events. You can see the two big logos there, they are the Hong Kong Government bureaux, and also for the Hong Kong IT community they are very supportive too. We have over 30 IT associations supporting us.
As I said, we cannot do it alone and we need a lot of partners, and more partners to work together to promote IPv6 in Hong Kong.
Next I'd like to mention the Hong Kong Government's support for IPv6. Of course, I'm not representing the Hong Kong Government, but this is what I know that the Hong Kong Government has been doing for IPv6. The Hong Kong Government's network backbone, Internet gateway and portal are IPv6 enabled. Their websites, around 230, are on IPv6. Their Internet mails for 60 bureaus are running on IPv6. The procurement equipment must be IPv6-compliant and the Hong Kong Government issued IPv6 security guidelines.The IPv6 NTP will be ready by the end of this year by the Hong Kong Observatory. One of the milestones is that recently the Hong Kong Government allocated HK$1 million to promote IPv6 awareness to the general public as well as SMEs. I believe it is a strategy to create a kind of end user demand, so as to push the ISPs and the enterprises to deploy IPv6 faster.
For the industry, a brief update. So far, we have seven ISPs in Hong Kong providing dual stack services to corporate customers on a limited scale. We have PCCW to roll out IPv6 fibre direct by the end of this year; and we start to have hosting companies locally to offer IPv6 web hosting; and, of course, Hurricane Electric to offer an IPv6 tunnelling service.
Again, I want to say that what we have been doing, we need a lot of effort from different parties and resources, so we are looking for collaboration and cooperation, not only locally but from the region. So if any one of you or any country would like to do something together, we can work together.
Last, I want to introduce this IPv6 cartoon image ambassador. His name is Bitbit. It is designed by a very famous local comic artist, Siu Hak. He will represent us to have some fun stuff for IPv6, so he is a good choice for us to use to promote IPv6 to the general public. He says, "IPv6 now!"
Miwa Fujii: Thank you, Ping.
Now, all the presentations related to the opening of this full-day IPv6 Transition Plenary are over. Now I would like to break for morning tea. Please come back to this room before 11.00 am. We will have a panel discussion 1.
The panel discussion 1 is Content Providers and CDN, Lessons from the World IPv6 Day and the Way Forward.
APNIC 32 Conference
Tuesday, 30 August 2011
11:00 (UTC +9)
IPv6 Transition Plenary
Miwa Fujii: This panel discussion is moderated by Dean Pemberton from InternetNZ. So Dean, please start.
Dean Pemberton: Welcome, everybody. We were just saying before that this panel discussion is very similar to one that was held back in APNIC 31 at APRICOT, but back then we were looking forward to IPv6 Day and explaining what we were thinking was going to happen and all that stuff, and it's quite good now to be running a very similar panel, but after the event, so we can find out what actually happened and what the experiences were.
I would like to start off with our first speaker, who is Erik Kline from Google just explaining about their experiences through World IPv6 Day.
Erik Kline: Hello, I'm Erik Kline. Thank you for allowing me to speak here.
I'm Erik Kline, for the last four years, I have been half of the full-time IPv6 team. As of yesterday, I'm only a third.
I was going to talk about our experience with World IPv6 Day. At first, we figured it would be just another day serving IPv6 to the Internet and then, upon reflection, we realized that we could do more and we should, so we began to do this pushdown for broken users, so if you had broken IPv6 when you went to google.com, if we detected it, we literally did this little HTML Java script pushdown. This is a screenshot, I don't know what I did but I broke my Mac. This is not simulated, this is real. When it happened to me, I thought I should capture this.
We internationalized this in more than 40 languages and it says if we think there's a problem for World IPv6 Day and there's the thing that says, try it now to see if your connection is ready, and that took you to this page. This was also internationalized in more than 40 languages and it did some testing to see if you're IPv6 ready and if you had a problem, you could click on that link and it would take you to an IPv4 only help page.
So that even if you were broken on dual stack, you could still get to help in more than 40 languages for diagnosing your connection.
Also, the Chrome guys, after we were talking to them for a long time about how to do correct IPv6 connectivity detection and so on and so forth, they said, "We're just going to do this fast fallback thing, we're going to give an IPv6 connection a 300 millisecond head start and if it doesn't complete within that, we'll start a back-up IPv4 connection."
As you can see in this graph, this is the graph of dual stack impact. It's the percentage of connections that are likely to fail if you add a AAAA to a host, regardless of whether they're trying v6 or v4, you don't know because they failed. This is the additional percentage and you can see it brings Chrome's reliability or its dual stack impact down to 0.01 and dropping, so its reliability is up in the four, four and a half nines territory.
A Google engineer pretty quickly afterwards put the same code into Firefox, so Firefox is at 7 or 7 beta or whatever it is, has the same fast fallback feature in it.
What did we actually see on the day, what happened inside the company and so on?
We actually got a lot of traction from inside the company, a lot of folks started, a lot of teams started doing IPv6 and it wasn't just us. Toolbar.google.com added AAAAs, Talk and Mobile Talk became IPv6 ready, Google Analytics has AAAAs in the Google IPv6 program.
Google public DNS, which you may know as 8.8, got IPv6 VIPs, they're somewhat less memorable. If somebody has an IPv6 vanity prefix, let me know.
Some other teams didn't quite make the change window, but they did add AAAAs after, so adwords.google.com now has IPv6 and so does IMF POP nsmtp.gmail.com. This is mailuser agent SMTP. So essentially, all of your gmail connectivity can be v6 enabled now.
When the day was over and we were rolling it back, we let the teams know, "Hey, we're rolling this back, but you don't have to roll back if you don't want to."
Some teams decided that they were going to leave AAAAs on, notably YouTube and Mobile Talk. We are serving AAAAs for YouTube videos, not YouTube.com, but the videos inside the page. I'll have AAAAs for all of the world except some of the most problematic networks.
We were already serving quite a bit of v6 traffic.
You can see between the vertical thick white lines World IPv6 Day, the effect when we added YouTube videos and kept it on thereafter. We were already serving most of the v6 Internet. This was kind of just another day from that perspective for our network.
We did see some awesome stuff in some roll-outs, so props to 2516 for a really fast roll-out. This is a graph of the v6 native percentage of AS 2516 and they went to I think about 12 per cent by the day and now 15, 16 or more per cent of v6 native inside their ASN. That was really good news; that was not expected.
But the brokenness, what did we see that was bad?
Well, we learned of a consumer electronics device that plays YouTube videos. It never tries IPv6. It doesn't have IPv6 capability, but it does ask for AAAAs and if you answer it, lo and behold, it has some kind of a problem. That was kind of broken, but there's an update for that.
Some mobile users couldn't get to Google voice but we found and fixed that. That's sort of the lesson for the event, one of the lessons for the event was that you can try and do a lot of analysis and testing, but until you really do it, things don't get fixed until you make them break and it's kind of unfortunate, but you also can't necessarily predict all of the interactions of things until you really try it.
One day was definitely not enough in the case of some of the things that were broken. Actually, leaving them enabled allowed the engineers to see the problem prolonged and to be sort of, dare I say, forced to fix it. Several of us, all here at the table, really want to do a World IPv6 Week, we want to roll AAAAs on for the site and we want to leave it on thereafter.
We need heavy, heavy access participation for this to be a success, though. I think World IPv6 Day showed that content providers can move pretty fast. Six months or less, but the user, the number of users largely on the Internet did not really change, so we need to have more IPv6 eyeballs.
What happened to brokenness across the Internet?
This is again the dual stack impact graph. This is excluding Chrome, because Chrome has that fast fallback and it causes the graph to go down. If you exclude things like that, it's kind of, sadly, the same. We can't necessarily serve AAAAs to the entire Internet because of this brokenness, it is 0.06 per cent. That's outside of any reasonable SLA.
There is this interesting case of Japan. I don't know how well it shows here, but the green graph behind is the latency delta, the scale is on the left side of the graph. This is looking at clients inside of Japan, the whole country, not any specific ASN, clients that ask for a AAAA and the latency impact that they receive when they actually successfully connect to a dual stack host. This is not people who fail to conduct within 15 seconds, that is the orange line below, .83 per cent.
The latency impact for serving a AAAA to someone who asks for it in Japan is 870 milliseconds.
This whole graph -- by the way, the average includes Chrome and Mac OS10.7 Lion, which do some smart things and if you remove them, the graph is worse, so possibly this is the Windows TCP reset behaviour, Windows actually has TCP reset behaviour that is documented in RFC 5461, section 4.2, that basically causes it a 1,000 millisecond delay when you send it TCP resets.
We would like very much to serve AAAAs to the world.
With 870 millisecond latency, we probably can't serve AAAAs to these networks.
So what's next?
As I said, we would like to serve AAAAs to the world and we're at the point now where we can fully automate the Google over IPv6 program. We can ultimately decide not just which data centre you should be going to but also which protocol you are probably best using to get there. The exact parameters for when to serve AAAAs haven't been decided yet, only I think real operational experience will tell.
We need to consider that we'll add AAAAs when there's a certain percentage of IPv6 users, when there's a lower percentage of brokenness and when the latency impact of doing so is not significant.
As I mentioned before, we very much want to have a World IPv6 Week in the first half of 2012. We want to roll in stick, we want to turn on AAAAs and leave it on, but we need access providers to contribute some significant number of IPv6 users.
This is still the adoption graph. This is very dismal. This is 0.3 per cent.
The only thing that's encouraging about this graph is that 6to4 in Teredo is decreasing; the overall level of IPv6 is basically the same.
I'll finish with this message and that is that France is, as a country, or I should say as an economy, is 3.5 or 4 per cent in v6 native and Japan is 1.5-ish per cent native. There's still a chance for any country out there to come together and say we want to be the country that is the first country to 90 per cent v6 native or even 10 per cent v6 native. There's still a chance for that. Summon your national or economic pride and be the first economy that can say, we're the first economy to be more than 50 per cent v6 native.
That would be fantastic.
Erik Kline: I don't know how you want to do questions. Do you want to do it later?
Dean Pemberton: If there's any quick questions for Erik in particular, then we can handle them now, but I will be taking some questions for the entire panel later on as well.
Do we have any questions specifically for Erik?
Erik Kline: Thanks very much.
Dean Pemberton: Thank you.
Next we have Donn Lee from Facebook.
Donn Lee: Thank you, Dean and Miwa.
I really like this venue that we're at, you have the beach right over there. My favourite visual for IPv6 is that you can put an IPv6 address on every grain of sand on that beach. In fact, you could do it for a hundred beaches, and you would still have more IPv6 addresses.
That's a lot of numbers. It's a big number that's hard to grasp.
Go ahead, stick your toe in the sand, lift it up and look at the tiny grains of sand. I encourage you. It will be a great week for you.
So I'm here to talk about World IPv6 Day at Facebook.
The first is why does v6 matter to us? I know that I don't have to tell you that the Internet is a wonderful thing. With a college student's budget our founder and CEO, Mark Zuckerberg, was able to start Facebook from his university dormitory and, just seven years later, drive Facebook to become a top 5 website.
That is some severe power. So it's an open model.
I call it a free model. It's very inclusive and it fosters a lot of creativity.
That is why v6 matters to us, because it is an architecture, it is a model of the Internet that has proven to be successful and it's going to be the future Internet for the generations that come after us.
This is the traffic graph on World IPv6 Day itself.
Not very interesting, except that it says that, yes, we participated and there was a lot more traffic then than what we had on Facebook before the day.
These are statistics that we have measured.
0.2 per cent of Facebook users are capable of reaching us through IPv6. I believe the next milestone is 1 per cent. 1 per cent is one out of 100 users. We have over 750 million active users, so 1 out of 100 users is a significant number of users. That's a lot of users to us. I think that 1 per cent milestone will be very significant and we're already one-fifth of the way there.
These are statistics for dual stack brokenness, 0.02 per cent that we have been measuring. It seems to be going down. It's not conclusive data, but we have been gathering this data for months through non-intrusive browser testing of all Facebook users on a rotational basis every day. On World IPv6 Day itself, there were over 1 million IPv6 users.
Of course, we didn't want to just turn on and surprise those 0.02 per cent of broken users, because it does add up to a significant number, even at 0.02 per cent. So we provided this message. We worked very hard on this message and we argued over every single word and how many words there were and what should be in this message. It took three days, probably four different teams involved and this is what we settled with.
We even debated whether "IPv6" should be in this message because, if you think about it, we're trying to message ordinary users, not network savvy users, not people who know what an IPv6 address is or could even recognize the name. But the name of the event has IPv6, which is significant to the event, so we made sure it was at least there. But we wanted to convey to the user that the impact would be something that they should act on and that impact would be that they would have slow page loads, which is one of the symptoms of a dual stack broken user.
So the day itself went without any problems. We felt very encouraged by this and so we decided that to make some progress, we would permanently dual stack our developer website, which is Facebook platform and the URL is developers.facebook.com, it is permanently dual stacked.
We did have to turn it off for www.facebook.com.
That was always the plan. The biggest reason is the dual stack broken users. We don't want any users being unable to access Facebook and unable to take care of their crops, so it's very important that we turn it back on or we turn it off so they can have access again.
Also, we're not done with our code. There's plenty of places in our code that still need to be adapted for v6. You think about things like provisioning systems and log-in systems. So we were able to get permission from our management to turn on for one day, but the code still has to be updated in various places for v6 for permanent dual stack.
And another thing that we discovered was when we prepared for World IPv6 Day, we realized that mobile would not be affected, which is good, we wanted to contain the impact to broken users. But Facebook Connect, which is on 2.5 million websites around the world, it just so happened it would be impacted. There was no time to change that, so we went forward and said, OK, 0.02 per cent of users whether they're on Facebook or not, when they go to major websites that have the Facebook Connect web widget, that web widget will be broken. It will not be there unless they stay on the page long enough for it to load after the time out.
So you can think of it as, wow, it's kind of scary, but it turned out, hey, we got 2.5 million extra websites out there for World IPv6 Day.
This is the chart of brokenness that we have been charting after World IPv6 Day. I am getting some results that actually are below 0.02 per cent that I never had before. I'm not sure exactly why this is happening. Our code hasn't changed, instrumentation hasn't changed and our way of distributing it to users, these tasks haven't changed. Although if you look back to some of the post-W6D presentations, there were a significant number of websites that left it on. So I'm thinking one possibility is that users that were broken can still access the sites that took off the AAAA, but the ones that left it on, eventually, maybe after a few days, they tried to get to those sites and eventually had to fix their broken connection, possibly.
So we found these are the learnings for the day. We had a one hour test that really boosted our confidence.
We just surprised the world, we had what we call Facebook v6 hour, which happened before World IPv6 Day.
We just see what happened. We turned it on for an hour, didn't tell anybody and the website was fine. And users were not affected, so that was a big confidence boost.
I mentioned that we have to adapt our tools, back-end tools for v6. This is the largest effort still remaining for us.
Also, we had posted on our Facebook page that we were participating in January and also a few days before that, we're getting ready, it's happening. Users that follow Facebook are able to post comments to our blog posting and we found that there are users out there who are very passionate about v6 and expressed a lot of frustration that their ISP didn't offer v6 at home and how they could not be a participant as a home user.
So these types of things were very interesting and also encouraging to us that there are users that care about it.
We found that in deploying, in participating in W6 day, there was nothing to fear, that it can be done, the way to deliver a website for dual stack -- over dual stack, is known and it works and, sure, we're going to encounter bugs from time to time, but overall, there's nothing to fear.
In talking to the press, and looking at some of the mainstream reactions to W6D, so not the technical press, we found that it's very difficult for the ordinary users to understand what v6 is.
So I actually did a lot of thinking about this, because how do you explain IPv6 to the average Facebook user?
It's not an easy thing and more importantly, the importance of v6 and v4 exhaustion to the ordinary user.
It is not an easy topic to explain to them.
So I thought about basically that, you know, v6 addresses are numbers. Why is it that we have such a hard time with numbers? Why should the average user care about such numbers?
The first thing I found was that the acronym is kind of difficult. These are acronyms that average users understand. They're short, they're all letters and they're all capitalized. But look at ours. You've got this acronym for techies, I don't know what that is, as an average user -- and, OK, now you lost me. This is clearly not just geek speak, this is in the area of assembly code.
So, for better or for worse, this is our name. But I realized through W6D that it's a name that you don't often see in the headlines, and the press had a hard time trying to explain it, but also trying to put in the headline what World IPv6 Day is, without having something like this in the title, because the average citizen isn't going to understand it.
I thought about it and probably the closest thing that can come to this name would be this: "MP3"? OK.
That's about as close as I can get. My mum can almost understand this, but the average user can.
And for us to really understand how the average user feels when they see these letters, I'll show you this: "XO-2b". How do you feel about that? That is how the average user feels. That is a real name. It's a planet in the constellation Lynx. Right? So that's how the user feels when they see IPv6. It's just like us looking at this, which comes from my friend who's an astronomer at UC Berkeley.
So what is it about numbers? Because we do use numbers every day, so I thought about this some more.
You know, the average person uses postal codes all the time. We have to know our current postal code, we have to know the postal codes of where we're going. The average user does not complain about postal codes. OK, so there is precedence for people using numbers on a regular basis.
Another number that we have is the taxpayer ID or national ID number. In the US, it's also known as social security number. An American can recite their social security number or taxpayer ID in about 3 seconds. This is another number that humans have adapted to and have no problem adjusting to and using on a daily basis.
Well, we're lucky, because v6 and the Internet doesn't have to rely on users knowing numbers, because v6 is really a mapping to content. So I thought about mappings, what about mappings? What about mappings the numbers? Is there evidence that the average citizen uses a mapping? Well, there is. Here is one. We do this all the time, especially if you travel as much as I do, is that I want a particular piece of content, MTV, and I've got to figure out in my new hotel room what channel it is, so I'm doing a mapping from the content I want to this number that I really don't care about.
But I have to work with the number because I want my content.
This is another mapping I have to deal with.
A little bit easier because, thanks to smartphones, when somebody calls me, the first thing I can do is map that to a name. Then I don't have to worry about that number ever again. So I don't even think I can remember my mum's mobile phone number. I just use this all the time. This is another mapping that average users use.
So the Internet shouldn't have any problem with that, right? We do that mapping better than these with DNS. The user doesn't even have to type m-o-m when their mum's computer connects to the Internet. So I thought maybe mapping isn't the issue.
What is it about the numbers? It has to do with numbers versus content and numbers mapped to content and what goes on in the mind of the real user. This is a big number space that the average person knows. This is what I had to think. What numbers are big that could be exhausted that my mum can understand? Did you know that this number space was expanded? In fact, this expanded twice. Did you hear about World UPC Day or Item Number Day? No. Right?
In fact, there are five variations of the item numbers that all happened through migration, transition and upgrading of point of sale machines from the biggest retailers to the smallest family owned stores in the smallest countryside. All those upgrades happened across all those point of sale machines.
So whether it's a candy bar, a book, an iPad or your favourite beverage, you know the code is there, you know the number is there, but what do you care about? You care about enjoying your content, right? That's what it is about.
I can assure you that users do not care about our complaints about how difficult it is to migrate to IPv6, no more than this user cared about what happened with the item number expansion program.
So I actually on World IPv6 Day had experience with fixing a broken user, and this also illustrates the point about content and numbers. I wanted to see, are there broken users out there? We are told they are out there. We're measuring them, so I wanted to experience a broken user. So I found this user. Frustrated.
Macbook is not working, some websites are working. Oh, somebody has tweeted back to me that something is going on today. So that's why Facebook and Google, et cetera are not working. Right? So awareness. And a little bit of, OK, so there is some logic to what's happening to me.
Now, yeah, this other user tweeted me and said it's this thing called Internet IPv6 Day or whatever it's called. I don't know if this is a critique of our marketing. What's interesting about this tweet is that it continues with the frustration, but it says, "Why am I the only one affected? Everybody else is fine.
I don't see really evidence of other people. I'm the unlucky one."
So I found this user and I was really excited, I say, said, hey, I couldn't say I was from Facebook, I couldn't even direct him to our Facebook prepared help page that we had, but I said use Jason's site, which ISOC was pointing users to, go to test-ipv6.com, and success.
Now, here is an illustration of a user who had to know about IPv6 Day kind of, but really didn't have to go into the details and ultimately just saw it as a barrier to getting to the content, so again, it's our job to make sure that our transition is seamless to the user, as seamless as possible, because really what the user cares about is the content, not IPv6 itself.
So my talk is really boiled down to this, which is that IPv6 is the right answer and if you are not part of the solution, you are part of the problem.
V6 is a solution and everything else, the rest, the alternatives, are problems, so again I'll challenge you, if you're not part of the solution, you are part of the problem.
Also, I know that we can do this. We must do this.
The Internet is just too important.
Dean Pemberton: Thanks for that, Donn. Any questions particularly for Donn or can we save them to the end?
All right. Thank you very much.
OK, next up we have Jason Fesler from Yahoo!. We're starting to see some common things here about what the content providers saw on World IPv6 Day, so we'll let that continue and try and tease out at the end where we go from here.
Jason Fesler: Hello, I'm Jason from Yahoo!. My agenda is going to be fairly brief and I'll try and blow through some of the stuff you may have seen already. I do want to go over briefly some of the motivations for why we are doing IPv6 and our involvement with IPv6 Day.
Yesterday, we heard that there were definitely different motives for different players in the markets and I just want to show some of the perspectives of a content provider on this.
Next I'll talk about World IPv6 Day itself, our preparations for it, where we did communication wise and what we saw on the day of the event and last some ideas for what's next.
So motivation: why not just use NAT? People probably have seen variations of these before. If IP addresses were motorcycles, this is 15 years ago, you can go off road, on road, you can go fast, go over some bumps, you can go explore the country, do whatever you want. It's widely successful and it's going to be a bit more difficult to find a place to park, fewer places to go.
NAT is unfortunately the future for IPv4. You're stuck sticking to specific paths and if there's a bump, everybody is going to go flying at once. Not my ideal world. For us, geolocation becomes a big barrier. We use that a lot in our products for showing you the local content for news and movies and advertising.
Abuse. We heard earlier today, that ACLs are a problem in our world of NAT. If we were to block one NAT global IP address, we're going to block somewhere between 10, 100 or even 1,000-plus users. If it's just one IP, OK, that's collateral damage, if we have to start blocking a botnet, a wide range of IPs, that's very heavy collateral damage.
Lastly, we are concerned about performance with the NATs, poor exhaustion, CPU exhaustion, pretty straightforward.
So we know the well is dry. V4 ran out here. It is in the process of running out elsewhere. We have to deal with the fact that NAT is coming. We have no way around it, at least content providers do. Those of us in the room might try and deny and go IPv6, but we still have to prepare, but at the same time, we are promoting IPv6. We really do want to limit the damage that v4's future is going to cause.
That brings us to World IPv6 Day and our particular involvement in it. You guys are familiar with IPv6 Day already. It was a major one-day event, international, lots of players involved, 400 plus sites, we heard, 2.5 million sites. And it was a test flight for the Internet, it wasn't a lab experiment. It was the first real worldwide-scale experiment of its kind.
Yahoo! had decided to deploy the main site, not all the Yahoo! content, but just www.yahoo.com and the international variants for most of our markets. There were a few markets intentionally omitted either because of technology or because they are separate companies from Yahoo! Inc. But most of the markets were covered.
To do this, we weren't ready to deploy IPv6 in every location, particularly at the data centre level. We had to deploy proxies in our peering points. Part of this is a matter of time, part of this is a matter of code qualifications with certain equipment, et cetera.
We deployed seven locations in total, and in each of these locations, we also deployed outbound 6to4 relays to help mitigate the 6to4 disaster.
On that day, for us, it was just the HTML, we did not serve images or Java script or anything of that nature on that. The intent of the day was more to prove out that IPv6 won't cause any large-scale blowups at Yahoo! or across other players. It was also to help intentionally educate the broken users about their particular problems.
We knew there was risk involved. There is the broken users problem. Most of you are familiar with it by now. Our measurements show at this point it's about .022 per cent, 22 out of every 100,000 users. This is a stadium of 100,000. 22 of them were broken on that particular day.
We know about the time-outs. We found that if we didn't try and do some form of outreach the user experience was going to suffer. The news media gave some coverage about IPv6 Day, but it wasn't enough. We, like the others, had detection on our home page; if there were problems with the dual stack reachability, it would put this up at the top.
It did take them to our help site. The help site did have a test where they confirm their brokenness.
The test was available in 38 different languages and it was linked to heavily by our various social media outlets.
We did do trial runs. We did not want to shoot ourselves in the foot, particularly when everybody was watching. If we're going to shoot ourselves in the foot at all, let's do it when nobody is watching; blame it on the network people.
The first run we did was very informative. Nothing broke per se, but we did find that in.yahoo.com was rerouted to the United States. Our geoload balancing solution for India was detecting that IPv6 was not reachable, because of problems with the IPv6 connectivity for the health check. It did the right thing. It said send all the traffic elsewhere, but in the meantime, there was a 15-minute window where things were a little bit slower for India.
We definitely learned that monitoring is not the same in a dual-stack environment. The geoload balancer was working fine for days and days before the event, for IPv4. Until we put it to the real test, we didn't catch every possible error point, and this one is one that was caught in the trial instead.
We did a second run, instead of at the lull of our traffic, we did it at peak traffic, again to kind of stress things and see what breaks.
As we threw the switch, traffic dropped 5 per cent.
Whoa! It turns out it was 5 pm on the east coast, people were leaving the office. We learned before doing any major traffic shift, any kind of IPv6 turn-up or turn-down, or anything else for that matter, don't do it on the hour, do it a little bit off. That way you can isolate it from people going home. So don't start something big and risky at a traffic inflection point.
IPv6 Day came. Traffic shifted in the UK, but we didn't notice it. The proxies shifted, but the actual back ends didn't see it, and we at this point do not have external monitoring through our commercial suppliers for that particular service and as such, we saw nothing. Everything looks great.
Thank you to RIPE. We saw our UK traffic actually jumped in latency as we threw the switch. This was another case where the geoload bouncer didn't quite do the right thing. It turned out that the geoload bouncer had been re-imaged the week before and they forgot to enable IPv6 on it.
So that got fixed within a few minutes of it actually being seen. Traffic came back to normal and in this particular case, IPv6 was actually much closer to RIPE than IPv4.
So, again, always have more than one way to look at things. In this particular case, we're now aggressively seeking this external monitoring from our partner on that.
Another take away on this is practice does make perfect for major changes, schedule at least one test run. Don't do it when everybody is watching. It definitely paid off for us.
I have only a few statistics for World IPv6 Day itself. We kind of feared IPv6 Day. We didn't know what might break, router might go, as some of you may heard, "Kablooie!" It didn't happen, at least not for IPv6.
We didn't know if the measurements on brokenness were a hundred per cent accurate. We had some collaboration on those numbers, but we didn't have it in the real world quite yet.
The reality is it was just another day on the Internet, it was a Y2K. It was a lot of prep and no excitement. People didn't really notice. That was a good thing. We had very few support calls. We had roughly 10 coming directly to Yahoo!. Monitoring IRC network with some of the other participants, the general consensus was very little customer support traffic on that day.
The question is: were there few people broken or were people thinking, the Internet is down today, I'll try tomorrow? So that's definitely a downside of the 24-hour event.
In terms of page views, roughly 0.17 per cent of our page views during that 24-hour period were on IPv6.
This does exclude Yahoo! Japan and Yahoo! China, as they are separate companies from the rest of Yahoo!, so they're not represented here.
Breaking that down a little bit more, Europe was the clear leader, when you take into account those exclusions. US was at the bottom, sadly, Asia 0.1 there.
When we looked a little bit closer, we brought that down into unique users per location and the biggest numbers of users were France, US, Britain. From there, it trails off.
So with so little traffic, with few things going on, the question is did we actually move the needle at all?
I'm going to say yes, we did. We definitely saw roughly 2 million users on IPv6 that were unique that day. We saw the industry respond to the day, to the date in particular, we saw changes in Mac OS, in Safari, to deal with the broken users, changes in Chrome -- thank you, Google -- and Firefox -- again, thank you, Google. The roughly happy eyeballs implementations, we're looking forward to seeing similar changes happen in IE or Windows, we have no information on that at this point, but we're hoping to see something from that direction.
We also saw a number of the World IPv6 Day participants just leave their sites up dual stack, which is a good thing. The event started off with roughly half of them being IPv6 ahead of time and 60, 70 per cent afterwards.
Even the politicians took notice. This is an interesting barometer. Right after the event, ISOC was at a Conference and the Department of Commerce congratulated them on the event for how well it ran and for the job they did. I was at a similar Conference in the UK and we had a Member of Parliament come in and tell us pretty much the same thing. I'm not 100 per cent certain they were as in technically deep as we are here, but the fact they were paying attention was impressive.
So the question is: what's next?
We do have some informal discussions going on right now on what to do next, between content, between access, we really need this to be a bigger event, it needs to be a longer event, we need access providers to be much more prominent than what we did this last June 8.
Content, we definitely need more content, speaking on behalf of Yahoo!, I know we only did HTML this last time. This next time around we plan on doing a lot more than that. We definitely need more global participation. It shouldn't be just this weird group of people doing this thing, it should be more of an industry-wide event.
And last I just want to thank the Internet Society for helping drive IPv6 Day, for helping provide an umbrella for us all to work under in a friendly way and free of any kind of politics or whatever else. It's really important for the industry and they have also been really great hosts. Last, they gave us a date.
That's something that we have been missing for the last 10 years. This helps tremendously this year.
I'm ready for questions.
Dean Pemberton: Thank you. Do we have any quick questions?
No? OK. Great.
Dean Pemberton: Last up on the panel before we start going through at least the list of questions I have, is Gaurab Raj Upadhaya from Limelight Networks.
Gaurab Raj Upadhaya: Thank you, Miwa. Thank you, Dean.
I'm Gaurab Raj Upadhaya, I work for Limelight.
I think first, before I start, I probably have to point out that, different from my other three panelists, I really can't talk about our customer data, so it's not so much interesting statistics, but I'll give you our perspective on things.
What is CDN? Most of the people here, unless they're in network engineering or actively working in the Internet, probably don't know who we are or what we do. Basically, Limelight is a CDN, we deliver content transparently, so our name never shows up, it's just the customers who show up. We call ourselves massively provisioned, because we have a pretty massive network, as the stats on the slides show.
Primarily looking at around, you know, 5Tbps of egress capacity, of which we use 50 per cent and are upgrading at a rate of almost 100 per cent every year rapidly and have presence in many different markets. We run our own backbone between our sites or between our POPs, which means that we have a fairly large network architect team and network operations team that runs the backbone and we carry our traffic over that, instead of carrying it over the Internet.
I have a few slides here on the motivation of why v6, so I'll probably skip this because we have seen this already. One of the things for CDNs is that NATs are not going to be good, as pointed out earlier, because it complicates geolocation massively and also abuse tracking and TCP port exhaustion, as we know from a few years ago, the Google Maps presentation. So it's not good.
At the same time, as CDN, we really can't tell all our customers we have to do this. This is what is to be done. We can't really enforce it. The requests or the push is always from the customer. We can just be ready when we are ready, but until somebody says, OK, I want my content on v6, I really can't go on and say, OK, I'm going to turn on v6 for you. That's quite different from the content owners themselves, which can determine when they want to do it, whereas we are quite dependent on somebody else agreeing to do it. That is a bit more complicated for us.
Having said that, our network has been v6 ready for almost three, four years. The work started in 2008.
The network itself, all the plumbing, as we call it, the plumbing of our network, has been dual stacked since 2008. All the peering transit, everything we do on the network side is dual stacked, for many years now and we have been peering on v6 for quite a few years now.
We dual stacked all our AS machines and servers in 2009, which means all our platforms, everything has dual reachability over v6, both internally and externally so it has been there. Back in 2009, we actually tested Netflix over v6, the Netflix Watch Instantly over v6.
This was not doing a AAAA record on the main Netflix domain name, but we had a sidecar, ipv6.netflix.com, so that people who wanted to use v6 could use it and it also helped us test our infrastructure, whether it was ready to handle v6 or not. So using a sidecar we were able to work through it, it worked quite well.
Coming to World IPv6 Day, basically, when ISOC came to us and actually we hosted the ISOC IPv6 World Day website on our CDN, so we were in it quite early, because we are hosting the ISOC World IPv6 Day website, we better be damn ready to do it. So we looked at what was missing.
As I said, the plumbing was already there, we went out, we viewed everything with our ops team and made sure that all our machines, everything was there, instead of integrating our v6 DNS into our v4, we kind of created a separate architecture which is just v6 based, mainly because we realized that v6 connectivity is not as dense as v4 is and so we had to put -- we couldn't really say the v6 is going to work the same way as v4, so we had to separate it so that we could control both of them separately, based on how our egress worked for v6.
We then continued to improve v6 performance upgrade links, asked all our peers, transit providers, partners to do dual stack on all our PNIs and all our public access point peerings and continue to monitor.
One of the things we also realized at that time is there are not many third-party tools that will monitor v6 for us. There are not many agents out there. There are many for v4 that sell this as a service for monitoring reachability and things like this, but not so much for v6.
The other one was identifying interested customers.
As I said, the plumbing is ready, we got our own website on v6, but that's not the bulk of our traffic. It's the customers. So that is a bigger issue for us. It seems like the customers don't want to risk going to v6, but at the same time, they don't seem to realize the risk of just staying with v4, because there is no more IPv4 to use in future, so there are risks on both sides and I think the big thing for us going forward at least our business side is to try to convince our customers of the bigger risk that v4 has run out and they have to go and take the risk of at least starting to play with v6.
So the planning and non-intrusive provisioning of v6 put up the sidecar for almost everyone on our internal network, so that if the customer says, "I want to test v6," we say, "OK, this is your URL, you can go and do it." Not anywhere in production or not anywhere publicly visible, but we still have it for everyone now.
Internally, if any of our customers say, "I want to test v6," we say, "Use this soft domain and there is the v6."
That doesn't need to go public at all.
Then another one we really had to work hard as I said earlier, there is no third party monitoring agents that we rely for v4 to test v6 reachability and performance from the end site and that is something we are still working on, you know, not really there 100 per cent yet.
There are a few things we had to look at, anticipating the issues, whether to scale or not.
Because it was unknown. As we saw, the Formula One crash and the picture earlier -- those pictures were really good, I should have put some more on mine. But that was it. So we had to anticipate those problems.
We knew that there is going to be asymmetrical routing issues, we had already seen that and that's how it was slightly different, so we also had a panic button by introducing the ability to tune out the AAAA records really fast by reducing the TTL on our DNS and the big plan was how to measure.
Are we going to roll out net flow v9 all over the place, collect the data? That is something we have still not completely worked out. How do we measure, what we store. Now we store everything, but we don't know what is measurable and what's worth measuring and what is not.
So as I said, we hosted -- this is the World IPv6 Day statistics. As I said, I can't really talk about a lot of other customers, so I have taken two that were willing to share the data. We hosted the ISOC website and thanks to ISOC for that, trusting us with that. The absolute numbers are really little, so I put it in percentages.
The amount of v6 traffic to the ISOC World IPv6 Day was 0.53 per cent of the IPv4 traffic that hit that website. That is only World IPv6 Day. Prior to that, it was around 0.46 was IPv6 compared with the rest of it being IPv4.
Interestingly, NASA, one of their websites went dual stack on the World IPv6 Day, and we saw that 22 per cent of the traffic going to their website was v6 and it has remained actually quite close to that number, almost around 20 per cent now since then. Where is that traffic coming from? It's probably mostly R&D networks, US Government, military, we don't know, but definitely that is a very impressive or encouraging number there.
Overall, the total v6 objects or v6 requests that come into our network is 0.00029-something per cent of our total v4 requests coming into the network. The number is low -- that is for the World IPv6 Day. If I look at it now, it is even lower. That is why I said it's not worth putting it on a graph.
We had 3Mbps of sustained traffic with a peak of around 9Mbps and most of the traffic was still in the US, then Europe, followed by Asia. That was our thing.
When you compare with the 2.5 Tbps we see, this 9 Mbps are really insignificant, but it is growing.
That's the main thing, I think.
We had some specific problems, it should be MTU black hole in there. We realize that one of the carriers was using 6PE and they had some empty users and the same carrier also had some really weird asymmetrical routing. We solved it by starting v6 with them.
Some external software we used on our systems were not passed with v6, but that was something we detected very early and were able to solve it. Our fault, we forgot to put a v6 default router on one of our routers, but thankfully that was towards the end of the day when the day hit that part of our network.
Before the v6 Day, the net ops people went around and made sure we could ping all the boxes on our network with v6, so the plumbing was there and there was nothing wrong, on the day itself.
After that, what we did, we have done the scale problem, saying what works, what doesn't work. One of the things we have seen is as vendors have put more and more 10G ports on a single line card, they have probably not put enough cam on the line cards, which means that as the number of prefixes grow in both v4 and v6, we actually saw in a few places cam partitioning on the router, so that was a big concern for us.
The other one was our load balances worked well at least because we had tested them previously in 2009, and that was the advantage of having it dual stacked for a long time, we knew our plumbing works quite well.
I think I will say that nothing went wrong, it went sort of as planned.
The other one is still we don't see as dense a connectivity in IPv6 as we see in IPv4 and that is still a big problem. We just can't say v4 and v6 are going to be exactly the same, when it comes to EPS traffic.
The focus now is identifying our critical paths, what do we do, how do we go ahead? The plumbing is done, what next?
Preparing for a longer permanent ... interval. Like Erik said earlier, maybe we will have an IPv6 Week next to test the plumbing better. One of the things the net architecture team is working on is to convince our business side that IPv6 should be an opt out feature instead of an opt in feature, but that of course is going to take a bit more time.
Dean Pemberton: Thank you. All right.
I would like to thank all of our presenters. I have some questions down here, but I want to throw it open to the audience first. Does anyone have any questions that they would like to ask our presenters about their content delivery side of World IPv6 Day?
Masakata Ohta (Tokyo Institute of Technology): I have questions to some of the presenters who mentioned issues with NAT. You said that there is a geolocation problem ... but it's shared by non-NAT DHCP based dynamically assigned IPv4 addresses already. So NAT does not make the problem worse and if addresses are assigned statistically, the port number may also be assigned statistically and then according to 48-bit static number is much easier than recording 128 bit, so maybe large-scale NAT still causes a problem because it's large scale, but NAT does not have to be large scale and even with large-scale NAT, if a single IP address is shared by a small number of people with similar geolocations then large-scale NAT is not a problem. So I think it's just a cooperation of NAT, not a problem of NAT itself, that there are several issues. What do you think about that?
Dean Pemberton: That's a question for our panel for what they think about your statement.
Donn Lee: Can I ask a clarifying question? I feel like I missed part of that, but are you suggesting that you have more distributed smaller pockets of NAT, distributed and maybe fragmented to more devices and smaller geographical areas?
Masakata Ohta (Tokyo Institute of Technology): It's a possibility but even with large-scale NAT, a single IP address is shared only by a small number of fixed set of people with similar geolocations.
Erik Kline: You're making an implementation assumption there. You're assuming that that's the case, right, and that may very well be the case for some implementations, but there is no guarantee that will be the case for all implementations.
Masakata Ohta (Tokyo Institute of Technology): Then even with IPv6, an IP address may be shared by people with different geolocations, so it's merely operational issues of IPv6 and/or NAT.
Erik Kline: How exactly are IPv6 addresses shared?
Masakata Ohta (Tokyo Institute of Technology): If there is a NAT box here and it covers a very large geolocation, then IPv6 box can share, can serve the same geolocation and can be dynamically assigned randomly, then the geolocation program is the same.
Erik Kline: If someone wants to move around the IPv4 prefix to a home, which is presumably somewhere between a /64 and a /48, and I think in Japan it's largely a /48, but if you want to move that around on a minute-by-minute basis or an hour-by-hour basis between Osaka and Tokyo, I mean, yes, there will be a problem. You can in fact make geolocation problems.
I don't think people actually operate this way, though, so I don't think this is an operational reality.
Masakata Ohta (Tokyo Institute of Technology): So I think we can agree that it's an operational problem.
Dean Pemberton: OK. Thank you.
George Michaelson (APNIC): I have a question from the Jabber. John Curran from ARIN has a question for the panel. Can you explain the reasoning for World IPv6 Week rather than World IPv6 Turn-on-for-Good Day?
Erik Kline: I'm not sure that there's a difference. I'm not a marketing person, so maybe we should call it what John suggests, but the intent is to roll and stick. We turn on AAAAs and they stay on. The point is that if you are a network device, if you are an operating system, if you are an application or an operator of any kind and you break dual stack or you break IPv6, you're doing it wrong and when we turn on AAAAs collectively and leave them on, that will become the case. Right now, you get sort of a free pass, but eventually there will be none of that. So, you know, I suppose we should take John's marketing advice.
Jason Fesler: I will say that if we do IPv6 Week, it does allow for participants to get a little bit easier buy-off from their management to participate if they're not able to buy off rolling and sticking. As for us, we're definitely talking about sticking, though.
Donn Lee: I have one comment about the NAT question earlier. For me, it's an architectural thing. We have to either say that the Internet has -- the architecture being direct and open, has worked, and we want to stick with that or we have to say that we're going to go to more brutal architectures, force fit them, hope they work or change the connection model and I think it's architectural decision. I think we have to decide are we going to keep the current model, because that is what IPv6 requires, adherence to the open and direct model without NAT.
Masakata Ohta (Tokyo Institute of Technology): I'm not sure you are talking about the DSCP dynamic IP address assignment is architecturally bad or lack of transparency is bad. Which do you think is bad?
Donn Lee: I'm talking about inserting a device that doesn't exist today between the user and the user's content or between a user and a user. There is a device that large-scale NAT is suggesting that doesn't exist between users today that will be inserted and that model doesn't exist today and it hasn't, except at their home, which essentially is a user and that's only because we ran out of v4 addresses long, long time ago, so we had to do NAT at home.
Masakata Ohta (Tokyo Institute of Technology): But I know several ISPs which already use NAT.
Donn Lee: I'm sorry?
Masakata Ohta (Tokyo Institute of Technology): I know several ISPs using NAT before IPv4 address exhaustion.
Donn Lee: But they probably were not doing NAT four years ago or six years ago. I understand --
Masakata Ohta (Tokyo Institute of Technology): About 10 years ago.
Donn Lee: OK. Why?
Masakata Ohta (Tokyo Institute of Technology): I don't know.
Donn Lee: Large-scale NAT will happen. I'm not in a dream world here. I know it will happen. I'm saying that, like it was said yesterday, we have to make the transition as short as possible and, yes, we can use NAT, but for how long? What are you doing to do IPv6 today? When you leave this Conference, what will you do? Will you do your part? Will you be part of the solution or will you be part of the problem?
Lorenzo Colitti (Google): Just to add something to that discussion, it's true that there are ISPs that are doing large-scale NATs today, it's true there are ISPs that have been doing large-scale NAT for 10 years, there is one in Italy called Fastweb. They used to do NAT using bogon addresses, then the regulatory authorities said that they had to do one-to-one, and instead of assigning public addresses to their users, they assign one-to-one NAT addresses to their users, so that users still don't have public IP space. There will be people who do everything. The point is the industry as a whole does not in general use NAT or large-scale NAT today. We can expect that the industry as a whole will use large-scale NAT five years from now.
I think the question is, how many people do we want to rely on only large-scale NAT five years from now and how many will we want to do v6 and large-scale NAT and which way are we going. Yes, it's true, there are people who just want to do NAT, but I agree with Donn.
The real solution is IPv6. Some people will just want to do NAT44.
Dean Pemberton: OK. Thank you.
Randy Bush (IIJ): There are people who want to do stupid things for any number of stupid things. OK? The Internet is about intent connectivity, it is about smarts at the edge, not smarts in the middle. It's about transparent networks, et cetera. These kluges are death. OK? And, you know, Erik and Lorenzo have shown you slides measuring the death. Google is going to blacklist Japan. This is not something we should be very proud of.
Yan Ma, I think what you folk are doing in China is great. It's embarrassing to us, Yan Ma. So there's no excuse for NATs. You're abusing the network, you're abusing the users.
Dean Pemberton: Speaking about the users, I have a question about the 0.02 per cent of the kind of unbroken dual stack users. How significant is that number and should we care about them and are they a reason to turn off the v6 after World IPv6 Day?
I liked Donn's analogy for the barcoding and how that barcoding got extended. I think that's really good.
I'm going to steal with pride.
I was thinking, if 0.02 per cent of stores couldn't scan the new barcodes, would they have actually stopped the deployment or would they have just rolled on and slowly those stores come along?
Do you think that number is significant and the follow on question, what does that number have to drop to before we can turn this on and leave it on?
Donn Lee: If you ask me, the network engineer, I don't think the number is a show stopper. I think the number is small. But if you ask people that care about various parts of the business at Facebook, user experience or advertising or the integrity of the site, the integrity and up time of the site, then those numbers start to matter, because if they turn it on, they automatically take a hit and don't get a benefit, that's always been the situation, but I think that there are several pivot points that could make this number a non-issue.
One is that really we didn't see any measurable problems with World IPv6 Day, because if you think about it, .02 per cent, the way I explained it to my site integrity people, if you have a monitor that has 3,000 pixels, it doesn't even register as one pixel on a graph, so you can't even measure this. It's very difficult to incorporate that into your SLAs. I know theoretically it's happening, but it won't appear on your graphs.
The other pivot point is when is there more than 0.02 per cent of users on NAT because that might be something that could take up the chain, there are more large-scale NAT users than there are v6 users.
So, OK, there's another number that we can work towards. And then there is the kind of philosophical one that what if a large mobile phone gets released next month and hits the shelves and there's tens of thousands of units and they all have a dual stack bug, it would be nice that we had the AAAA out there because that phone would not have been released, they would have failed testing, because we had the AAAA already out there, but because we didn't have the AAAA out there, it was released in the wild and now it's nearly impossible to get out of the wild.
Alastair Johnson (Alcatel-Lucent): I make a product that does carrier grade NAT. I make it not because I think it's a great idea, but because it's a necessary evil for many of our customers. I would just like to comment really that most of our customers that are interested in NAT because they need it for longevity of IPv4 services all have IPv6 plans, very serious ones, quite often they go hand in hand and they are deploying them in parallel.
So it's not that I have seen many customers that have said, "We are going to NAT and we are going to forget about v6." I think that is at least positive for the operators that are deploying NAT.
One other point. There is reasonable precedent for established NAT in large networks in the mobility space.
Pretty much every telco somewhere or another is NATing their mobility network, so we do have reasonable experience with this.
Dean Pemberton: Thank you.
Miwa Fujii: Jason mentioned that if you do a similar world IPv6 activity again, you hope you have more access providers participate in the event. Do you have any strategy or tactics or suggestions to achieve that goal?
That is number 1 question.
Number 2: a few speakers mentioned the importance of talking with end users and it's quite interesting to learn about that and especially this morning Ping mentioned that the Hong Kong Government is approaching to outreaching to the end users to understand the importance of IPv6 ready to purchase the IPv6 ready equipment. Do you have any suggestion what kind of activities or event or method would work to achieve this goal?
Dean Pemberton: I think that's a very good question.
A lot of times the access providers claim that one of the reasons they're not rolling out IPv6 is there is no content and now we have a situation where the content providers made it available, and the uptake wasn't there because the access providers hadn't come along. Any ideas?
What's the carrot for the access providers? Next time we have IPv6 Week, Month, Quarter, how do we get it so that the access providers have come along? So that once the content providers enable the websites, we actually get that 20 to 30 per cent growth because the access providers have built the networks out?
Jason Fesler: There are a couple of aspects to it. ISOC is starting to round up people right now and getting them in a room and getting them to beat heads and talk about the next steps for IPv6, including a potential IPv6 Week or whatever. There's outreach that we have all collectively done for World IPv6 Day, we are here now talking about World IPv6 Week to get attention.
I don't know that there are other better forums for this specifically.
Erik Kline: I think we have a number of large access providers who want to do this. We're going to get them all together. We're going to try to issue a press release and if you want to be on it, you want to be on it and if you don't, then your competitor is doing it.
We are trying to find a large access provider in each market and if you want to be up there with your competitor doing next-generation Internet access, then you clearly have the social pressure to join, because your competitor is going to.
Dean Pemberton: The other idea may be to not so much just appeal to the access provider but try and appeal to the end users. The problem with that is that all they care about is drinking their beer rather than scanning it, right, but what if the beer cost half the price if you scanned it versus the manual process? Has there been any thought about, even for a short amount of time, offering premium content via v6 versus not on v4?
Erik Kline: I don't think so. We have no interest in playing games with our users, we just want to serve them the content. One of the issues with users, of course, is making sure that they have gear at home that can do IPv6. Ordinarily, you wouldn't have to engage the user.
We shouldn't have had to engage the user, but everyone has devices in the home that aren't even going to ask for a v6 prefix or address or that kind of thing and for that reason, I think there will have to be some kind of campaign. Maybe an ISP that really wants to could roll out CPEs to their customers that are v6 capable, but I don't see very many lining up to do that.
The other choice is to simply wait for the natural turn in home gateways -- I don't know what that is, two, three, five years -- before we have IPv6 capable devices. And who knows what bugs those have and whether or not they actually work with the way IPv6 is deployed on the operators side.
Dean Pemberton: Thank you.
David Farmer (University of Minnesota, ARIN AC): I want to encourage a World IPv6 Week or World IPv6 Next Step. It is important that it happen within 12 months of the last one. We need to keep the snowball rolling. If the snowball stops, we're all going to be in trouble. If we can make it World IPv6 Turn-On-Everywhere Day, that's great, but if we can't, we still need World IPv6 Week to keep the ball turning, to keep the snowball rolling down the hill. Because we'll eventually get enough momentum to just let it go and turn it all on, but we got to keep rolling, keep the momentum, if we're not ready for perfect, that's great, but keep it rolling.
Randy Bush (IIJ): I think we're indulging in optimistic fantasy. I work for a company that's had IPv6 13 years so far of commercial deployment. It has not caused our competitors to do anything until recently, only a decade or so later, and you heard earlier what they're doing and I don't think any of us are proud of it. I think this is all wonderful and the American idiom is preaching to the choir, but what are we actually going to do to get effective change at the customer?
Erik Kline: I think there was some social pressure for World IPv6 Day, Randy. There were people that were in the room when we talked about World IPv6 Day that opted out, that later came back and said, "Maybe I should do this, because so and so and number 1 and number 2 and number 3 are doing it and I'm not and maybe I should do it too."
Randy Bush (IIJ): I didn't expect to see you and Donn and Jason all up there, so that's progress on the content provider end. As Dean said, this is a surprise, it's a reverse. We used to say that the problem was there was no IPv6 content and so the access providers wouldn't do anything until there is. And now we have the content, but, you know, have we seen much positive change in the access providers?
Jason Fesler: We're here to hatch the egg. The thing I look forward to is having a number of the content providers and the top content, top 50 per cent, could be done by a few number of people all on dual stack.
I want to give the access providers a reason to go.
IPv6 might actually be a little bit cheaper than continuing to have more and more NAT. We still have to deal with the long pull of the end user and the CPE, but I'm hoping that we are at least giving the ISPs incentive.
Randy Bush (IIJ): In IPv6 Week, why don't you turn off v4?
Donn Lee: Well, it would be great to have some of these upstart websites that have a CEO who has got v6 passion to no degree that he or she just decides to do something like that. That would be just awesome. Maybe not a top 10 website, but maybe one of these young upstart websites that are doubling like every two months. That would be nice. I have often fantasized about that, maybe we should start a company, a website that's so popular that we can do that.
So I think with the access piece, I think we need to talk to access people. I can't speak for them. I have talked to them and they all have different concerns or beliefs, perceptions, but we have to engage them and we have to have them up here and we have to ask them the questions.
Lorenzo Colitti (Google): I think what happened with World IPv6 Day is that a critical mass of a few large websites got together and said, "We're going to go this, we're going to do this or bust," and then just people followed. In retrospect, three months later, it was, "Yeah, of course we could have done that, but when we started, we didn't know if people would commit to it, we didn't know anything, we didn't know if we could do it."
A lot of people had made plans, and in the end, we delivered and 500 other websites just tagged along.
So I think the single most important thing is that the industry breathe a collective sigh of relief, saying, "We turned it on, our companies didn't go bust, the Internet didn't break, everything happened as normal."
Access networks have not had that moment yet. So I think that we can do this. I share some of Randy's concerns, but I think that the access industry can be led by a couple of early adopters in this way and I think that the sort of roll-on effect of competitive pressure can actually help. I think that's really part of what we're trying to do here and we are talking to the access providers and there are people who have really production-grade trials. I mean, Internode comes to mind and I have Comcast at home and it's really solid so I think we shouldn't write that off, I think we should keep talking to them.
One thing that I didn't hear anyone on the panel say -- and I'm sure somebody forgot -- did you forget to say, "If you're an access ISP and want to play, come talk to us"?
Sala Tamanikaiwaimaro: First of all, I would like to say thank you, it was a very interesting panel. Just to introduce myself, I'm Sala and I'm speaking for myself as a netizen in the Internet universe. The question I have is in terms of the context of transitioning to IPv6 and preparations for transitioning for those who have yet to transition, where do you draw the line or where does one draw the line in terms of traffic management and blocking an entire country off or somebody off? That's just what I wanted to put to the panel.
Erik Kline: I don't think anybody has talked about blocking a country. In no sense is access to any of the logic or resources denied in any way, right? It's just about advertising whether it's there or not that causes it to be used. There is no access control list. There is no denying. There is no blocking of any kind. It's really no different than, hey, we should serve you out of a data centre in your country rather than across three oceans.
Tony Hill (ISOC AU): Fascinating to listen to the analysis of IPv6 Day, and the message I have taken away so far from these presentations is, firstly -- and not just from these presentations, but from talking to people around the industry -- the sweating on just getting the v6 to work, making sure it does; secondly, the sweating on whether the v6 is going to break the v4 in some way, and I think we seem to have passed both those hurdles.
What I haven't heard yet from this panel, and I would be interested in, and I think maybe we are going to hear some more of that today as well, is from each panel member, what the positive thing that they learnt from this IPv6 Day was? Did they take away positive messages, did they see advantages?
Dean Pemberton: We have got probably three more questions and then we can go through and ask that of the panelists.
David Farmer (University of Minnesota, ARIN AC): I just want to respond to Randy real quick and he was asking about what has changed, what's all that.
There was quite a bit of press around this. This is a room of engineers and we are talking about what we saw on the Internet and stuff like that.
There is a whole number of effects that World IPv6 Day had that we haven't talked about in this room. I'll just give you a personal example. My senior IT management team, without prompting from engineers, came back and said we need an inventory of IPv6 readiness for all of our applications at the University of Minnesota.
Without all of those articles, without all the trade press and the regular press, that would have never happened.
I'm serious. The snowball is rolling. We need to keep giving it a little bit more momentum, a little push, a push and a push and we'll get there.
Erik Kline: On the content side, on the website operators side, there are only so many times you can offer AAAAs and take them away, we up here only have one more push of the snowball, right, turn it on and leave it on and after that, it's got to be up to other effects.
Judith Duavit Vazquez (PHCOLO): I joined the board of ICANN in October.
No one can answer this question. Because they're not here. Where is Microsoft? A big part of the problem is Microsoft. I think we have to get someone from Microsoft to the floor in APNIC and other RIRs, moving into the future and get them involved and getting the users, because they do control the PCs and I still see many PCs in front on this floor, we got to get them on IPv6.
Erik Kline: Bing.com did participate and xbox.com has AAAAs and --
Judith Duavit Vazquez (PHCOLO): That is on the content side. We are talking about the user side and --
Erik Kline: Windows 7 has iPv6 --
Judith Duavit Vazquez (PHCOLO): Yes, this is Asia and it takes time for us to get -- no, you don't have -- you're Google, please don't answer the question. I said this isn't for you.
Erik Kline: I know some of the folks at Microsoft and I know that they have done work, they have certainly contributed. I don't know what it is that you feel is missing. Can you state what it is that you think is missing, what they need to do?
Judith Duavit Vazquez (PHCOLO): Yes, when you look at even Windows 7 today, you still have the option of IPv4 or an IPv6. When one sets up, you're on a Mac, so I have seen basically I'm very concerned about the user side, in fact, maybe this is not the place for us to discuss it.
Erik Kline: Maybe we can talk offline.
Judith Duavit Vazquez (PHCOLO): Yes, yes, offline, thank you.
Dean Pemberton: Andy, and then one last question.
Andy Linton (Victoria University): Andy Linton from Victoria University in Wellington. So that's the hat I'm wearing at the moment.
I teach network engineering in a university and one of the things that certainly hasn't come across to me with my colleagues is that there's any awareness of IPv6, you know, from things like the World IPv6 Day.
I still have colleagues in network engineering at my university and I suspect it's very similar in lots of others, who are still talking about class A addresses, class B addresses, class C addresses and they still really aren't teaching anything that's got anything to do with the Internet of the 21st century, so one of the things I would like to see any future work on World IPv6 Week push on is to have a world IPv6 -- put outreach to universities in all our countries so that they are teaching IPv6 at least on an equal footing with IPv4 and perhaps even teaching it to the level of, "We are going to teach you about IPv6 and by the way, there is this legacy system called IPv4 which you may need to know about."
I think if we can get that emphasis happening, you know, one of the groups of people who are going to come out from universities, work in our industry, who are going to make a difference, not just in the network areas, but in commercial organizations and so on. So that's just a thought for you to throw into the mix.
Erik Kline: I think perhaps the most effective thing to do would be to find all of the professors in your university and find all of their Internet access providers and have them deliver IPv6 to the professors' homes and see if it changes the curricula.
Dean Pemberton: I personally want to get back to just talking about IP and not worry about the version number.
We don't speak about BCP v4. Eventually, we can get back to just talking about IP.
One more question and then Lorenzo, but that's it.
Kuo-Wei Wu (NIIEPA): I think from the v6 Day, what we learned, basically we can see in the two different sides. For the v6 Day, actually we tried to ask the content side to turn on the v6 services. We actually didn't touch the ISP because the ISP is more complicated. Basically, ISP they need to think about other issues, it is very difficult for them to jump in before they have a customer, you know.
But for the content side, I will say if we have lesson learned from v6 Day, I would say we should change the name instead of v6 Day, should be the v6 Turn On Day. Because a lot of people participate in v6 Day, they thought, "This is the event. OK. On June 8, I turn it on. After June 8, OK, not my business.
I turn it off."
I think if we are going to learn from this event, in the future, I don't like the v6 Day, I don't like the v6 Week. I don't like the v6 Month. I really like this is v6 Turn On Day.
Lorenzo Colitti (Google): Do you want to answer that?
Erik Kline: Yes. We agree. We want to turn it on and leave it on. Everyone up here wants to do that and I think that's basically the next event for us, a dollar sign event.
Donn Lee: I think one point is that if we had a name that said it was like a flag day, then maybe it could be more impactful. I think that's good input.
Lorenzo Colitti (Google): I claim responsibility for the World IPv6 Day name, it was my fault. It's not a good name. It was the best that we could come up with.
A week just seemed more than a day, but the problem is if we call it Turn On Day then people say, "But we can only do it for a week. Then we can't participate."
One thing I wanted to say about this, I just got an email. We have a lightning talk slot today to talk about the ideas we have at World IPv6 Week. We have about one and a half minutes to talk, but maybe we can put up a couple of slides saying what our current thinking is and then, given that you have our current thinking, you can come talk to us and give us your ideas and give us your feedback on what we are trying to do.
I'll be putting up some slides today at the 4 pm slot.
If you are interested in World IPv6 Week, come to the lightning talk session at 4 pm.
Dean Pemberton: Awesome. I think that's great. I think the idea about finishing off by going through and each of the panelists explaining the one small positive take-away point that they had from IPv6 Day, I think that's really good.
Erik Kline: I think the thing that I, Erik Kline, individual, not wearing any other hats than that, learned was it's really amazing what can happen when a few people who are actually passionate about doing something come together and then other people connect to that and it just becomes a multiplicative or exponential effort.
You can go back to the Google over IPv6 Conference in June 2010, the very last session that Randy chaired, Cameron Byrne, Mark Townsley and Ron Bursman all brought up these ideas that became World IPv6 Day and that was where that idea happened and it sort of germinated for a while and eventually folks came together and said, "I think we can do this." It was the power of people who just decided that we should do this because we want to, that multiplied and escalated and it became this worldwide thing that caused so many things to happen that we had no idea could even really materialize.
That's very encouraging. I really hope that we can see the same thing on the access side.
Donn Lee: It shows that if you make a commitment, how powerful that is. You put a date on the calendar, I have a date, my team has a date, my company has a date. All of you have a date that we have to work towards. That was the power that I kind of knew, but when I saw that happen, when I saw the peering sessions being turned up, being requested from us, sometimes the day before the event, but certainly the week before the event, I was able to mobilize people and say, "Hey, you know that backburner project called IPv6, we have a day, we don't want to get embarrassed in front of the whole world, we've got to write code."
So because of that, I saw a tremendous amount of work being done fixing bugs, writing code, peering sessions being turned up, awareness throughout our organization and all of your organizations, the press.
So having a date is powerful. So we can leverage that again.
Jason Fesler: Yesterday and today we heard a lot about fear being the reason for turning it back off. I think that we have passed the one day, we have passed the fear point. We have proven that nothing is going to blow up, as was said, we are still employed, that was a success criteria.
Within Yahoo! we had a huge attitude change within the company after that day. It's no longer, "I don't want to work on that, I don't want to subject my property to that, I'm afraid it's going to blow up," to being much more positive. It's cleared a lot of hurdles for us to keep the movement going.
Gaurab Raj Upadhaya: One of the things for us was that we actually got a lot more traction with customers and partners and peers because of the World IPv6 Day. Given that internally we were ready for a long time, but we could go to customers and even peering sessions that were not being brought up and say, "Hey, you know, there is the World IPv6 Day and we've got customers on our network with v6, we should do this," and there was a deadline before which it had to be done. That was a big driver, such a big driver that for a few days, where the entire architectural team was just busy with v6 sessions that had been on hold for ages and ages.
That was the most positive thing out of it, there is a lot more awareness about it, general awareness, both with our customers, which is the content providers, as well as access networks.
And, you know, big transit providers and I think the biggest thing is there are more people who are taking it more seriously than a bunch of engineers at some companies. I think I can probably say that from all of us, that a lot more people know about it and it's a lot easier for us to go and say, "We are ready. Are you ready?"That's probably the most important thing for us coming out of it.
Dean Pemberton: Awesome. Thank you. With that, I would like to thank all of our panelists and all of the people from the audience who have contributed questions. So thank you to everyone.
Dean Pemberton: We now have a break for lunch and the next session is panel two, which is going to focus on ISPs and vendors and will start at 2:00. So if everyone could come back in the room just before 2:00, that would be great. Thank you very much.
APNIC 32 Conference
Tuesday, 30 August 2011
14:00 (UTC +9)
IPv6 Transition Plenary
Miwa Fujii: Hi, everyone, welcome back to the IPv6 Transition Plenary.
We will start panel discussion two. This session is ISPs and Vendors, Lessons from the World IPv6 Day and The Way Forward. It will be chaired by Philip Smith from APNIC.
Philip Smith: Thank you very much, Miwa. As Miwa was saying, this is the second panel session about the lessons learned from World IPv6 Day and how we will try to move things forward.
This morning we heard from the content providers and the content data networks. After lunch, we have brought a couple of ISPs and a couple of vendors up on to the stage to give you their impressions of what happened from World IPv6 Day and hear about some of their recommendations and suggestions about how we will move forward.
I have asked each of them to do a presentation first, then after we have each of their presentations we will have plenty of time for questions and hopefully some answers as well.
First up, we have Edwin Purwandesi from TELKOM Indonesia, who will talk about his experiences.
Edwin Purwandesi: Thank you, Philip and thank you to all the guys here for giving me a session for sharing our experience during World IPv6 Day.
I think there are many lessons from previous sessions, mainly from the last session, regarding how IPv6 tests in access and also in content as well. So I think our company experience should be a realistic service experience, because we worked on preparing the service for the end users.
We created a bridge from Internet access from the service to the end users. We have a capture from World IPv6 Day and also some time after that, so I can show you some capture to figure out what happened during the World IPv6 Day and the time after that.
TELKOM Indonesia arranged this World IPv6 Day together with Metro Ethernet Exhibition Forum, so we called this session WIDEX, World IPv6 Day, Ethernet Workshop and Exhibition. This is our building in Indonesia where we held this event.
As you can see here, we put the IPv6 test logo to make awareness to people surrounding the building and also the people who came to the event.
On day one we had a presentation and demo regarding IPv6 security devices from our partners, from Juniper Networks, Alcatel Lucent, Cisco, Huawei, IVIO. This session was attended by ISPs, academics and government officials. Since TELKOM, our company, is a company owned by the government, most of our customers come from the government side or academics.
On day two we had World IPv6 Day in Action, by launching an IPv6 proxy, which was open for the public.
If you want to try this proxy, you can use it right now; I will tell you the IP address.
Also we launched an IPv6 tunnel, but for this tunnel it was only open for our customers. Also we put three IPv6-ready websites on the official World IPv6 Day websites. So we had www.telkom.co.id, telkomspeedy.com plus other coms.
This is the capture of our proxy setting during the World IPv6 Day. This idea to launch this IPv6 proxy for our customers is to reach the newbies of Internet customers, because if we ask them to set the tunnel or something that is quite complex, I think they have difficulties.
So we put an IPv6 proxy for this level of customers, so they just set the proxy direct to our proxy, and after that they got masking IPv6, so if they reach an IPv6 website they will acknowledge as IPv6 original.
These are the three websites that we made ready for IPv6 access. We have the three websites, and this is the capture of the test that was done on that day, by 8 June.
Also, as mentioned by some speakers at the last session, maybe the end users do not care about whether they use IPv6 or IPv4, they just want to access the Internet with good speed and good content.
This is one of our strategies, to ask them to try IPv6 proxy and IPv6 tunnel. We held a quiz for them and offered them to win a prize, one iPad 2 and one Samsung Galaxy Tab.
You must know that the most famous question asked by users during the World IPv6 Day is not about the proxy setting or about the tunnel setting, but they only asked about when will the winner be announced by us? That is a good point, because they were aware of this, even though they only were attracted by the iPad and the Samsung. But it's OK, this is a good starting point to make them aware regarding these IPv6 activities.
We also put a fan page on Facebook, so we have this number of users, we have interactive discussion within this Facebook fan page, to answer some questions from the users that used IPv6 proxy and IPv6 tunnel.
This is some data that we gathered from a live session during World IPv6 Day. At point 1, more than 12,000 visits to telkomspeedy.com. This is our most famous website, because this is a website for our broadband customers, so they visit this website for upgrades of firmware, for upgrade of services and things like that.
We did not shut off the IPv6 capabilities on this website after World IPv6 Day, so we put it always on until right now, so we have data until one month after the event, that we have 400,000 users accessed to this website.
7,925 users used proxy IPv6 to access the IPv6 website on the Internet, to access Google, v6, to access Yahoo! v6. They use our proxy so this is the number of customers that used our proxy to access the IPv6 resources on the Internet.
By September we got this number, and it was quite a surprise for us, because we thought that after 8 June they will stop using these facilities, but in fact they still use both the IPv6 proxy and the IPv6 tunnel. 114 users used the tunnel, and more than 1,500 for one month after the event.
91.5 per cent of our customers used Windows operating system when accessing the IPv6 proxy, and others used Mac OS, Linux and others.
This number is still under 10 per cent of TELKOM broadband users. I think this is 4 to 5 per cent of all our broadband customers.
About the quiz, 1,585 submitted the answer to the question, but only maybe one-third answered correctly.
394 answered correctly and with a good answer, so correctly and good quality. This shows the fact that our users actually know or are familiar with the IPv6 term, so this is a good fact. But if we compare to all of our customers, all of our TELKOM broadband customers, I think we must put in some hard effort in the future.
The IPv6 tunnel and the IPv6 proxy that we launched that day are a part of four solutions that were prepared by us for our customers. We have enterprise solution, native-retail solution, tunnel-retail solution and NAT-retail solution.
These solutions might be different with other ISPs, since the condition of each ISP is quite unique, so we have this solution fit for our customers, but maybe other ISPs have different solutions.
This is the enterprise solution that is already done. I mean, if the customer asks our company to prepare an IPv6 solution for enterprise, we have a solution right now. We upgrade our PE router to have 6VPE capability, and we can redirect customer traffic from their location to a dual stack gateway with 6VPE router.
As I said before, if, right now, an enterprise customer needs this solution, we can give it right now.
This is the ideal solution for a big number of our customers, because in this scenario we have to provide end-to-end service for retailer. So this is the next scenario that we have to prepare.
This tunnel-retail solution is done right now, done by 8 June, within the World IPv6 Day event, because this one is used to serve our customers to get tunnelling to IPv6 network. So this configuration is the basic configuration that we had already done by 8 June, during World IPv6 Day event.
NAT-retail solution is the solution for special resources that couldn't be upgraded to IPv6. Some devices, because they are obsolete in terms of firmware or software, they could not be upgraded to be IPv6 ready, so we created a solution for devices like this.
This one is a list of our devices or our subsystem.
That refers to the preparation of IPv6 transition or migration. We have these devices in our network and we must create a roadmap or a plan to migrate or to upgrade these devices to get IPv6 capabilities. So we created a category, as you can see in this presentation, terminal, modem, access, transport control and transport. For the native IPv6 requirement we must ensure that these devices can be upgraded to get IPv6 capability.
There is also the PE core group application software.
This slide shows the portrait about our readiness condition. For some devices we have still zero per cent, some devices we have 50 per cent, and the others we have got 100 per cent already.
This is our global or total roadmap, to become IPv6 ready, a telco IPv6 company ready by 2013.
We describe all the needs that have to be done by 2013, to get the telcos IPv6 ready.
For infrastructure roadmap, we put a list of our systems. You can see in this slide that for some subsystems we right now get ready, regarding IPv6 implementation. For Internet gateway, we have already done. Also for some products, like IP transit and also dedicated Internet, we right now can support or can prepare the service for our customers.
But it's still by request, because we haven't put this service or features as a default service, so if they ask about IPv6 services, we can prepare it for them.
Thank you for this session.
I think we have spread the word about our efforts during World IPv6 Day. If we can deliver a good service for IPv4, I think we must do that as well for IPv6.
Philip Smith: Thank you very much, Edwin.
Next up is Pitoon from CAT Telecom in Thailand.
Pitoon Piluwasandhalai: Good afternoon, everyone. First of all, thanks to Miwa for the opportunity to share our experience about IPv6 in CAT Telecom case study. My name is Pitoon Piluwasandhalai from CAT Telecom. You can follow our Twitter @catipv6.
About CAT Telecom, we were founded on 14 August 2003 and we changed the name from the Communications Authority of Thailand.
This is the main business and customers, for overseas calls from Thailand to other countries and VOIP also. The network infrastructure service is L2/L3 layer. We are a network provider service for the ISP in Thailand. CAT Telecom operates as the Internet gateway to the Internet backbone, and we have a domestic exchange also for ISP.
Another way we provide broadband service to the customer, the broadband we have are wireline and wireless, as we will ensure we are one provider in Thailand for CC operators.
The focus for the CAT Internet gateway and services, we have two services we provide to the ISPs, which we call CAT-THIX. This service is to serve for the Internet gateway peer to upstream provider and peer linked to the ISPs in Thailand.
The other one is called CAT-NIX, which serves as the domestic Internet exchange for the ISPs in Thailand.
We also provide broadband service to customers, to two separate customer groups. The CAT corporate Internet service is reserved for business, and most of them are government customers. Speed-up we start at 10 MB to 1 gigabit and we offer five packages plan to our customers. The differentiation of packages is for international traffic and domestic traffic.
CAT broadband Internet, we service residential and some businesses organized by the copper wire technology and fibre optic technology. This service is based on downstream and upstream Internet use.
This shows CAT's position on the Internet. We are one provider in tier 2 and we have a tier 3 provider connected to our network. We have a CAT ISP to serve the corporate Internet broadband suite and we have another ISP connected also.
CAT Telecom has many links to upstream providers and linked to the big content in the world, such as Google, Yahoo!, Facebook, et cetera, and peer to peer to the Asia Pacific region providers, for example, Malaysia, Singapore and Taiwan.
CAT IPv4 resource usage: we are the last member of the APNIC and we managed to tie the IPv4 assignment for our service, that means we assigned for broadband Internet and static, we assigned for the corporate Internet and some package broadband services, if the customers need fixed IP address.
CAT IPv6 resources: we got one IPv6 prefix from APNIC in 2000. We promote CAT IPv6 Working Group to make awareness of our staff, and how CAT goes to the IPv6 and what is the best practice to deploy IPv6 transition and migration for all services.
What is the impact, how CAT prepare readiness for IPv6, to make an action plan and deploy IPv6, that is the primary function of the CAT IPv6 Working Group.
On the other hand, we have a CAT staff representative to join Thailand IPv6 Forum to promote IPv6 deployment in Thailand. Another function of the forum is a consultant to MICT, which is the Ministry of Information and Communication Technology, about IPv6 policy plan in Thailand.
CAT and PSU, which is the Prince of Songkla University in the southern region in Thailand, is a project to get the architecture so that we can develop the first CPE in Thailand that is IPv6 supported.
CAT IPv6 milestones: of course, between 2000 and 2010 we study and we learn and we create IPv6 testbed network and we develop CPE IPv6 support, to provide IPv6 CPE to our customers.
This year, we have the first dual stack Internet facility to serve our customers, for example, DNS web with no less portal size and so on. We do not wait for transition, we will deploy IPv6 native links to customers in the first quarter of this year. Maybe we will have a trial of IPv6 multicast for the customer for local content streaming.
This is a roadmap for the future. It is a four-year plan, to force top level management to make decisions and promote the CAT IPv6 deployment direction.
As I said, we do not wait for transition. In this year we deployed IPv6 Internet services to the customers starting with CAT corporate, and provide dual stack and native link IPv6 also and now have started CAT broadband wireline and wireless service. We think we will be deploying at the beginning of 2013.
Right now we are finding the best solutions, best practice for our customers, and at the same time IPv6 applications should be deployed for test drive by the customers.
This is the transition technologies and CAT Internet service categories. CAT Telecom prefers three-phase transition to IPv6 for Internet broadband service. The first phase we deploy the LNS, large-scale NAT solution.
This solution is helping postpone the IPv6 transition and helps solve the problem for IPv4 exhaustion. We select 6rd for deployment in the second phase. We will start at the beginning of 2013.
In the case of the Internet corporate and Internet transit and VPN also, we begin in 2011 and done for deploying dual stack.
What we have done about the IPv6 solution: of course, we have done the dual stack and we have promoted the dual stack Internet server facility to our customers this year. For example, in the S-mail service and CAT IPv6 portal website, we have many links provided for peer link to upstream providers, for example, Google, Hurricane Electric, and we have more than 4,000 IPv6 prefix routes.
In addition, we have an IPv6 peer to the domestic exchange and the Thai academic network also.
In the interim, we do IPv6 support for our customers and free-to-air content to trial the IPv6 service. We are implementing in this year a tunnel broker solution.
In the focus about the CAT IPv6 portal, the first thing is how we can create knowledge about IPv6 to the customer and how we start IPv6 deployment to the customer. This is the main website to answer IPv6 questions from the customers. I think maybe it is making a new community for the new potential customers.
Of course, the website is certified for the IPv6 Global Forum and for getting the IPv6 Ready logo.
This is the diagram of the dual stack Internet. Our server facility, we start with a virtualized server and provide native IPv6 links to the new customer, which is expected to be complete at the end of 2011.
On World IPv6 Day, we were proud to be present with MICT and to present the event at the Amari Watergate Hotel in Bangkok, Thailand, and to invite the government to join the event. The objective of the event, we sent a message to the CEO about IPv6 awareness and promoted IPv6 policy in the country and share IPv6 transition from vendor/providers also.
We are a sponsor of the dual stack link to the event and we announced the CPE IPv6 support to the customers.
This is a picture of that event, the CAT IPv6 booth, and the grand opening by MICT.
Of course, at that event we applied for the IPv6 Ready logo from the IPv6 Global Forum for the CAT website and applied the Google Analytic program for the measurement of clients access to our website. Google will send the summary report to us by email.
This shows that at the CAT IPv6 booth at the event we provided dual stack Internet service, and the right side shows the Internet CPE access for IPv4.
This slide shows the on-screen free-to-air prototype for IPv6 applications, and the next slide shows how to access the CPE IPv6.
This slide shows an iPhone connected to the CPE on the IPv6 page, and browse to the CAT IPv6 website in the live picture also.
This shows that 6to4 automatic traffic is captured a lot for that. This shows the traffic on the Internet, the orange bar shows the 6to4 traffic.
The CAT CPE IPv6 support, the project between CAT and PSU is a joint project and provides support to the CAT customers if they need IPv6 access. This has been certified by the CPE Global Forum, and you can check it on that URL. It is the first CPE in Thailand for IPv6 support.
This is the basic concept of the 6to4 community: should we have a 6to4 relay in the network service, or maybe you can connect to the default gateway IPv6 address at 126.96.36.199. This is a global address of 306S, to allow multiple providers for the 6to4 gateway.
This shows how to manage the CPE. You can access the CPE with a web browser and we have a dynamic DNS feature and can monitor CPE. You can see SSID CATIPv6-WiFi.
If you connect, you can test with the IPv6 website, you can see the IPv6 address, that is a prefix from the 6to4 automatic tunnel. If you access the IPv6 test, you can see the IPv6 6to4 prefix and some streaming prototypes.
Maybe you can access the website catscreen.hgw6.cattelecom.net and click on it and maybe see some channels. This is a channel from free-to-air, from the Internet. This is a Hong Kong channel, I think.
Some applications about machine-to-machine on IPv6, maybe you can use Apple, because this is the protocol for access, which I installed in my office. You can use AFP protocol to install it. I installed it in my office in Bangkok, and you can connect. You can install it, connect it behind the CPE.
That's all from me.
Philip Smith: Thank you very much, Pitoon.
Philip Smith: Next up we have Hans Liu from D-Link, who will talk about D-Link's experiences and lessons from World IPv6 Day.
Hans Liu: Hi, I'm Hans Liu from D-Link.
Today I would like to share our experience on World IPv6 Day as well as our CPE device deployment and development.
This chart might look familiar. After a decade, we have been told about IPv6 transition and where we are.
I think we are still in stage 1, trying to move on to stage 2. We all know that IPv6 will not take over one night and we are all aware that different ISPs may have a different strategy in offering IPv6. So we support all protocols, stage 1, stage 2 and stage 3. Therefore, a lack of CPE devices probably might not be a good excuse any more to prohibit or hesitate you to the transition.
Five years after IPv6 conversion, World IPv6 Day was held, and in D-Link we participated in this day as well.
However, it was a little different from the content providers. We do not think the users are going to visit our website to test whether their IPv6 connectivity is OK or not. Instead, in addition to providing IPv6-ready gear for World IPv6 Day, we think we can contribute in a different way.
In addition to our devices, we think it is very important to have our support service ready for IPv6-related questions, especially on the day, though according to the data I have, we did not really have many support calls from customers regarding IPv6 questions on that day.
However, we spent some time training our customer service and support centre to enable them for IPv6 questions. I think this is what we can do as a hardware device vendor.
Let's take a look at our development. At D-Link we have been aware of IPv6 for a very long time. We joined the IPv6 logo program at the very beginning, since 2003, and in 2008 we shipped our first IPv6-ready router to the retail market.
Last year, we participated twice in Cablelabs IOT events to ensure that our device is good for cable ISP and cable networks.
This year we also had our DHCP server, the first DHCP server embedded inside your box to get IPv6-ready certification. Right now, more than 23 SOHO consumer AP access points and routers are certified with IPv6-ready logo phase 2.
This May, we also participated in an IOT event hosted by UNH-IOL. This was a test event mainly focused on IPv6 CPE router devices, based on RC6402, I think.
Luckily, we passed it. I think we are the first hardware device vendor who passed this test particularly for IPv6 home gateway.
What we have been doing: as I mentioned, we support all protocols for three stage, we support 6rd and DHCP options, we support DHCPv6, DHCPv6-PD, PPPOE, IPv6 for native dual stack; and for those ISPs who already want to go directly to phase 3, we support DS-Lite and DHCPv6 options already.
In addition to those, we also support some IPv6 management protocols as well as some security enhancements.
We support IPv6 firewall, which was a firewall that requires users to manually configure. However, recently we also started to support simple security, which was introduced in RC6092.
We also support features like ULA, IPv6 DDNS and IPv6 UPnP for Windows 7. I think Windows 7 already supports IPv6 UPnP, although they do link local only by ULA.
We also support LLMNR, which is link local multicast name resolution, for Windows and Bonjour multicast DNS for Macs.
Moreover, I think there were questions this morning in the session that we do not have to take sides between the IPv4 NAT and IPv6. Actually, we run them both by using a box like this. This year at D-Link we are going to announce our 2 gigabit connectivity, which allows in several of our high-end routers we will enable the user to have a 2 gigabit wire speed IPv4 NAT plus IPv6 routing performance, kind of equipment in the retail market.
Then deployment: at D-Link, as we promised and as we committed to the IPv6 community, we will keep going promoting IPv6 local program and we are going to keep promoting IPv6. We have put the IPv6 logo on to the gift box. So if you are looking for an IPv6 router, you just go to the retail market and see if you can find the IPv6-ready logo on the gift box. With every D-Link device with the IPv6 logo on the gift box, it means the device is IPv6-ready logo certified and I think the device can do the firmware upgrade to the latest firmware to enable all three stage kind of connectivity support.
Also, it is important for those ISPs who like to enable their service directly to users, we also have these kinds of programs and we promote our IPv6 implementation and our technology to ISPs. Right now, we are co-working with a few ISPs on their IPv6 deployment, mainly in Canada, US and Germany.
This slide, I would like to go through our IPv6 supported products. Like you can see here, you can have the very entry level kind of CPE device, DIR-615, 600, the entry level device, through the middle end to the high end one like the DIR-825 device. Basically, all of our devices support IPv6.
This year, we have some new devices, like DIR-645 and the coming DIR-827. The 827 will be the router that will provide 2 gigabit wire speed, NAT and IPv6 routing performance.
Also, for APs, from the consumer APs, from even the pocket APs, like the DAP-1350, a small device like this for you to travel around the world, it supports IPv6 already, and we also have IPv6 APs for small business as well.
PowerLine: we still add more and more devices and more and more developments from the PowerLine market.
We have the world's first PowerLine wi-fi integrated device with IPv6 support.
Recently, a lot of people are asking, "OK, we know there's a CPE, there's a router from D-Link, what else do you have?" Although the NAS devices are not under my product line, I'm not in charge of it, but I think later, in Q3 and early Q4, we will have several NAS devices and several IP camera devices that will support IPv6.
So here comes my presentation. Are there any questions? Or we can do it later in panel discussion.
Philip Smith: We will hold the questions during the panel discussion.
Hans Liu: Thank you.
Philip Smith: Thank you very much.
Finally, we have Alastair Johnson from Alcatel Lucent.
Alastair Johnson: Thanks, Philip, for the introduction.
I am from Alcatel Lucent.
For those who don't know what we do, we are a full service vendor and span more or less all of the layers that a telco wants to work with their vendors, from optical systems, SONET/SDH, ATM, submarine cable systems, through access products, the wireless domains, the wireline domains, D-SLAMS, Node Bs, GPON and so on.
We have our IP routing platforms, which are IP/MPLS platforms, and we have a professional services team, applications team and so on on -- so pretty much everywhere.
I am going to talk a little bit about what we saw on World IPv6 Day and some of the things that happened inside Alcatel Lucent around IPv6, because it is kind of the imminent steamroller heading for all of us.
From a timeline perspective for World IPv6 Day, we started talking about it internally in 2007. Before 2007, v6 was a nice to have, some of our customers were asking for it, but very few were actually doing much with it. It lived in engineer land, in test labs, in segments of the networks that were looked after by the engineers directly and not necessarily servicing customers.
Coming along in 2007, I was working in Australia at that stage and we had a major customer start talking to us about what to do with exhaustion. They saw the writing on the wall. We started seeing that across Asia at about the same time.
Internally, we started looking at IPv6.alcatel-lucent.com as how can we get IT to do this.
You will see that, from the enterprise perspective, it's been an interesting exercise for us getting that one going.
In 2008 we started working on our carrier grade NAT products, not because we think they are the greatest idea and we want to make a lot of money from them, but because customers have said they need them, they want them, they have a necessary evil they must solve. At the same time we ported our Biras platform to support full dual stack subscriber management, so that is the ESMv6 acronym there, and that was to meet broadband networks rolling out in dual stack.
We did a lot of research with some of the universities and some of our customers around what traffic was like on their networks, both in terms of tunnel v6 traffic and what the impacts of carrier grade NAT would be.
In 2009, the Broadband Forum really started working in anger on TR177 and 187, which are the forums for the standards published by Broadband Forum for IPv6 and broadband networks. A couple of my colleagues got quite busy with IETF around the CPE standards and around solving some of the gaps that came out of the Broadband Forum documents. At about the same time, our European and North American customer base really started talking to us actively as well.
Moving along to 2010, universal recognition was needed. Everyone that I worked with, as I mentioned earlier, has pretty serious plans for v6. We launched quite a few products, we made sure we went through and closed any feature gaps we had.
We had quite a few customer starting trials in 2010 and we started seeing our first RFPs coming to us with IPv6 as a mandatory requirement. If you couldn't meet IPv6, the response was basically thrown out. That was an interesting one, to see that finally as a mandatory, unavoidable requirement, with "you must demonstrate".
In 2011, where we are now, World IPv6 Day. We made ipv6.alu.com a reality, we saw real deployments in customers. We have customers with real carrier grade NAT, with real dual stack subscriber management, with real dual stack VPN services out there.
We realised we had focused an awful lot on the residential customer base and core networks and nobody had talked a lot about the enterprise and business customer needs. That is coming up this year. We are seeing a lot of business oriented ISPs going, oh dear, what do we do? The mobility world carried on, but it has opened up with LTE, that v6 is there, we should do it, we are building new things, let's put it in properly. They have had a long and illustrious experience with carrier grade NAT.
On World IPv6 Day we had a lot of customers approaching us wanting to do joint marketing exercises around World IPv6 Day, which was interesting. We had never seen that happen before.
In terms of what we did in the enterprise to get our alcatel-lucent.com web page up and running with dual stack, we started in 2007 and made some progress in January 2011 when World IPv6 Day came along. In the intervening three years when nothing happened, I think that's kind of been typical of many people's experiences, particularly in the enterprise space.
Our corporate IT group, which is obviously very decoupled from the realities of the Internet that we get to deal with on a day-to-day basis, simply did not see this as a priority. There was no budget, there was no demand, there was no content, there was no access -- all of the usual arguments we have heard.
In February, we got internal agreement to participate in World IPv6 Day. That wasn't very hard.
IT actually said, "We will deliver it."
At 5.00 pm on 2 June, IT said, "We can't do it." We went, "Ah." Actually, those weren't quite the exact words. So, on 3 June, a Friday, plan B commenced. What we did was we got it running on the 8th and it was a success, or some measure of success, but it was done.
How did we do it? This was our initial architecture for alcatel-lucent.com. We have some CDNs, routers, firewalls, load balancers, switchers -- pretty typical web service for an enterprise -- sitting in one of our datacentres in Chicago. Unfortunately, a lot of this is legacy. As the name implies, my company is two very large vendors that got smashed together in 2006, so there is a lot of legacy floating around, particularly this datacentre.
We thought it should not be too hard. We know the border routers are capable of IPv6, we know the firewalls are and we know the load balancers are. So what we should do is dual stack the DMZ infrastructure, including the DNS, we'll put a v6 VIP on the load balancer towards the existing v4 real service and we'll publish some AAAA records. Sounds pretty simple.
So, what happened really? Well, Naperville is a legacy datacentre, as I mentioned, and it has a lot of older stuff in it. We knew the DMZ was mostly OK, but there were a couple of challenges.
The next challenge was allegedly no v6 transit.
That was pretty quickly checked with our transit providers there, which are the AT&Ts, Sprints and Verizons of the world, and it was not going to be a problem, but it just wasn't implemented.
CDN is used for delivery of the main website and a lot of our support sites, and this became the biggest stumbling block from IT's perspective. They said they could not turn off the CDN and the CDN was not able to be deployed with v6.
We had some business and political challenges.
Enterprise IT is pretty decoupled from the business units, and how dare we, as a business unit, tell them how to run their networks? The support teams were unwilling to risk our primary website, being a major branding thing -- the same reasons that Google and Facebook have reservations. Of course, they also didn't like it because we might breach SLAs with customers that pay us a lot of money, and if they can't download a patch like that, there's a problem.
Of course, support. IT said, "We don't understand this v6 thing. We sit on so much legacy v4 space that we've never bothered to learn it because we'll never need it."
What did we do? We went into panic mode. A couple of us in the business unit and in the CTO strategy team really wanted to see this happen, so we put together a minimal implementation. It was an off-net solution from us. We didn't get dual stack DNS and I had to give Patrick Gilmore a call to help us out. The CDN guys came through and helped us get our AAAA records published pointing towards one of our own servers.
The architecture started looking a bit like this: we put a reverse proxy in a friendly ISP in Canada and published a AAAA that pointed to it and it reverse proxied back to our main servers. It was not very elegant and definitely not a complete solution, but it was enough to get us up and serving dual stack content.
We have had this sitting there for a while. It was on ipv6.alcatel-lucent.com but it wasn't published for Alcatel-Lucent or www.alcatel-lucent.com. So this got us through -- real simple -- the patch reverse proxy.
Initially, because it was ipv6.alcatel-lucent.com, we had some rewriting going in there to change all the literal URLs. It worked pretty well, testing and everything went OK, so we published the AAAAs.
A small problem we hit, which was just the side effect of doing it too quickly. We were proxying, as you might have noticed in the Apache config, back to www.alcatel-lucent.com. When the AAAA went live, the proxy proxied to itself.
That was about five minutes when we went, oops, that's not good. So we fixed it up to point to the correct origin servers, and business as usual, went ahead, no one noticed, it just happened. There was a bit of discussion on our internal mailing lists and social networking about what we were doing and why.
Observations: we don't really have interesting numbers like Google and Facebook, but we got 26,000 hits after we took out the probing and testing traffic to the dual stack site. We saw 333 unique v6 hosts and saw 9 hits via 6to4. The New Zealand hit was me testing from a machine in New Zealand, and one hit via Teredo.
Everything else was native, as far as we can tell.
Like others have commented, France was the largest economy to visit us, partially because we are a French company with French employees but also because of free.fr. They were the sheer majority of the traffic in France. USA was second, Canada was fourth; not too surprising, we have a large R&D facility in Canada as well. We saw a lot of unique hosts that made only a few requests. As far as we can tell, those were probably researchers just checking to make sure we were there and were answering queries. They didn't look or match the profile of a regular browser.
We saw what appeared to be Verizon wireless LTE clients connecting to us on IPv6. I have never quite figured out if that was someone testing from Verizon wireless or if that was just traffic that happened to come over the LTE and happened to be dual stack.
Unfortunately, we had to turn it off. That was because to get the AAAA record published we had to sign a waiver with our CDN provider to go on to a limited CDN platform. So that meant we were waiving our SLA contract with them, and unfortunately the business would not accept that for a long term. There is full commitment internally to continue this forever, there is no concern about that, as long as we get it all lined up with our vendors.
By the end of 2011 we hope to have it done properly.
We are building a new datacentre to retire Naperville.
One of the first assumptions in the datacentre has been it's going to be native v6.
We mandated, after some arm-twisting with our procurement teams, to include IPv6 as a mandatory requirement in all of our procurements to our vendors, so that's for our transit and our IP VPNs that we use for our WAN.
Internally we play around. We use some 6over4 stuff to tunnel traffic between our labs and various buildings that we have around the place. We know that we have a lot of things to clean up. Unfortunately, it is a very large company, very distributed and lots of legacy, so it takes a long time to tidy up.
General observations: customer interest went ballistic after World IPv6 Day was announced. My colleagues and I were constantly on planes, talking to customers, v6, v6, v6, what do we do, how do we do it, when can we do it, what do we need to change in the networks, where are the problems; everything that's been talked about ad nauseam in these forums.
During the two months prior, we had so much interest from operators it was unbelievable, I couldn't keep up, nor could my colleagues. They wanted to know what we were doing and what they should be doing, which was a little worrying, given the prevalence of these forums.
From TAC perspective, we saw no increase in tickets, no one had any issues, no one logged any faults. Our managed services, NOCs, didn't really see anything either. Admittedly there wasn't a huge amount of visibility but I went and checked the ticket databases and didn't see much.
Overall traffic volumes didn't really change in any meaningful manner. We didn't have anything splitting out the v4 and v6 traffic stats prior to this, so we could not see with much visibility. It sort of came and went. People were worried about it, they invested a lot of time to make sure it went smoothly, and it did, and that's really good.
General observations across a couple of domains.
I work mostly on the core side, one of my colleagues works mostly on the broadband side. These are the five areas we work with, with customers. Residential is the most focused upon domain. People say, we have to get it to the hundreds of millions or billions of end users in their houses.
The standards have evolved pretty well. Broadband Forum has had their documents published for some time, the IETF has published and is now revising many CP router specs, CP security specs. It is the most problematic domain as well because of the variations of devices in the home, including the OSs and the CPE.
There are some difficult ideas still perpetuating.
We all see Owen's posts on NANOG about the prefix length. That's still one that comes up: we are only going to assign one v6 address to our customers. Why would you do that? On it goes. No matter how many times we try to say, "Do it properly," people will still try to break it.
We have some live trials happening. I don't know if anyone has moved stuff into full production but we have a lot of customers with real live trials.
The enterprise area has had less immediate focus until this year. I have been picking that up with a lot of our service provider customers asking, "How do we get our gig customers or customers with 10 VPN sites, how do we get them v6 ready, especially when we run out of v4 to give them?" For customers that don't have a residential v4 service pool to steal resources from, using NAT, they are really struggling with what they do.
It's not as easy as you'd think. I don't even want to describe how painful getting v6 internally on our corporate networks has been, dealing with enterprise IT.
It's challenging for ISPs who have to provide services in areas that are exhausted.
I spoke to a customer recently who said, "I can't use ROC1918, I can't get any more v4 assets and I need to build a new MPLS core and I need something to number my links and loops; what do I do?" There's no easy answer to that. You can't yet say, "Use v6 there," but I think that's the next thing we should force the IETF to work with.
Mobility domain: it's cool because it's a high turnover for CPE. People change their cell phones every year or every two years -- in my case I seem to drop them in the toilet or something like that. New CPE come out pretty quick with v6 capability. We can thank some of the guys in the room here for helping that happen.
Apple, Google and Nokia have driven that, so it's there.
The majority of the traffic is HTTP and only to a few destinations that are starting to support v6, so they are saying, "That's great, we can buy less carrier grade NAT if we dual stack our mobile networks."
The backbone has been in place for a number of years. Scaling is now the main challenge. As Gaurab mentioned earlier, vendors like me sell you a 10-port 10 gig card and we have to make sure it is going to do 100 gigs of v6 through it.
There are occasional transmission requirements in there -- 6PE, 6VP, et cetera -- but generally the backbone is well understood and well under control and has been for probably five years.CPE is the most universally challenging domain, with wireline CPE not cycling quickly. My mother still uses a 10-year-old ADSL modem from when I was working for a telco in New Zealand. It's a problem. You can't convince people to change devices if they have to pay for them, and a lot of telcos have done the business case and said it is going to cost $100 million to swap the CPE so they don't want to.
There's increasing availability. What we saw from D-Link is fantastic, with so many devices supporting v6.
Some of the other CPE vendors I've worked with are really improving their support as well. We see a lot of customers doing trials with the FritzBox and with Linksys and quite often with DD-WRT. But it's getting there.
The other challenge is there are too many implementation variations. As a community we haven't done ourselves much favour by having 6rd, 6to4, 6in4, dual stack, PPPOE, IPOE, slack numbered and DHCPv6 and DHCP-PD numbered. Getting the CPE vendors and BNG vendors, like us, to test all the interop scenarios is a channel. That's interesting and the service providers I work with are still struggling in those domains too: How do we design it and get a CPE that supports what we want?
That's pretty much all from me. I'm happy to talk to anyone, or take questions.
Lorenzo Colitti (Google): I heard that 6rd exists because of certain DSLAMS not supporting v6. So it was necessary. I don't know who makes those DSLAMS.
Alastair Johnson: We make DSLAMS and they support v6.
There seems to be a perpetuating myth that the DSLAM must support v6. I have never encountered a DSLAM out there which will only glean a v4 customer and only forward v4 packets. Very few of them look at the ether type or anything like that.
What the DSL issue has been classically is the inter-one VLAN architecture and the DSLAM needing to insert something like option 82 into the DHCP packet as it traverses, and no one did that for v6. So that is also there in our DSLAMS.
Lorenzo Colitti (Google): Is it supported now?
Alastair Johnson: Yes. I can ship you a DSLAM or a GPON OLT with it. It's a good point, and 6rd was developed to work around those challenges in the layer 2 access domain. There's very large DocSys deployments that have a similar challenge with pre-DocSys 2 compliance where they don't really support v6, and that's another area where we see 6rd being used.
If anyone wants to discuss anything about our products, feel free to come and chat with me, I'm here all week.
Philip Smith: Thank you, Alastair.
We will move into the panel discussion.
Philip Smith: The first question is from Jordi.
Jordi Palet (Consulintel): In your presentation, you mentioned that for introducing IPv6 in your one links, you said 6over4, right?
Alastair Johnson: 6in4.
Jordi Palet (Consulintel): Okay. Because that's very confusing for people. We really need to make an effort, especially in public presentations and documents, to avoid confusing people who are still learning about IPv6. It's even more confusing because you have in the slides 6over4, you said 6over4, but the people doing the transcription write 6to4, so it's a complete mess.
I know for the transcribers it is really difficult, but we need to take care with that.
Alastair Johnson: I agree. Internally, the 6over4, 6in4 and 6to4 and various other transition debate causes no end of pain, particularly for our product units that are not IP SMEs. The wireless team gets thoroughly confused.
Jordi Palet (Consulintel): Just in case some of the people here were not in the training on Sunday, 6over4 means using IPv4 multicast to build IPv4 tunnels and typically you don't have IPv4 multicast support.
Alastair Johnson: You are right, and that is indeed an error. It is 6in4, just protocol 41 static tunnelling.
Philip Smith: Any questions for the panelists? All four of them have given interesting presentations so I would like to throw open the floor for questions for any one of them or all of them.
Lorenzo Colitti (Google): A question for CPE manufacturers, particularly. What I use at home is a D-Link, and I love it because it works and I thank you very much for having all the v6 support in all your product lines, it's like living in the future. You just plug it in. The question after that is I had to go in and click buttons and turn it on.
You specifically and CPE manufacturers in general, we can't get there until it's on by default, because nobody changes the settings on their CPE. How do we get to a place where we can get CPEs doing IPv6 out of the box, like they do for v4? What would it take to make you feel comfortable with this and what would it take for other manufacturers to be comfortable with this?
Hans Liu: I think, yes, that is quite an annoying question. I recently have been approached by several IPv6 communities to ask me to put IPv6 by default enabled. However, from our point of view, somehow, like most ISPs have IPv6-ready, even if they have IPv6-ready, it's more like a trial service.
We had a lot of internal discussion about this, whether we should put IPv6 by default enabled or not.
However, my management team still have concerns that if there is a trial service, if your ISP actually provides an IPv6 service. However, the pie might be small, the user experience might not be good; what about if we enable IPv6 by default and the user has no idea they have been redirected to the IPv6 network, and they suddenly might complain and call our device support centre, saying, "Originally, I had another box from another vendor that works pretty well, and right now I have a new D-Link box and it goes like everything was broken." Things like that.
I know Lorenzo is sort of pushing on this, and honestly it's not under my control.
Lorenzo Colitti (Google): It sounds like what you didn't say but what somebody else said is "Our competitors aren't doing it so we are not doing it either."
If that's the case, I think what some people said -- another time when people said that kind of thing without actually saying it, was World IPv6 Day.
Hans Liu: Yes, of course.
Lorenzo Colitti (Google): "Our competitors aren't turning on IPv6, so we're not turning it on either."
By creating World IPv6 Day, we basically removed that concern, because everyone was going to turn it on on the same day.
I guess the question is: have you talked to your competitors and have you asked them if this is a possibility?
Hans Liu: Yes, I did.
Masato Yamanishi (SoftBank): I am sorry to say this, but whenever I hear a discussion between a vendor and Lorenzo, I hear the same comment: that many vendors say that IPv6 is already supported on their gear, and basically they are happy to support it for vendors' new equipment, but many carriers of ISP already have existing equipment. In my company's case, the oldest disk ram was installed in 2003, so it doesn't have enough CPE power or enough memory to support IPv6.
CPEs are also saying, "I'm afraid we cannot live with this situation," but just saying, "IPv6 is already supported." I cannot believe it's not so simple problem. Sorry, this is my comment.
Philip Smith: Would the ISPs on the panel like to comment about their experiences?
Edwin Purwandesi: In my company, we invite many vendors to do collaborative works in our lab, so we call our lab an oasis, out in Bandung, which is a city near Jakarta. So we open to all our partners and device vendors to come to the lab and try the device within a compatible system. We put all of the devices there and we prove together whether the devices work or not, regarding to IPv6 capabilities. That's what we have done in our company.
Alastair Johnson: I will take that from the vendor perspective a little bit. It is true we have challenges supporting v6 on older hardware. In my team, we try to back-port as much as possible on to older platforms. We usually end up sacrificing scalability, simply because of CAM or RAM or whatever is the particular limit where we are heading is, but we can usually support it.
In the case of the DSLAM side, on our DSLAMS and certainly some of the other vendor DSLAMS I have worked with, it depends on the deployment model. If you can live with transparent cross-connects in the DSLAM, the DSLAM doesn't care. It doesn't look at the packets, it just forwards them in a DSL LT and out on an ethernet NT. So that can be used to work around, but it depends on your network architecture.
Putting on the brutal vendor hat, you are talking about a product that is 8 years old. We know 8-year-old products, Windows XP, et cetera, have challenges supporting v6 as well. At some point you will simply have to upgrade.
Philip Smith: Do we have any other comments or questions from the audience for our panelists?
I would like the panelists to give us a one-minute summary of their observations and what they would like to see happen next.
Edwin Purwandesi: I think we still need work for IPv6 to get to start the transition period, especially for introducing IPv6 access services for end users. So maybe the previous one is IPv6 transition for us as a provider, but we need to start transition for our customers without confusing our customers themselves.
So I think we must create a simple transition strategy to be presented to them. Thank you.
Pitoon Piluwasandhalai: For the issue of IPv6 for CAT Telecom, dual stack interconnect work should be done for the dual stack interconnect work, and some broadband services we plan to deploy in the immediate two years, I think. We started some service, and that was CAT corporate, in the first project to deploy IPv6 in Thailand.
Hans Liu: As a device vendor, I think what we can do is keep going, complete our features there, as well as product lines, so we will have our full digital home enabled with IPv6 in this Q4. And we will also enable the other devices, such as our set-up box IP cams, NIRs and the other consumer digital home kind of products IPv6-ready after our support to our CPE router devices.
Alastair Johnson: On the network vendor side, stop making it optional, make it mandatory. We still have too many customers that seem unwilling to make v6 a mandatory requirement.
And just continue deploying. Take an opportunity to revisit your network architecture and see whether what you are doing makes sense or were you driven in that direction because IPv4 painted you into that corner?
Maybe you need to change your architecture while you are refreshing products or elements in your network.
Continue to push us as vendors to make it work as well as possible as well.
Philip Smith: I think that brings us to the end of this session. I would like to thank the four panelists for their closing comments and thank them for their presentations earlier in the afternoon as well. I would like to thank you all for your questions and for participating in this second panel session.
The next session begins at 4 o'clock, which is 27 minutes from now. We have the v6 lightning talks at 4 o'clock and then we have three more presentations and a panel discussion after that.
Thank you all very much.
APNIC 32 Conference
Tuesday, 30 August 2011
16:00 (UTC +9)
IPv6 Transition Plenary
Matsuzaki Yoshindou: We have three speakers here. The first is Shin Shirohata-san from Tokyo 6to4 project and he will talk about operational experience at 6to4/Teredo relay on World IPv6 Day.
You have seven minutes and I will ring the bell at 6 minutes past once and 7 minutes past, then, ring ring.
Shin Shirohata: Good afternoon, everyone. I would like to say about our operational experience with 6to4/Teredo relay on World IPv6 Day.
First, I would like to explain about Tokyo 6to4 project. Tokyo 6to4 project is a volunteer based project that operate 6to4 and Teredo relays in Tokyo.
The mission of this project is help to build IPv6 enabled Internet by providing 6to4 and Teredo relay until commercial IPv6 Internet service is widely available in Japan.
Before Tokyo 6to4 project, there is no 6to4 rerouter in Japan, so Japanese 6to4 and Teredo relay users use 6to4 Ontario relay in United States or Europe. So it caused poor experience of 6to4 and Teredo.
After we launch it, traffic could be handled by rerouter in Tokyo. Here is an example of RTT before and after. For example, this is RTT to www.kame.net, a native IPv6 site in Tokyo. Before we deploy 6to4 relay in Tokyo, RTT to this site is about 200 milliseconds. After deployed, RTT is now less than 10 milliseconds.
This is network diagram of current deployment. We have two sites in Tokyo, NTT Automatch and KDD Automatch and we connected to three Internet exchanges across Tokyo and we have 73 IPv4 peering session and 43 IPv6 peering session. So it covers almost half of Japanese ISPs.
Also, we are using PC router for 6to4 and Teredo relays and we use PC router and Cisco router for PCP peering.
Also, I would like to share about the traffic trend on World IPv6 Day.
This graph shows long time trend of our 6to4 relay.
You could see traffic decrease in last November.
I think this decrease is caused by implementation changes on Mac OS X. Before this implementation changes, 6to4 is preferred the native IPv4 connection.
After this change, after this upgrade, native IPv4 before is preferred than 6to4 connection, so amount of 6to4 traffic will decrease since last November.
Here is traffic trend across World IPv6 Day.
Traffic table is almost same as World IPv6 Day, about 20 to 30 megabits per second.
So we didn't have a special impression of World IPv6 Day and we don't see any issue on 6to4 and Teredo in our network.
Here is the number of unique source IP addresses observed at our relay. Left line shows number of clients in May, middle graph line shows clients in June and right line shows clients in July. On World IPv6 Day 20 per cent increase since May, but no big difference.
Here is the ratio of traffic. Left graph shows ratio of TCP traffic and right graph shows UDP and total traffic.
You could find ratio of PTP rated traffic is quite huge. For example, bittorrent is about 36 per cent and JXTA PCP protocol is about 11 per cent and HTTP traffic is about only 0.8 per cent, under 1 per cent.
This is one month before World IPv6 Day. On World IPv6 Day, ratio of PTP rated traffic will decrease, Bittorrent traffic only 5 per cent and JXTA is about 8 per cent. Compared to PTP traffic, HTTP traffic and SSL traffic will increase.
Ratio of HTTP traffic will increase from 0.8 per cent to 1.4 per cent and SSL is increased from 0.2 per cent to 0.8 per cent. Ratio of empty packets, that means TCP packets which don't have any payload, is increased.
Here is TCP failure rate. Before World IPv6 Day, TCP failure rate is about 4 per cent. But on World IPv6 Day, that failure rate is increased to about 48 per cent. This is a very big increase.
To conclude, I would like to conclude this presentation, on World IPv6 Day, we didn't see a major issue on 6to4 and Teredo operation itself and we observed 20 per cent more unique IP addresses since last month.
Some users use 6to4 to obtain IPv6 connectivity and it caused increased TCP failure rate on World IPv6 Day.
We encourage users to use latest version of software that supports policy table, which would increase unnecessary 6to4 traffic.
That's all. Thank you.
Matsuzaki Yoshindou: Next speaker is Lorenzo.
Lorenzo Colitti (Google) (Google): Hello. I won't introduce myself. Let's go.
So a few thoughts that we have been having. We are all in the room, come talk to us if you have opinions, ideas, suggestions, about how we can follow up on the success of World IPv6 Day.
Work in progress, absolutely zero commitments have been made. This is currently worthless, but we're working on it.
What went well on v6 Day? It galvanized the industry, caused a lot of progress. It brought it to the attention of management which we didn't have before.
It made us feel as an industry that IPv6 can actually work. We turned it on, our networks didn't break, we are still employed, the Internet was functioning as normal. Nothing happened, which is good.
And one of the key properties other than this was the success was measurable, participation was measurable. If you signed up for World IPv6 Day as a website, the ISOC measured you and you had a red or green light in the web page and people could verify that you participated. This made sure that progress was real and not just paper and press releases.
One of the problems, first of all, we turned it off after the day; and, second, we had very little participation in access networks. IPv6 adoption in the Internet is still about 0.3 per cent.
What are the problems we are trying to solve at this point? At this point, the technical issues are mostly understood. They're not solved, but most of the operators have an idea of how to solve them, in most types of networks, you have 6rd, you have cable. The problem is still an organizational one. Everyone is dependent on everyone else. I have crossed out there is no content, that was one of the main things that was a problem, but I think we have kind of moved past that.
There is no access or at least no big access. There is no CPEs, at least no CPEs that do a IPv6 by default.
There is no deadline, no common coordination, no timelines and there is insufficient mind share in product, in marketing. IPv6, we don't know what that is. IPv6 Day helped a lot here.
How are we thinking of fixing these problems? We're trying to organise another event in the first half of next year, maybe April, maybe June, depends a lot on the participants.
We are thinking of having three categories, websites, they get to commit to enable v6 on their website and leave it on. They leave it on for good.
Access ISPs or mobile providers will roll out IPv6 to X per cent of their users. And CPE manufacturers of devices will enable IPv6 by default in their devices.
This will be measured independently, so content will be measured -- again these are all ideas -- by ISOC or RIPE or whoever wants to. Access providers can be measured by a collection of large websites if they agree to that or perhaps by third parties, if they have a representative user base. I know that Geoff has ideas on that, involving ads, I hear. And CPE manufacturers could, for example, undertake to pass a CPE test in the default config and in competition, each category would be encouraged and it would be open to everyone.
We want to start with a core group of initial players who are willing to publicly announce their goals and willing to be measured against their goals publicly.
That was the key to World IPv6 Day, that it was publicly measured.
Then nail down the details. These are flexible, because it depends what the participants can actually commit to. On the content side, this is mostly easy.
We did it once, we know that we can do it again.
I don't think there will be any problems if a large number of us just decide to turn on AAAAs as part of this event. That can just happen and we can leave them on.
It kind of depends on what the other side of the equation want to do, the access networks and the CPE manufacturers. Remember the CPE manufacturers are a critical part of the problem because a lot of ISPs say, "We have no CPEs," and a lot of the CPE manufacturers are saying, "There are no ISPs that we can run our stuff on, so we're blocked." So there is a chicken and egg problem between the ISPs and CPE manufacturers as well.
Or at least there is a perceived problem which we need to address.
The idea would be to publish a press release with the initial core group, say in late 2011, to kick it off and give the Internet six months of notice that this is happening.
We have talked to a bunch of people. The content side is mostly well represented because we were the ones who already did the initial World IPv6 Day. We have talked to a few access providers, we are looking for more. If you're an access provider and you have a v6 roll-out, come and talk to us.
We have talked to a couple of CPE manufacturers that consumers buy. In general, people have said to us, yeah, we could probably do something in that timeframe, but, you know, we have to do X or Y and, again, we are willing to entertain different suggestions, because the event hasn't been nailed down yet. If you want to participate and you need some changes, then let us know.
So please come and talk to us if you think you could help, if you want to participate, let us know.
Matsuzaki Yoshindou: You have one minute 30, so you can take questions.
Lorenzo Colitti (Google): I will take one question.
No questions. Please come and talk to us and participate.
Matsuzaki Yoshindou: The final speaker is George Michaelson.
George Michaelson (APNIC): I am here to talk about the realities of IPv6 deployment. It's all about fashion. What do we really want to know? Well, I want to know if I could be happy about this stuff. I really like being in my happy place, because it doesn't look good if I have to go somewhere else.
So what do ISPs really want to know? The big question is: are the customers ready?
So how about the content provider side? The big question is: are the clients ready? Because it's about the money.
OK, so how about this third arm, the one we often don't talk about. What do the public policy people want to know?
Well, they want to know if they have to play nasty or if they have to play nice. Actually, they really want to know how they are doing against everyone else.
Because if they are doing well, they want to make something of it. OK, so if those are the questions, what do we know?
Well, we know about the kind of ways that v6 clients behave. There are two kinds. There are the ones that say, "I'm out there. I do it." They're really great.
Then there are the ones that say, "I might do it, but I would rather not." They're a bit of a problem.
So we have also got a bunch of data. We have been doing this measurement exercise using Java script and using a flash technique, and the really nice thing about this is that it samples for any given client a whole set of behaviours that is about that client. It's about everything we know about that client within the limits of the test.
It can find things out about dual stack. It can find out about tunnels. It can find out about if you are absolutely brought to the brook what can you do. We think it scales and we think we can actually make trend observations out of it.
So it comes to this core question: is the Internet uniform or lumpy? What's the distribution like? Can we see differences going down into the economies? Which really goes to the question: are the promotion campaigns working?
To give you an idea of the background, it's this scale of tests that are being applied, so the numbers are large. We are trying to limit the impact of small economies, because relativities with small sample sets aren't good.
So what did we see? So we get a measure of the relative economy behaviour. And it looks good, doesn't it?
OK, so what we are actually saying is the latent -- this is the graph of the latent ability to do IPv6, not "I am doing it" but "I will do it if you make me." If you look at that graph, you'll see that it said 25 per cent, because we get all these figures about 0.03 per cent. Oh my God, we have doubled, we have hit 0.06.
The latent capacity out there is already high, it's 25 per cent. 25 per cent of the customers, the people that pay your wage, are already able to do v6, if you bring it to the door they will give you money.
25? You said it was 0.4. Well, that is also true.
Because the de-preferencing behaviour means people won't do this.
So then we have this thing about tunnelling, which is if you do this, it looks really bad. But if we stopped doing all the ad hoc, if we did the upgrade, if we spent the capital investment, they will use it.
OK, so what does it look like? So are we going to do some comparisons? Here is Russia. You will notice the line in the middle of the graph there, that's World IPv6 Day, so there was an appreciable spike of behaviour in terms of capability around then, but what is more interesting is there is a high variability of above 20 per cent v6 latent capability coming out of here.
Sure, this is skewed by where the numbers come from. We know that we are really measuring the deployment of Windows platforms, but it is interesting, 40 per cent of people in this economy are prepared to say they could do v6.
America, not such a pretty story. You don't want to read too much into the lines going up or down, but the bottom line here is that it is a slightly suppressed market at the moment. It's harder to pull v6 latent ability out of end points in the USA. Maybe it's Canada's fault.
Australia is doing pretty OK. New Zealand is doing pretty OK. Bangladesh is absolutely on the nail at that median point, it's doing really, really well. Maybe that's about the age of deployment in that economy.
Fiji is doing pretty damn well. So this idea that it's about the developed economies doesn't have to be true, maybe there are really interesting v6 stories in the emerging Internet, in the regional Internet, in the local Internet.
This is something we can talk about. South Korea is not too bad. So where are we going with this? What's the real story?
This is where I would like to be. I would like to be in my happy place. But if I'm going to get there, we need to do a bit more work. The idea is that we want to go on with this measurement. We want to try and get better figures, more accurate figures. Though it's kind of a bit of a game, the reality is we think we have the basis of a measurement technique that can actually inform the space and get a dialogue going with regulators, with planners, in standards bodies, with ISPs and talk about the economies effect, what happens at national scale, what's the capability out there that would justify the capital investment, because it doesn't matter if we are talking about tunnelling or software solutions or CPE upgrade, it's about the money.
Matsuzaki Yoshindou: Thank you.
Miwa Fujii: Thank you so much for managing the lightning talk. Did you have actually a chance to ring a bell?
All right. So the next session is the last panel discussion, the World IPv6 Day data analysis, what we observed and way forward. So all three panelists and also the moderator, Tomoya Yoshida, Multi-Feed Japan.
Tomoya Yoshida: Thank you very much, Miwa. This time we have panel discussion about World IPv6 Day data analysis point of view, what we observed and the way forward.
So where we already had several questions this morning, we were focusing on the content providers and including the idea of the World IPv6 Day.
I remember at first, the name of the World IPv6 Day was AAAA Day, something like, and it was very successfully than World IPv6 Day on 8 July.
In the previous session, one is from the ISP's point of view, almost successfully but one important input from ... thinking about old equipment of the ISPs is very important issues.
So anyway, so this time, we are focusing on the data analysis, what we have seen between the World IPv6 Day, or before and after, and what we should do to the next step.
So we have three speakers and each presentation will be 15 or 20 minutes, please. We will get individual specific questions each by each, but after that, we are going to the panel discussion.
One thing I would like to ask the speakers, please speak more slowly, because most of us are non-native English speakers, so please speak more slowly.
OK. So let's start please, Geoff.
Geoff Huston: Thank you. Good afternoon. My name is Geoff Huston. I'm with APNIC.
This is all about measuring IPv6 Day. So what I got in my inbox was precisely this: this is what I meant to talk about -- all these statistics and measurements and analysis and technology and performance and what worked and what didn't work and what happened on World IPv6 Day. Yep.
So I look at the data and I go: this is difficult.
Because I'm trying to see what I saw on June 8 that I didn't see on June 7 or June 9. What was so different from where I was looking about World IPv6 Day? What changed? Nothing. Absolutely nothing. It was the same as the day before and the day after.
Now, I could sit down right now, if you want, because that was the measurement. But let me explain why I saw nothing extraordinary. Because I am trying to measure you people, all of you out there and all of your 1.7 or 1.9 billion fellow Internet users. On the whole, all of you did nothing that you didn't do the day before or the day after. So I didn't see anything different because, as users, you did nothing different.
So what were you doing that day anyway? Bearing in mind it was much the same as you were doing the day before, much the same as you're doing today, even.
Because client events and what clients do is not a one-day story, it's all about trends and very long term.
So, to fill out my time, because I can, I'll give you this story about longer term behaviours and trends.
I have seen a lot of other work on this too and these are very interesting questions because the first kind of question comes from the content provider and the question from the content provider is: should I go dual stack? So these are the kinds of questions that are in your head, when you go: will I enable that AAAA record?
If I do, how many customers will flip over to use it?
Will they be tunnelling and using access from hell?
Will their experience be crap or will it be bright and wonderful? If I turn it on, am I going to piss my customers off? That's the real question out there.
So we thought maybe we could help this. In fact, we thought how about if we take the pain for you? If you allow us to measure your client's IPv6 behaviour without you in preparation for this day doing anything at all.
There is this cute little thing called cross-site URL fetches. So we also observe that Google have this really quite neat package called Analytics and we thought aha, if we muck around a little bit we can feed back into Google Analytics IPv6 capability for your clients and your customers. We'll do the test, you get the answers. No new tools needed.
Then we thought about ads. Because it's one thing to deface your own website and do all these tests, but we weren't getting enough and we thought what's the world polluted with? It's polluted with ads.
Now, the beauty about ads is that you can put flash in it and the beauty about flash is you can do all kinds of really cute things in flash, including a whole bunch of scripting.
So we had this ad; really quite cute, isn't it?
Don't you like all those colours? The marketing department was just right there. We built this, the capability of the test, into the ad.If you ever see this ad on your screen, for God's sake, don't click it. It cost me a whole lot of money that I have to pay to Google. Do not click the ad, right? We thought if we said on the ad, "Don't click here," you would, so no. This is more a really boring ad. The thing is that the ad works by simply displaying on your screen. It's the impression. And impressions are really cheap. $25 a day gets us at least 25,000 impressions, and if you do the ad words and so on right, from all over the world.
Every impression, as long as you are patient, if you see that ad, wait until the end, OK, it's a really fascinating ad, wait until the end. As long as you wait until the end, you will have been comprehensively tested.
We do notice that some of you were very bad people.
Because you're ad intolerant and you blow away the ad as soon as it comes up. Naughty. About a third of you do that. So we see a fair deal of aborted tests in this.
But for the rest, it's actually really very, very cool data.
So you have seen this a few times today, and I don't know the numbers behind what Google are doing, but what they are seeing are long-term trends around the 0.2 to 0.3 per cent growth in terms of end user capability that prefer v6 in dual stack.
If you mimic that particular experiment, because we do a whole lot more, we actually see a very comparable line. But we are doing a whole lot more than this. We are not just doing a dual stack test, we are actually doing a test that involves IPv6 only. So one of the URLs only has AAAAs, so that as well as the IPv6 capability, we can look at a few other things as well.
That graph also has some more information because we know the v6 source address.
Some source addresses start with 2002. You're using 6to4. Some start with 2001:0. You, you poor bastards are using Teredo and life isn't worth living.
Fortunately, on the whole, the Teredo folk are bouncing around 0.00 per cent in this test. Really lucky. And even, fortunately, because 6to4 is horrible, that is also in the low fractions of a per cent. The folk out there in operating system vendor land have finally got it right. If you're going to do v6 in preference when you're doing dual stack, use native mode v6 because auto-tunnelling sucks.
The general trend, it's kind of up and to the right, but convince me that from 0.2 per cent to 0.4 per cent is a dramatic increase over a year.
These are small numbers. I'm sorry. We need big numbers.
So it's kind of decent data and it aligns with what I'm seeing in other sites, but it is still small.
I don't see tunnelling and that's a good thing. But I can look a bit harder. So the next test is it's all part of the same thing, I give you a URL that only works in v6. So even if you prefer v4 in dual stack, if you have any kind of working v6, I'll see it.
It's around 3.5 per cent. Interestingly, it's not getting any bigger. That's almost a year and it's flat.
There is something else down there in the bottom right.
All of a sudden, since the middle of July, there are a number of folk running Unicast v6 who preferred to use v4 in dual stack. Something changed. The change was a combination of two things. It was the release of Mac OSX Lion and also I believe Chrome beta started doing it, the thing that I have heard referred to as happy eyeballs. Now, when given the choice between v4 and v6, your browser will actually do the DNS and send the first send, but it's a race. It will do both protocols in parallel and whichever one gets the SYN-ACK back first wins, the other one just gracefully stops. That means there are a small number of folk and you can see them in the bottom right, where v4 was faster.
Interestingly, all the rest of these folk, since July, there's a fair deal for which v6 was quicker. So the good news is, I suppose, inside of that, that, yes, some folk have reverted to v4, but an awful lot of folk have stayed on v6 and that v6 is actually working really very well.
So that's the first kind of sort of grand picture.
The folk who could 10 times more than the folk who will.
But where's Teredo? Because since Vista, IPv6 on Microsoft systems has been turned on by default. That's what the marketing department of Microsoft said and they were telling the truth. It's been on.
But weirdly, they have also turned it off at the same time. Because the way Vista and Windows 7 work is that if your only v6 interface is Teredo, it will never consult the DNS for a AAAA. Aha, we thought, we can't measure that. And then we thought, oh, yes, we can.
Because in browser land, you can put the address in the URL, no DNS. So we did.
Here are the results. When you coerce the customer, unwillingly, to use v6 by simply avoiding the DNS, it's not 10 per cent, it's not 20 per cent, it's massive, up to 34 per cent to 35 per cent of the client base will actually successfully fetch that URL. That's the breakdown. The blue line is that they actually use the Teredo protocol, the blue line further down is they use 6to4 and the green lines at the bottom were the Unicast folk that we always knew about, that 0.3 per cent. So there's a whole bunch of folk who are coercible. If you put it all together, there's an awful lot of folk who actually are running v6, most of them are running the stack in a covert mode. It's Teredo. But the stack behind it is active, activated and running.
Down there below are the ones running 6to4 and IPv6.
The total picture, it's sort of almost by factors of 10, 0.3 to 0.4 per cent -- if I'm being optimistic; let's be optimistic, everyone else is -- is running v6 and willing to use it in preference in dual stack. Around 4 per cent are going to tunnel using basic 6to4 and around 28 per cent are running Windows machines with Teredo turned on by default.
But that's not all. Because we also do a whole bunch of looking at this via TCP dump. We capture every single packet. We can then do some analysis of what failure looks like. The first kind of measurement was actually measurements about: if I turned on dual stack, how many folk wouldn't get anything at all? That's a really hard question and devising measurements to test it are also difficult, because it's hard to tell folk who are impatient and just walk away from the ad versus folk who have sat there and sat there and nothing is going to happen. So we changed our methodology a few times in looking at this and I'm getting a failure rate of around .1 per cent, but I don't believe it. I still think I'm getting a few folk who are walking away. So I'm getting failure rates of around 15 in 10,000. It needs an awful lot of folk to test it to get 15 folk reliably failing, but I'm still not convinced. I think the true rate is more like what I heard from Igor at Yahoo! that it's around 3 per 10,000 -- .03 per cent failure rate. To my mind, that figure seems better.
But I'm measuring the packets. I'm looking at the packets. I can do more than this. Because I at the server am seeing the incoming SYN, the incoming ACK and so on. So what I'm really interested in is naked SYNs.
Aren't we all?
What I'm interested in are SYNs that come in and I send the SYN ACK, because I like SYNs, but I don't get the next ACK. Those are the folk that I'm really fascinated in because those are the folk who can do v6, but something between me the server and them is stopping it. So they're trying and failing reliably.
So this is a really fascinating graph of the percentage of naked SYN failures in 4 and 6. There is a certain amount who fail in 4. I think they're the folk who were bored or whatever. They walked away quickly. I just never managed to complete. So there's a very small fraction that fail in 4. But, wow, in 6, 7 to 8 per cent of all connection attempts don't. They hang. They die in some horrible death of time out.
Could you set up a service where the reliability or the unreliability was 7 to 8 per cent? I know when I'm using WiFi services where the packet drop gets to be astonishingly bad, 5 per cent, life is hell. 7 per cent of all connections fail currently in 6.
What are they? Well, this breaks it up by protocol type. Most of the failures occur when folk use 6to4 tunnelling. So that red line at the top varying between 10 per cent to 16 per cent, is actually 6to4 failing.
Further down there, the blue line is actually Unicast v6. That's still 2 per cent. I have no idea how and why that 2 per cent are failing. They're delivering me naked SYNs, I'm sending them the SYN ACK, I love SYNs, I'm seeing nothing more. 2 per cent.
Now v4 was fractions of a per cent, so that figure is worrisome. The green one down the bottom is Teredo.
More on Teredo.
That's not true. Teredo isn't that good. No, no, no. Teredo is way more complicated and way worse.
Because for Teredo to set itself up, initially, it has to send an ICMP packet through the NAT, through its relay, through a server, through to me and I have to get an ICMP back. So it's not just the SYNs any more, I should also be looking at ICMP packets. So let's look at Teredo again with a new light.
That's the failure rate. The big green line. The one that goes between 40 and 50 per cent. Teredo sucks.
No one, no one on this planet could set up a reliable service using Teredo like this.
So what's going on? Between 2 and 5 per cent of v6 Unicast fail. That's not good enough. Unfortunately, I can't tell you why, but they do. I have no idea what's going on with that Unicast v6 failure. It doesn't seem to be covert 6to4, it doesn't seem to be old Apple port extremes, it doesn't seem to be any of the things that we suspected. They just seem to be anonymous v6 Unicast source addresses where it's just the SYN.
12 to 15 per cent of 6to4 connections fail.
I suspect it's because most folk are running some kind of firewall filter that says protocol 41 incoming is evil. Because auto-tunnelling is auto-tunnelling, a whole lot of folk don't even know they have 6to4. So they kind of go to this v6 test and out goes a 6to4 packet through protocol 41, the firewall says, well, that was good. In comes the response on protocol 41 and the firewall says, you're evil. Because it's auto, the customer, the end site never even knows that they're filtering out legitimate traffic. That's OK. That's their fault.
45 per cent of Teredo connections fail. This is a conventional UDP in IPv4 NAT traversal. The customer is doing nothing wrong. The customer is just fine.
Teredo uses a NAT busting protocol called STUN. That protocol seems to have a failure rate of 45 per cent.
Now, this morning, you and I both heard this thing called carrier grade NATs. Yes? I heard it. You heard it?
Now, let me understand this. How does an application negotiate its way through a NAT? STUN?
Yes. What's the failure rate of STUN? Are you worried yet?
Has anyone else truly measured what actually happens through NATs? Because some folk, some very, very large folk with millions of customers, are putting forward plans or indeed are doing it right now, to put these boxes in the middle between you and where you want to go. And the measurements we are seeing coming from this experiment tend to indicate that if you push just a little bit -- and Teredo doesn't push very hard -- the failure rate of our standard NAT traversal protocols is stunning.
So what can we say about the performance and robustness of dual stack content? You know, quite frankly, as we have heard a fair bit, it basically works. You can put up AAAAs and because Microsoft only use Teredo, only turn on STUN when there's no DNS, because of course all your URLs use the DNS, you will be OK. And the latest Chrome and OS line make it even better. Because there's no long delay and time out if there are problems. Because those ones really are quick. They select the fastest straight off. They're good. A very, very small fraction of your clients may experience some trouble because they're running old stuff and some of them, really small, may fail, but it is amazingly small. So that will work.
But there's a deeper message here about combining your technologies in dual stack. And the message is actually, if you think that your clients can just tunnel around you -- and I'll go a fair way out and say even 6rd is not immune -- tunnelling has its problems and auto-tunnelling has an amazing amount of problems. It's not a useful mainstream transition tool. If we want v6 to be out there, the access providers and CPE must -- must -- run native mode v6. There is no other option.
Tomoya Yoshida: Any questions and comments?
So the next speaker is Emile Aben from RIPE NCC. He will talk about RIPE Lab's experiences on World IPv6 Day.
Emile Aben: While we are fixing this, I can already introduce myself, I'm Emile Aben from RIPE NCC and what I'm going to talk about is the measurements we did specifically for World IPv6 Day and it's not only me doing these measurements, it was the whole team, our R&D team at RIPE NCC doing this.
So what did we learn from this? We did a whole bunch of measurements that I'm not going to go into.
IPv6 eye chart. You can read it all on RIPE Labs. What I want to talk about today is the active measurements that we did. We had a measurement network, this was all using components that we already had from various places. For the vantage points, both RIPE TTM boxes which is RIPE NCC service and we had collaboration with the CAIDA research organization, they also have a measurement infrastructure called Ark, and from these sources we measured 53 participants in World IPv6 Day or sites that were already dual stack, because we wanted to have a baseline.
So we measured a week before World IPv6 Day, until a couple of days after and what we measured is DNS, so A and/or AAAA records, ping (6), trace route, HTTP, both over v4 and v6 and this is what our measurement network looked like.
Of course, as the RIPE NCC, we have a bias to the RIPE region, so there's a few more boxes in the RIPE region than elsewhere, but we had some on all continents except Antarctica. Maybe the next meeting or the next v6 Day or Week.
We are a little undermeasuring at the Asia Pacific region, so if people have capacity for measurement boxes for an eventual next v6 Week -- let's call it that, right -- please come talk to me.
One of the first lessons we saw from this is control your on/off switches. So your on/off switch for something like World IPv6 Day is DNS, right? You just announce a AAAA record and your v6 requests start coming in. It is a little more complicated, in that DNS is really heavily caching, so what the clients do is they go to their caching resolvers and these things cache all kinds of things, they cache answers, but they also cache the lack of answers. So this caching of lack of answers, you can configure it in your authoritative zone and we thought, nice, let's measure this right before World IPv6 Day.
And this graph shows what we actually saw there, which surprised us a little, because we thought, OK, maybe people do negative caching up to an hour, but we actually saw people having their settings so the negative caching was one day or even measured two days or a couple even more. So what this means is that for these sites, if they were going to announce their AAAAs, there was a good chance that the people going to these websites were not going to see it until the day was over.
So these are actually measurements of the AAAA records that that we saw during IPv6 Day, which you can see in the middle. The green here is where all of our vantage points saw a specific site announcing their AAAAs. The light blue is where we saw only A records, so they didn't announce AAAA records and all the various shades in between are 10 or 20 or 80 or whatever percentage, so you can see on World IPv6 Day, this nice line going from blue to green and in the end going to blue again for some, not all.
But you can also see the negative caching here.
These three you can see, it's a longer period, so they were sort of eased in. It took a little longer for all the clients to see this. Of course, you have the opposite, your normal TTL on DNS zones and some people just had that at a really high value, so you can actually see when they stopped announcing, the clients still saw AAAA records while they already stopped it in their authoritative zones.
I think the lesson we can take from this is that you know your on/off switch. If you look back at this graph, all the big parties, they have these really nice cuts from blue to green, so they know what they're doing and in case of a rollback, and if you look really carefully, you can see one here, you can quickly roll back and go forward again, so if something is wrong, you don't have to deal with these clients that are still using caches that have faulty information at that point.
So next thing, test and monitor. So this is something that you don't want to happen. On World IPv6 Day, we saw this. On IPv4, this website www for a large tier 1, worked perfectly fine. On IPv6, we got an HTTP404, which is a layer 7 problem, right, because you get this back on layer 7, so this has nothing to do with IPv6 as such.
You can also see the picture next to it, which is the IPv6.large tier 1, worked perfectly fine, so this is just somebody screwing up their Apache config, didn't put the right thing in the server alias or the equivalent. But their v6 worked.
This is another one that we saw. At the end of World IPv6 Day, people turned down their service. So at about 3 am, we see HTTP responses going to zero, ICMP going to zero, you can see that here, but for hours, they still announced AAAA records and so what happens in this case, if you are a dual stack client, you get an A and a AAAA, you think, hey, they're on v6, I'm going to go to their v6 service, which is turned off. You get into this classic 20 second time-out, horrible user experience. So this is stuff that you don't want to happen.
I don't want to pick on these two, because we're all humans, and nobody is perfect.
So test when you deploy something. The more real life your tests are, the less likely you'll do bad things.
Second thing, monitor your infrastructure. I think these have been in various talks earlier today already.
The more real life your monitoring, the more things you catch. The other one that I think is interesting is actually people to people reachability, so it's not machine reachability, but there's avoidable situations like this. These were things that we monitored, we saw, which came up at various forums, people tried to contact the people responsible for this, but the situations were fixed, sometimes hours after, which is really bad. It's bad publicity for you.
One thing, of course, is have your contact information up to date in the RIR databases, so the classic Whois -- some people don't have that, unfortunately.
The other thing is, you can monitor the web, there is a NANOG, there are all kinds of NOG mailing lists, there is Twitter, there are all kinds of social media or whatever that you can monitor to find this type of stuff out.
I think that's an important one, be reachable, not only technically, but also on a personal level.
And the other thing is that what we wanted to have is more of a global view, because what I talked about before is just individual cases of things we saw go wrong, of which there are very few, luckily.
So if you look at the performance here, this is a nice -- this is the average over 10 minutes of the relative performance, so you take your round trip time on v4 and your round trip time on v6, the average of a period of 10 minutes and that's one data point. So this is a nice histogram of that, to the left IPv4 performs better, to the right IPv6 performs better.
The nice thing here is this is centred around zero, so this is equal performance. So this was the most common case that we found. They perform the same.
On the left-hand side, you see that it's a little fatter. There are a few more cases where v4 performed better than v6. It's not very much, but it is visible.
Instead of looking at this v4 side, this left side, which in the future is at best going to stay the same, but likely is going to go worse with all the NATs and whatever stuff people want to put in there, there is also a right-hand side and the fact that it is already there means that with dual stack, you have two chances to get the best performance from between two end points.
So this is something you can exploit in things like realtime apps. For instance, if you want to build a competitor to Skype, which does IPv6 for a change, that is a big thing missing, or for instance in gaming, where ping times is everything. If your ping over v6 is faster than your ping over just v4, you're doing a great service to your gamers as a games developer.
Another thing we notice is partial reachability. As you know, the Internet is a collection of interconnecting networks and these interconnections can be different on v6 and v4. What we noticed is that for some vantage points not all destinations were reachable.
One question we asked ourselves is: are our vantage points representative? Of course they're not exactly at the edge, these are typically boxes that people put in data centres and stuff like that, so we are not exactly at the edge. It's 50 points and there is 1.2 billion Internet users, so that's one thing, but we encountered actual network partitioning and that was supported by a nice Wikipedia article, and you see the link here.
What you have to keep in mind here is that this also happens in v4. This is not new. But this is something that people have to be aware of, I think.
The other thing -- and we also saw similar graphs like this before -- is the long-term effect on content.
So we got raw data from Dan Wing -- who is not on the Google white list, as far as I know, and as far as Dan knows -- so this is a lower bound, actually. So on 8 June, in his data, we saw 3.8 per cent of the Alexa 1 million websites, so what's what everybody uses to measure content, so 3.8 per cent on the day itself.
But the long lasting effect was that if you compare before and after the day, you actually see 0.4 to 0.7, which is, OK, it's a 75 per cent jump and you don't have to use a magnifying glass any more to see that, so that's good.
The other interesting thing we noticed is that around 3 July, we saw a similar jump and when I looked into that data, it looked like this was caused by a single hosting provider in Germany, so one lesson from this is that single parties can actually make a huge difference here.
The other lesson I would hope to get from this is that this was actually caused by World IPv6 Day.
I tried to contact this party, but their sales people wouldn't get some free publicity out of this or something, I don't know, but they refused to tell me or they just didn't know. That's the other possibility.
If you do a linear extrapolation from this, you need an IPv6 year to get to 100 per cent. That's kind of depressing. So if you do exponentially, which is much nicer, it's an IPv6 Week, which is what's been discussed, so let's hope it's exponential, but of course this is extrapolation based on two data points, so this is not science, this is making numbers up, unfortunately.
Another long-term effect that we looked at is the percentage of ASs announcing an IPv6 prefix. The upper graph here is from beginning of April until two days ago. What percentage of ASs is announcing an IPv6 prefix? You see before World IPv6 Day, this is steeper than after.
The first observation is this is going up and this continues to go up, which is good, but before World IPv6 Day, it's steeper. So I can think of two things to explain this. It is people pushing their deployments earlier so people had plans and then there came this nice v6 Day, so they could go to their managers and say, "Hey, we want to deploy now," and they actually did it.
The other thing is that this is also the summer vacation period, so it could just be caused by the summer vacation and be picking up in two or three weeks.
The lower graph is actually the daily increase and what you can see nicely in this graph, first of all, is that there is this periodicity, there are five days where you see large growth and then two days no growth.
People are not deploying IPv6 in the weekend, which is good.
Again, you see this big difference before and after.
There are a couple of outliers there that I haven't really figured out what these are. If somebody knows what event happened here, which is August 9, then I would be happy to know.
So for all the measurements we have a web interface for these measurements, v6day.ripe.net. We did a bunch of analysis far more detailed than I could go into here, which is available from RIPE Labs and we have the raw data for all of our measurements available. If anybody is interested in looking at the raw data or seeing how they performed or want to do data analysis on this, it is available.
So conclusions, what we learned. It was boring. It works just fine. It's not really interesting to measure. But make sure that it's properly tested, just like v4 and properly monitored, just like v4. The dual stack gives you two chances for best performance and I think the non-technical one is that days like this just work. They raise awareness, give people a target to work towards and I think we are ready for a next v6 Day, week, month, year, eternity. With "we" I mean both RIPE NCC as an organization that wants to measure this and if people have ideas on what to measure for a next one, then I would really like to hear this. But I also think the Internet at large, the community, everybody is ready for this, for a next event.
So that's it from me.
Tomoya Yoshida: Thank you. Any questions? Specific questions?
Erik Kline (Google): You asked for ideas for things to measure and does your measurement infrastructure permit you to measure percentage of v6 native deployment by origin ASN?
Emile Aben: That would be more -- it would typically be something similar to what Geoff has. We actually also have a similar infrastructure and we could do that, and you can do it as well, right?
Erik Kline (Google): I can, yes, but I'm not necessarily a neutral third party.
Emile Aben: That's true.
Geoff Huston: We can do it.
Erik Kline (Google): Right, so there's two of you, so that's brilliant.
Emile Aben: So there's two neutral parties offering, I think.
Sala Tamanikaiwaimaro: I would like to ask a question to both Geoff and the current speaker.
Tomoya Yoshida: Please, after all speakers finish. Is that OK?
Sala Tamanikaiwaimaro: It's not question and answer time?
Tomoya Yoshida: Later we will have a panel discussion.
Any other individual questions to Emile?
OK. Thank you.
Tomoya Yoshida: OK, the last speaker is Martin Levy from Hurricane Electric. He will talk about some IPv6 deployment topic, including the IPv6 World Day.
Martin Levy: I am fully aware I am the last speaker of the day. That's an important thing to do. But I also have to do one other thing as well, which is I always take pictures of speakers, I never really take pictures of what it looks like from here. You are now immortalized, including somebody waving in the back. You will be tagged on Facebook.
Hi. It's a whole day of v6 and I'm the last guy, Martin Levy from Hurricane Electric with a bunch of statistics. Pretty much you may have seen all this, because everybody has given all the same data. Now, following two scientists and the like, the good news is my graphs are fuzzier.
Two basic things. Talk about the global deployment report which is again our view of what the world looks like at the BGP level and some other levels and then just some of the things that we saw on World IPv6 Day.
But to be honest -- and I know that this is being key because no one has done it today, no one has actually covered: why do we need IPv6?
I'm kidding. Because there's no way we should be talking about that at this point in time. But I just want to make sure you were awake.
All right. Let's look at a graph you just saw, but I have annotated it in a slightly different way, but we are seeing really the sort of the same thing. You go back and look at three-plus years worth of v6 at the BGP level, we saw an inflection point, you know, around the time just before IANA run out and we saw, though it's not as easy to see at this scale, a slight levelling off after World IPv6 Day.
So from a statistics point of view, I can tell you that as a company, we were very busy prior to World IPv6 Day. It's not like we're not busy now, but from a measure, a simple measure of v6 interest, that was a pretty impressive jump between IANA running out and World IPv6 Day, and all of the press associated with that, it got a lot of people motivated.
So let me just take you aside here, because we have talked about the pluses and minuses of v6 all day.
I just want to thank all of you.
Martin Levy: I haven't told you why yet.
We are a commercial company, we are not an RIR.
Nothing against -- we are a commercial company. We sell services and as those people that know, we have been pitching and talking about the fact that we have been IPv6 ready for a long time.
The customers have listened. As a company, v6 makes us money. So I thank everybody who has done anything to promote v6, because we have benefited.
Now, can I talk about this from my competitor's point of view? Maybe. But I'm not really responsible for that. But the bottom line is being ready for v6, it has a positive ROI, so this graph, our sales guys like this.
Next graph, you have seen this one as well. I told you, this is all data you have already seen: Alexa 1 million list, absolute interesting peak. We looked at it again from a BGP point of view. We weren't surprised at this. The fact that people even for most of these hosts didn't leave them on surprised us, but again we have been plotting this for ages, it was unbelievably a boring graph. And then all of a sudden, on World IPv6 Day, up and then down. But so be it.
SANOG is surprising there. As I said, you've already seen this graph today. Let's look at what we can show you that's a little bit unique. Global IP backbone runs through Asia, through the US, through Europe, moves a lot of traffic, and moves a lot of IPv6 traffic. Ratio wise, uninteresting -- and you have already heard somebody else say this today -- it's a very small percentage. But it's there.
So this is what we saw on World IPv6 Day.
Literally, instantaneously, as expected, a jump in bandwidth. Basically, we ended up World IPv6 Day at about 5X, the amount of bandwidth that we had been running v6 prior to that.
If you look at the graph, it's nicely curved, you can tell that it obviously has a user base. Although the world, you know, has 24 time zones, the reality is that there was still some focus on certain time zones.
But what's really interesting about this obviously it goes away, with some latent traffic I'll talk about in a moment, but there's a couple of key points.
This is measured -- by the way, for those who want to know how we measure this, it's all flow statistics.
It's v6 only and it's v6 TCP to port 80 and 443 only.
So no DNS traffic, no nothing.
There was a nice talk that Geoff gave prior at NANOG in Denver. You gave a DNS talk and showed some reverse DNS, I think, was -- was that George? Twins. Can't see them.
OK. Sorry. Back on subject. There's a couple of interesting points from our point of view. Notice that zero UTC was not quite the best time to do this. It's the right time to coordinate people, UTC is what it's about. Zero UTC, everybody can understand that. But the reality is that it came on with gusto and you look at that end of the graph, it was going strong and you didn't want to see it turn off. Why did people turn it -- I know why people turned it off, but the point is that it just was going with gusto. But it didn't go away.
Look at the longer term view and what you basically see on our IP backbone is about a 3X gain in v6 traffic from before to after. Now, different people have got different numbers here, because if you look at content plays, it's about specifically their content. If you look at eyeball networks, it's specifically about their eyeballs. I'm talking about something at the backbone that's servicing both of those, an intermediary, if you want to look at where we stand.
But bottom line is, this is a good thing. The other thing about this is that it's all as far as we can tell, human driven. It has wonderful cyclical cycles. If you look at it longer term, you can see those weekends, and you basically can see that this is users using v6.
Again, this is pure TCP port 80, 443, again, we are just looking at that. OK, not everything is rosy. We are the guy that sits there and goes, "Rah, rah, rah, 6to4, rah, rah, Teredo."
Let's make sure we talk about this in the right context, because I just got sliced and diced by Geoff on this one, so I would be very careful to go against Geoff on the quality of this thing.
The reality is this. We run a large number of 6to4 relays that have Teredo relays built in. We have been running them now for years. We have modified code, we have modified servers, we have shipped more servers out.
We shipped a bunch of servers out prior to World IPv6 Day and the traffic just keeps growing.
On World IPv6 Day, again, starting to sort of re-edit the statistics and look at purely the traffic that's going through it that's actual TCP, and actual port 80 and 443, there is the jump. There is the continued traffic. So those relays just got another bunch of traffic that showed up. It isn't going away.
Now, I'm going to ignore the subject matter of whether it should go away. I'm simply saying it's there. We don't create it. We are just relaying this stuff and we are relaying it around the world and, yes, we saw it everywhere.
Interestingly enough from the Teredo point of view, it biases closer to where Microsoft runs their Teredo servers -- not relays, these are relays, not servers, and I did not split out -- the Teredo traffic is actually pretty small. That's the good news.
But that's not a real view of how much Teredo traffic is out there and it's unrelated to World IPv6 Day but the reality is that if you look at non-port 80, port 443 traffic, you see a lot more Teredo traffic and you can attribute that to file sharing, bittorrent, et cetera, which is not a subject for today.
The latent part of this, very clever. As you look at smart code, in fact, the type of code that I'm sure was written inside Geoff's banner ads, you can actually realize that if you're a content player or, more importantly, if you are writing an application that owns the client and the server side, that you could do something if you saw a 6to4 address and say, yeah, maybe that's not a good use of v6. OK.
Let's look at some non-traffic statistics that we saw. There was a lot of press about World IPv6 Day and in fact there was a lot of press for four, five or six months and then there was a lot press during that day. Look what happened to our tunnel broker accounts.
Tunnel brokers are specifically for people that don't have access to v6 but want to get connectivity somewhere in the world.
This is a massive jump. We are seeing about 3 or 400 people a day that sign up for this puppy. And, by the way, some of them run it for a day and then go away, that's the natural. We have worked out now how to sort of clean out some of these accounts. That happens with everything, people play with something, they go away.
But this was the biggest amount that we have ever had on any day. To hit just shy of 1600, not bad.
Remember, we are measuring this -- I'm sorry, don't remember, that's the wrong phrase. Let me inform you, these are services based in California, so therefore these days are based on California time, so if you were to look at zero UTC, that was about 5 pm, that's why we see stuff starting the day before. But people were interested in v6. The PR side of World IPv6 Day worked.
I'm going to show you one other piece of information. This is an old one. This is three-plus years ago. This is from when Google first announced IPv6.google.com which was coincident with IETF and it was really just a blog posting I think originally. We saw a jump on that day as well. But a tiny jump.
You're talking about 20 more accounts.
So in a way, we have come a long way from March 2008 from people being interested in V8, general public.
The ugly. What did we see? As I said, we had a bunch of customers. Those customers panicked. We had a range of customers by the way, customers that spent four, five months planning and those that spent 15 minutes.
Now, some of the 15-minute guys pulled it off, which is good because they should be able to. Some of the many month guys only just pulled it off. So let's look at what we saw. Again, you would have heard this before, but this was a real life test and what was more interesting about World v6 Day was that the CIO had heard about this v6 thing.
Prior to that, it was just an engineer playing around on a spare server. But now the CIO, the executive, was actually sort of looking over their shoulder and going, "You better not break anything."
So we saw PMTU issues, predominantly because of people having misconfiguration-s, we saw ICMP 6 blocking, predominantly because people were looking at how they'd done things in v4 and thinking, we will do it in v6. Even the people that spent many months planning for this, these were filters that only disappeared 24 hours before World IPv6 Day kicked in.
Interesting point, and why we are the magnet for this, we actually know why we are the magnet, everyone and their uncle that had a Teredo issue phoned us, including people we had never heard of who weren't our customers, and we found out why. We knew about this.
This is a different level of Teredo failure than Geoff talked about. There are many levels of Teredo failure and you just work to get as little as possible. But the bottom line is in the Teredo world, you need to start the whole transaction off with an ICMP 6 message all the way to the server and it's all this bunch of fake addressing that goes along with this. It's pretty ugly.
If you have a pretty nice robust hosting environment that you have had for years for v4 and now you are starting to add v6 to it, all the filtering that you have, if you copied it over to v6 will probably make v6 on Teredo fail. This need to get a ping all the way from the servers through multiple layers of control routers, firewalls, load balances, and so we had a pretty heavy duty financial institution that was just looking at me going, "I have a CIO that wants Teredo to work." And I'm telling them, "No problem, just open up ICMP pings all the way to your servers." And he goes, "I can't do that. We have never done that. That's not what we do." They had to sit there and work on how to deal with this subtlety of two different worlds. This is because of Teredo but it is also the reality of where we live and the transition, this is a good example of the fact that I agree that the sooner we get over transition technologies, the better.
But these are the types of things that people dealt with on that day. Other than that, as people have said, this was essentially an easy day. Just another day.
But that's what I wanted to talk about, that's what I have got, nice fuzzier graphs than everybody else.
Tomoya Yoshida: OK. Let's go to the panel discussion.
Sala Tamanikaiwaimaro: A very quick question. In terms of the different transition methodologies -- and this is a question posed to the first two speakers, because I have heard the point of view of the third speaker, he's actually answered -- what, in your view, in terms of transition methodologies, would you rate as the best two and why?
The second question: you criticize Teredo in terms of clean transition methodologies, and the second question, I suppose, is to the third speaker, in terms of Teredo, whilst there has been an influx in terms of traffic generated, my question is: has there also been simultaneous measurements on the security threats or potential security threats related to Teredo?
Geoff Huston: What transition technologies or methodologies work? Well, quite frankly, none.
Honestly, all of them don't work. Partly the problem is that we don't even know how badly they don't work.
When I criticized Teredo for having a 45 per cent connection failure rate, let me make this perfectly clear. The fault is not Teredo. The fault is worse than that. The fault is that the NAT negotiation protocol called STUN, that tries to make the binding in UDP promiscuous, fails 45 per cent of the time.
Any service provider who is deploying a CGN is in deep, deep dodo. Because the only thing that is going to reliably work through that thing is Unicast single directional traffic and if any poor sod user tries to do anything fancy, at least 45 per cent of the time, they're going to die.
Any ISP looking at that would have to look at that with horror. They're not there to make the customer's life agony. Theoretically, the deal is the customer pays them money to make it all work.
We have wedged this network really, really badly.
You're saying how can I suck oxygen? To be brutally honest, unfortunately, and, you know, I'll be brutal, we shouldn't be here. We should have done this job five years ago and we should have done it quickly. Because none of these transition technologies when you have no more v4, work. You were meant to do this when there was plentiful v4 and the dual stack would have worked.
You're now trying to make v4 work when it can't. And it can't.
And you're trying to deploy v6 at the same time.
I'm tempted to say I'm going to do a law degree, good luck, I'm out of here, but I'm not. I'll hang around.
We'll see how bad we can make it.
Tomoya Yoshida: Emile, do you have any suggestion?
Emile Aben: That's for the auto-tunnelling, I completely agree and for the non-auto-tunnelling, things like 6rd, the problem there is that you do something temporarily, which costs you time, resources, extra equipment, whatever, which is not working towards the end, the end is dual stack and then v6. This is in essence -- if you cannot do it otherwise, you have to do it, of course, but it is throwing away money, basically.
Martin Levy: Before you ask that question, they have the last part of the last question.
Unfortunately, I don't have the right answer for you. As a backbone, we can measure flows at the highest header level. I can say a packet went from this port to that port, maybe. But I can't tell you anything about what's in it and that means that, to go back and talk about Teredo, you wouldn't ever want to design a network that specifically chose to use Teredo. Far from it.
But what's important about the security aspects of or any of the these tunnelling environments is that by deploying a native v6, at least on the LAN segment, where you have Windows machines or any machine that has Teredo enabled on it, you are essentially disabling Teredo, native will disable 6to4 as well. So therefore, don't look at it from that point of view, simply look at how can I deploy native and get around this problem lock, stock and barrel and then use classic filtering and firewalling.
Owen Delong (Hurricane Electric): Just to follow on to what Geoff said, I heard Geoff a couple of times during this Conference talk about has anybody measured just how broken NAT is and I think one of the reasons nobody has measured how broken NAT is, is because anybody in their right mind uses NAT as an absolute last resort when you have no other choice. If you're using NAT for some other purpose than a last resort where you have no other choice, you are asking yourself for additional trouble and headaches. Don't do it.
Geoff Huston: I think the real message I was trying to put forward is not quite that. What I was trying to say is that measuring failure has been an eye opener.
Realistically, what we are finding is that some of the things that we thought worked, don't. And I can ascribe reasons to some of it, but some of it is the protocols don't match the task and this use of STUN is a failure.
I'm now tempted to wonder if ICE and TURN are just as horrible. I think they are.
So let me go right to the heart of why, because it's a really interesting question. How did this industry stuff this up so badly? Because the IETF had too much attitude 10 years ago. Because when the one chance came to standardize the behaviour of NATs, we were too arrogant and we said NATs are evil. We're never going to standardize them. Every single NAT is different. So any poor person trying to write NAT traversal protocols is in a horrible place, because NATs don't all behave the same. But our applications assume one network. And every time we deploy another NAT, we make that little bit of the network different.
While they were your home, it was self-inflicted damage. Huh huh huh, you're doing this to yourself.
But the first ISP that puts in a carrier grade NAT is damaging all of their customers at once.
How have we got to this point? I don't know, but it's not a good place.
Geoff Huston: It's all too late. We should just flag Day v6.
Geoff Huston: You heard it here first. As of September 1, v4 is over. We are all running v6. It's the next flag day.
Lorenzo Colitti (Google): You are free to make that bet.
You might win. Good luck.
Sala Tamanikaiwaimaro: The point is if the industry has failed, then we are here. The question I have is, even in the midst of the mess that we are in, what is the cleanest transition that you have seen that you would recommend? That's one.
Thirdly, what's the solution?
Martin Levy: Sorry, we are all trying to say exactly the same thing in the microphone. Dual stack. It's the right way to go and there really is no other discussion at this point.
Tomoya Yoshida: Any questions? OK. So thank you very much for the speakers and please give some small comments for the next step, so what we should we do for each by each.
Geoff Huston: You know, there's this group of folk that appear to go to car races just to see if they're going to smash. I don't know about your motivation, but I think I'm here for a reason not dissimilar. I'm trying to make light, I think, of a hideous problem. We really have collectively managed to ignore this to the point where it is way too late and we're about to do a phenomenal disservice to our users and a disservice to everything we built over the last 15 years.
The best I can say to you is for God's sake, don't prolong this mess. A fast transition is all we have left. If you're not actively in where you're working doing v6 now, why are you making life worse for the rest of us? This is honestly a really bad place to be and if you think transition is going to even last for five years, I don't think we're going to have an Internet at the end. The only advice I can do is to say if you think you're going to make money out of the Internet, you better be doing v6 now. Because otherwise, there is no future there, but I'll have a damn fine car crash to look at.
Emile Aben: So one thing I would like to see in the near future is that IPv6 equates quality of a network so -- what I got from the vendor talks is that it takes quite a while for equipment to get upgraded, so you could see the correlation, people that are not upgrading their equipment, using old stuff, they have to stay on v4.
For the people that can upgrade stuff, they will have v6 and it should not be hard for them to actually deploy it, so I hope in the near future we are going to see this in organizations having v6, being equal to that's an organization with quality.
George Michaelson (APNIC): I have a question from Jabber.
This is to Emile and Harvey. Did they notice -- I think that's Martin Levy. Did they notice any rise in email traffic on World v6 Day like any box using v6?
Emile Aben: We haven't looked into email specifically for v6 Day, but I did a little side thing on looking for various mailing lists like NANOG on how much v6 addresses there are in headers and it's fairly constant actually. But I didn't do any research specifically for v6 Day and if people are interested in these numbers, I would be happy to make that into a RIPE Labs article or something.
Martin Levy: I have somebody who can look at this graph and say exactly what we are seeing. Yes, there is a rise. There was definitely something going on around June and it seems to have been growing, but there's an overall trend on our backbone from basically about April or May, where there was no change over -- actually, let me do this.
A graph because I can't show you because it's off our internal system, but the answer is that we see port 25 traffic pretty consistent over the last two years.
Then about May this year, we saw a jump and a month over month graph. So we are seeing something. That's worth looking into. But I don't think there were many people that focused on it specifically for World IPv6 Day. It may have been an interesting afterthought for people, enabling their servers and suddenly realizing that the servers are quite happy to use AAAA records. Pretty clever. Hadn't looked at that.
Emile Aben: I can confirm he didn't make it up. There's really a nice graph here.
Tomoya Yoshida: Well --
Martin Levy: Sorry, I missed the summary. Dear competitors, please continue to ignore v6.
Tomoya Yoshida: Thank you. So, yeah, we found we still have many things to be fixed to IPv6 world, but we can move to IPv6 world. Since this is the end of the panel discussion. Thank you very much for joining the panel discussion and this is the end of the session. Thank you.
Miwa Fujii: Thank you very much all speakers. We have Tony Hill here. He arrived around noon today and he would like to make some closing remarks to you, so Tony, please come over here.
Tony Hill (ISOC AU): Miwa probably told you earlier this morning that I'm chair of the Asia Pacific IPv6 Task Force. I just wanted to really close today by saying thank you very much to everybody who has been involved in organizing today and particularly to Miwa who has put all this together. I think she deserves a round of applause.
Tony Hill (ISOC AU): Now that you have experienced that session, I hope you have found it enormously valuable.
I think that having all that information in one room in one place is very hard to get in any other way. I have made a comment that trying to do World IPv6 Day is a bit like a surgeon trying to do open heart surgery on someone while they're sitting at their desk and continuing their job.
I think the conclusion that I have out of listening to the presentations today is that the Internet community actually did achieve that major goal, they showed that IPv6 was capable of working in that environment and so that's a major milestone and a big achievement and I think you have heard how that allows us to go forward.
Geoff made a very good point, that we only have a very limited time in which to make this work from here. I guess the only point I would make is that the process of communication and collaboration amongst all the players is very, very important. If you do this the wrong way, it could go very badly for you and for the Internet. So keeping up the process of collaboration, listening to your colleagues, and learning from their experiences, very important and that's what the IPv6 Task Force is all about.
So thank you very much for your attendance and I hope you have enjoyed it. I'm between you and drinks.
Miwa Fujii: Thank you, Tony. Thank you everybody for participating in this session and particularly thank you to the speakers and moderators for your support for this plenary.
We learned a lot and probably we'll carry back this message like to be part of the solution, not to be part of the problem and let's stop this prolonged mess and maybe we'll continue our IPv6 activities for the next meeting as well and I hope we can hear many more success stories and business cases in the next meeting.
So before we completely conclude, there are a few housekeeping notes. NRO NC on-site election will be held tomorrow in Policy SIG. Voting opens at 11 am and closes at 4 pm.
Next message, social event at Busan aquarium. The aquarium is located five minutes walk from Paradise Hotel on the ocean front. APNIC staff will meet delegates in the foyer of the hotel main building, this building, at 6.45 pm and walk towards the aquarium together.
Please make sure you take your delegate badge with you into the aquarium. The social event starts at 7 pm sharp. So please do not miss it.
All right. Thank you very much, everybody.