Transcript

Arth Paulite - APNIC 28 Conference Network

Thursday 27 August 1800-1900

ARTH PAULITE: Hi. Good afternoon. My name is Arth from the APNIC network team. I would like to give you an overview of the APNIC conference network.

We're actually running seven access points from here up to the Secretariat and we are running wireless A, B and G and we also have an IPv4 and IPv6 running on this network. China Unicom has provided us a 20 meg link of IPv4 and CERNET provided us a 10 meg link for IPv6. There are some stories behind the network which already passed. We had one router that didn't work. Ones which are broken, and my laptop was broken yesterday after fixing the B6 and fixing the wireless, so I would like to thank Cisco for providing a back-up router, for CNNIC for giving us the spare seats, for Jonny Martin to help us to tweak the wireless in this conference.

That's all from me.

And by the way, we didn't block Facebook here. We're not blocking anything.

That's all. Thanks.

GEORGE MICHAELSON: Thank you, Arth. It's a miracle that we can survive these days on a 20 meg upfeed on 4 and 10 on 6. I'm really surprised we have been able to survive with so little head room. And even more surprisedI can call 20 meg "so little'.

So, I get the joy of closing.

MARTIN LEVY: I have a question.

GEORGE MICHAELSON: I get the joy of taking questions from floor. How incredibly rude of me. Please excuse me. Martin?

MARTIN LEVY: Martin Levy, Hurricane Electric. Thank you very much for the networking by the way. I truly mean that.

Not criticizing it, but is there any way we could start adding the sort of social bar area to the wireless networking -

GEORGE MICHAELSON: Seconded. Do I see a third or a fourth. I see a fourth. I see a fourth. I see a fifth.

MARTIN LEVY: Anyway, you know what I'm saying!

GEORGE MICHAELSON: OK, any more questions?

OK, thank you.

Matsuzaki Yoshinobu - EDNS0 and Equipments

Thursday 27 August 1800-1900

MATSUSAKI YOSHINOBU: Hi everyone. I'd like to talk about EDNS0. It's not a new topic, but I would like to remind you about this issue. So, EDNS0 is a new capability to our big DNS UDP. And our usual DNS is a UDP, but it's restricted under the 500 byte. But using EDNS0, you can use 4,000 byte. And modern DNS software supports EDNS0 by default. And why EDNS deploy? Because we added IPv4 quality code for each zones, like any ccTLD zones, and also, nowadays, we support DNSSEC, and some of the TLDs now find its zone like BZ, CZ, PRO, SRO, TH, and there are also some zones that support DNSSEC signs.

OK, and I have a case when .org was signed, then a user complained that they could not access some .org websites. They could not resolve the host names. And at that time, they have... their internal cache server behind their Firewall. And at that time, we could see the host names on our ISP's cache server. So something went wrong around their cache DNS server. So we asked them to send us something and then, I found the user sent an EDNS0 query to the DNS servers, and this is the default setting of most DNS servers. So the DNS content server deployed a big UDP DNS packet. But, this also was brought by the Firewall, because they have a policy that do not arrow fragmented UDP. And at that time, .org was signed so it was bigger than 2,000, but it is fragmented. So the cache server wouldn't get an answer.

And even worse, the user filtered the TCP 53 as well. So it's too bad. So, we suggest our UDP 53, because of the fibre, and also our fragmented IP packet. And, arrow TCP 53 - of course, it's a speck of DNS. And worried about it, you can check by your server, because there are some sites that provide such kinds of checks. If you see this host name, they say, OK, you are enabled DNS 0, and you can use 4,000 capabilities or something. And this checks the cache DNS server and the content server's relationship. And you'd like to check your client and the cache relationship, you simply call your DNSSEC enabled TLD, like 3 or Info or something. Then, if you can get the answer, it's OK. But if you can not get an answer, you probably have some problem, so please check by yourselves. And in the near future, DNSSEC introduced to this legion, I think. So, please check before that, otherwise you will have names with really big issues in your site. OK, that's it.

***APPLAUSE***

GEORGE MICHAELSON: Thank you, I think there is a wonder synchronicity given what I will be talking about at the end, because we are indeed examining the consequences of deploying DNSSEC in our refer sites. So we may very well trigger the situation that you are reminding people to think about. Thank you.

SPEAKER FROM THE FLOOR: One question. Do you have any idea due to the EDNS0 impact of how much is then being converted into TCP query against DNS servers?

(Pause)

Let's talk later.

MATSUSAKI YOSHINOBU: Yeah. Personally I prefer UDP because of the cost of TCP.

GEORGE MICHAELSON: I will make the observation later on that relates to that if I can remember the issue.

Andy Linton - IPv6 Readinesss Survey

APNIC 28

Thursday 27 August 1800-1900

ANDY LINTON: I thought I'd do this talk, because there hasn't been enough talk about IPv6 over the last two days, so just to give you a little bit more. Last week in New Zealand, we ran three meetings in Auckland, Wellington and Christchurch to try to persuade people that v6 was something that they needed to start think being now, and as part of that, Internet NZ did a survey on IPv6 readiness in New Zealand. It's interesting to compare the results of this with the member survey results that were done earlier in the year by APNIC, where you're asking members about things. This is asking people who aren't necessarily members or haven't really done much about v6.

So... how do I do this, George?

GEORGE MICHAELSON: Just hit keys at random.

ANDY LINTON: Oh.

GEORGE MICHAELSON: Are we good?

ANDY LINTON: Yeah, I just wasn't hitting it hard enough. OK, survey method was to go to the top...

GEORGE MICHAELSON: Hit on that. Pause.

ANDY LINTON: Now we can go. The survey method was to go to the top 100 chief information officers of companies in New Zealand and Government departments. It was done by e-mail and there was phone follow-up. We got responses from 37 of them. Which maybe says something in itself. And all the respondents, all of them had and were enterprises with 500 desktops in their organization. Of the ones who responded, 60% of them were from public service, so Government departments and local Government. And 40% were out of the private sector.

OK. As the saying goes, there's good news and there's bad news. The good news is that more than half - at least adequately informed about what IPv6 is and how it works. So, if you look at the bar graphs here. 10% were very well informed. 43% adequately informed and so on, and 70% of them were aware that all IPv4 addresses will be allocated. And the question was asked in the survey was 2012. It would have been nicer if they asked 2011, but they certainly are aware that this is a finite resource and will go away.

The not so good news - management are not generally aware of IPv6 as a risk or opportunity. So you've got 8% not sure. 65% not especially aware. 24% adequately aware. So, like the people who are aware or very aware, 27%. That's a big percentage who don't see this as an issue.

And over half had no plans to have an IPv6-enabled website, public Internet services including e-mail and DNS. So there's a 54% number there and that's from people you would hope in Government departments who would see this, or who would have had some sort of message passed to them down the tree. So it's quite a large number who have no plans to do this.

And roughly the same percentage had no plans for IPv6 in internal networks, so I think that's pretty scary for us. And if you look at the graphs here, the distribution, it's going to be two years before even a very reasonable number of them begin on this thing. And those who do have plans, 43% will support both IPv4 and v6 in parallel. And I'm not sure what that means for the other 57%. They're going to switch to v6 immediately, so I hope that that works out.

Is there any light? There's a bit of saving grace here. The majority of the networks support v6 are upgradable, but there's quite a lot of don't knows there, so not a huge amount of preparedness, certainly not on people's radars to a great degree.

There's a couple of gotchas here in spot the gaps. For most, IPv6 is not a factor regarding purchasing. So 51%, it is not especially important. 19% not important at all. That's 70% are saying - well, you know, if our vendors don't give us v6, we don't really care. And just another look at the 41% who say it doesn't feature in their future IT requirements.

There's plenty here. Spot the gap here too. 84% say their telecommunications providers have not briefed them on their plans to support v6, so New Zealand telcos are not well down the track here. If they're planning to do IPv6, they're not letting people know, so they don't see it as a major thing that they need to communicate. And it's still kind of interesting that 49% of people were going to rely on outsource support, and 24% have no support. So there's not that many have support in-house.

And there's a couple of comments there. I'd highlighted some of them. So the other one I haven't highlighted, which I think probably should be highlighted was the one - "I am unaware there is no problem for us." I'm not sure what that means. " We think that we have plenty of IP space." You know, they're not asking the question about what about other people. So I have plenty of IPv4, so therefore it's not a problem. So it's kind of interesting. "We are not going to be upgrading to the top-end product just to get the IPv6 functionality." So there's a number of problems there. And you can have a look at those comments. I'm sure that the slides will appear somewhere. And that was really just the last one, was it? Yeah, that was the last one. So just interesting for me to compare it with the readiness survey or the survey that's in the APNIC member survey, which has quite different figures from this because you're sort of preaching to the converted people who understand what's going on.

And perhaps these questions, some of these questions could be incorporated into some of the stuff that you ask the members and so on. So there you go.

GEORGE MICHAELSON: Thank you, Andy.

Jonny Martin � Three Steps to WiFi Happiness

Thursday 27 August 1800-1900

JONNY MARTIN: OK, hello, everyone. I'm Jonny Martin, here to talk about WiFi happiness, specifically three steps we want to use to get there. So the reason I'm doing this is that I see at a lot of conferences, even the basics of a WiFi network seem to get forgotten about, so I thought rather than providing a complex presentation, we'll go for a simple one this time and hopefully in the future we'll see more and more of this being done.

So WiFi happiness - well, providing WiFi for a high density of users, and by high density, I mean, you know around so 100 to maybe 300 users in a confined space, such as this room. You could spend a lot of time designing, network testing it to achieve such. Some people do that. A lot of people don't, so we're going to talk about three things we can do to make our WiFi happy. The site survey, use of non-overlapping channels and turning the power levels down. None of this is rocket science, but we need to do it.

GEORGE MICHAELSON: Jonny, rocket science is not actually rocket science.

JONNY MARTIN: Interesting! OK, there's actually a fourth step in here that I'll put here. The first thing is to always use good quality professional access points. By that, I mean something like we've got here. We've got Cisco 1200 series access points. Another one I've used in the past is Proxim access points. The reason you want to do this is because if you use cheap access points, Delink or consumer devices or whatever, they'll typically only allow you to use five or ten clients attached to them. So when you have a couple of hundred people sitting in a room, they're going to fall over and not work. So a good access point will easily handle 70 to 100 access points to it, and looking at this one here, it's constantly had 70 to 80 people attached to it, so it goes to show that they can handle it.

So the first thing here is a site survey! You don't necessarily have to dress up for it, but it can help. So you want to have a good look at each room. And just think about where you're actually going to place the access points. The things you want to think about are placing them as high as you can so you get a good line of sight to the clients. In this room, you can see they've all been raised up, so it is a pretty unobstructed line of sight to the access point. Up don't want to have them sitting down low like so, because if they're at a low level, then all of the wireless are going through lots and lots of people and glasses and water and what not to get to through to the laptops. You want to try to stick them as far apart as possible. Obviously the closer they are together, the more interference you're going to get and the more noise in the spectrum.

The other thing you want to do is check for existing access points in the building, and if you can, turn them off. So obviously, the things I'm talking about here, you don't always have control over everything. You sometimes have to compromise, but you want to start trying to get it right.

The second thing you want to do is use non-overlapping channels. So if you're using 802.11 b/g, that runs at 2.4 gigahertz, only use channels 1, 6 and 11. If you use any other combination of channels, then you're going to get overlap between the channels and interference, so things just won't work very well if that's the case. You want to use 802.11 a. That runs on 5 giga hertz, so it won't interfere with your b/g. The good thing about 802.11 a is that it provides us 19 non-overlapping channels, so the big thing you want to do is run, if you have dual radios in your access points, run up both b/g and a at the same time. Because we only have three non-overlapping channels for the b, if that runs out, don't be tempted to put in more b radios in the same room. Fill it out with 802.11 a. So most new laptops will run a, b and g. It's up to the individual client which one they prefer to use, but you'll get a good number using 802.11 a. And that will make your life easier. And the big thing with selecting where you want to put your access points is that up want to prevent any access locations in the room from seeing multiple access points on the same channel, so if we count the access points, we can see in here there's three, so they're going to be on channels 1, 6 and 11 and wherever you are in the room, you shouldn't be seeing any two access points on the same channel.

Now the thing to think about with that is when you have walls, there may be access points on the other side of walls. If they're thin walls, the signal is going to go straight through them. All right, number three there - we want to turn the power down. So we only need about 2-5 dBm of transmit power. Now, looking at that, most access points will easily do 150 miliwatts, which is around 20 dBm, so you want to turn the power right down. If you don't do that, you will have problems. So the reason we do this is to reduce the overall RF level in the room. So that's it, three simple things to do, and if you do those three simple things, then making a wireless network work and keeping the people using it happy is pretty easy.

One last slide here that's looking at some simple tests you can use to actually test your network. Up don't have to go out and buy big expensive spectrum analysers, you can do the basics. So you can see on the top graphic there, there's a ping that's been done just across the local LAN to the router in this case. And you want to send some big packets - in this case, they're sent at a rate of 10 a second. You probably don't want to go too much faster than that, and you're looking there for a nice consistently low round trip time and no loss, obviously. If you're doing one 32-byte ping per second, you can see quite a good response, but the network could still be performing quite poorly, so you always want to make sure that you're generating a wee bit of traffic.

The other thing you want to do is use a wireless scanner, so there's a program here called KisMAC from Mac and that allows you to see all of the access points, the channels they're operating on and what's happening. So, I'm not quite sure who created the random network SSID to shut off auto connect ib, but that's not a terribly smart thing to do because it's just creating more noise in the room and it's on channel 11. We have another access point on channel 11, but we did quite well with the channel selection here, because if you look at the power levels that we're seeing, so in the average column, from where I was sitting when I took this screen shot, we were seeing about 30 dBm of signal on the random SSID which is 11. 19 on the free public WiFi, but the APNIC 28 one, we're seeing 36, so we can see that we've placed our actual channel 11 as far away from the existing channel 11 as we could. The only other thing to say here is make sure if you're testing these things, you check all around the venue, so you're not just checking one point. And you make sure that you check things once the room fills up with people and laptops. So three and a bit things that if you do, the network should perform pretty well and you may not need to tweak any further after that. So thank you.

GEORGE MICHAELSON: Those of you who don't realize it, Jonny was not actually sent here to help improve the APNIC WiFi network, but he selflessly volunteered and spent a large amount of time working to help with the reconfigure. So quite apart from a great talk, we owe him a great thanks to help to get the network running. Thank you.

***APPLAUSE***

JONNY MARTIN: No worries.

Randy Bush - Use of Default in the DFZ

Thursday 27 August 1800-1900

RANDY BUSH: Okey-doke. So, I have a second life. I'm actually a researcher. I do computer science. OK. And we study, among other things, visibility. What's the routing graph of the Internet? What's the AS topology? How do we debug? How biased is our methodology? We did some years back, some research for ARIN to enable them to diagnose what ASes were filtering newly allocated addresses, what's known as bad bogon ASes, and you can see a presentation from APRICOT 2007, two and a half years ago. ARIN never deployed it, but we continued to measure to see how long it takes to get filters removed. It's actually quite a long time. But we had this tool for measuring topology in the Internet. And one-day, when we'd finished a measurement, my friend, Olaf, said, Randy, why don't you announce a /25 and see how far it gets. So, we announced the /25 to NTT. NTT said that they would pass the /25 only to customers, they wouldn't give it to peers. So we went and we looked the RIPE routing information service, we looked in 700 BGP measurement points from a large CDN. And it showed that 15 ASes saw that /25. This is kind of what we expected. A /25 shouldn't propagate very much.

When we used ping sourced from the /25 to all the ASes in the Internet. Well, almost all. 1,024 ASes could get packets back to the source. This means 1,000 ASes had some sort of route back to the source. So if you lose them to see the Internet, you are off by a factor of 60. OK? And by the way, one was as good as the other. You looked in route-views, you looked in RIS, you looked in 700 BGP servers and you got the same bad answer. How much of this was due to default? So we created an experiment. And if you read the NANOG mailing list, you will remember the framing. What we did is that BGP - say we want to see if AS 42 has default? If we put an announcement that has 42 already in the AS path, BGP loop detection means that 42 will say, "Oh, I've already seen this, do not listen. " So we could - what's known as poison the announcement so that 42 wouldn't get them. Everybody else in the Internet would get it, but 42 wouldn't. And we used two prefixes. An anchor prefix, which is an old established prefix for many years, and we used a test prefix, 98128. So we poisoned the test prefix for 42 and we pinged from the anchor prefix, and if we got a response, we knew the test was there. So we know it is live, and then we pinged from the test. And if they responded to the test and we knew they didn't have a BGP route to it, they had default. So we tested some 20,000 something ASes in the fault-free zone. And the Mac didn't work. Hello!

So, it is not a default-free zone. In fact, it is 69% of the ASes in the Internet have default. That's 70% of the Internet, 70% of the default-free zone is not default-free. 25% is default-free, because 6% were mixed. For some of their prefixes, they returned the fault and for some they didn't. Or, you know, it could have been a measurement error, etc.

Now, what's interesting is that it's not the default-free zone. So, let's call it the BGP speaking zone. This is another way of looking at the same thing, and we used some division of the ASes in the Internet and stub ASes and small ISPs and large ISPs, but even 17% of large ISPs had large defaults.

Somebody - Yoshida-San, took the list of Japanese ASes and our measurements and looked and Japan is the opposite. 60% default-free. 36% default. So, you know, things are not the same all over the world. And he's going to use our data to go and survey the rest of the APNIC region and we should have results in a couple of weeks.

OK, so and I just think it is culture. I think it is the operator culture that's different in the different areas. So, our ways of measuring the Internet are broken. Looking in route views, looking at the RIPE information server does not tell you if they can get packets to you. OK. Looking at the control plane measurement does not tell you the data plane result. Looking in just route views or RIPE information services is just as well, one or the other or a hundred different BGP feeds. You get the same bad data everywhere. Researchers should be very, very careful using these routing service data for lots of analysis, like when Renesys gets up here like they have and says "Oh the Taiwan cut - we looked at BGP announcements and the Taiwan cut changed traffic this way," they have no idea what traffic did. They just don't have any idea. You can go... hello! OK. I'll hit the arrow if I can find it. Space bar? That's space bar?

GEORGE MICHAELSON: Maybe you don't have another one in this preso?

RANDY BUSH: The other one that I don't have is this one. There is the URL. If you go there, you can type in your AS number and it will tell you what we think, what our analysis said about your AS, and you can type in, "you're wrong", "you're right", "here's why", etc. Please do that. We've had about 300 results so far and we're right about 97% of the time. Terry?

TERRY MANDERSON: Randy, I love this presentation. The reason why I love this presentation is it finally categorically points people to observe that what's in the RIB does not match the FIB.

RANDY BUSH: No, it doesn't. It doesn't do that. It doesn't do that. What was in the RIB and the FIB was default.

TERRY MANDERSON: I don't see a 0.0.0.0/0 route in BGP.

RANDY BUSH: Many of the ISPs were people...

TERRY MANDERSON: So you're saying...

RANDY BUSH: No, stop. Many people who went to the URL and looked up their AS said, we have default and when they looked in the routers and could not find the default statement. And went and looked in the RIB and found that their upstream was giving them default and they didn't filter it.

TERRY MANDERSON: OK.

RANDY BUSH: So let's not say why this happened. We have categorically measured what happened. Saying why - do not jump to conclusions.

TERRY MANDERSON: OK, you didn't say that sentence earlier.

RANDY BUSH: No, there's lots of stuff we have behind this, this is a lightning talk, not an entire storm!

GEORGE MICHAELSON: So, I have one cheeky question that can probably be given a short answer. Is it arguable that this much presence of default is actually a good thing? And that it has beneficial effects in terms of overall stability of the good. But in other talks that you've given, you've mentioned that the churn in the routing plain is poor measure for the good put of useful packets.

RANDY BUSH: Yes. This helps packets get there. It will have good effects and bad effects. It helps packets get there. But think about what it means, for instance for BGP routing security. So, it's got good things and bad things.

GEORGE MICHAELSON: Yeah.

RANDY BUSH: And I was surprised that I found out that one of my sites had default. Not IIJ.

GEORGE MICHAELSON: So food for thought and an entertaining presentation. Thank you.

***APPLAUSE***

George Michaelson � What's DNSSEC Going to Cost APNIC to Deploy

Thursday 27 August 1800-1900

GEORGE MICHAELSON: So, this is an APNIC internal presentation, which I'm putting up because it is topical. It relates to the DNSSEC topic, but it is not an approved presentation. This is not policy, this is personal opinion only.

So the question I was asked as an internal research feed to my tech manager is - what is going to happen when we turn DNSSEC on? You know what, I'm going to put this mic down. That's better. And I unusually said, you're going to have to be really scared about this. I mean, people are saying 600 times the traffic load and everything cuts to TCP and the world ends. But it turns out that's not entirely true. It's not quite like that. So the questions I went and looked for in our information systems were - how many people could do this, potentially? And because we host two different kinds of DNSSEC, some on behalf of the other RIRs and our own, and because the RIPE NCC has been doing DNSSEC, I think for at least five years now, four or five years, a very solid track record of providing this, and we host that in Asia, we actually have a position to measure the relative use of DNSSEC on their services and on our services.

And it turns out about 40% to 45% of all the queries we're seeing had DNSSEC as a request aside. They said, we could do this if you're prepared to answer. Now, that is a lot more than we expected.

When we last looked in 2008 it was only at 25% to 30%. The second thing, which I can't easily show on one side is that this is not a constant growth curve. This is more like an arbitrary event. Logistics - not logistical supply curve, but the logistics of when you do things in the DNS. If you deploy new DNS code and it happens to set a bit in your resolver that says, "I can do it." We're going to see it. It's not because people actively do this. At least, I don't believe so. It's gone down as well as going up. I'm noting that it has peak above 40%. It's around 35%. So the second question in my mind is all well and good. If we were asked to do DNSSEC, how much is it to answer?

Well, this is the non-DNSSEC response profile size we see on our own service, and the interesting thing is that there are only two types that really matter here. There's the good answer in red, and there's the bad answer in green, and you'll see that although it looks like it is two radically different peaks on a graph, they're pretty much around the 100 bytes to 150 bytes. There's not a lot of cost in saying yes or no before you turn on DNSSEC. It's all the same thing.

So, it just looks to be pretty much no cost to say yes or no before you do DNSSEC. So when you turn DNSSEC on, you grow an extra limb of size here. I'm sorry that the scaling doesn't show well, but what I've grown here is that we now have a huge bubble of response between the 300 and 400 bytes, which is the good response, and then right over on the right-hand side, we have around an 800 to 900 byte NX domain response that we're giving that did not exist before, and that is essentially the cost of the RR SIG and the NSEC data which just has to be appended in DNSSEC to give a complete binding appropriate response. Now, I would love to claim I'm a DNSSEC expert, but I'm not. I'm not even a DNSSEC expert at heart. And if you got up and said, ah, but we're going to do NSSEC or we could limit the information or any of these things, I would have to say, I don't know. All I observe is that this is what I'm seeing on the wire.

If you turn on DNSSEC, you actually can get five times more data than you had before just to say no. That seems a little expensive. So, then you get to the basic maths. How many queries do we actually do in our particular information space? Well the answer is 99% of what we do is PTR. I know we host CCTLDs, I know we have zones and we training it, it is all PTR data. So, if we're doing PTR, and if 60% of them haven't set the bit, then that means that most of those 60% are going to lie in that 150 byte cost whether I say yes or no, so what I care about is the 40% max who say they will set the do bit. And it then turns out about 30% of that is NX Domain responses. We have a lot of unregistered reverse, which means we give a lot of negative answers. That means 30% of the 40% goes up to 800 bytes, and it's approximately the 60/37/13 split, and if you do the arithmetic, and it's not mathematics, it is arithmetic, and I can do arithmetic, it's a two times growth. Which is kind of cute, because two to three years ago right back at the envelope statement with the IPG, it was going to cost us two to three times. And that was based on completely different arguments, so for the wrong reasons, I was right the first time around. So the basic answer is - for us, and this is not a general advice for everybody, but for us because of the nature of our traffic mix, we think we're going to see a two times traffic growth for every zone we turn on DNSSEC. It's just all about PTRs and NX Domains, and that's a good prediction that I can then make a big fool of myself with, because at the next meeting, we will have turned on DNSSEC and I can do a lightning talk and tell you what we actually saw.

That's it.

***APPLAUSE***