Associate Professor, Computer and Information Science, University of Pennsylvania
Secretary of State, Indiana (R); President, National Association Of Secretaries of State (NASS)
Director, Cyber Security Project, Belfer Center for Science and International Affairs, Harvard Kennedy School; Contributing Editor, Lawfare
Washington Bureau Chief and Senior Vice President, CNN
This is the first session of the Hacked Elections, Online Influence Operations, and the Threat to Democracy symposium.
The panelists provide an overview of the complex problems facing election cybersecurity and offer recommendations to protect the integrity of elections from cyber threats.
This symposium convenes policymakers, business executives, and other opinion leaders for a candid analysis of the cybersecurity threat to democracies, particularly to the election systems themselves and the subsequent attempts to shape the public debate through disinformation and online commentary.
SEGAL: Good morning. I’m Adam Segal. I direct the Digital and Cyberspace Policy Program here at the Council, and I just want to say a few words of thanks and a few words of welcome.
First, let me thank Marisa Shannon and Laura Bresnahan and Alex Grigsby for helping put this together and making sure that it runs smoothly. We want to thank PwC, which has provided some funding to help us run this and a number of other programs in the Digital and Cyberspace Policy Program.
Like many of the other think tanks, we are doing more and more work in this space, on information operations and Russian information operations in particular. Just this month we published a “Cyber Brief” by Keir Giles of Chatham House giving some policy recommendations to liberal democracies on how they can counter Russian information operations. So if you haven’t seen that, please take a look. It’s available on the website.
Also, last month we ran—we rolled out a cyber operations tracker. That’s the website you might have seen when you walked in. It is a list of publicly-revealed state cyber operations, so espionage, doxing, more disruptive and destructive attacks. It goes back to 2005. We have approximately 200 known incidents, and the plan is to add more as they happen and as they become more known to us. Please check that out. It’s updated every quarter. If you have an incident that we don’t know about, please let us know and we will—we will go ahead and add it.
Please find me today if you have ideas and suggestions about how CFR can be helpful in this space, what we should be doing. If we’re doing something that we shouldn’t be doing or there are things that we should be doing that we’re not, please come and find me.
And thanks for spending the day with us today. I think it’s going to be a great discussion.
FEIST: Good morning, everybody. I’m Sam Feist. I’m the Washington Bureau Chief for CNN.
And we’re going to have a conversation this morning on hacking into our election systems. The next panel, the 10 a.m. panel, is going to focus on hackers and other—how hackers and other mischief-makers are trying to influence public opinion and influencing the public, and how that might change votes. But this panel is going to focus on how mischief-makers might actually try to change votes and hack into our election system. So we’re going to talk about the security of our election system, voter-registration system, tabulation systems.
And if you think back and think, well, they might be able to hack here or hack there, but can they really impact an election, we can just look to the election that happened last night in Atlanta. If you haven’t—if you’re not familiar with what happened, it is yet undecided. The Democrat is ahead at this moment by 729 votes in one of America’s largest cities. So any mischief could affect any election. In fact, one of our—one of our panelists, who happens to be Connie Lawson, the secretary of state of Indiana, told me about a race this morning in her state that was decided by just one vote. So these things do matter.
So I’d like to start by introducing our panel. I mentioned Connie. Connie Lawson is the secretary of state of Indiana, and is currently the president of the National Association of Secretaries of State.
LAWSON: Good morning.
FEIST: To her right is Matt Blaze. He is an associate professor of computer and information science at the University of Pennsylvania, recently helped organize the DEFCON voting machine hacking experiment to test the vulnerability of our election voting systems.
And to my right is Michael Sulmeyer, who is the director of the Cyber Security Project at the Harvard Belfer Center. Michael previously worked as the director of plans and operations for cyber policy in the Office of the Secretary of Defense.
So thank you all for joining us. We’ll spend about a half an hour visiting and having a conversation. And then, at about 9:15, I’d like to open it up and have you all ask questions for our panel, and we’ll continue the conversation until about 9:45.
So I wanted to start with Connie. Connie’s responsible for the voting systems in the state of Indiana, but also works with secretaries of state across the country. So would you describe our voting system, in your opinion, as currently safe from hackers and mischief-makers, or are you particularly concerned? Where do you fall on the continuum?
LAWSON: Well, first of all, you know, obviously—and I know people in the audience have heard this before—there’s no evidence that any votes were tampered with in the 2016 election.
I think that election security has always been a priority of secretaries of state. And I think that the email that every chief election official received in August or September of 2016 changed the way we do business. And so we’re making especially cybersecurity a priority. And we’ve done a number of things, working with the Department of Homeland Security, the FBI, and—to become—to make sure that we get the information we need.
So the number-one activity since the 2016 election for the National Association of Secretaries of State and the secretaries of state has been to improve the communication between the intelligence agencies in the United States and we as chief election officials, so that we can get the information we need in order to prevent and/or react quickly if there should be a cyberattack.
FEIST: So just going to put you on the spot: Are you comfortable at this point—knowing that no system is perfect, but are you comfortable that we have done as much as we can do? Are you comfortable that if there were an election tomorrow in the state of Indiana that it would be safe?
LAWSON: Well, about the time you say you’re comfortable, that’s when you should be worried. So I’m never going to say I’m comfortable about it, but I’m always going to say that I’m going to be very vigilant. But I do believe that we are doing everything we possibly can in Indiana to make sure that our elections are safe.
I am very fortunate. Not every state has the support from their General Assembly. My General Assembly appropriated $1.4 million so that we could make sure that our system is secure. We migrated our data. We’ve done a number of things to secure our outward-facing websites. We certify our voting machines for use in Indiana. We have what we call the Voting System Technical Oversight Program. So we know where every machine, every type of machine, every serial number, every tabulating machine, we know where it is and when it’s in use in the state of Indiana.
So I feel good about what we’re doing. And we’ve been told by DHS in Multi-State Information Sharing and Analysis Center that we’re doing the right things.
FEIST: Matt, do you feel good?
BLAZE: Well, you know, I mean, I feel good that Connie is doing the best that can be done, but what I worry is that the best that can be done is almost certainly either not good enough today, or the honeymoon is going to end very, very quickly.
So, you know, a little bit of background from my perspective. I’m a computer scientist. I’m a technologist. In 2007, I led teams that were contracted by the states of California and Ohio to do a top-to-bottom review of their election system technology, including the voting systems and the backend systems from the vendors used in those states, which turn out to be the same vendors used in the other 49 states. And what we discovered in 2007 was that these systems were riddled from top to bottom with exploitable security vulnerabilities in virtually every component of the systems. And the—some of those vulnerabilities were sort of coding errors, you know, bugs in the programs that could be fixed. Some were more architectural, particularly in the so-called DRE systems, direct-recording electronic voting systems, the touchscreen voting machines that record voter selections electronically in their internal memory and the systems that process those.
You know, interestingly, you know, we know that those can be exploited. In many cases they can be exploited with no more physical access than you would need as a voter or as a poll worker at a precinct, but there’s been no evidence that they’ve actually been exploited in any election. So, you know, we have to kind of walk a fine line between saying, look, this technology very desperately needs to be improved and, you know, falsely telling people that our elections are illegitimate. So I don’t want to—you know, I don’t want to say that our elections are illegitimate, but I don’t know how to prove that they aren’t, because in some of the cases the technology that we’re using doesn’t really tell us. And that concerns me greatly.
FEIST: I want to bring Michael in. What do you think the biggest vulnerability of our voting system is?
SULMEYER: Well, first, thanks to the Council for putting this on and for having us here.
I was at DOD, so I never feel good about anything. (Laughter.) I never feel comfortable.
The challenge is one, it strikes me at least, of risk reduction, not elimination. So you have to set your standard, your objective in some way that’s reasonable. And so you’re always going to have some level of uncertainty here. The challenge and the opportunity is to reduce that risk as much as possible. It’s nice to hear the General Assembly in Indiana wants to help you do that with some appropriation.
The challenge that I really see is it doesn’t take much to have an effect on the vote count. You don’t actually need to have national-wide intrusions, right? Reducing the risk of gaining unauthorized access, that’s what—the risk we’re trying to reduce, gaining unauthorized access. You do that in a couple key jurisdictions and get the timing right, you can change a count. You can make things a lot more difficult for the folks who are trying to make sure that our elections are conducted in a way that’s as high-integrity as possible. You can really complicate that effort in just a couple key ways.
So that’s my perspective on it from my experience. It doesn’t take much, but we’ve got to reduce that risk as much as possible.
FEIST: Yeah, Connie.
LAWSON: Well, I just want to make sure that everybody understands that the last election that we questioned was the 2000 election, when we were virtually using paper and punch cards. And so, if you think about the way we do elections today, you know, I’ve been a county clerk, and I’ve been on the ground, and I’ve run elections. I did that for eight years. And I will tell you that there are security measures that our local election administrators take that make it nearly—well, make it very impractical for someone to get to our—to our voting machines.
First of all, these machines are kept under lock and key, and most of them have a visual scanning of the facility. So we know who comes and goes. They use logins. So we know, again, who comes and goes. We do public tests. And once those public tests are run before an election and, you know, we know that the votes are recording properly and that there are no votes that will be present on election day before someone actually comes to vote, those machines are sealed. And when a bipartisan team arrives on election morning, they cut the seal from the machine and they record the number. And one of the first things the election administrators do at night when they get the results from the—from the precinct level or the vote center level is they look to make sure that the serial number on the lock that was cut off the machine is actually the serial number that was placed on the machine after the public test. And so—and a bipartisan team, again, delivers these results.
So is it possible? Yes. Is it practical? I would say no. And there are many physical aspects of these voting machines and tabulation machines that take place and have taken place for years that it just seems—I mean, we don’t put them out in the middle of the courthouse and say have at it.
FEIST: So I covered the 2000 election and the Florida recount. Those are 35 days I will never get back. (Laughter.) But as a result of the Florida recount, the federal government—and state governments, but federal government—spent billions of dollars to help replace our—many of our election machines. The Florida system used punch card—paper ballots that had punch cards that were, as we know, sometimes not always easy to read, and that was one of the issues. But we replaced them with, to a large extent, these electronic touchscreen ballots that didn’t necessarily have paper records at all. They were completely electronic. In fact, in Indiana, what percentage right now are those machines?
LAWSON: We have 92 counties, and about—there’s 50-plus that use the DREs. However, the DREs that we use do have a(n) audit trail, a paper audit trail, inside the machine. It’s a—it’s a mirror image of the ballot. It’s not a voter-verifiable paper trail, but there is a paper trail.
FEIST: So did we make—did the Florida debacle make things worse, Matt?
BLAZE: It made them different. It essentially shifted us from a system in which you could have—kind of very vulnerable to small-scale retail mishaps to one in which small-scale retail mishaps probably have become less critical since the Help America Vote Act, but the—we pay for that by exposing ourselves to catastrophic failure in ways that we previously weren’t. Our elections are far more dependent on the integrity of software. And that’s something that we simply don’t know how to do.
FEIST: So if we were—if we had all the money in the world to design our system today, what sort of equipment, machines, system—what would you—if you were in charge of voting in the United States of America, how would you have Americans vote to get us the safest possible outcome so that at the end of the day, the day after or the week after election, the losing candidate or anybody else can’t come and question it and say: This vote’s not right?
SULMEYER: I’d hire Matt.
FEIST: OK. (Laughter.)
BLAZE: That’s a fine idea.
SULMEYER: Thank you. Yeah. (Laughter.) The two things I would just say is you’ve got to have paper backups, some way to an audit trail on every machine. And you’ve got to have a way to turn off the WIFI on some of these devices. I’m with you on physical access. I don’t have concerns about anybody rolling into the courthouse and having at it. But wireless access through a network is a problem. Some of the machines we looked at in a report called Hacking Chads we found couldn’t even turn off the wireless. It was not possible to turn it off. That’s a security problem.
FEIST: Connie, if the legislature gave you $20 million, $200 million, what would you buy?
LAWSON: I have no idea. I need—I need the experts. But I could certainly be doing, you know, a lot of research. I would say, you know, the most important thing is education. For our local election officials. We—you know, in a number of states our governors have set up cybersecurity councils. We have one in Indiana and we’re working with our local elected officials. We’re running phishing email campaigns so we can educate them on, you know, what to notice, what not to notice, or what to click on—what to notice before they do that. we are working on multifactor access so that, you know, their passwords are stronger. So those are the things that we’re doing in the state of Indiana. And I think most secretaries are doing that as well.
But I would just say that the very first election I ran as an election administrator was in 1989 in Hendricks County, Indiana. And we used lever machines, which are pretty—they’re probably—there might be one in a state museum now in Indiana. So I’m dating myself. But I will tell you that it wouldn’t make you feel very well if you saw the way those results were taken in. You know, we would get a written total from the precinct. And you have a tally sheet. And I remember sitting on the floor, with this huge tally sheet, and numbers get transposed and you’re adding all these up. I mean, this—it was a disaster. (Laughter.) It really was. I mean, I think we finally ended up with a result that was fair and correct. But 2:00, 3:00 in the morning you’re still working on these paper tallies.
People are not that patient today. I think the worst thing that could do would say that we would have to go back to all paper. What we need to do is think about how we can make our technology work the way we need for it to work.
FEIST: If you were—if you had billions of dollars, what would you do?
BLAZE: So, you know, it’s funny. I’m in the one branch of computer science that has—most of my time is spent pointing out how terrible computer science is at building reliable things. (Laughter.) And we really are truly terrible at building reliable software systems. It is literally the first problem of computer science is, you know, we don’t know how to build programs that don’t have bugs in them. And that, you know, may at some point in the future, there may be some breakthrough that makes that less of a problem, but it has not yet happened. You know, arguably this problem is getting worse rather than better as we build larger, more complex systems.
So what’s the solution? Well, the best solution that anyone has come up with for elections is a concept invented by Professor Ron Rivest at MIT called software independence. That is to say, you know, we’re going to use software. It has all sorts of benefits to have computerized election systems. But we don’t want the integrity of the election to depend on the integrity of the software, because that’s simply a Herculean task. So the technology that exists today that has this property of software independence is a combination of two existing things that we can do today.
One is what’s called precinct-counted optical scan ballots, that is ballots where the voter marks a ballot, or maybe uses a ballot-marking device to create a paper optical scan ballot that’s fed into a reader at the polling place that records the selections and keeps a tally, and then captures the physical ballot and stores it in a locked box. And that technology has the advantage that it maintains an artifact of the voter’s choice, that the voter actually marked.
The second thing you need to do is make sure that the software that’s doing the tallying is—has not been tampered with or doesn’t have bugs in it. And that can be achieved with a technique called risk-limiting audits, where you do a statistical sample of the polling places, do a manual count of the paper ballots, and ensure that that matches the electronically recorded results. If it matches, great. If it doesn’t match, then you know you’ve got a problem and you have to do more of the recounts. You know, the combination of doing both of those things properly gives you this software independence property that just eliminates a wide swath of potential vulnerabilities that are really hard to counter in any other way.
FEIST: But why—I watched the British elections this summer, OK? Reasonably well-developed nation, the United Kingdom. Held an election for parliament this summer. They use paper ballots. They’re tabulated at each constituency. The people who voted for Candidate X, there’s a pile there. There are poll watchers that are checking. They count, the recount, they write them all down, and then someone stands at a microphone and reads off the results without ever touching a computer. The only one who seems to add them up are the television networks back in London, where they literally add them up and do the arithmetic, but that’s it. What’s wrong with that? Isn’t that—isn’t that foolproof? Why do we have to get all fancy? Seriously?
BLAZE: So, you know, there’s nothing wrong with that. But the United States has the—
FEIST: Are we just impatient?
BLAZE: Well, we are impatient. You know, we’re Americans, and, you know, we’re an impatient people. But the more serious problem is the U.S. elections are the most logistically complex in the world, right? We vote on more contests on a single ballot. We have more different ballots. You know, we have school board elections and the dog catcher election and referenda and, depending on where you are, you know, bond issues, and so on. So, you know, in England they’re voting for a—it’s a parliamentary democracy. They’re voting for a single representative in general in these elections, or maybe one or two issues. You know, here we might—you know, I vote on about 20 different things in Philadelphia.
FEIST: All right, Michael. You work for the Department of Defense. The word we have not said—Adam mentioned it earlier—we have not said the word “Russian” on this panel. But that’s the backdrop for this, at least right now. Do you believe that the Russians—or any other bad actors, but we’ll use the Russians for this—tried to hack our election, want to hack our election, are actively trying to break through all of Matt’s fancy systems? Or is this really just something that we’re—a problem we’re overstating?
SULMEYER: Do I believe that foreign intelligence services would love to gain unauthorized access, or hack, into systems that would reveal information? Absolutely. Would foreign intelligence services love to be able to gain access to systems to try to change tallies? I think in their dreams, they’ve love that ability. It’s hard for me to see a proposal being discussed in the Kremlin and the security services and they say, no, no, let’s let that one go.
FEIST: We don’t hack—we’re not trying to hack their elections, are we?
SULMEYER: I think that—who knows. (Laughter.) But the point is that I think it’s—that they may want to be able to achieve these outcomes is predictable and understandable. Being able to see the causation from intent to actually being able to realize an objective, that’s the tricky part. It’s not always that there’s an Dr. Evil plan hatched and then everything falls perfectly into place. It’s usually let’s see what happens if we try moving some pieces around the chessboard, right? Send a bunch of phishing emails, see who clicks, see what that reveals. Once you’ve gained unauthorized access to one system, what does that open up? A lot of times you have to play on the line of scrimmage and audibles, and it doesn’t always work according to a playbook.
FEIST: The voting machines—we’ve spent most of our time so far talking about the voting machines. But I just want, Connie, if you can walk through, for those of us who don’t count votes at a local and state level, just walk me through when if I go into—if I’m voting in Indianapolis, and I’m a voter, I go into a polling station, I push my vote on a machine. What happens between the time I vote and the time that the secretary of state’s website reports the total? Just walk me through the—who is in control of those numbers and how does the vote—the information about that vote move from my finger all the way up the line?
LAWSON: Well, once the vote is cast, obviously it’s up to the election officials—a bipartisan team of election officials at the precinct level in Indianapolis, to bring the results back to the county level. And then the county totals—
FEIST: And how do they do that?
LAWSON: They will do that depending the type of machine. On a DRE, it would be a recording device. I believe on the optical scan it is as well. I’m not as familiar with those. But they bring those results back.
FEIST: Is it in a key fob or do they write a number down?
LAWSON: It’s in a—it’s some sort of electronic device that they bring back. And then it’s run through a tabulation machine. There’s a machine that reads the device, the USB port or whatever. And then the precincts are totaled together. And then the counties call the results, fax the results, we call the counties—
FEIST: So they get a total—
LAWSON: We don’t—it’s not connected to the state in any way, each county.
FEIST: So a million, 200,000 votes in this county. And somebody—well, that’s a lot, but anyway—
LAWSON: That’s a lot. (Laughter.)
FEIST: Yeah. Anyway, and someone calls your office on a landline?
LAWSON: Yes. Yes. But the results are not final for 10 days. So, because remember, folks are able to cast a provisional ballot, if they got to the polling place, for example, and they forgot their photo ID. We have a photo ID requirement in Indiana. They have 10 days to go to the clerk’s office on the county level, take their ID, and the provisional ballot is then counted.
FEIST: But then someone on the telephone back in your office hears the votes and they type them into a computer and the computer does the math, and then it gets published on the website.
LAWSON: That’s right.
FEIST: And that’s how the world knows about it.
LAWSON: That’s how the world knows about it. But, again, the counties have the opportunity to do their audits. They make sure that the results are final. And they don’t actually certify the results to the state for 10 days.
FEIST: OK, so, Matt, if Michael’s friends in these foreign intelligence services are trying to make some mischief, what are the points of failure—we talked about the electronic voting machines. But what are the points of failure in any secretary of state’s system, from vote to report?
BLAZE: So I worry less about the secretary of state-wide system than I do about the counties. There are 3,000 counties in the United States, roughly. About 2,500 of them have responsibility for running elections. Which means that we have somewhere in the neighborhood of 2,500 to 3,000 different local election administrators. Some of them are quite good at protecting their systems. Some of them are less good. You know, there’s pretty wide variance among them. You know, and this has nothing to do with intentions or goodwill. You know, this is simply a matter of, you know, very widely different capabilities.
To the extent that our voting systems have been secured—and, you know, we’ve seen horrible—when we look, we see really horrible, exploitable vulnerabilities in them. But to the extent they’ve even been designed against a threat, it’s a threat of conventional corruption. Somebody trying to, you know, get themselves elected mayor or, you know, sell votes, or what have you. Nation-state adversaries were not even in the threat model that these systems have been designed against.
And so when you think about the capabilities of a national intelligence service, like the GRU or, you know, really of any country, they have capabilities that certainly include everything that a corrupt candidate may want to do, but also, you know, they’re going to have additional capabilities. They’re going to potentially do supply-chain attacks, where the equipment that gets shipped may be tampered with even before it’s received. They may do attacks against infrastructure that’s being used. So they have additional resources and capabilities.
But that’s actually not the most serious problem. The most serious problem is they actually have an easier problem to solve than someone who wants to cause a result to go a particular way. A state adversary may be absolutely satisfied with simply disrupting an election, casting doubt on the legitimacy of the result, causing chaos on election day. And that is significantly easier than causing a determined result. So they both have more capability and a wider range of things that may satisfy their goals.
FEIST: OK, so Michael, elections are run by states. They’re run by counties. They’re run in towns. Is this a policy issue for the federal government in the same way, after the 2000 recount mess, the Congress got involved and we had billions of dollars appropriated for new election machines that are now causing another problem? But is this a federal issue? And, if so, what should the federal government be doing to address this before perhaps the next presidential election in three years?
SULMEYER: The federalism questions here are thorny. There’s no doubt about it. I think that for the federal government to say federalism is too difficult, so we’re out—good luck to the states and locals—I don’t think the federal government can take a pass.
What I would like to see is some sort of a playbook, right, that the federal government be able to put together for best practices and counsel. My colleagues at Harvard put together a playbook for campaigns. I see no reason why the federal government couldn’t provide an updated playbook for states and local authorities on these issues as well.
FEIST: So we saw at the end of the 2016 election, or late in it, the Department—President Obama’s Department of Homeland Security seemed to get involved in this game in a different way. Do you feel like the current President Trump’s Department of Homeland Security is interested in this issue, as interested in this issue, taking a leadership role, continuing Obama’s? How do you see them playing right now?
LAWSON: Well, there’s—they’re playing a huge role. Department of Homeland Security is working with the secretaries of state and other chief elections officials on a number of items.
First of all, every chief state election official is going through getting their security clearance so that we can get up to secret, not top-secret but secret, information. I—for example, I’ve just received my interim clearance. Then the tier two of that clearance will be our staff, the staff that needs that information. That’s number one.
Number two, we created a Government Coordinating Council so that we can determine what level of—what the definition of the critical infrastructure means for states. So we have the Government Coordinating Council, which is a large group of NASED people, National Association of State Election Directors, the secretaries of state, the local election officials, the Election Assistance Commission, the DHS, NIST. All of those agencies are involved.
And we’re talking about all sorts of things; for an example, communication. We’re—there are seven pilot states that are—MS-ISAC, Multi-State Information Sharing Analysis Center, out of Albany, New York, is giving these states monitors so that they can be monitoring internet activity on our election systems. Not all election systems are on the state system. And so it becomes a little more complicated than what you might think. And so we’re doing the seven pilot states, and hopefully by the primary of 2018 every state will have a monitor on their internet activity so that we can be informed of that.
FEIST: Michael, do you have a sense that the Trump administration is making this a priority? Are they doing what they need to do? Or, you know, how would you grade their efforts so far, knowing that we’re still three years away from a presidential election?
SULMEYER: It’s hard to tell on an administration level. I mean, just hearing some of the specific DHS people talk about how they want to be helpful, I think those folks who are in the specific offices that would work on this, they certainly see it as a priority.
The question about how fast they can push through clearances, I’d rather they be able to do a drug deal with the rest of the intelligence community to just declassify certain information and call you rather than work through an entire government process to figure out how to give clearances to everybody. But life’s imperfect.
FEIST: What do you think? How would you grade the federal efforts? And what should the federal government be doing now, while we have time?
BLAZE: Well, you know, first of all, we’re three years away from a presidential election.
FEIST: One year away from a—
BLAZE: We’re 11—right. We’re 11 months away from the midterm elections. So there isn’t, in fact, a lot of time for the next nationally significant election. There are extremely capable people in DHS and NIST. You know, obviously I can’t speak to the administration’s posture on this, but certainly there are very, very capable people who need to be empowered to assist.
FEIST: Do you think they’re empowered?
BLAZE: I have no opinion.
FEIST: OK. On that note, I want to open it up to questions from everybody here. So raise your hand and then wait for a microphone. The microphone will come your way. And then also say your name and your affiliation, please.
Q: Hi. Russell Wald with the Hoover Institution.
I have a policy question that’s probably better for a later panel. But since we have everyone up here, I’m curious. Given the ubiquitous nature of the vulnerabilities, it seems to be that deterrence by defense does not seem really feasible.
So this question is probably more likely for you, Michael. What is a good policy that says to adversaries U.S. elections are sacrosanct and the risk is too high for another nation-state becoming involved?
SULMEYER: The challenge is, if you’re going to leave bags of money on the lawn overnight, and then try to deter and talk tough about, hey, don’t come take that money, and then you’re stunned the next morning when the money’s gone, deterrence is not quite the model there.
I think there actually is a lot that we can do across the board to just bring the money inside, and forget locking the door for a minute. No defense is perfect. I completely agree, right. But before you talk about deterring and imposing costs, I think you do have to look at are we really just talking about not wanting to put more resources into architecture and very unsexy things that would actually make it much harder to hack?
FEIST: In the back. Yes, ma’am.
Q: Great. Thank you. I’m Elmira Bayrasli. I’m the CEO of Foreign Policy Interrupted.
We haven’t talked about what happened—we’ve been talking about the actual voting, but we haven’t talked about what happens before the voting, and particularly Facebook and social media and how people are influenced by—
FEIST: So I’m going to just push pause only because that’s the next panel. So we’ve been—this panel is really focused on voting, voting systems, and such. And the information war is—
Q: But I do have a question for Connie, because I think that this is something that, not only on a Facebook level or social-media level, but what are states doing? Are they looking at this? Is this something that states are getting involved in?
LAWSON: In the information—
Q: In looking—in looking at the influence of how elections are being influenced.
LAWSON: No. I don’t think the states are involved in that. Obviously, we’ve known for years that foreign nations have tried to influence people’s opinion here in the United States regarding candidates and how they should vote. And so, I mean, obviously we do voter outreach. We encourage the accessibility of our voter registration, our, you know, accessibility to—I think Indiana may be the only state in the country that has an app. It’s called WAYEO, Who Are Your Elected Officials? And you can find out from school board to president how to contact your elected officials.
So, you know, we’re doing everything that we possibly can for people to get the correct information, but I don’t have control over Facebook or Twitter or someplace like that that puts out the wrong information.
FEIST: Yes, sir, in the third row. Just wait for the microphone. Thank you.
Q: Mike Mosettig.
In addition to security, the other election issue going on at the moment is suppression. To what degree does the risk of technology or technology-imposed risks in terms of wiping people off registration rolls and some of these other things that seem to be going on at the moment? I mean, in other words, before people even get to the voting booth.
FEIST: So this is actually—we haven’t talked about—we talked about the voting systems. We haven’t talked about the voter-registration rolls. So let’s talk about that for just a moment.
Either. Go ahead. And how safe—how safe Connie’s systems are? I’m sorry, you happen to be here, so you’re the example. But how safe Connie’s systems are so that when someone walks in to the voting booth their name is on the roll and they’re actually allowed to vote, that a mischief-maker hasn’t erased my name before I get there?
BLAZE: So, I mean, you know, this—I have no idea what’s going on in Indiana. You seem great—(laughter)—and I’m sure your systems are terrific.
FEIST: Pick Colorado.
BLAZE: But, you know, this is definitely a point of vulnerability, particularly for a nation state interested in disruption. In many states and many jurisdictions, the poll book at the polling place is an electronic device. Those devices are often—often have security weaknesses in them. If your name isn’t there, you’re, at best, going to be casting a provisional ballot.
FEIST: How often are, for either of you guys, how often is there a paper backup? If I walk in, even though the poll worker, you know, use an iPad to check my name, does he or she have a book under the table in case everything goes haywire? Likely or not likely?
BLAZE: There are 2,500 different answer to that question in different counties.
LAWSON: They should, yes. Yeah. Let me just say that when we were notified last fall before the election that there were two IP addresses that had been responsible for getting into the Illinois voter registration system and getting into a small county in Arizona that had actually then had allowed that IP address to have access to the Arizona Statewide Voter Registration System, so we checked our voter registration system from January 1 until that date.
And so, you know, Indiana has 92 counties. We have 6.7—I see Professor Mike McRobbie is here—we have 6.7 million residents in the state of Indiana, 4.8 million registered voters. We checked 15,500,000 logins into our system. And the reason we had to check that many is because that’s how busy the counties were. They were looking at petition signatures. They were, you know, registering voters, candidates were filing their declarations, all these activities, absentee ballots, everything was going on. So that’s how busy these systems are. And that’s why states are looking at things like multifactor access into the Statewide Voter Registration System.
We are implementing a timeline. So, for an example, if it’s after midnight, maybe it’s just the supervisors of the elections that have access to the Statewide Voter Registration. All those things we’re looking at.
FEIST: In Indiana, though, if I were to go vote, before I push that button in Indianapolis, if I walked up and their iPads quit working, is there a paper book that—I keep going back to paper, paper sounds so sophisticated right now.
LAWSON: Yes. There is a—they would have a poll list, yes.
BLAZE: So, you know, I’d just like to add one thing to that. You know, the data breaches, on a large scale, are literally daily events, right, they don’t even get reported unless they’re on the scale of Equifax or the Office of Personnel Management.
The only reason we haven’t seen—if we have not yet seen a large-scale data breach of voter registration databases, it’s because no one has seriously tried. You know, the individual states and counties, whatever their best efforts are are going to be no better than what the Office of Personnel Management or Equifax or any of the long list of equivalently complex systems that have been, you know, catastrophically breached are.
And, you know, we may be in a honeymoon where it hasn’t happened yet, but it’s only a matter of time.
FEIST: On that comforting note, let’s take another question.
Yes, ma’am, right here in the second row. Just wait for the microphone, please, and state your name and affiliation. And here’s your microphone.
Q: Oh, here we are. Hi. Can you tell me how many states have that commendable independent system that you talked about or how many are working on it? And conversely, which states, in your experience, are the most vulnerable?
BLAZE: So there are a few states that are using both exclusively precinct-counted optical scan plus risk-limiting audits. Virginia just decertified all their DRE machines. They have risk-limiting audits, but I learned recently that they actually happen after the certification period rather than before, so there are some adjustments that need to be made. Colorado has made significant headway there. So there are a few states that are starting to pick up on this, but they’re the exception rather than the rule.
LAWSON: But I will say that NASS, the National Association of Secretaries of State, have a winter meeting and we’ll be talking about risk-limiting audits. And we’ve got the Belfer Center coming to talk to us about tabletop exercises and incidence response and cyber. I mean, all of those things are on the table. It’s not like anybody is ignoring those. So I want you to know that those have been a definite priority of chief election officials prior to 2016, but it’s been a heightened priority since that time.
FEIST: Michael, when you go to an event like that, the Belfer Center, for example, what is it that—what’s the most important thing you’re telling the secretaries of state? What are you saying, either that you have to do or you’re trying to scare the daylights out of them? I mean, what’s the message?
SULMEYER: Well, fear is always a great motivator for pretty much anything. But I think in this case, there’s enough awareness, you know, among the secretaries of state you don’t need to really do that. You’ve got to give news you can use. I think everybody now is geared up to the reality of what’s at stake. So the most helpful thing that I’ve seen my colleagues bring to the table is, again, something like a playbook, something like an implementable, actionable best practice that the secretaries and other colleagues can use.
FEIST: Yes, sir, second row. Wait for the microphone, name and affiliation.
Q: Good morning. Adam Ghetti with Ionic Security.
I’ve spent the better part of the last six years of my life doing nothing but data security and data integrity for some very highly targeted clients, including the federal government. One thing we’ve been discussing all morning is the security of the voting systems and the voting process. But the intent should be that the constituencies of our democracy trust the results. Ultimately, that’s the goal, right?
So while there’s a lot that can be done and is being done to ensure the security of the system, security of the process, what I haven’t heard discussed this morning is, is there a way to have a common, private, only-voter-verifiable and reconstructable audit overlay on top of the results that can ensure the trust and integrity of the outcome of all of the things, such that any of the lack of integrity and the rest of it may not necessarily cause the outcome that a nation state adversary might be seeking, a la voter disruption or changing of a candidate? The audit is independently verifiable.
And I know, Matt, really, particularly in your case, I didn’t hear you mention anything about secure multiparty computational overlays or oblivious pseudorandom function overlays to do some of these things. So I’m interested as to why.
FEIST: Just do it in English. (Laughter.)
BLAZE: Yeah. So, you know, the basic problem is that the systems that do that are extraordinarily complex and they essentially make our elections more dependent on the integrity of underlying software systems, particularly as part of the vote-casting process. So you may actually have the effect, when you look at the overall usability, of decreasing the confidence in our elections if the vote was counted rather than increasing them.
The other problem is it’s a really heavily over-constrained problem. We have—you know, we want elections to have transparency and we also want them to have a secret ballot. We want it to be impossible for somebody to learn how someone else voted. We also want it to be impossible to prove how you voted, right, because, you know, we don’t want people to be able to be coerced into revealing that.
That’s a pretty difficult set of things to achieve. Instead, you know, I think we have to rely on making systems simple and publicly auditable, you know, with processes that include, you know, the chain of custody of the ballot, you know, a public ritual where we do the risk-limiting audits and so forth. And in practice, that’s likely to do much better than any fancy cryptography. And I say that as a fancy cryptographer.
FEIST: Matt, just—you’ve used the term risk-limiting audit a few times. So just for those of us who don’t live and breathe this stuff, just explain that a little bit.
BLAZE: So the basic idea is that you—once you’ve captured the paper ballots and electronically counted them, you want to make sure that the software that counted those ballots hasn’t been tampered with or doesn’t have bugs in it that’s reporting an incorrect result.
So you sample the precincts and the various races, do a manual recount in every race of a statistically significant sample and verify that what you hand count matches what you’ve electronically reported.
FEIST: Do it every time.
BLAZE: And you do that every time.
FEIST: And you do it every time before the votes are certified by the secretary of state.
BLAZE: That’s right. And if you discover a discrepancy, then you have to do more hand counting.
FEIST: Is this happening in Indiana? And if so, how does it happen and what triggers it?
LAWSON: We don’t do risk-limiting audits right now, but we have our VSTOP, our Voting System Technical Oversight Program, working on that right now. We’ve had conversations with Colorado. And I just heard on a call this week that New Mexico is working on that as well. So we are going to be doing that. That’s something I definitely support, I think we should.
I will say that after the 2016 elections, we’ve had a congressional district and a state senate race, both were in a recount. And we recounted an entire congressional district. And the results were the results.
FEIST: Exactly the same.
LAWSON: They were the same.
FEIST: Exactly the same. People in Atlanta may be calling you for advice if it’s still 729 votes when we finish this session.
Yes, ma’am, in the very back, go ahead.
Q: Hi. Astri Kimball from Google.
When you talk about the federal policy responses, do you think the United States is doing enough to invest in training the workforce of the future to design the systems that are going to be the most secure? And what can the U.S. government do to get the workforce to do this?
LAWSON: I can’t answer.
FEIST: And, Michael, you want to take a stab at that? You’re Mr. Policy Guy.
SULMEYER: Thanks, Astri. Great question.
No, the government is not doing—I don’t think the government is doing enough. It’s not that, though, there’s ignorance about it; I think the biggest problem is that there’s no singular set of skills that will solve everything or equip everyone to do everything technical.
There’s a lot of this kind of information that you can learn just factually, dare I even say free YouTube videos to just learn about how different systems work.
FEIST: You said that for her, YouTube?
SULMEYER: Yeah, YouTube, yes, exactly. Yes. So I think, you know, there are some types of skills that, yes, require a large type of federal investments to steer people towards, but that actually doesn’t need to be the way to really solve the STEM crisis as a whole.
FEIST: Question? Yes, sir, front row.
Q: Alan Raul, Sidley Austin.
First, a comment on the thorny question of federalism as a possible constraint on the federal government’s responsibility here. I would note that Article 4 of the Constitution obligates the federal government to guarantee a republican form of government to every state. So if the elections are compromised, I think that would be called into question.
But my question concerns—the discussion has centered around whether there’s any evidence of impact on U.S. elections in terms of hacking the voting or the results. What do we see internationally? I mean, there are a lot of other elections held and presumably many of them are subject to electronic procedures as well. And the nation state adversaries might have an interest in chaos, uncertainty, and maybe even in installing particular candidates in other countries. Do we see any evidence around the world that’s relevant to us?
BLAZE: So probably the largest country that uses electronic voting systems is India. They have a custom-designed voting machine. Questions have been raised about the security and integrity of the design they’re using. It is a paperless DRE system. But, you know, to a large extent, the U.S. has been on the leading and bleeding edge of rolling out computerized election technology post Help America Vote Act. And so, you know, I think we’re seeing—you know, we have to look inward as well as outward to see what’s going on.
And I will also say that the situation very much reminds me of internet security in the 1990s when, you know, technologists were basically warning that, you know, these systems that we have on the internet, in general, are going to be—you know, are insecure and going to be attacked. And for a while, you know, people were saying, oh, you’re just, you know, Chicken Little saying the sky is falling. And then, you know, sure enough, the sky fell and, you know, we really haven’t been the same since.
The situation with electronic voting systems is remarkable reminiscent of the situation with general internet security in the mid-1990s.
FEIST: The example you mentioned that there were issues in Illinois and Arizona in the last election, did we ever actually learn who the hackers were, who was trying to penetrate those systems? Have we actually—has that actually been discovered and announced?
LAWSON: I have information on that, but I don’t think I can say.
FEIST: Oh, you don’t have your clearance yet. It’s OK.
LAWSON: I don’t have my clearance, yeah.
FEIST: Just kidding. Sorry, whoever is watching in Russia. (Laughter.)
LAWSON: Yeah. No, I can’t say.
SULMEYER: It’s a question by Alan. And I think, A, you had me at Article 4 of the Constitution, that’s good. And then in terms of where else, you know, you can look at where are things vulnerable, which is India, and then you can look at where is an attractive target for people who want to be in this business, and this is Eastern Europe. I mean, the Russians have an interest. And so not surprising that they would be poking around.
I think if there are some aspiring grad students who are watching, a good master’s thesis, you know, could be trying to do some comparative studies in the Baltics, looking at Ukraine, some recent elections, to go get some travel support money and investigate.
FEIST: So we’re not done yet, but I have yet to be convinced that any of these systems are superior to paper, so far. I mean, this is—as we have been talking for the last 45 minutes, paper sounds better and better and better. I mean, the idea that many or all of the systems in India are DRE systems in the world’s largest democracy, anyway, I’m troubled by it. But that’s neither here nor there.
Yes, sir, in the middle.
Q: Thank you. Fred Roggero from Resilient Solutions, a drone safety and security company.
I’d just like to ask the panel questions, either, about, if there are similarities from the financial markets and cyber precautions we take there or the coming autonomous transportation markets that we’re going to be looking at? Are there any lessons learned from those that could be applied to the voting cyber issues?
SULMEYER: Well, for my money, the financial sector, I think, has invested the most for the longest amount of time because they realize they had money to lose, right? So they took it upon themselves to defend themselves. And I think the federal government has actually had the best type of relationship with an information-sharing-analysis organization that is managed by the financial sector. So there’s definitely some good lessons to be learned there.
The question is, though, you start getting across different industries, what’s been the government’s role in requiring different kinds of transparency, reporting about intrusions, right? There is some defense authorization language requiring defense contractors to report to the military when there have been certain kinds of intrusions. We can start to think about how transparency and reporting can lead to better practices going forward.
BLAZE: You know, I’d also point out that in the financial industry there’s a really straightforward feedback mechanism that tells you how much you should be spending on security because, you know, we know how much we stand to lose. And you can do pretty straightforward risk calculations that tell you what your budget for a, you know, given exposure is.
In the case of, you know, the integrity of elections, that feedback system doesn’t really exist. So, you know, unfortunately, we spend far more on election campaigns than we spend running elections themselves. And, you know, we have—most election operations are within counties. The budget for voting machines and running elections competes with the budget for fixing roads and building fire stations. And so, you know, probably the most important lesson we can take from this is we need to think about how much we value the integrity of election systems in understanding, you know, how much work to put into this.
FEIST: Connie, you mentioned that Indiana has invested some money recently, which is, I presume, a good thing. Do you sense that the other 49 secretaries of state are in a similar situation to you? Or are more of them underfunded or not funded either at the state or the county level? Is this really where the rubber meets the road?
LAWSON: I would say I’m very fortunate as a secretary of state of Indiana to have the support of the general assembly. I don’t know that it’s common practice across the states to get the appropriation that I was able to get to modernize the system. But I think, as the attention continues to be drawn to these issues, that the states will step up and fund. But, obviously, there are some folks who would like to see the federal government step up as far as the funding goes as well.
FEIST: Article IV.
Q: Dee Smith, Strategic Insight Group.
I guess this is really a question for Matt. To what extent would we know if these things have occurred? I mean, there are cases of, you know, system intrusions hiding out for 500 days in the—in the corporate sector. And that’s question number one.
And question number two is, if we put in all the various things we’ve been talking about, are you convinced that that would secure against this sort of thing?
BLAZE: So the answer to the first question is it depends. Unfortunately, in a lot of these systems the audit trails are just as vulnerable as the other aspects of the system, so there may not be good forensic evidence of a successful intrusion. In other cases there may be signs. We saw in 2016, certainly, there were indications of attempts.
And, you know, I would say that we can’t—with the current design, we cannot be universally confident that it hasn’t happened. And it’s probably only a matter of time before it will.
The combination of risk-limiting audits and, you know, an optical-scan paper artifact of the voter’s record gives us a pretty good assurance, you know, within a statistical certainty, that the count of votes cast is accurate. It doesn’t help us with the other piece of that, which is the disruption piece, voter-registration systems, and so on. Those we have to do the same hard thing that we do with any other online system: training, put resources into it, keep systems up to date, monitor them, and so on. And, you know, we just have to love them enough to pay enough attention.
FEIST: Yes, in the back. The microphone’s right beside you.
Q: Thank you. Tim White, Spectrum Group.
Is there a critical mass of public concern about the issues that you have been explaining this morning? Or, in fact, does the public, generally, either not care, or believe that the tradition of electoral systems in this country have always kind of been nibbled around the edges and, well, this is just another way to do it? Because without that critical mass of public awareness—not press awareness, and not an awareness on those involved professionally in the process. But do people really care that much?
FEIST: What do you think, Michael? No?
SULMEYER: I don’t—you know, I wish there was a broader mass. It’s stunning, I know, to most I guess who were at Harvard; I guess the rest of the world isn’t like Cambridge, Massachusetts. There, everyone’s focused on it. (Laughter.)
But I’m concerned also about a leadership deficit on this topic. No one wins political points by talking about it. This is not a partisan thing, but you need a leadership level of it to generate that mass. It can be bottom-up, but it could also be top-down. So, to talk about how an election is rigged before the election, right, is difficult when you’re trying to actually improve people’s confidence and get to ground truth on this.
LAWSON: I think they need to—you need to come to the Indiana Secretary of State’s Office and answer the telephone, and you would understand that people are concerned about it. We get calls every day.
FEIST: What do they say?
LAWSON: You know, what’s going on? I just read—I just read in the paper about, you know, the Russians hacking the state of Illinois. Is Indiana good? Is there voter registration fraud in Indiana? You know, just whatever is in the news. We get calls. We get a number of calls. I get a report every week on the types of calls that come into the office. And this is an off election year, and I will say I have had just as many calls regarding the security of our elections this year, this off election year, as I have ever had. I mean, I’ve had more. I said just as many; I’ve had more. So I think the public does care.
FEIST: Front row.
Q: Audrey Kurth Cronin from American University.
Following up on that question, is there not a sense that there is a politicization of this issue, where some people feel that concern about elections reflects more upon a particular type of election, or our latest presidential election, or a particular election locally? How can we truly remove this issue from the partisanship and the polarization right now that we have domestically?
BLAZE: Yeah, I mean one—I think we’re in a rare moment where it’s more bipartisan than it has been in the past. You know, in every election there’s a most recent loser, and the risk is that it looks like the people complaining about it are simply upset about their candidate losing. You know—but, you know, after the 2000 election, we saw bipartisan interest. It led to the passage of the Help America Vote Act, which arguably, you know, had some significant problems with it, led to some bad technology, but it was at least a bipartisan effort. I think we’re—I’m optimistic, which is rare for me, but I think we are at another one of those moments, or approaching another one of those moments.
LAWSON: You have somebody over here that’s been raising his hand for a long time. (Laughs.)
FEIST: Yes, sir. Thank you.
Q: It’s actually quite an unimportant question with the time. Edwin Williamson from Sullivan & Cromwell.
I just wanted to go back to the—to the comment that was made a little earlier about the great vulnerability of data breaches. Are these just information theft issues? In other words, is the risk just a risk of loss of privacy? Or does the—in the data breach, do you also get an ability to manipulate or disrupt?
BLAZE: So the goal of particularly a nation-state adversary who attempts to breach a backend system, particularly one with voter-registration information, is—would be disruption. And so they’re likely to want to, first of all, ensure their future and continued access to the system. They may want to delete—they may try to delete legitimate records. They may want to add spurious records to make it appear that there has been widespread registration fraud, disenfranchise selected voters in some way that advantages them, or simply, you know, cause havoc and delete everything, or cause systems not to be ready on election day. So, you know, particularly when we look at this from the perspective of a rival intelligence service attempting an information operation against us, we have to look at a very broad spectrum beyond merely leaking data.
LAWSON: I would just like to add that in June, at a U.S. Senate Intelligence Committee, one of the undersecretaries of DHS stated that 21 states had been targeted. And so that was a surprise to all 50 states because we had not been told that we’d been targeted. And so the language that we use is so important, because what does “targeted” mean? And so then just this last week, I believe, Chris Krebs from DHS said that you need to remember that just because someone targeted or tried to get in state data doesn’t mean that there was an actual breach, and that when we use the word “targeting” it really meant they were scanning, they were trying to get in.
And so it made us all very concerned. Some states—I mean, Indiana was not a state that was targeted, even, but it took three or four months after that to find out the 21 states that had been targeted or scanned. But we know of no additional states, besides Illinois and Arizona, who had been—had gotten a breach.
FEIST: Make sure I didn’t miss anyone over here. OK. Yes, sir.
Q: David Martinez, State Department.
I agree with the moderator that I fail to be convinced thus far that the safest and most secure way of guaranteeing the integrity of the results would be a return to paper ballots. But I also worked briefly in state government for my home state of New Mexico, where I was able to observe polling in practice and saw that that also presents some vulnerabilities. But I was wondering if, Connie, you might be able to expand a bit on what some of the costs—either political, logistical, or financial—of a return to paper ballots would be. I think you made a very important point about how, in a 24-hour news cycle, we simply don’t have the patience to wait for that counting. So that may be a political cost. What are some of the other costs of a return to such a model?
LAWSON: Well, I can’t imagine. Obviously, I think it’s a balancing act. You know, we need the technology, but we also need to verify what we’re doing. And I couldn’t answer the exact question regarding, you know, the financial impact, but it would be—it would be a large one, I would think. I know the hours that it took me to—and my staff to count that first election that I ever administered, in Hendricks County, Indiana, and that was a small special election. So I can’t imagine what it would be in a county like Marion County, Indiana, for an example, our largest county—in Indianapolis, Indiana—can’t imagine what it would be. They count their absentee ballots centrally, and they have over 300 teams of people, bipartisan teams, who count just the absentee ballots. So can you imagine what it would take in addition to that to count the actual paper ballots?
FEIST: At the end of the day, paper ballots may make it more complicated for the Russians, but they don’t necessarily lead to a more accurate count, or do they?
BLAZE: So when you say paper ballots, I mean, I—
FEIST: Going back to the Stone Age, as—
BLAZE: Well, so you’re talking about 100 percent hand-counting.
FEIST: The British system. The British system.
BLAZE: One hundred percent hand-counted ballot(s). So, I mean, I think everybody in the, you know, election security world is pretty uniform in advocating paper ballots. The question that we’re talking about is whether or not they’re completely hand-counted. You know—
FEIST: As opposed to the optical scan with a paper backup.
BLAZE: As opposed to—right. So I think, you know, we need—optical scan has the benefit that the—you get some assurance that your ballot is being fed into the system and a record of it is being made as soon as you submit it, and that’s an increase in integrity. The problem is we are also now dependent on software. And so that’s why you need the risk-limiting audit backup behind that.
FEIST: Right. So we’re just about out of time. We are at the Council on Foreign Relations, where we always end on time. I want to try and end with a simple question and as close to a one-word answer as possible. I’ll even give you your choices. Looking ahead to 2018 and 2020, are you optimistic or pessimistic that the system will be materially safer than it was in ’16? I’ll start with Matt, and then I’ll come down.
BLAZE: I’m convinced that the threat actors will be emboldened.
FEIST: Mmm. (Laughter.)
LAWSON: I’m optimistic.
FEIST: Well, all right. Perfect. (Laughter.)
On that note, thank you very much. Thank you for coming today. (Applause.) The next panel, if you’re joining, starts at 10:00. So thanks very much.