The Hacker Mind Podcast: Reverse Engineering Smart Meters
May 13, 2022
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
After hearing a talk, a Dallas-based hacker set out to find out what was going on inside the smart meter attached to his home, and what he found was surprising.
Since then Hash started a reverse engineering wiki site called Recessim and created dozens of YouTube videos in a channel of that same name to chronicle his adventures. He joins The Hacker Mind to talk about his journey, about mesh networks, and even glitching. Like any true hacker, this isn’t his day job; this is his passion.
The Hacker Mind is available on all podcast platforms.
[Heads Up: This transcription was autogenerated, so there may be errors.]
Vamosi: Beginning over a decade ago, in an effort to deal with global warming, countries around the world push to adopt emission control regulations. Environmental effects caused by pollution. This included the rapid adoption of smart meters. For example, in 2009, the Obama administration provided financial incentives to utilities in the United States. It's estimated that within the first year, over half the homes in the United States had spark meters installed. Similar initiatives were undertaken in Australia, India and Japan, all with the intent to roll these devices out within a relatively short amount of time. And again, smart meters were positioned squarely as making the environment more friendly by knowing how and when energy is being used by individual customers.
UK: To transition to a renewable energy system, we need to be smarter in the way we use our energy in our communities and in our homes. Smart Meters provide the data to support a cleaner future, showing how much energy is being used and how much this will cost you. Smart Meters open a whole world of new, useful insights connecting the energy we generate from renewable sources to the way we use energy, making our city cleaner, greener and healthier. We need to install Smart Meters now to make this happen.
Vamosi: But as someone who wrote a book questioning the security of our mass produced IoT devices, I wonder why no one bothered to test and certify these devices before they were installed? I mean, on the one hand, we're saying it's a national priority. And on the other hand, we're saying security, that's a secondary concern. Part of that was in thinking that these were embedded systems. It will be hard for anyone off the street to come along and flash them with malicious code or otherwise exploit any existing vulnerabilities. They were designed to be left into the field for years and otherwise not maintained. But we all know how security by obscurity works in the end. So I reached out to someone who shared that curiosity in wanting to know what was possible with smart meters already in our homes. And in a moment, we'll hear how he took it upon himself to learn all that he could about smart meters.
Vamosi: Welcome to the hacker mind and original podcast from for all secure. It's about challenging her expectations about the people who hack reliving.
I'm Robert Vamosi. And in this episode, I'm going to be discussing the world of reverse engineering hardware. And what it's like to hack sensors such as a lidar, or even a smart meter. It's all about the journey. And it's all about the hacker mindset.
Vamosi: Electricity, we probably don't think enough about what it means to have power in our homes, in our businesses, in our lives. And yet, everything we do requires electricity, either AC through the wire or DC in the form of a battery when it's gone. Well, that's probably when we think about it most. In February 2021. During a harsh winter storm, the state of Texas experienced massive blackouts. Here's ABCs Good morning. America.
GMA: Part of the Lone Star State hitting record low temperatures not seen in a century. And people are cranking up the heat, which is taxing the state's electrical system, leading to cascading power outages and rolling blackouts this morning, more than 3 million waking up without power. The cold weather and loss of power turned deadly. At least 20 people have died, including an eight year old girl and a woman dying from carbon monoxide poisoning after a car was left running in a garage to help create heat.
Vamosi: I've spoken before in other episodes about how truly massive blackouts are unlikely within the United States because of how the electrical grid is divided up and how states can borrow power from other states. But in this case, this was Texas and Texas is special. Here's Chris Cilizza: from CNN to explain.
Cilizza: Now the primary fall guy and all sides seem to agree is responsible here is the Electric Reliability Council of Texas also known as ERCOT. It's an independent organization that operates the vast majority of Texas power grids and the way it's never been to have reliability in your name if you're not reliable
Vamosi: So basically the state of Texas couldn't generate its own power, nor could they borrow power from other states. That's because of the way Texas established itself as independent from the rest of the country. So these are again, that understand
Cilizza: what's happening right now in Texas and who's really to blame you have to go all the way back to 1935 when President Franklin Roosevelt signed the Federal Power Act, which govern electricity sharing and sales between states basically what the Act did is it allowed the federal government to regulate states who brought power in from outside their state lines as it is bought and sold during times of need. So Texas, never found a federal intrusion of any sort set up its own power grid system split between the northern and southern parts of the state to avoid any federal involvement that eventually led to the formation of ERCOT 1970. This is a strange fact. There are three power grids in the United States, the Eastern power grid, the Western Power Grid, and Texas. Yes, you heard that right. Texas has its own power grids because well, it is Texas.
Vamosi: So this is great when it works. But when it doesn't, well, it doesn't.
Cilizza: Reality is that Texas is an electricity Island, which isn't a problem. Until the lights go out. You don't have enough power in the state to turn them back out, which is where Texas finds itself. There's no question that ERCOT itself does bear some blame here too, when your only job is to manage the power grid and that power grid fails miserably. That's the problem.
Vamosi: So all this detail about the power grid in the state of Texas is background for this episode. Important background?
Hash: Yeah, so I mean, my name is Hash Salahi. I just go by hash for the most part because it's the easiest thing to remember. You know, a lot of times it's security researchers or things like that, this puppet on it. You know, I probably reverse engineer would be the one that I would like, I just enjoyed that as a name for kind of what's what's being done because it's broader. it's broader than just security and it's, and it's kind of just to discover how something works and maybe there are security implications that come from understanding how something works right.
Vamosi: Hash initially didn't start out hacking smart meters. He started out small with a sensor in a robot vacuum cleaner.
Hash: I think it was because it was posted on Hackaday. The challenge showed up. And so I saw this challenge. And so I didn't have anything to do with LIDAR before that. I've always liked robotics, you know, like short circuit and things like when I was a kid, I thought I'm gonna be a robotics engineer when I grow up, you know, after seeing that movie, but it got posted on Hackaday and I had seen the Xbox controller one and I felt like that was a little outside of my capability. I've always been more embedded, you know, you are at serial ports, things like that. And the Xbox One seemed like a higher level software type hacking and I wasn't quite comfortable with that level of software. So when that came out, I I didn't think anything of it when the vacuum one came out. I thought I could do this like I could buy it. I had the money to buy it. If you if you want the challenge you want $400 which was the cost of the robot so I thought if I can when the robots free, and so money was tight enough for 400 bucks, like I had to think like okay, tell my wife I got a young kid, like how am I going to balance this thing, right? And, and then I ended up I ended up doing it, you know, kind of figuring it out and in the process of figuring it out and being more involved in this kind of community online that was into robotics and stuff. That kind of sent me down that path of like, hey, maybe it'd be more fun to reverse engineer this, but it was just like happenstance, like I wasn't in the light RS or anything I just saw that challenge is like, I think I think that's in my wheelhouse. Embedded microcontroller. These other things were uncomfortable, so I'm gonna give it a go.
Vamosi: Neato is an autonomous vacuum. cleaner that uses sensors to map out your home. It uses light detection and ranging or LIDAR in particular.
Hash: And my first experience with that was when I reverse engineered this LIDAR that was on the Neato. There's a Neato robotic vacuum. And there was a LIDAR module on top of it and it was right around the time that the Xbox, that motion controller for the Xbox got reverse engineered so that people could use it for things other than the x box. The robotics community put out a bounty to reverse engineer this LIDAR module because the robot was $400. And normally, those LIDAR modules were like eight to $10,000. So it was a low cost way for people to use LIDAR which is light detection and ranging like to map rooms. And so I ended up reverse engineering that and winning the challenge and so there's some articles online about that. And, I created a wiki just to post as I was reverse engineering this robot and I used wiki spaces, which is some company that's out of business now or whatever. And like 100 people join this thing, and everybody started putting in stuff about what they were doing and reverse engineer the robot. And then the company went out of business.
Vamosi: Wiki space, got bought out and then shut down in January of 2019. So Hash took it upon himself to continue.
Hash: When I created this, I'm gonna create my own, I'll host it myself. I'll build my own media wiki so that I'm not beholden to some other company that might go out of business. And I'm using a free service. I'll just pay for it. And I'll let other people post for free. I'll kind of build that thing up myself. And see if people join and that's what I've noticed. Now more and more people are joining and sharing their projects and either working on smart meters or just sharing whatever they are reverse engineering.
Vamosi: So it began as a challenge posted in Hackaday and curiosity to learn more about LIDAR. That's how hash started down the road. of reverse engineering. The site he created is called Recessim.
Hash: The challenge is finding a dot-com that's available, which I think is all our challenges. So what we're Recessim. It's a Latin word for moving in reverse. So reverse engineering. I looked for something that you know, that was kind of unique and different. And so my idea with that as a channel name, and kind of it started also as a wiki was a place that everybody that's working on reverse engineering stuff could have a spot to, to post things, and to share it. I noticed it seems like it's very fragmented. Everybody's got their own little blog where they post stuff. And there isn't like a Wikipedia of reverse engineering, but it's super beneficial if you can see all that in one place or see what people are working on.
Vamosi: It is true that embedded security has been underground. Yeah, there are tracks at conferences, but it hasn't yet exploded into the mainstream. So Hash has been collecting the knowledge that he's gained, and sharing it with others.
Hash: Like I've always been interested in taking stuff apart and working on things and way back when I was younger, you know, I buy like old cell phones from surplus places and try to take it apart and understand it and I had an amateur radio license when I was really young, as well. And so I was always kind of into you know, wireless stuff. It's always seems kind of magical, I guess to people, you know, wireless transmission and everything else and how it works. And so I always wanted to know that pretty in depth and just how, how systems work.
Vamosi: So here we are: The tinkerer, the word hack, literally means to take things apart. But it's more than that. It has to include that innate curiosity about what makes things work.
Hash: So I guess I've always been taking apart things and trying to work on them and stuff. I just never really published anything. And I think when when I did the robotic vacuum and it became kind of more public and I started, I created that wiki and started putting it out there. I kind of felt like it was giving back a bit to the community that I had kind of taken a lot from like when I was growing up by being IRC channels, and I had found the internet, all this information that was available. And now it was like how can I contribute back to Tibet and put material out and help other people maybe join in the same way I did by not traditional means like, it wasn't really college or anything that got me into electronics. It was all this discovery on the internet that brought me to it. And so I think a bit of the desire is to do something like that. Right.
Vamosi: So one takeaway with hash is that he truly is a hacker in the old school sense. He has a day job, and it's not this. He does this hacking on nights and weekends.
Hash: My day job is like probably the complete opposite of this. So I generally don't talk about my day job or my day company or whatever it is. I don't kind of mix the two and I'm in charge of global sales operations, channel operations and go to market strategy. So nothing to do with electronics whatsoever other than we sell, you know, technology products, right, and cameras and things like that. And so this is purely because I like to be kind of in the weeds on technology and things like that. And, and I don't get that I can't satisfy that desire during the day. You know, in that job. It's a different kind of thing. So I do it nights and nights and weekends and vacation. days.
Vamosi: I first became aware of hash after the Texas freeze, in particular hash and published a video on YouTube showing him driving around the streets of Dallas. In particular, he was reading the data off of the various smart meters learning when they were last shut off.
Hask This is a map of Dallas, Texas. And each one of these red dots represents the location of a smart meter. I was able to receive this data while driving down the freeway with a software defined radio and my laptop, the smart meters transmit their location information along with the amount of time that they've been running since they last experienced a power outage meters that are low to the ground like this one have only been running for a few days. meters that are much higher, like this one up here have been running significantly longer. This one for 1783 days. This video was picked up by major mainstream media and it clearly shows that ERCOT didn't really know what information was available. It couldn't say where and when specific outages occurred, but ash could so after the power outage happened here. I was reading some articles and there was this and so there were all these questions being asked to these power companies and providers like hey, where did you choose to turn off power? Why some areas didn't lose power at all. Some people lost power the whole time. I lost power for like eight hours at a stretch. It's freezing cold inside my house. The whole pool was frozen. You're trying to just not like you know pipes are breaking like you're trying to not lose everything you have. And these companies are saying well we can't tell you where power went out because that's it's actually like national security level. Information is what the article said like the power brighter said we can't tell you because it's essentially a matter of national security. Something security with the grid. It was this Austin article that they got some quote from somebody and so I started looking and I had realized that in the broadcast data from the meters, the meter is broadcasting constantly more than once a minute. In that broadcast that it sends out in plain text is its uptime. The last time it experienced a power outage. So you could drive around essentially. And if there had been an outage, you could drive around and see all the meters that had a really short uptime, meaning they lost power and the ones that have a really long uptime means they didn't experience that outage. So I went driving around down the freeway. I logged all that data. I loaded it into Google Earth and I plotted it to see okay, who had an outage who didn't there's some areas you see it's really low but other ones you see are high I went to some you know more well off areas and you see they're really high like they never have a power outage. And so I thought that it was interesting that the power company people thought that like they first of all, they thought that that information isn't even available. They probably still think that and second of all they're saying it's some kind of matter of national security and anybody can just drive around and receive it. So both of those things I thought were kind of funny.
Vamosi: The Texas freeze really wasn't his first interest with smart meters. Nor was at the start of our story. Our story actually begins a few years before with the famous Dallas hacker association.
Hash: So I moved. I'm in Dallas now. I moved here at the end of 2014. And I was in California before that, but not like in Silicon Valley or anything. I was in the Central Valley like in Fresno and so not like a high tech area. And so when I moved to Dallas, I've been here a while. I thought what if there's any groups of people that get together that are in security or anything like that, like something that would be I've always been kind of interested in hacking and stuff like that. And there was a group called the Dallas hackers association that met up and so I ended up coming out and hanging out and they would just do these kinds of fire talks and people would get up and talk about whatever. And one guy at one of the things he was talking about protests, and there was this it maybe it was around it was there was some protests or something going on, I guess there's always things like that, but and they mentioned that, you know, in times like that they might jam communication, or shut down cell phone communication or these other things to try to control protests. And so we kind of started just like musing, you know, over a beer of like, well, what other networks How could you set something up? And it was the ideal blue two things and stuff like that. And I thought the smart meters are communicating like what if you could build little pagers that will communicate over the network that the same smart meters have essentially established? Like what if you could piggyback on that network somehow? Because what's the odds that they would black out a whole city and to stop communication? So it was just purely like a thought experiment? So I started looking into smart meters and saying, like, I wonder maybe I'll give a talk about just how they work at this little local kind of thing. And as I dug into it, I found things that were interesting like one of my videos, I talked about this geographic routing protocol, how they send data in and I thought, wow, you know, they're aware of where they are. And they're aware of sending a message like you could probably use that to say, Well, I'm here and I want to send a message to there and maybe the network will carry it across somehow. And so I ended up giving a talk a few months later, I just dove into these things and tried to figure out what I could and I gave this in person talk I never posted online. I was kind of worried if I said anything online, just the forces of B were going to come and shut me down for even looking at these things. And everybody was kind of glued to their seats. You know, it's I was talking about this and the data and whatever. And so, at that point, I then that was like a few years ago, I gave that talk and I never talked about it again. I didn't publish anything online. No one essentially knew I was doing any of this work and I was kind of just working on it in my room. Then you get to the point and then I decide I think I'm gonna start making some videos about this. And that's when we had this kind of great freeze event in Texas where they lost power and all this stuff. And I kind of thought, you know, I don't really know if these guys know what the hell they're doing. Like maybe I should just risk it and just start publishing this stuff to raise awareness. And to just have other people join in and take a look and say, How well are these systems built? You know how, how safe and secure are they? Yeah, just to take a look. And so it kind of started ramping up I would say then about that point.
Vamosi: So a lot of us came home one night and there was a shiny new smart meter on our house. And I remember asking questions, who were the manufacturers? And were these things even vetted by a security community? Turns out they weren't.
Hash: So it wasn't to analyze the security of smart meters. It was just like, I wonder how these things work. And could you craft a message that these meters would carry across and I kind of equated it to like, freight hopping like if there's a train going down, and a you know, a hobo or something hops on this train. The train keeps going. The train isn't affected at all by this person hopping on. So you can essentially piggyback on this thing with no adverse effect to air. And you know, and off you go. And so I thought, well, what if you could craft a message that would go a few hops that would expire? You know, would the system ever even know that that message was carried across? You know, could you make use of this for some low speed kind of communication as this massively deployed network? That they're only using smartphones? And so that that was kind of one of the initial things that drove it.
Vamosi: By the way, given all this hacking house hashes, the relationship with ERCOT is an energy company.
Hash: They they haven't called me I did reach out to the EFF and I spoke to some lawyers that they have just to kind of understand the landscape of what I was trying to do and how you can do it in the means that is, you know, that is kind of above the board because, you know, I think the idea is that you want to people need to be able to look at these things for everyone's benefit. Because, you know, it's when things are sold, and they say it works this way. If you can't check it, like, you know, you're just trusting somebody else is saying it and that's a bit of a challenge. And so, what I didn't want to do is is make it that I'm just completely rogue, and I dumped the firmware and I posted on for everybody to see and you do these things that you're not that legally, you know, as kind of a society we say we don't want to do, but we still need to look at these things. So how do you do it in a way where more and more people can do it and I try to guide them as well to say, look, you know, don't post illegal things. Don't share something if you get it, but how can we all get a copy of it and look at it, which would be an illegal way of doing it? You know, they haven't reached out to me and my power is still on so I'm square in that respect. Something like this,
Vamosi: I'm always curious about the barrier to entry. To this research. It's not network security where you can download a spec or code and analyze it on your Linux box. No, this requires a little extra effort.
Hash: Like if you see my lab behind there it's like scopes and function generators and power supplies and all kinds of stuff. So for me, I'm the most comfortable like in the embedded zones, circuit boards and chips and everything when it gets to the abstraction layer of some massive bit of code and how do you even navigate that and everything? Like I'm trying to get better at that because obviously there's a lot on that side right? But yeah, I'm just super comfortable. Like if we need to solder 100 wires onto some tiny chip to solve the problem. I'll do that probably before I'll even look at the software route. Because I'm just so comfortable working in that space. So that's why when I saw that thing, it drove me in that's kind of the same with smart meters. A lot of embedded parts, some wireless aspects. I wanted to get better at software defined radio. And I'm kind of comfortable learning those, like spending the time to learn those things. And so that's where I felt like maybe attacking this not on the network side and the IP side and is the backhaul secure and all this stuff that people are traditionally looking at. That's a traditional ICS attack vector. Like they come in over a network and then they take over the stuff I was like, what if you could come in over the other side and take over things because people probably aren't looking at this other side because the barrier to entry is high. It's complicated stuff that people usually aren't looking at. It's easier to look at the network side right? So
Vamosi: you need to buy some additional equipment, some specialized equipment. So what is something like that set you back? How much does it cost to get into embedded systems these days?
Hash: $5000 plus, I mean, you can do it with some cheaper stuff, like cheaper scopes and things but I tend to like you know, I enjoy it and it's hobby so if I'm going to do it, I want to have fun and so I'll I'll spend some money when I have it to to get some stuff so I mean and then there's random stuff like a microscope over here and other things because I wanted to start looking at silicon and, and diving in kind of to the very the most basic level, but when you start adding in all the software defined radios and everything, it adds up pretty quick. So I would say it's it's above, maybe a weekend hobbyist of someone that just wants to look at something, but it's well below a state level or corporate actor. So it's you know, it's I would say it's accessible to someone who wants to do it. But it's not you're probably not just going to have the stuff sitting on your desk unless you're interested in electronics orwhatever
Vamosi:. And then there's the small matter of hardware itself. You need to physically obtain a smart meter.
Hash: Because you know, hardware hacking, you got to be able to get the hardware like it's awful. Hard and hardware hack with no hardware. So I thought like, and the thing is, depending on your motivations, anyone can get smarmy, like if you go look at the side of a building or your house, they're not locked, right? There's a little security tag they put to see if someone is tampered with it but they are not locked. You click that thing with some wire cutters. You just pull it right off and you have a smart meter. So you know we have to remember like, it's not like this hardware is not accessible. If you're a malicious actor, that hardware is everywhere and it's free, and it's easy to get and no one's watching it and they're not going to know if you took it. If you power it up. That's a different story. We could talk about that. But, it's easy to get but I wanted to talk about it publicly. Like in the end I wanted to publish videos so I have to watch everything I do. You know if I'm sharing things, if I'm sharing the right things, all this stuff so that I can talk about it publicly encourages other people to do it. And also to do it legally, you know within the bounds that we can't and so I went to eBay because eBay is the place that everything seems to be available. And I mean you can search for smart meters. There's every brand, every single one you want to buy is there and they're like 25 bucks. And sold by the lot like a lot of them like it's not onesie twosie I mean there's people that says 40 available 60 available, a lot of 240 all still new in the box that you can buy. I mean, they are completely out there. So it's easy to get hardware for anybody that wants to get it and it's super affordable to get it like 25 to 50 bucks a piece.
Vamosi: That's curious. I wonder for what reason would you need to buy 25 smart meters on your own if you're not a utility?
Hash: Yeah, I mean, I have no idea. I posed the question a few times in my videos like, I'm not sure what the reason is. I would venture that it could be companies that buy surplus equipment. It could be electronic recycling companies where they don't have an agreement with them to destroy the stuff that they're picking up. They don't care what happens with it and so they turn around and sell it on eBay. There could be any number of reasons why it's there. I you know, it could be an enterprising employee that says I'll haul that stuff off the road away and they put it on eBay you know some of the stuff is like still in the factory boxes like from the fact like Landis and gear box that's I have in the garage of like meters GE like it's you know, it's like, you know, probably if you decode the barcodes that are still in the box, I could get some shipping information and whatever else from them, right. But I'm still not sure why they're all there, but they're there. And then there's other devices too. Like you'll see I have a video on a collector, which is one of the devices on a pole that's like the crown jewels, because there's a lot of information in there. And somehow that ended up on eBay, you know, is probably I speculate that that was maybe a package that the one that I got that could have been stuck in customs and then they just let it go. And somebody from there got it, put it on eBay. You know, it's it's interesting that there is so much hardware available, then
Vamosi: there's the option of buying something off of eBay. As I said in a previous episode, you don't always get the software when you do that. It's proprietary with each device. So with smart meters, though, it's all embedded right? There's nothing floating above that that's proprietary
Hash: For the software on the smart meters. So the Smart Meter network, you got to think of it in like kind of maybe three pieces. There's the meters themselves, and there's the mesh network they're on and the software associated with that kind of operator. Then you go up. Once you leave the mesh network, you go up to a device on a pole called the collector. That device is basically the bridge between the internet and the smart meters. And so it can either go over a private way and a cell modem or a wired connection or whatever but now you're kind of on to the traditional like, you know, network, then you go back to a headend system that is inside of a utility that's actually aggregating all that data. It's used for the billing, it's how they send signals to turn power on and off to meters and these other things so in that respect like you have a head end system, they don't have those on eBay. So I don't have one of those yet, but if they do, I'll get one. The next level is the collector. There's been a few different ones that have been on eBay for different manufacturers and I bought a few of them. Virtually all the stuff I bought before I ever made the first YouTube video because I was I thought if my source gets dried up, I won't have what I need to continue talking about the so I kind of just essentially stockpiled everything before I ever made the first video that I did have a little foresight on and so those they have that kind of bridge software running in them and so it depends probably on the manufacturer, how much intelligence they put in the collector, and how much they leave at the head end. And that's probably a security design of what they're, what they might put out there and encryption keys and things like that. And then you have the smart meters and so the software on the smart meters that's just 100% on its firmware. If you can glitch it and get it off, you got a full copy of what it's running that collector because I got one that had an SSD drive in it. And so you just pop that into the computer and it turns out it's run in Windows, and everything's written in dotnet. And so it's fairly easy to take a look at that code. So that proved to be fairly interesting.
Vamosi: And there must be only a handful of companies that make these smart meters
Hash: I think there's definitely a different range. I haven't analyzed a ton of them and so it's unfortunate probably for Landis and gear that they happen to be the brand that's on the side of my house, which is just where I happen to go to look to say what should I buy? And so I went outside and I said well, what's going to be near me I need to receive the RF data from something near me. And so I need to use the system that's around me so I look and there it wasn't the same for encore, the energy company like it's unfortunate for them that they just happen to be my energy provider or fortunate, I guess, depending on how you want to look at it, you know? But so I've looked at those. And then I've seen other ones that have been advertised that are, you know, that people tell me about that are available in other areas. And so I think there are some where it's very simplistic to get the data. There's a guy I can't remember his name. He created this software called RTL AMR for automatic meter readers. And there's some meters that they just broadcast out the power usage completely plain text, super simple. And you can decode it using an RTL SDR, a $30 SDR, right. And so people would go to his site RTL Amr, and they would say, Hey, can you support this meter? And I think he's kind of maybe over that project or adding support and so he would say, No, we're not going to add that or no protocols and understood and so when I was initially searching for this meter, I saw Lana gear, the focus, whatever it was, it was like a request on his GitHub to add support for it. He said, I'm not going to add support. It's a more complex protocol or whatever. And so that was kind of my first look. And I said, Okay, I guess I'm gonna have to figure this out myself because nobody else has done it so far. Because it's not there's nothing published about usually those mesh side protocols, you know, just because it's proprietary to every company.
Vamosi: A mesh network is a network of individual nodes fully distributed. They self organize and self Configure. So if a new node is introduced, it gets integrated quickly. This is how smart meters talk with one another. They bounce messages off each other. And given this dynamic nature, testing, smart meters therefore can be problematic. I mean, you have to have a lab. I mean, you have a lab and you have the hardware. But the moment you turn on that smart meter, you may have a problem. You might black out your own home, you might block out your neighborhood. So there's something else you need to do before you turn on one of those 25 smart meters that you purchased. Right? You have to create a special environment and isolated network to do the analysis, when you turn it on.
Hash: So if you just power it up, it's going to try to contact the network and reach out. And so you know, that would be a problem if you're experimenting with it. And you're interfering with a network around you. And so the first step that I wanted to do was to not do that. And then to also make sure that when I'm creating videos and showing the stuff that that it's done in a way that kind of shows other people a way to go about it, and so, I used a thing called a Faraday cage,
Vamosi: A Faraday cage is commonly used in security research. It's used to shield RF signals coming in and going out. Often, it's a physical mesh of conductive materials, and it could be as simple as a bag or a pouch or a full room. The point is, the signals from the outside really can't get in and the signals on the inside really can't get out. They're trapped within the physical mesh. The science behind it is cool. Basically the conductive material causes the electrical charges to be distributed, so they cancel each other out. And this principle is useful in deflecting lightning strikes.
Hash: The Faraday cage basically contains all the RF energy, and so I feed an antenna into there for my software defined radio, and it allows me to listen to the meter that's in there. I also took it one step further, and I took apart the RF circuitry that the Smart Meter uses on that mesh side and I disabled the power amplifier that it uses to amplify the signal to transmit so there's like a, an RF chip that transmits out some energy and then there's an amplifier on it that really boosts it to the kind of full one watt and so I just disabled that amplifier. So then I'm inside of a Faraday cage. I've kind of made it as quiet as I can, so that it's not going to interfere with anything, and then I start listening to it individually, to try to figure out what is it doing? What's it trying to find? How's it? How's it booting up all those things?
Vamosi: It's an interesting question. If all of these smart devices are interconnected, someone could manage to insert a message into the network. Someone could shut down the network, at least in theory. So behind the glass on the circuit board, are smart meters really smart are really, actually kind of dumb.
Hash: I would say they have two different processors on him. The latest ones have like two megabyte of flash memory, like I mean, they're, you know, they're doing a lot of stuff like it's not as simple. It's not as though there are some smart meters that are very simple. They just ping out here's the power usage. A truck drives around, it just receives it. But these things I mean, they do build they'd like build a self healing mesh network of all these meters, where they're broadcasting the information, the ones around them are hearing it they're constantly sending out like, you know, kind of packets that are announcing their state their time, how they're running all this stuff so that they can you know, send this data back so so they are like I would say they these ones I'm looking at they're they're very complicated like their devices to analyze the circuit board and to try to create schematics only things I did. It just comes down to, you know, methodically looking at things one at a time. So like I said, you always need multiple pieces of hardware to reverse engineer something. So I got one meter and I just stripped all the components off the board. So I just like a soldering iron and heat gun and just took everything off a clean slate circuit board, and then use a continuity tester with a meter to just literally probe out pins where does this pin on the processor go or one to 100 go to the next one, where does it go? And then start drawing that in some schematic capture software and create a schematic of the board that then I can look at and say okay, what are the interesting pieces? What are some interesting lines? Is there a way to put it into a test mode? Does it have serial port access into something special, like I found on some of them, so but it just you have to kind of attack it, you know, at the most basic level and, and just build up from there. So it seems complicated. I think when you look at it from the big picture, it doesn't feel complicated when I'm doing it. It's tedious and time consuming, but it doesn't feel complicated you know?
Vamosi: Hash started creating some videos on YouTube a few years ago, and the bulk of these are honest, smart meter research. The way they play out is it seemed like they were in real time. They were real time progress reports. There's a genuine sense of discovery here. And curiosity that runs throughout the series.
Hash: There's a very faint kind of vision of where to go. But it's essentially real time. You can think of it as real time. Choose your adventure kind of story like I have kind of an idea of what my next video will be that I'm going to do right now. But beyond that, a bit of it is the discovery and what happens and comments people leave and questions that are raised and you know, things that are discovered along the way that says hey, what would be interesting like right now I'm working on glitching the processor to dump the flash memory. For a long time I was working purely on analyzing the wireless network. And so I thought, okay, we can probably learn more about the wireless mesh network. If I can get the firmware and understand how is it frequency hopping? What are choices that are being made? Because I've kind of felt like I've gone as far as I can with just a blackbox analysis of how it's working. So I need some more information and how can I get that? And so people would ask me, they say, you know, do you know about glitching? Or do you know how to glitch a processor to dump stuff and I've always been interested in it, but I hadn't actually done it. So I thought okay, well let me see if I can figure it out. And I'll start making videos and I usually like to make the videos even before I know I can figure it out. Because in it, it also kind of drives you to spend the time to figure it out. You've kind of put your name out there like I'm going to do this thing. And so yeah, so for the most part, it's like I might not even know in a video if I can do the thing that I said I'm gonna do in the next video. It's like a matter of trying to figure it out. So that's why sometimes the videos aren't exactly every week or something right? It's like because hell I got to try to figure out whatever the heck it is I said I was gonna do in the next video.
Vamosi: One of the videos is devoted to something called glitching. You might hear this in the gaming community and in robotics, I really wasn't sure what it means.
Hash: Yeah, I mean, so when you do so there's a few different ways to glitch things. I don't know everything about all of them. And so I'll kind of just tell you what I've done so far right. So they generally you can glitch the power to something which means you kind of momentarily dropped the power the power rail of a microcontroller, or even glitch the clock, which means that if a clock is running, say at 10 megahertz or something, you kind of like you might still keep it at 10 megahertz but you might you know it those are usually spaced that waveform spaced out like 50% so it's high 50% of the time low at the present time. And you might make like an extra little clock pulse in there, which basically, the processor isn't expecting to receive a change of state that quick. And what that can do is, you know the processor when it's running the chip, it's like a state machine. So everything's happening, and it's all based on this change of state of the Clock Line. And if the Clock Line changes state faster than it's expecting, like a program counter might increment, but the rest of the stuff down chain from the program counter might not actually have enough time to process what you're saying. So in the case of glitching a clock, you're saying I'm going to try to maybe cause something to happen with the program counter or some other piece where the downstream parts of like maybe setting some bits for security or for checking some bits don't actually get a chance to process and it just steps to the next line of code, which might say like if you know the variable gets set to zero, I've run my check. If the security is set, I set it to one and then I evaluate it, if you're able to glitch past the go check what it is and you set it initially to zero. Then when you do your check, you end up checking and go oh zero that means there's no security you somehow cause it to jump past that with power glitching which is what I started doing. You're doing the same thing, but you're trying to basically drop the voltage line of the processor down incredibly fast and raise it back up incredibly fast. So you don't cause the chip to reboot. You don't cause some brown out detector, which is what usually when voltage dips to go off and reboot the chip, but when that drops, things like writing to SRAM or writing to other memory locations or updating variables might not take place or it might get corrupted, or some program counter might get messed up. So essentially, you as the chip trying to execute its instruction you're trying to glitch it to the point where there's actually not enough power to execute that next instruction. So it goes to do something but it can't actually complete it, but then it keeps stepping on so it's like the program counter kind of keeps incrementing but what it told the chip to do, there wasn't physically enough power. Like if you were just too tired to get up out of your chair like your brain saying we should get up and go get food and your body's saying, I'm not going anywhere. It's kind of that same thing. So it can be it's not an exact science. So you do kind of have to say okay, I'm going to glitch I'm gonna, I'm going to profile the circuitry, I'm going to profile the board, all these different things, and then and then set your timing. So there's a bit of that you have to figure out but once you do figure it out, you can pretty consistently like I can consistently make my development board dumbbell is flash memory, I can consistently corrupt a certain variable. So as you figure out more about it, you can consistently, you know, do things without it taking a super long time to achieve the result.
Vamosi: There's also this concept known as site channel. It's pretty cool, just by looking at the Rise and Fall of energy usage. You can see when a chip is actively computing and when it's not. And given the strength of each rise and fall, you can also estimate the results. So this is sometimes used to figure out passwords and credit card details as they're going through any point of sale. It's pretty cool stuff.
Hash: I've played with that a bit with some test boards. The guy Colin O'Flynn that makes this chip whisperer which is one of the devices I've been using. They have an incredible like, set of kind of training Jupyter notebooks that they've built and defend boards that they send with stuff and so in that sense I've played with their power analysis, so the side channel kind of attacks or side channel analysis. What's interesting is when a chip is processing, you can see a lot of times what is happening by looking at the power line of the chip. So if you look at the power line incredibly closely, you'll see it kind of dip up and down as it's doing things like if you haven't multiply something, you can literally see the, you know, kind of the fact that it was executing these different instructions and so when you start to say, maybe an encryption routine that's running or something else, you can kind of see that the the classic query, is to say, I'm going to connect it over a serial port, I'm going to try a password. And so depending on how it checks the password and the code, it's writing, maybe it checks each letter your sending to see if that letter matches or not. Maybe it takes the whole thing and checks to see if it matches. And so depending on that if you can see the powerline you can see how far did the processor go in its code before it rejected me. So if I send the letter A and it rejects me in one millisecond, but I send the letter B and it rejects me in two milliseconds. And then I send C and rejects me in one millisecond. Again, I can say well, there's something interesting about the letter B. So now maybe I try B A, B, B, B C, and you take something that was maybe a long password that would have been incredibly hard to figure out. You can figure it out. In a minute or two. Right. And so and that's something that usually the developers weren't thinking about, because they weren't thinking someone could see their code or could see how the if statements or anything else were working. So they're not trying to protect against that. And so it's very easy to end up with code that if you can see that stuff is incredibly insecure. But people are assuming you can't see that and you don't have access to the code. So I don't have to worry about how that check routine was written. So so there's there's a lot of those things that are uncovered, like issues with embedded systems and even if you don't have the firmware, if you have the firmware, it's easy to be like this routine is completely broken. Here's how I get past it. But even without having that there's ways to figure it out.
Vamosi: Not everything is proprietary in his research. There's a fair amount of free tools to be had.
Hash: Yeah, so when I first started looking at these, I was using a device by Michael Osman, you know, Great Scott gadgets and it's called the yardstick one, and it basically it's a device that uses the same RF chip that the smart meters use, so you can set it to a channel so I can say there's like 240 channels that these things operate on between 902 to 928 megahertz, they're spaced out every 100 kilohertz and they kind of they kind of come in from the edges a little bit so 240 or so channels, so you can set this thing to one channel, hook it into USB in your computer and it'll receive the data and I wanted to do it with software defined radios and and pull everything in like from a wider spectrum, but I couldn't make that work. Like it was incredibly complex to try to do that. And I didn't know enough about digital signal processing and all these other things to do it. But with this little device, you could just tell it, go to this frequency. Use these parameters and it will start spitting out data but you're only hearing one tiny slice of all this stuff. So I would just hear anytime a meter happened to hop to like nine oh 4.6 megahertz. I hear whatever it is sent. And that was kind of enough to get a feel for what's happening on this network because they're transmitting so much. And they spread pretty much equally across all the frequencies that I could hear. But obviously that doesn't give you a full picture because if you're trying to understand what these meters are communicating back and forth, you kind of need to hear the full, you know the full bandwidth all the time. And so, and the frequency hopping thesis, you know, the meters are hoping all the time. So every 700 milliseconds, they hop to a new frequency. And they're all synchronized around each other. I'm not quite sure how. But what I've read is that it's their hopping sequences based partially on their meter ID and partially on the CRC, which is essentially like their network. Id, that they combine that and that comes up with the hopping pattern. So when you know when say the energy company next to me wants to put a new meter on the network, they set that CRC for their network, it means that the other energy company that's right across the street from me selling power to those people using the exact same meters, those two networks don't interfere with each other. They kind of hop in a different pattern. They ignore the messages that aren't for their CRC, which is also how I found that there's ways to do testing without even putting things in a Faraday cage because you can set it to a test CRC and these other things and then they won't interfere with the network. But you kind of fast forward a bit and I was trying to figure out a way to receive all of this data at the same time. And I'm searching the internet. I'm watching every YouTube video I can of how people deal with frequency hopping stuff. And I was building things and I spent a boatload of money to build this computer that's behind me with a crazy graphics card to try to do parallel processing and literally receive all 240 channels simultaneously and run it through some GPU accelerated Chanalyzer thing. I mean, it's a talk that I gave it a new radio conference. Not that long. ago. And in the end, I still wasn't successful. Like it was just and I was so I went it was near Christmas time. I go into the living room. I'm kind of feeling defeated. And back to searching for YouTube videos to try to get ideas and I come across. Jacob What's his last Jacob Gilbert giving a talk at the GNU Radio Conference about this Frequency Hopping Spread spread spectrum toolkit that he had managed to have Sandia Labs open source and put out there and I'm watching this video and it's like showing how they're able to tag any packets. I'm just getting goosebumps being like oh my god, this is the thing. So I ran in and tried to build all these different modules and get the new radio setup. And I managed to get it and it starts dumping packets that it's receiving. So no matter where it is in that range, it grabs the packet. It dumps it to the screen. And I was like a holy cow like you know, this is like the Holy Grail right here. It didn't need that massive computer. I can run it on a laptop so I could drive around if I wanted and that's what then led me to start creating my own smart meters block that goes into the radio the decodes this land is in Atlantis and gear protocols specifically right
Vamosi: when you do get around to watching hashes video, you can see a stylized sticker from time to time within the frame. It's from an organization called Hacking is not a crime. And this simple statement, well, it really sums up hashes position.
Hash: Initially, it was a cool sticker that I got at the Dallas hackers thing and I thought that's cool and I had it sitting there and my kids were looking at they're like that hacking is a crime. And I was like it's actually not like holy cow. You know, like, let's talk about it because it has been so stigmatized and even all the videos I post comments constantly on tick tock all these other places like you're not allowed to take those apart, you can't touch them. It's a federal crime, you're gonna go to jail, you know, all this stuff. And I was like, Look, I bought it on eBay. It's mine. I took it apart like there's nothing illegal about taking things apart and trying to understand how they work. That became I guess more of a focus to tell people and to kind of try to help them understand that like taking something apart isn't a crime and trying to understand it's not a crime and so and so then I ended up they asked me like, Hey, would you like to also be you know, kind of an advocate for hacking, not a crime, because I think the stickers everything, we're getting attention. And so that's how I kind of came to that. And you know, and if you've been around a while, you know that electronics used to be very accessible. A lot of stuff shipped with a schematic. It came with the source code, like at some point companies decided it's better that we just keep all this encased in and turn all these things into additional revenue streams and everything else. And the idea of anyone looking at anything, just vaporizes, like it's people who work on software like they want to write apps, you know, like they're 18 levels removed from designing their own product right like they're using someone else's platform. They're using someone else's everything. And that's what I like about embedded hardware. It's like you're right at the base level. That's why I have a microscope and I've started looking at the silicone and everything else. It's like, you want to be down at the ground level, you know, and so yeah, that was like a big motivator. For me, I guess.
Vamosi: So along with the idea of hacking is not a crime. There's also this right of repair. In episode 14. I talked about the idea that you should be able to look at the electronic devices in your life without voiding the warranty without getting in trouble with the manufacturer. I wonder if any of that comes into play when looking at smart meters?
Hash: I don't. I don't know if there is anything there I focused on essentially like some, some basic things like not distributing like I've already dumped the firmware and a different version of this meter. But I haven't you know, I haven't shared it with anybody. I've just done my own analysis and some reverse engineering tools like binary ninja and things like that. And, and so I've just focused on essentially, what are the most basic things like we can't distribute software we don't own but if I do own the hardware, and I get a copy of the software, I can take a look at that. I just can't publish it. So I'm not sure if it's right to repair anything like that. I mean, breach repair would be interesting if somebody published a schematic I guess so these things would be helpful. But you know, companies tend to fight tooth and nail like those guys are. What's his name? The one guy that's the most vocal on YouTube, something Rossmann I want to say he's very vocal on the right to repair as relates to Apple equipment, you know, Apple kind of hardware and stuff. And so yeah, it's the kind of fight anybody is looking at. Yeah, I guess they just don't want it. You know, I don't think they see a benefit in it. I'm not sure.
Vamosi: And given that Hash has been looking at smart meters for a while now, has he reported any of these vulnerabilities and what does that experience been like?
Hash: So I haven't, I haven't reported any vulnerabilities because I haven't. I haven't found anything I guess yet that I consider like, you know, like, if I can create a packet and shut off power, or, you know, change firmware or something over a wireless connection, and I find a vulnerability there, which is I work on it more and more you get closer to stuff like that, like you're able to dump the firmware and analyze it and things like that. Some of the stuff where it's all plain text and everything else that's being broadcasted out there. I look at it more as raising awareness to the public because the people that design this system, obviously know that they're broadcasting stuff in the clear, so I don't feel a need to disclose to companies like here's how your product is working. It's like they should know how their products work. The public doesn't know how their products work, right, like and so I kind of like I probably have less of a responsible disclosure stance, the more I work on these things. I don't like the idea of going to the company and asking for permission to publish information that is that people should know about. Now, if it's something that can be used in a malicious way, I think you have to look at how much you disclose. But I'm very for disclosing something the moment I find it, like I can send a packet and shock power in Dallas. Like that would be useful information to show now. You know you you wouldn't say here's the packet and here's the software. But if there's these vulnerabilities, I think they get addressed because of public attention. If you allow it to be quiet in the background, it you they'll just drag it on for however long because these are systems that are hard to update, right like rolling out firmware updates to hundreds of 1000s of meters across the city and and doing all the testing to make sure things don't break like there has to be a motivation to want to do that. And, and public outcry is usually a pretty solid motivator, you know,
Vamosi: this is the spirit of educating others, which I fully support.
Hash: Well, I would say I'm trying to get more people involved. So a lot of the videos and the stuff I do are to try to make it more accessible to try to make it interesting because you know, smart meters like they aren't, you know, they aren't sexy, you know, they're not it's not a Bitcoin wallet where maybe you find a million dollars or something. But I think it is important. I think it is interesting. And I think you do learn a lot that can be applied to other things that are interesting. And at the same time, you're kind of making sure that systems are secure, and it's a fun thing to analyze. So I've been doing more to try to involve people and to come up with lower cost ways. Like I'm trying to come up with a way now that anyone could dump the firmware out of a meter for like 10 bucks. So like buy the meter for 25 by a Raspberry Pi pico loads of code on it and it will almost be like a mod chip on an X box or something. It will dump out the code so that you can analyze it and Diedrich which is free. So to try to get more eyes on it but in a legal way, like if they buy the meter if they buy this thing if they put it on and get a copy and don't share it with anybody. They can be looking at it too. And and I have like a discord channel where people are in there and so a lot of people that are discussing it and looking at it and find it interesting so I think that piece you know, it's fun to try to get more people involved just in reverse engineering in general and, and in these kinds of things, you know, and also it's more eyes for me, like I can only figure out so much myself, you know, and there's a lot of people that know a lot more about network protocols or mesh protocols or all these things. And so I try to do what I can to, to make it accessible and make myself available.
Vamosi: So this has been a process over a series of years. Remember, this is not half his day job. It's his passion project.
Hash: I kind of flip flop around based on interest in and keeping myself interested in it. So a lot of times people say like, well how can you work on just this one thing or for so long? And it's there's so many that's what's interesting. There's so many different angles, you can come at it. And the desire is kind of always learning. So I wanted to learn about software defined radio, but it's much easier if you have a project that's the way I learned. So it's like okay, well how can I do this crazy thing. So it took me like a year to figure out how to do the frequency hopping stuff. And then okay, I want to learn more about you know, writing C++ code. Okay, I'll create a block in the software defined radio thing. You know, I want to learn more about how to glitch Okay, well, I'll try to glitch this processor. So it's been one kind of focus of attention. But it's been coming at it from every kind of technological aspect you can and I would say the probably the final pieces. These devices just like anything you know, the longer they live in the field, the more vulnerable they become to attacks over time. Anyone in the security community understands that, like, if it's only if it's useful life is only six months. It has a different profile, you know, than if it's going to be out there for a year, two years or five years. Well, smart meters are 15 plus years, that they live in the field. So 15 years is a long time for technology to withstand attack, because the attack space evolves over time as well until the cost of attack drops every year. And that's why someone like myself 10 years ago when they were looking at these things, probably couldn't have afforded the software defined radios and all the stuff needed to attack it because those were many 1000s of dollars. Now the same software defined radios are hundreds of dollars, right? And so all of a sudden, you know, the attack is much easier than the processors that are used in these things. They're kind of frozen in time so glitching attacks and stuff that don't work as easily on the latest embedded processors in smartphones do work super easy on the stuff that's in smart meters, right? So I think that there is a piece that because these aren't because there are no bitcoins waiting inside of smart meters, they assume no one's going to attack them. But as we see in the world, there's a lot of motivation for attacking things. And so these are iView low hanging fruit in that sense, and so I do feel like the more people I can rally along with myself to help attack these, the more awareness it raises that maybe 15 years is too long to have them last in the field. Maybe they should only have a five year life. And then they got to build a cost model for the power companies around replacement every five years. Right. It could just be something like that, but but without people attacking them you just leave this super easy attack vector for other people to come in and, you know, ransomware that starts on meters or something else kind of crazy like that.
Vamosi: I'd like to thank hash for coming on the show and talking about his journey in reverse engineering. As I said, this is not yet mainstream. So if you're looking for bleeding edge research, such as the work that hash is doing, check out the embedded security and hardware hacking tracks at any major convention. And definitely check out the retest them, wiki page along with the recheck them, channel on YouTube. Finally, if you want to know more about hacking is not a crime. Check out their website, and you're in luck. The next episode of the hacker mine is all about hacking is not a crime. I hope to see you then.
Hey, if you enjoy this podcast, tell a friend. I bet there are others who like commercial free narrative information security podcasts. I have so many stories about hackers who are making a positive difference in the world. You won't want to miss out.