...making Linux just a little more fun!
Rick Moen [rick at linuxmafia.com]
Mon, 17 Jul 2006 08:28:39 -0700
----- Forwarded message from Rick Moen <rick at linuxmafia.com> -----
Date: Mon, 17 Jul 2006 02:50:25 -0700 To: conspire at linuxmafia.com From: Rick Moen <rick@linuxmafia.com> To: TAG <tag@lists.linuxgazette.net> Subject: [conspire] Compromise of a Debian Project hostEarly in the morning (European time) of last Wednesday, July 12, the Debian Project figured out that one of its shared Internet hosts, "gluck.debian.org", had been security compromised, and immediately took it down to be studied, rebuilt from trusted program files, and back within a day. It look like, as with last time this sort of thing happened, they detected the compromise pretty much immediately -- probably courtesy of monitoring from the intrusion detetion software "AIDE". As before, the package archives weren't penetrated. ("gluck" currently fills these roles via DNS aliases: "cvs", "ddtp", "lintian", "people", "popcon", "planet", "ports", and "release". The machines where packages are created and cryptographically signed are much more heavily restricted.)
Their quick detection and correction are worth noting. So is the avenue of compromise (detailed below).
Also worth noting is that, if you use your security token on a compromised machine anywhere, it's equally prone to be stolen regardless of whether it's a strong or weak password, or a public SSH keypair, etc.
Debian believes in transparency on security matters, which is why the earlier (2003) compromise of "klecker", "gluck", "master", and "murphy" was immediately and extensively analysed in public, on a set of pages maintained by Wichert Akkerman: http://www.wiggy.net/debian/explanation/
...which I then wrote about, here: http://linuxgazette.net/issue98/moen.html I'm looking forward to a similar disclosure about the 2006 compromise.
Meanwhile, there's this Debian News article: http://www.debian.org/News/2006/20060713
At least one developer account has been compromised a while ago and has been used by an attacker to gain access to the Debian server. A recently discovered local root vulnerability in the Linux kernel has then been used to gain root access to the machine.
At 02:43 UTC on July 12th suspicious mails were received and alarmed the Debian admins. The following investigation turned out that a developer account was compromised and that a local kernel vulnerability has been exploited to gain root access.
This reminds me a lot of the infamous circa-2001 compromise at $LINUX_FIRM, where a developer's shell account at public shared server shells.$DOMAIN was stolen because his SSH credentials were stolen from him at yet another shared-resource machine (probably a college shell server), the bad guy SSHed into shells.$DOMAIN masquerading as him and rooted & trojaned that host, and then that same bad guy stole an SSH token exposed there by a careless $LINUX_FIRM sysadmin, and used it to invade the corporate network and compromise everything there on account of Swiss-cheese internal security on the corporate LAN, hanging out there until out of boredom he logged onto the internal IRC server and taunted the $LINUX_FIRM CTO.
The Debian Project didn't have that order of disaster, detected their breach(es) much sooner and more competently, and did not have the crown jewels (master software archives) exposed to risk from shared developer boxes.
Stealing a developer's SSH login credentials, in itself, just gets the bad guys in (as a grunt user). What they really want over the long term is root-user authority. Ergo, they must find some way to _escalate privilege_. In this (last Tuesday's) case, it was a very recently discovered 2.6 kernel bug:
The kernel vulnerability that has been used for this compromise is referenced as CVE-2006-2451. It only exists in the Linux kernel 2.6.13 up to versions before 2.6.17.4, and 2.6.16 before 2.6.16.24. The bug allows a local user to gain root privileges via the PR_SET_DUMPABLE argument of the prctl function and a program that causes a core dump file to be created in a directory for which the user does not have permissions.
The current stable release, Debian GNU/Linux 3.1 alias 'sarge', contains Linux 2.6.8 and is thus not affected by this problem. The compromised server ran Linux 2.6.16.18.
If you run Linux 2.6.13 up to versions before 2.6.17.4, or Linux 2.6.16 up to versions before 2.6.16.24, please update your kernel immediately.
Note that the 2.4.x kernels are not affected. (2.6 kernels have been a bit harder to keep security-fixed, for reasons that would take a while to explain.)
I'll keep an eye out for a write-up similar to Akkerman's 2003 pages.
----- End forwarded message -----
Rick Moen [rick at linuxmafia.com]
Mon, 17 Jul 2006 11:03:52 -0700
----- Forwarded message from Rick Moen <rick> -----
Date: Mon, 17 Jul 2006 11:02:26 -0700 From: Rick Moen <rick> To: conspire at linuxmafia.com Subject: Re: Compromise of a Debian Project hostI wrote:
> Their quick detection and correction are worth noting. So is the avenue > of compromise (detailed below). > > Also worth noting is that, if you use your security token on a > compromised machine anywhere, it's equally prone to be stolen > regardless of whether it's a strong or weak password, or a public SSH > keypair, etc.
One of the reasons I look forward to the write-up is to see the rationale for what they're doing to improve security prospectively, and why. Quoting http://news.com.com/Debian+locks+out+developers+after+server+hack/2100-7349_3-6094335.html :
"At least one developer account has been compromised a while ago and has been used by an attacker to gain access to the Debian server," Schulze wrote.
The developer said the attacker then used a recently discovered vulnerability in the Linux kernel to gain root--or admin--access on the server.
"An investigation of developer passwords revealed a number of weak passwords whose accounts have been locked in response," Schulze wrote.
Encouraging shell users on shared machine to use strong passwords is perhaps worthwhile, but failures to do so on "gluck.debian.org" might well have been fundamentally irrelevant -- depending on how and where the developer account got compromised.
Let me give an example: Let's say I let about a dozen of my friends and acquaintances have shell accounts on my Linux box. Let's say I'm being something of a security fascist, and configure my local password management to require eight-character passwords of mixed lettercase including at least one digit and one special character, with dictionary words disallowed, mandatory password changes every two months, and an enforced prohibition against re-using any of the prior 20 expired passwords.
Friend George, like basically all the rest of us, can't remember very many passwords accurately, and so he tends to use the same passwords for his accounts on multiple systems. One day, one of those other systems has a security breach. The bad guys manage to steal George's password on that system. They see from his .bash_history file that he SSHes to linuxmafia.com, and on speculation try SSHing into my machine using the same username/password George uses elsewhere. Because of George's memory-assisting shortcut, it works.
Or:
Trying to be even more of a security fascist, I disallow incoming password authentication entirely, permitting use only of prearranged SSH public keypairs. George uses such a keypair, and uses it to SSH in from a remote host -- which for entirely local reasons becomes root-compromised by the bad guys. The bad guys use their surreptitious root access to install a "trojaned" SSH client, rigged to capture outbound sessions' security credentials and secretly convey those to the intruders. George in all innocence SSHes into linuxmafia.com, and his private key and passphrase get captured. The next day, the bad guys are able to enter linuxmafia.com, masquerading as George -- and then seek there a path to root escalation, as they did on "gluck".
My point is that these paths to compromise would work against my machine _even though_ I was doing everything right. From what we know of last Tuesday's Debian compromise, such might well have been the case there, too -- which would make Martin Schulze's better-passwords effort (quote above) almost completely irrelevant to the problem.
My Internet systems live with essentially the same risk that "gluck" faces, and my planning must include the possibility of compromise through theft elsewhere of user-access credentials, even if I implement my own security policy flawlessly. Short of adding a second category of authentication (e.g., SecureID tokens or biometrics), it's not a problem with a ready solution.
Oddly enough, Jon Corbet at Linux Weekly News is probably feeling like Cassandra right now: On July 12, 2006 -- the very same day that "gluck" was compromised -- he castigated, echoing correspondent Paul Starzetz (http://lwn.net/Articles/191089/), two authors (for "rpath" and Ubuntu) who'd described a 2.6 kernel bug as (merely) a "denial of service attack" bug, when very obviously it was a likely path to local root escalation: "Denial of reality vulnerabilities" (http://lwn.net/Articles/191080/ -- readable by non-LWN-subscribers starting Thursday, July 20). And that was the exact kernel bug used to crack root on "gluck".
I'll not quote extensively from Jon's editorial for the obvious reason that it's a subscription-supported magazine (and deserves all of our support), but I will quote one very important sentence:
Lest it seem that rPath and Ubuntu are receiving too much grief: as of this writing, five days after disclosure, rPath, Ubuntu, and Red Hat are the only distributors to have fixed this problem.
If you're running a 2.6-kernel-based system with remote shell access, especially from multiple users, you should take careful note.
(Feedback to the LWN editorial reveals that Fedora Core 4 and 5 had 2.6.17.4 kernel updates in the testing area on July 12, Gentoo had updates to 2.6.16.24 and 2.6.17.4 on July 6 and 7, and Debian "stable" aka sarge's default 2.6 kernel, 2.6.8, didn't include the vulnerable code. Other 2.6 distros may remain unfixed, and are of significant concern.)
----- End forwarded message -----
Benjamin A. Okopnik [ben at linuxgazette.net]
Mon, 17 Jul 2006 14:10:21 -0400
On Mon, Jul 17, 2006 at 08:28:39AM -0700, Rick Moen wrote:
[ snip tasty newsbit ]
> Also worth noting is that, if you use your security token on a > compromised machine anywhere, it's equally prone to be stolen > regardless of whether it's a strong or weak password, or a public SSH > keypair, etc.
I'm afraid that my brain's gone a bit fuzzy on this concept; perhaps that's why it's sounding so dire - perhaps completely out of proportion to what you're actually saying here. I suspect that I'm misunderstanding the meaning of "security token", above - are you talking about just typing a password, using two-factor auth, or perhaps both?
If it's the latter - is there a solution other than carrying your own laptop wherever you go? Just carrying an, e.g., Knoppix CD leads down highly problematic avenues if you're borrowing someone else's machine for a quick round of email.
* Ben Okopnik * Editor-in-Chief, Linux Gazette * http://linuxgazette.net *
Benjamin A. Okopnik [ben at linuxgazette.net]
Mon, 17 Jul 2006 14:13:53 -0400
On Mon, Jul 17, 2006 at 11:03:52AM -0700, Rick Moen wrote:
> I wrote: > > > Their quick detection and correction are worth noting. So is the avenue > > of compromise (detailed below). > > > > Also worth noting is that, if you use your security token on a > > compromised machine anywhere, it's equally prone to be stolen > > regardless of whether it's a strong or weak password, or a public SSH > > keypair, etc. > > One of the reasons I look forward to the write-up is to see the > rationale for what they're doing to improve security prospectively, and > why.
[laugh] I need to be a bit quicker on the keyboard. While I was typing my question, Rick was already busy answering it. Now that's what I call fast service!
* Ben Okopnik * Editor-in-Chief, Linux Gazette * http://linuxgazette.net *
Rick Moen [rick at linuxmafia.com]
Mon, 17 Jul 2006 11:59:28 -0700
Quoting Benjamin A. Okopnik (ben at linuxgazette.net):
> I'm afraid that my brain's gone a bit fuzzy on this concept; perhaps > that's why it's sounding so dire - perhaps completely out of proportion > to what you're actually saying here. I suspect that I'm misunderstanding > the meaning of "security token", above - are you talking about just > typing a password, using two-factor auth, or perhaps both?
Genuine two-factor authentication is a proper fix for the problem, because it means you no longer have to trust both ends of the connection. Ordinarily, one of the designed-in limitations of the SSH model is that its security is only as good as that of both endpoints; with an RSA Security SecureID fob (or equivalent), a fingerprint reader, or the use of one-time keys, you can instead be SSHing in from an untrusted host, e.g., PuTTY on J. Random User's virus-infested Windows box or a cybercafe terminal.
My point, however, was that people have a tendency to assume they're fixing the authentication problem merely by using better single-factor authentication -- stronger passwords, public-key crypto keypairs, whatever -- but that such is false assurance, because any such token can be stolen while you're using it on an (untrustworthy) compromised originating host. The quotation from the Debian guy suggests they might be going down that mistaken path -- though it's difficult to tell.
Technically speaking, the generated crypto steam from a SecureID fob can be stolen when used, too, as can the one-time key you generate to use with the OPIE or S/Key one-time extensions to OpenSSH -- but not usefully, because they're only valid once and hence useless to the eavesdropping bad guys.
> If it's the latter - is there a solution other than carrying your own > laptop wherever you go?
Carrying your own laptop is ideal in the sense that (unless someone gimmicks it when you're not looking) it's entirely under your control. The next best thing is... debatable. The easiest solution would be, indeed, to carry a Knoppix or similar disk around -- with the obvious hole in that model being at the hardware level, e.g., the cybercafe machine you're using having a keylogger dongle that you didn't notice.
A bit more trouble, but a slightly more foolproof remedy, would be to run a second SSHd on a non-standard port, configured via PAM to accept only one-time keys. You could then use compromised cybercafe machines, etc., without exposing tokens reusable by the bad guys for the simple reason that you're using one-time tokens. All your data on the compromised machine could get stolen, right down to the last keystroke, but not your remote access.
This would also be, as you say, less of an imposition on people whose machines you're using. On the minus side, managing one-time keys is a genuine pain in the neck. You end up either using PalmOS apps such as PilOTP / PalmKey[1], or carrying around a sheet of one-time keys in very small type -- and having to generate new sets of keys frequently.
Meanwhile, I'm congratulating LWN's Jon Corbet for his uncanny prescience:
From rick Mon Jul 17 11:36:33 2006 Date: Mon, 17 Jul 2006 11:36:33 -0700 To: linux-elitists at zgp.org Subject: Running 2.6 kernels? Time to check patchlevels Jon Corbet, are you feeling like Cassandra, yet? I note, in last Wednesday's LWN, the editorial "Denial of reality vulnerabilities" (http://lwn.net/Articles/191080/), tsk-tsking a couple of security advisories' mischaracterisations of CVE-2006-2451 in 2.6 kernels from 2.6.13 up until just before 2.6.16.24 and 2.6.17.4 as merely a "DoS vulnerability", when the bug created an obvious path to local root escalation. ...which was in fact exploited on gluck.debian.org (aka cvs, ddtp, lintian, people, popcon, planet, ports, and release), the very same day: http://www.debian.org/News/2006/20060713 Per http://lwn.net/Articles/191166/, the hole was noted June 19, and cleared for public discussion on July 6.[1] Mirrored at the largest known repository of open-source PalmOS apps: http://linuxmafia.com/palmos/
--
Cheers, Your eyes are weary from staring at the CRT. You feel
Rick Moen sleepy. Notice how restful it is to watch the cursor
rick at linuxmafia.com blink. Close your eyes. The opinions stated above are
yours. You cannot imagine why you ever felt otherwise.
Kapil Hari Paranjape [kapil at imsc.res.in]
Tue, 18 Jul 2006 06:49:34 +0530
Dear Rick,
On Mon, 17 Jul 2006, Rick Moen wrote:
> Carrying your own laptop is ideal in the sense that (unless someone > gimmicks it when you're not looking) it's entirely under your control. > The next best thing is... debatable. The easiest solution would be, > indeed, to carry a Knoppix or similar disk around -- with the obvious > hole in that model being at the hardware level, e.g., the cybercafe > machine you're using having a keylogger dongle that you didn't notice.
The Knoppix solution needs to be augmented with some mechanism to carry (a) your private key and (b) the public key of the host(s) that you wish to log in to.
Otherwise, you end up using passwords with untrusted hosts.
Regards,
Kapil. --
Rick Moen [rick at linuxmafia.com]
Mon, 17 Jul 2006 18:33:58 -0700
Quoting Kapil Hari Paranjape (kapil at imsc.res.in):
> The Knoppix solution needs to be augmented with some mechanism to > carry (a) your private key and (b) the public key of the host(s) that > you wish to log in to.
Quite right. That's why I keep those on the USB pendrive in my pocket.
Benjamin A. Okopnik [ben at linuxgazette.net]
Mon, 17 Jul 2006 22:23:22 -0400
On Mon, Jul 17, 2006 at 11:59:28AM -0700, Rick Moen wrote:
> Quoting Benjamin A. Okopnik (ben at linuxgazette.net): > > > I'm afraid that my brain's gone a bit fuzzy on this concept; perhaps > > that's why it's sounding so dire - perhaps completely out of proportion > > to what you're actually saying here. I suspect that I'm misunderstanding > > the meaning of "security token", above - are you talking about just > > typing a password, using two-factor auth, or perhaps both? > > Genuine two-factor authentication is a proper fix for the problem, > because it means you no longer have to trust both ends of the > connection.
Thanks; you had me a bit worried (mostly, about having to invest lots of time researching some kind of a solution.) I spend a fair amount of time on the road, and not having secure SSH access would be a Really Bad Thing.
The fact is that 99%+ of the time, I'm either using my own laptop or banging on a keyboard from a machine inside a Sun center; these places are usually firewalled, the machines are kept reasonably well patched, and they get completely flushed and rebuilt once a week. Nevertheless, the more secure the communication system I use, the happier I am.
> My point, however, was that people have a tendency to assume they're > fixing the authentication problem merely by using better single-factor > authentication -- stronger passwords, public-key crypto keypairs, > whatever -- but that such is false assurance, because any such token can > be stolen while you're using it on an (untrustworthy) compromised > originating host.
[nod] At this point, even cracking 8-char passwords is getting to be a reasonably accessible task on commodity hardware. Single-factor auth is getting progressively less useful as time goes on.
> Technically speaking, the generated crypto steam from a SecureID fob can > be stolen when used, too, as can the one-time key you generate to use > with the OPIE or S/Key one-time extensions to OpenSSH -- but not > usefully, because they're only valid once and hence useless to the > eavesdropping bad guys.
That was my (admittedly slightly rusty) take on it as well. I thought you were saying that this had somehow become more vulnerable, and was confused. Thanks for the explanation.
> A bit more trouble, but a slightly more foolproof remedy, would be to > run a second SSHd on a non-standard port, configured via PAM to accept > only one-time keys. You could then use compromised cybercafe machines, > etc., without exposing tokens reusable by the bad guys for the simple > reason that you're using one-time tokens. All your data on the > compromised machine could get stolen, right down to the last keystroke, > but not your remote access.
This is a scenario that would suit me to a tee, actually. Pretty much the only reason I would be logging in from untrust(ed|worthy) hardware is 1) while travelling and 2) somehow separated from my laptop, and 3) if there was some relatively urgent reason to check my email at that specific time. Well, to be absolutely honest, "relatively urgent" is not an absolute: #3 could mean "I haven't looked at my mail in a couple of days". Anyway, I would have no concern for any data on the machine I'm using, other than as some variety of a guest's responsibility. [ Re: PilOTP / PalmKey[1] ]
> [1] Mirrored at the largest known repository of open-source PalmOS apps: > http://linuxmafia.com/palmos/
..and now downloaded to my machine for installation on my Palm. Thanks!
* Ben Okopnik * Editor-in-Chief, Linux Gazette * http://linuxgazette.net *
Rick Moen [rick at linuxmafia.com]
Mon, 17 Jul 2006 20:31:32 -0700
Quoting Benjamin A. Okopnik (ben at linuxgazette.net):
> [nod] At this point, even cracking 8-char passwords is getting to be a > reasonably accessible task on commodity hardware. Single-factor auth > is getting progressively less useful as time goes on.
Occasionally, I devote a few cycles to worrying whether brute-forcing 7-8 character SSH passwords is a realistic concern -- and then dismiss it as nothing of the kind by orders of magnitude. For one thing, I think I'd notice my logfiles suddenly overflowing /var like the old Zuider Zee[1] coming south to visit Amsterdam in a bad mood.
The point of my earlier post (just to make sure I was clear; not saying you didn't get it) is that the bad guys don't need to break passwords; they just steal tokens when you use them on compromised systems, instead.
[Running a second SSHd with the OPIE or S/Key one-time auth modifications:]
> This is a scenario that would suit me to a tee, actually.
Being the lazy bastard that I am, I logged to disk a number of threads from the old secureshell mailing list about such implementations, and then included them on the front page of my SSH clients FAQ, http://linuxmafia.com/ssh/ -- but have never so far tried them.
So, I'd be interested in how you worked out the details -- so I can save myself that work! ;->
[1] http://en.wikipedia.org/wiki/Zuider_Zee And yes, I've been there, and do know of the Afsluitdijk and Ijsselmeer. It's morbidly funny that the Dutch came down and demonstrated to Venice how to solve their similar problem, using a working example floodgate, only to see that proposal, um, sink under the weight of Italian politics.
Benjamin A. Okopnik [ben at linuxgazette.net]
Tue, 18 Jul 2006 09:34:49 -0400
On Mon, Jul 17, 2006 at 08:31:32PM -0700, Rick Moen wrote:
> Quoting Benjamin A. Okopnik (ben at linuxgazette.net): > > > [nod] At this point, even cracking 8-char passwords is getting to be a > > reasonably accessible task on commodity hardware. Single-factor auth > > is getting progressively less useful as time goes on. > > Occasionally, I devote a few cycles to worrying whether brute-forcing > 7-8 character SSH passwords is a realistic concern -- and then dismiss > it as nothing of the kind by orders of magnitude.
I'm not sure about that, Rick. If you use [a-zA-Z0-9], which is what most people do, you're only looking at about 3.5 x 10^12 possibilities for a 7-char password - and ~200 x 10^12 for an 8-char. These days, claims of 15 million passwords per second (on a 1GHz CPU) are fairly common on the Net (e.g., 'http://www.forensics-intl.com/breakers.html'); that gives us, what, about three days for the first one, and less than 6 months for the second one? That's worst-case scenarios, of course. Crackers can be pretty patient - and they don't have to do anything except start the process and wait for the results to pop out.
If we move to "all printable characters" as a field, it gets somewhat better - ~6 x 10^15 for 8 chars - but CPUs are constantly getting faster. And, if I decide to get just a little paranoid (as a proper sysadmin should), dedicated hardware isn't all that expensive - at least for government agencies with $$$BUDGETS$$$. "Well under $1B" is a figure I heard being loosely connected with No Such Agency, and that was about five years ago.
I agree that it's not a huge problem yet... but when I was teaching security classes, I managed to crack more than half my student's passwords by the end of the week. It wasn't a normally-programmed part of the class, but it made a damn effective demo.
> [Running a second SSHd with the OPIE or S/Key one-time auth > modifications:] > > > This is a scenario that would suit me to a tee, actually. > > Being the lazy bastard that I am, I logged to disk a number of threads > from the old secureshell mailing list about such implementations, and > then included them on the front page of my SSH clients FAQ, > http://linuxmafia.com/ssh/ -- but have never so far tried them. > > So, I'd be interested in how you worked out the details -- so I can save > myself that work! ;->
Well, since I don't have a 24x7 connected machine of my own that I can
use as a server, playing with this would be purely for my own education
(and thus, a relatively low-priority task; I can think of many different
bits of learning that I want to pick up that come ahead of it.) I may
get a wild hair someday, though. 
> [1] http://en.wikipedia.org/wiki/Zuider_Zee And yes, I've been there, > and do know of the Afsluitdijk and Ijsselmeer. It's morbidly funny that > the Dutch came down and demonstrated to Venice how to solve their > similar problem, using a working example floodgate, only to see that > proposal, um, sink under the weight of Italian politics.
I've sorta wondered about that. Why is it that Italy itself hasn't sunk a few miles and tilted up all of Europe and Asia accordingly? Is it because there's so much gold in Chukotka [1] acting as a counterweight?
[1] Easternmost region in Russia. All that Alaskan gold and oil? Hah. It was just a tiny crumb of the humongous pie. Although the people that opposed Seward were idiots.
* Ben Okopnik * Editor-in-Chief, Linux Gazette * http://linuxgazette.net *
Rick Moen [rick at linuxmafia.com]
Wed, 19 Jul 2006 17:25:22 -0700
Quoting Benjamin A. Okopnik (ben at linuxgazette.net):
> These days, claims of 15 million passwords per second (on a 1GHz CPU) > are fairly common on the Net (e.g., > 'http://www.forensics-intl.com/breakers.html');
To stdout or a local filesystem, yes. But fed one at a time to a remote sshd on a busy PIII on a DSL line? And you'll note what I said about my /var falling over, from several million sshd connections in a week or so, being more than a tad suspicious.
Theoretical lab attacks are interesting, but not to be confused with reality.
Anyhow, you'll have noticed having been required to use public key auth. ;->
> ...when I was teaching security classes, I managed to crack more than > half my student's passwords by the end of the week. It wasn't a > normally-programmed part of the class, but it made a damn effective > demo.
It was a lot more relevant before shadow passwords, though.
[Venice and the stupidly rejected Dutch floodgate:]
> I've sorta wondered about that. Why is it that Italy itself hasn't sunk > a few miles and tilted up all of Europe and Asia accordingly?
The main reason Venice hasn't circled the drain and disappeared is that they realised that tapping the local aquifer for industry was a really bad idea, and stopped doing that. Venice now has stopped sinking quite so quickly -- but is still at the mercy of any severe winter storm surge combining with high tide. Which is an absurd situation to persist, since they know how to fix it.