Leo “worst engineering staff in the business” Laporte showed us all his “Engineering” chat during iOS Today. You can see the raw video above where he scrolls backward chronologically.
We have reproduced the chat in its entirety below.
Key Points
- Look at Patrick’s contributions. He is a worthless sack of shit that waddles around the Shithouse with his hands on his hips. He provides literally zero assistance as Leo’s sole full-time “software developer” during this entire problem.
- Bruce makes fun of the one-horse farm town Petaluma that Leo Laporte makes his employees move to so that they are stuck, like Jason Howell. It’s a great way to keep them there while paying them just enough to get by, but not enough to save up to move away.
- Leo thinks it’s an attack by his “trolls,” but in reality, Bruce and Russell are just incompetent in setting up their load balancers.
- Leo doesn’t use LastPass anymore. Screw Steve‘s recommendations, right Leo?
- Russell Tammany is the only half-smart one who works for TWiT, and he doesn’t work for TWiT. He’s a contractor.
Sample screenshots from the video above are available below the chat transcript.
Chat Transcript
— Thursday, February 18, 2016 —
Bruce Chezem
heroku recommends CNAME at DNSimple among other providers
Russell Tammany
[hidden by sausage finger] sure if it was an attack or just amazon pulling away a load balancer, it could have been too much load against [hidden by sausage finger] elb though… do we have any really high traffic metrick from heroku right before?
Patrick Delahanty
[hidden by sausage finger] would be odd that our traffic was naturally high during our off hours.
Russell Tammany
[hidden by sausage finger] giving me an error but may have to delete the A record first which I want to have propagate first too
[screenshot that doesn’t display in time]
@LeoLaporte can I do a test with munchcast.com just to see if that’s the case?
Bruce Chezem
I’m AFK for 15
Leo Laporte
looks like Heroku is handling the increased load
our load time is up 500% but it’s loading
Russell Tammany
the way amazon elb’s work I don’t know they would show any metrics for a dns reflection attack
Leo Laporte
yeah
Russell Tammany
my guess is amazon blocks udp 53 against an ELB anyways
Leo Laporte
but you can see the dam break after your fix
Russell Tammany
so the traffic never makes it on to heroku’s ec2 instance
Leo Laporte
thank you amazon
and thank YOU @Russell
[screenshot that doesn’t display in time]
Patrick Delahanty
Coming up on AT&T now. 🙂
Leo Laporte
very nice work @Russell
Russell Tammany
no problem. can i do a test with munchcast.com deleting the a record and setting a cname to see if that works in dnsimple’s console?
then i will set it back
Leo Laporte
sure do whatever you need to do
and expect a little something extra in your pay packet this Friday
Russell Tammany
that’s not necessary I am just glad that it wasn’t more complex, I have no idea what goes on behind the heroku curtains either
Russell Tammany
ok well that’s interesting, you get the same error when there is no A record
Russell Tammany
but the error tells you to make an alias record instead
Bruce Chezem
that jives with their suggestions https://devcenter.heroku.com/articles/custom-domains#add-a-custom-root-domain
Russell Tammany
yeah. https://support.dnsimple.com/articles/alias-record/
I’m guessing that the A record pointer to the ELB was done at softlayer DNS before we switched to DNSimple
Bruce Chezem
yes, but I thought Bear made changes about a month ago to that
Russell Tammany
and then Bear or whoever forgot to revisit that after to make it an alias after the migration was successful
Bruce Chezem
when we finished up all the DNSimple stuff
Russell Tammany
I just remember something about heroku needing dnsimple instead of softlayer
Bruce Chezem
k….well I’m getting dressed and heading into the thriving metropolis that is petaluma
Russell Tammany
ok looks like the ALIAS record functions properly
tested with munchcast.com pointed at cnametest.twit.tv which I pointed back to the original munchcast ip which doesn’t seem to respond anyways, but just as a dns test
Russell Tammany
so because amazon “can” reset their ELB infrastructure at times, we should make that change tonight (delete the twit.tv A record and then setup twit.tv ALIAS ehime-4607.herokussl.com)
and then monitor for issues
i have to go to a client that i’m a bit late for
Patrick Delahanty
Other clients?!
Russell Tammany
yeah, but I really like working for wineries. it’s so chill
Russell Tammany
the most stressful thing that happens is their label printer machine burns out a printhead or the windows 2000 system running it doesn’t want to boot anymore
FYI @LeoLaporte this hipchat room is set to privacy public, so anyone at twit can join room and see history so just a reminder to change the heroku password later (and put it in lastpass admin accounts or something if you can)
or change the room to private when you are in it
or both
I’m afk for a bit
Leo Laporte
let’s change it to private
now private
and I will change the password and put it into lastpass enterprise
FWIW I am off lastpass and using 1password now
Leo Laporte
Is it fair to say that this was a DDoS that worked due to a misconfiguration on our part?
Russell Tammany
maybe, could have just been amazon changing out the heroku elb behind the scenes too, do we know for sure that there was high network traffic against the heroku servers? with newrelic or something?
Bruce Chezem
@Russell @LeoLaporte I put the creds in Heroku
i meant lastpass….i have heroku on the brain
Patrick Delahanty
Looks like it’s down again. 🙁
Russell Tammany
got to be some sort of attack then
Leo Laporte
Yeah
Russell Tammany
need to switch it to the proper alias record now then, and maybe needs a ticket with heroku setup?
Russell Tammany
ip’s changed again
i’m going to just do the alias setup, figured it would last until tonight where i could be more cautious about it but guess not
Russell Tammany
@all alias record is setup now to point to all three front end elbs
instead of the a record to just one of them
Leo Laporte
Nice
Russell Tammany
Name: elb065150-471723135.us-east-1.elb.amazonaws.com
Addresses: 23.21.76.215
54.243.32.107
23.21.95.158
Aliases: ehime-4607.herokussl.com
Leo Laporte
So would I be correct in thinking that it would take a fairly heft attack at this point to bring us down?
Russell Tammany
ALIAS twit.tv 600 ehime-4607.herokussl.com
Leo Laporte
It’s my best guess that this is a copycat Attack coming from our trolls who have far fewer skills than the GRC attacker
Russell Tammany
it should be more distributed, I’d imagine that heroku has monitoring setup to keep rotating servers but I’m not sure how their infrastrucutre is setup for ddos
Leo Laporte
Almost undoubtedly they don’t have the same bot army to command so they won’t be able to ramp up to 13 gigabits
At least that’s my hope
Russell Tammany
@BruceChezem can you open a ticket with heroku/ call them to see if they can check into that since I don’t seem to see anything from the heroku console side other than another block of error events
H13 Connection closed without response
all their load graphs look like they are for regular traffic
might be missing a dns reflection attack
Russell Tammany
might have to put cloudflare in front of heroku if it keeps up
Russell Tammany
from https://www.heroku.com/policy/security /code https://www.heroku.com/policy/security
DDoS Mitigation
Our infrastructure provides DDoS mitigation techniques including TCP Syn cookies and connection rate limiting in addition to maintaining multiple backbone connections and internal bandwidth capacity that exceeds the Internet carrier supplied bandwidth. We work closely with our providers to quickly respond to events and enable advanced DDoS mitigation controls when needed.
@all looks like we are back again?
Patrick Delahanty
Yep. Working again here.
Hopefully it holds this time.
Russell Tammany
if you nslookup twit.tv you’ll now see three ip’s
Name: twit.tv
Addresses: 54.243.32.107
23.21.95.158
23.21.76.215
Russell Tammany
and now when heroku cycles them to another ELB it will update there within 10 minutes
sort of just pasting that here for reference later to see if they change
Bruce Chezem
@Russell what did you mean above by ‘ips changed again’ ?
Russell Tammany
they were different a few hours ago
Bruce Chezem
would that suggest they were changed by someone or something or just reverted?
Russell Tammany
my best guess of what happened is this
months ago twit.tv was pointed to just one of the three elb ip’s which worked fine until someone attacked that one today, causing it to get shut off/overloaded at amazon
this morning I just changed the record to point to another one of the IP’s
that IP is now also gone from the elb
and there are three new ones (or at least one new one) in its place
I should have just done the alias thing this morning basically
but it wasn’t clear to me yet that it was any sort of attack as amazon can recycle ELB ip’s anytime for any reason
Bruce Chezem
at that is what they suggest….true?
alias
Russell Tammany
yeah heroku says use the alias
and from playing more with aws now I understand that you can’t ever rely on an IP address at amazon unless it’s an elastic IP tied to an instance
which the ELB’s are not
(elb = elastic load balancer)
Bruce Chezem
right….it makes sense
Russell Tammany
Also it appears heroku only gives us ELB in us-east-1
Bruce Chezem
I noticed that. Should we or they set up another region?
Russell Tammany
my plan for the redirect/metrics cdn.twit.tv was to have two regions, us-east-1 and us-west-1 both with ELB’s that have two AZ’s (availability zones) with route53 monitoring to trigger region failover
Bruce Chezem
should we be doing this directly with aws?
and back away from heroku?
Russell Tammany
We should ask how that works with heroku, they may move heroku, cname themselves if there is a region outage
since we alias to ehime-4607.herokussl.com and not ele065150-411723135.us-east-1.elb.amazonaws.com
heroku has the ability to point us dynamically at another region. it would just be a good conversation to have with them at this point
managing a ruby on rails deployment ourselves on aws is beyond my scope of knowledge 🙂
Bruce Chezem
where’s the bear?
Russell Tammany
I’d rather have support from heroku on it, but we need to be able to make sure they have mitigations for these things
as far as I know bear is flying back from SFO to PHL today
I’m not sure how involved he was in this infrastructure setup either. Wasn’t it all 4K designed?
Bruce Chezem
they were the architects but bear played a very active role in configuration
Russell Tammany
I’m going to update the existing heroku ticket just to let them know we implemented the alias record on twit.tv
Bruce Chezem
they’ll think that’s cool, I’m sure. 😎
Russell Tammany
updated, I have to get back to my other pile of crap here. will check back later
https://help.heroku.com/tickets/336350
Patrick Delahanty
Enjoy the vineyard
Russell Tammany
Do we have Heroku Premium support or just the free stuff? looks like we don’t have the premium support
can’t see it on the invoices which is probably why the ticket response is slooow
site still seems fast to me
Bruce Chezem
I haven’t been babysitting it, but I’ve been checking every so often….its been OK by my eyes too
Bruce Chezem
I don’t think we have premium support. I’ll look at support options in a few minutes….
it looks like next day is the SLA terms were on
Russell Tammany
let’s find out what they charge for premium support i guess
Bruce Chezem
@LeoLaporte @Russell Premium support is $1000/mo, 3 month minimum, 24 X 7, 1 hr response SLA
https://www.heroku.com/pricing
— Tuesday, March 1, 2016 —
Leo Laporte
where is the DDoS stuff @Russell
Russell Tammany
look in the Operations and Engineering room
— Friday March 4, 2016 —
Bruce Chezem
@Russell You needn’t be on that call Monday, thank you
Russell Tammany
Ok, no worries. See if you can get any secret sauce from them re: metrics analysis duplicates etc
Bruce Chezem
MM called and left a message for me this morning to call him back….he never calls me
Screenshots
TYFYC Leica Lens
Ken Sintek» Quote comment
Try IRC now. Bug in webchat kept 2 zombies from your IP online.
Lady Bird Johnson» Quote comment
Russell keeps telling them the cause but they still want to believe it’s a DDoS. But even then why is Russell getting paid to admin AWS when he’s not confident with beginner stuff?
Also why are they using such a jumble of services? If they had used Route 53 for DNS the setup process would have told them how to avoid the issue they had with ELB. Why do they need $1000 remote support from Heroku when they already pay these guys in the chat? They could make their lives so much simpler but everything is an amateur mess.
Jeff Bozo» Quote comment
when leo was talking about getting “something extra in your pay pack” was he suggesting he was gonna give Russell an extra $20?
also am I crazy or halfway though that log does Russell do a 180 and start agreeing with leo that it was an attack?
what a spineless piece of shit???!!!
GROW A FUCKING PAIR RUSSELL TIMONY!!!
Ken Sintek» Quote comment
Holy crap lecia. Your must have very little to do
Joel osteen» Quote comment
Is Russell boning Carley?
Jimmy Jam» Quote comment
def not Carly’s type. her type is a different kind of douche. she likes them emo punk hipster douches.
more in line with an @ G L E N N R U B E N S T E I N than a Russell Timony.
Ken Sintek» Quote comment
I don’t know Ken, I got a different vibe from teh social medias.
It would be kind of shocking since he looks like a garden gnome, but then again he’s pulling down the sweet Dude Named Ben monies.
Jimmy Jam» Quote comment
this is dumb
OMG Bad» Quote comment
I could care less about Russell, Patrick, Bruce, etc. But I am totally sick of Leo’s stupid voices/impersonations and all the talk about Hamilton. Gag me with a spoon.
Still a Sarah Fan» Quote comment
Uh. It’s I COULDN’T care less, Einstein.
holden» Quote comment
TYFYC, LL!
The fact of the matter is this proves that the TWiT ship has sunk and that all we are seeing on the surface Leo’s gas rising from the wreckage on the sea floor.
I have a case of ‘sporks’ that we need to drop off at the new building for fans to poke the TWiT beast later this year to see if it is dead. The case of sporks trick works better for the funeral home visitation of a crotchety, annoying old fart that has passed, but in the case of the CEhO’s empire, we need to see if this thing is dead or just a zombie.
NAKnight» Quote comment
Of course the TWiT Shit Ship has sunk. Why do you think they the gold digger create the second business “Art Is Anal”. It’s not to market TWiT so much as suck the final dribbles of life from it and use it to give an illusion of credence to the new business.
The move to the new location was based around running what is left of TWiT into the ground then turning it into a garden shed podcast done like most others, offline and then posted with minimal work done or even done like many do with Google Hangouts and fat boy sitting back and letting the guests yabber & argue while he sits back butting in just to give some crap advertising. Eventually the strip mall location will be closed or turned into a location for “Art Is Anal” and staff that moved to Petaluma will be kicked out on the street and told to build their own podcast studios.
Ferdinand» Quote comment
The fact that you have time to go through someone’s chat discussion frame by frame clearly indicates that you do not have a job and live in your mother’s basement.
GetALife» Quote comment
Speaking of facts. You are a cocksucker.
Now go jack off to Leo’s Instagram and don’t forget that you’re a cocksucker.
joe is a scriptkiddie» Quote comment
The only thing I listen to on twit is this week in computer hardware.
I just find it funny/sad how obsessed you are with Leo and everything he does.
GetALife» Quote comment
Dear cocksucker,
Most of us enjoyed old TWiT and were dismayed to see it become a wallet for some gold-digging whore. We are also disappointed to see Leo become a little bitch who put money and sex before the product that made him successful.
We are not obsessed with Leo or even his cunt of a wife. This site exists to post the raw brutal truth about the current state of TWiT and the failure that Leo has become. You can call it tough love, if you wish.
Constructive criticism has helped in certain cases but Leo has become too much of a cretin to pay any attention. So, this site exists as a shock therapy to him and the insecure losers who put up with his disrespect, sexual harassment, and humiliation. Losers like you, dear cocksucker.
This site exists because people care (or cared in the past) about the legacy of TWiT. And, as long as someone cares, there will always be hope that one day things might be better. Hope always dies last, as you know.
joe is a scriptkiddie» Quote comment
Leo continually blaming this site for his staff’s incompetence is telling.
It’s very clear in the Amazon ELB docs what needs to be done. NO FUCKING A RECORD!.
“As traffic to your application changes over time, Elastic Load Balancing scales your load balancer and updates the DNS entry. Note that the DNS entry also specifies the time-to-live (TTL) as 60 seconds, which ensures that the IP addresses can be remapped quickly in response to changing traffic.”
Note to idiots at twit. Setup according to docs, not randomly.
Loser@basement.dweller» Quote comment
This website and its authors would make a great paper for an abnormal psychology class paper.
GetALife» Quote comment
And you’d make a great pool boy for Leo and Lisa once they fire EffenDumb.
joe is a scriptkiddie» Quote comment
Exactly. Basically, Petaluma is the “company store” in which the Chief TWaT has trapped his indentured servants.
TOT (Tired of TWiT)» Quote comment
I’m pretty sure “GetALife” is actually the-one-and-only Molly. She would sometimes mention something about being in a psychology class.
Anyway, If it is you, I just wanted to tell you that you’re cute and sweet 😉 Will you go out with me, Molly?!
Oh boy! Another thing: That would be great if what “joe is a script kiddie” is saying is true. Are you really a cocksuck?! Please can you suck my cock too, “GetALife/Molly”?
Yummy! 😉
MollyFan» Quote comment
“Leo Laporte
sure do whatever you need to do
and expect a little something extra in your pay packet this Friday”
Yea Russel, expect something little, very little, like padresj donation little.
“Leo Laporte
It’s my best guess that this is a copycat Attack coming from our trolls who have far fewer skills than the GRC attacker”
Wait till 14:00 UTC April 24 you clown.
Veronica Mars» Quote comment
Well said Veronica!!!!
And I have my Popcorn ready & waiting for April 24
“Wait till 14:00 UTC April 24 you clown.”
That fat fuck Leo won’t know what hit
him!!!
Yup “Man Tranny” will be leaving your ass once u run out of monies!!!
OhNoYetAnotherTroll» Quote comment
He’s so obsessed with his trolls. Go back 5-10 years and you would never hear of any perceived “enemies”….he was liked and respected. Just shows how far he’s fallen.
Pibb» Quote comment
No. You’re very, very wrong there.
What any psychologist would do is realise that people were acting as if a great community, a great family, had been deliberately broken up. People show sadness, bitterness, unhappiness. They try to reach out to others to find a community of others who feel the same. They get together to ensure that the people who broke up that family and community don’t just get away with it without comment. They have their own little corner of the Internet, they make no money out of it, and it allows them a little of the community they used to have.
What a psychologist would actually do with this thread is say “The real people who have too much time on their hands are the people who read the articles just to confirm their own prejudices, and who then take the time to try to humiliate and undermine others”.
Psychologists would be fascinated by how fragile your ego is, GetALife, that you have to take time out of your day to post on sites like this just to show that YOU aren’t like us.
See, that’s always the paradox of the person who thinks they’re making us look bad and small: the only person who ever comes out of it looking small and pathetic is YOU.
evilpants» Quote comment