Leo “worst engineering staff in the business” Laporte showed us all his “Engineering” chat during iOS Today. You can see the raw video above where he scrolls backward chronologically.
We have reproduced the chat in its entirety below.
Key Points
- Look at Patrick’s contributions. He is a worthless sack of shit that waddles around the Shithouse with his hands on his hips. He provides literally zero assistance as Leo’s sole full-time “software developer” during this entire problem.
- Bruce makes fun of the one-horse farm town Petaluma that Leo Laporte makes his employees move to so that they are stuck, like Jason Howell. It’s a great way to keep them there while paying them just enough to get by, but not enough to save up to move away.
- Leo thinks it’s an attack by his “trolls,” but in reality, Bruce and Russell are just incompetent in setting up their load balancers.
- Leo doesn’t use LastPass anymore. Screw Steve‘s recommendations, right Leo?
- Russell Tammany is the only half-smart one who works for TWiT, and he doesn’t work for TWiT. He’s a contractor.
Sample screenshots from the video above are available below the chat transcript.
Chat Transcript
— Thursday, February 18, 2016 —
Bruce Chezem
heroku recommends CNAME at DNSimple among other providers
Russell Tammany
[hidden by sausage finger] sure if it was an attack or just amazon pulling away a load balancer, it could have been too much load against [hidden by sausage finger] elb though… do we have any really high traffic metrick from heroku right before?
Patrick Delahanty
[hidden by sausage finger] would be odd that our traffic was naturally high during our off hours.
Russell Tammany
[hidden by sausage finger] giving me an error but may have to delete the A record first which I want to have propagate first too
[screenshot that doesn’t display in time]
@LeoLaporte can I do a test with munchcast.com just to see if that’s the case?
Bruce Chezem
I’m AFK for 15
Leo Laporte
looks like Heroku is handling the increased load
our load time is up 500% but it’s loading
Russell Tammany
the way amazon elb’s work I don’t know they would show any metrics for a dns reflection attack
Leo Laporte
yeah
Russell Tammany
my guess is amazon blocks udp 53 against an ELB anyways
Leo Laporte
but you can see the dam break after your fix
Russell Tammany
so the traffic never makes it on to heroku’s ec2 instance
Leo Laporte
thank you amazon
and thank YOU @Russell
[screenshot that doesn’t display in time]
Patrick Delahanty
Coming up on AT&T now. 🙂
Leo Laporte
very nice work @Russell
Russell Tammany
no problem. can i do a test with munchcast.com deleting the a record and setting a cname to see if that works in dnsimple’s console?
then i will set it back
Leo Laporte
sure do whatever you need to do
and expect a little something extra in your pay packet this Friday
Russell Tammany
that’s not necessary I am just glad that it wasn’t more complex, I have no idea what goes on behind the heroku curtains either
Russell Tammany
ok well that’s interesting, you get the same error when there is no A record
Russell Tammany
but the error tells you to make an alias record instead
Bruce Chezem
that jives with their suggestions https://devcenter.heroku.com/articles/custom-domains#add-a-custom-root-domain
Russell Tammany
yeah. https://support.dnsimple.com/articles/alias-record/
I’m guessing that the A record pointer to the ELB was done at softlayer DNS before we switched to DNSimple
Bruce Chezem
yes, but I thought Bear made changes about a month ago to that
Russell Tammany
and then Bear or whoever forgot to revisit that after to make it an alias after the migration was successful
Bruce Chezem
when we finished up all the DNSimple stuff
Russell Tammany
I just remember something about heroku needing dnsimple instead of softlayer
Bruce Chezem
k….well I’m getting dressed and heading into the thriving metropolis that is petaluma
Russell Tammany
ok looks like the ALIAS record functions properly
tested with munchcast.com pointed at cnametest.twit.tv which I pointed back to the original munchcast ip which doesn’t seem to respond anyways, but just as a dns test
Russell Tammany
so because amazon “can” reset their ELB infrastructure at times, we should make that change tonight (delete the twit.tv A record and then setup twit.tv ALIAS ehime-4607.herokussl.com)
and then monitor for issues
i have to go to a client that i’m a bit late for
Patrick Delahanty
Other clients?!
Russell Tammany
yeah, but I really like working for wineries. it’s so chill
Russell Tammany
the most stressful thing that happens is their label printer machine burns out a printhead or the windows 2000 system running it doesn’t want to boot anymore
FYI @LeoLaporte this hipchat room is set to privacy public, so anyone at twit can join room and see history so just a reminder to change the heroku password later (and put it in lastpass admin accounts or something if you can)
or change the room to private when you are in it
or both
I’m afk for a bit
Leo Laporte
let’s change it to private
now private
and I will change the password and put it into lastpass enterprise
FWIW I am off lastpass and using 1password now
Leo Laporte
Is it fair to say that this was a DDoS that worked due to a misconfiguration on our part?
Russell Tammany
maybe, could have just been amazon changing out the heroku elb behind the scenes too, do we know for sure that there was high network traffic against the heroku servers? with newrelic or something?
Bruce Chezem
@Russell @LeoLaporte I put the creds in Heroku
i meant lastpass….i have heroku on the brain
Patrick Delahanty
Looks like it’s down again. 🙁
Russell Tammany
got to be some sort of attack then
Leo Laporte
Yeah
Russell Tammany
need to switch it to the proper alias record now then, and maybe needs a ticket with heroku setup?
Russell Tammany
ip’s changed again
i’m going to just do the alias setup, figured it would last until tonight where i could be more cautious about it but guess not
Russell Tammany
@all alias record is setup now to point to all three front end elbs
instead of the a record to just one of them
Leo Laporte
Nice
Russell Tammany
Name: elb065150-471723135.us-east-1.elb.amazonaws.com
Addresses: 23.21.76.215
54.243.32.107
23.21.95.158
Aliases: ehime-4607.herokussl.com
Leo Laporte
So would I be correct in thinking that it would take a fairly heft attack at this point to bring us down?
Russell Tammany
ALIAS twit.tv 600 ehime-4607.herokussl.com
Leo Laporte
It’s my best guess that this is a copycat Attack coming from our trolls who have far fewer skills than the GRC attacker
Russell Tammany
it should be more distributed, I’d imagine that heroku has monitoring setup to keep rotating servers but I’m not sure how their infrastrucutre is setup for ddos
Leo Laporte
Almost undoubtedly they don’t have the same bot army to command so they won’t be able to ramp up to 13 gigabits
At least that’s my hope
Russell Tammany
@BruceChezem can you open a ticket with heroku/ call them to see if they can check into that since I don’t seem to see anything from the heroku console side other than another block of error events
H13 Connection closed without response
all their load graphs look like they are for regular traffic
might be missing a dns reflection attack
Russell Tammany
might have to put cloudflare in front of heroku if it keeps up
Russell Tammany
from https://www.heroku.com/policy/security /code https://www.heroku.com/policy/security
DDoS Mitigation
Our infrastructure provides DDoS mitigation techniques including TCP Syn cookies and connection rate limiting in addition to maintaining multiple backbone connections and internal bandwidth capacity that exceeds the Internet carrier supplied bandwidth. We work closely with our providers to quickly respond to events and enable advanced DDoS mitigation controls when needed.
@all looks like we are back again?
Patrick Delahanty
Yep. Working again here.
Hopefully it holds this time.
Russell Tammany
if you nslookup twit.tv you’ll now see three ip’s
Name: twit.tv
Addresses: 54.243.32.107
23.21.95.158
23.21.76.215
Russell Tammany
and now when heroku cycles them to another ELB it will update there within 10 minutes
sort of just pasting that here for reference later to see if they change
Bruce Chezem
@Russell what did you mean above by ‘ips changed again’ ?
Russell Tammany
they were different a few hours ago
Bruce Chezem
would that suggest they were changed by someone or something or just reverted?
Russell Tammany
my best guess of what happened is this
months ago twit.tv was pointed to just one of the three elb ip’s which worked fine until someone attacked that one today, causing it to get shut off/overloaded at amazon
this morning I just changed the record to point to another one of the IP’s
that IP is now also gone from the elb
and there are three new ones (or at least one new one) in its place
I should have just done the alias thing this morning basically
but it wasn’t clear to me yet that it was any sort of attack as amazon can recycle ELB ip’s anytime for any reason
Bruce Chezem
at that is what they suggest….true?
alias
Russell Tammany
yeah heroku says use the alias
and from playing more with aws now I understand that you can’t ever rely on an IP address at amazon unless it’s an elastic IP tied to an instance
which the ELB’s are not
(elb = elastic load balancer)
Bruce Chezem
right….it makes sense
Russell Tammany
Also it appears heroku only gives us ELB in us-east-1
Bruce Chezem
I noticed that. Should we or they set up another region?
Russell Tammany
my plan for the redirect/metrics cdn.twit.tv was to have two regions, us-east-1 and us-west-1 both with ELB’s that have two AZ’s (availability zones) with route53 monitoring to trigger region failover
Bruce Chezem
should we be doing this directly with aws?
and back away from heroku?
Russell Tammany
We should ask how that works with heroku, they may move heroku, cname themselves if there is a region outage
since we alias to ehime-4607.herokussl.com and not ele065150-411723135.us-east-1.elb.amazonaws.com
heroku has the ability to point us dynamically at another region. it would just be a good conversation to have with them at this point
managing a ruby on rails deployment ourselves on aws is beyond my scope of knowledge 🙂
Bruce Chezem
where’s the bear?
Russell Tammany
I’d rather have support from heroku on it, but we need to be able to make sure they have mitigations for these things
as far as I know bear is flying back from SFO to PHL today
I’m not sure how involved he was in this infrastructure setup either. Wasn’t it all 4K designed?
Bruce Chezem
they were the architects but bear played a very active role in configuration
Russell Tammany
I’m going to update the existing heroku ticket just to let them know we implemented the alias record on twit.tv
Bruce Chezem
they’ll think that’s cool, I’m sure. 😎
Russell Tammany
updated, I have to get back to my other pile of crap here. will check back later
https://help.heroku.com/tickets/336350
Patrick Delahanty
Enjoy the vineyard
Russell Tammany
Do we have Heroku Premium support or just the free stuff? looks like we don’t have the premium support
can’t see it on the invoices which is probably why the ticket response is slooow
site still seems fast to me
Bruce Chezem
I haven’t been babysitting it, but I’ve been checking every so often….its been OK by my eyes too
Bruce Chezem
I don’t think we have premium support. I’ll look at support options in a few minutes….
it looks like next day is the SLA terms were on
Russell Tammany
let’s find out what they charge for premium support i guess
Bruce Chezem
@LeoLaporte @Russell Premium support is $1000/mo, 3 month minimum, 24 X 7, 1 hr response SLA
https://www.heroku.com/pricing
— Tuesday, March 1, 2016 —
Leo Laporte
where is the DDoS stuff @Russell
Russell Tammany
look in the Operations and Engineering room
— Friday March 4, 2016 —
Bruce Chezem
@Russell You needn’t be on that call Monday, thank you
Russell Tammany
Ok, no worries. See if you can get any secret sauce from them re: metrics analysis duplicates etc
Bruce Chezem
MM called and left a message for me this morning to call him back….he never calls me
Screenshots
