The Internal Bug Bounty Programme*

Please see this post before reading, for important caveats about the sources of information used to help construct this post.

Introduction

One of my favourite Security subjects is Bug Bounty Programmes, mainly because I was lucky enough to start work somewhere big enough to already have an internal programme in place and where I was wholeheartedly encouraged to get involved with its operation and help transform it into an external programme by its architect and implementer Dan Adams.

Internal Bug Bounty Programmes

The inspiration for this post came from this post on LinkedIn from the Product Security Engineering Manager of Lyft

As internal programmes are something I have experience with, but aren’t talked about publicly much, I figured it would make a decent blog post to start the New Year.

So, for those new to the idea of internal Bug Bounty Programmes, they are essentially private (invite-only) programmes, where invites are only available to staff members of the company offering the bounty payments (often an implicit blanket invite to all technical staff).

Doing this has some real big benefit and a few draw backs, so lets run through each in turn.

Learn as you go

Getting a Bug Bounty Programme right first time is very hard. An internal programme genuinely lets you develop robust processes and procedures ahead of an external launch.

If you’re upfront about the programme being a learning exercise then internal staff can be much more forgiving of slow triage, poor communication, vague or confusing scopes, and inconsistent or slow payments (all common problems with newly launched programmes). Making your internal hunters an important and valued part of your process can be the dividing line between “they listened to our feedback” and “they constantly keep moving the goalposts”.

Everything from your scope, your rules of engagement and your payment structure may need refining or even completely rewriting once you programme is live. That’s MUCH easier to achieve when it won’t cause disruption outside your organisation.

Flow Control

One lesson I learnt early from both Dan and from Bug Bounty Pioneer Katie Moussouris** is that one of the keys requirements of launching a bug bounty programme (or even knowing if you’re ready for one) is understanding your ability to handle the flow of incoming info the programme will bring. Having the resource to efficiently triage, remediate and pay bounties on new vulnerabilities is often a key differentiator between the launch of a successful programme and an embarrassing, expensive or unproductive one, but for many organisations launching as internal-only can limit that risk somewhat.

If you find you genuinely can’t keep up with the flood of new vulnerability reports, then being internal-only makes it much easier to change which classes of vulnerability are in-scope, refine the process or pause the programme altogether without upsetting the wider bug bounty community.

Unfair Advantage

When planning to launch a programme it’s not uncommon to lose sight of WHY you’re doing this. People try to create a fair and level playing-field so all hunters have the same chance of getting a bounty. If your aim is creating a challenge to find the best bug bounty hunter or simulate specific attacks then great, but it’s safe to say that ISN’T your aim. Most people are in it to collect the high quality vulnerability info they don’t already get from other sources, so why tie hunters hands behind the back.

The most successful internal programmes I’ve seem leverage the home-field advantage of internal hunters to go harder and deeper with less time wasted on dead ends by allow/encouraging hackers to review source code and design documents as well as going and talking with the development teams directly. Remember, even if you only care about public facing vulnerabilities and demand a PoC from a public network, that doesn’t mean your hunters need to be limited to doing their initial investigations from only public networks. It may be your website has a serious inject issue that’s currently mitigated by a WAF rule, but what happens when that rule is updated or you switch to a new vendor with a different vendor?

That said, it’s not uncommon to put the same Bounty Conditions on Internal hunters as you would on external ones, such as the vulnerabilities must be exploitable from the public internet.

Obviously achieving this may vary vastly from organisation to organisation, it’s not unusual for smaller organisations to have access controls that give all engineers the same flat access to all your source code repositories, design documents and network segments etc, but in larger organisations it may be necessary to allow hunters to apply to temporarily gain access to other teams’ resources to aid investigation.

Remember when it comes to paying, the value to you is in the information you receive, not the effort that was put into gaining that info, the fact internal hunters may have an unfair advantage and easier time doesn’t make their information any less valuable, so I’m therefore a big advocate of making internal payment structure identical to any external/public offerings you may have.

Dupes

One of the big annoyances for bug bounty hunters that has never really been addressed effectively is that of duplicate reports***. Most programmes only pay for the first reported instance of a vulnerability (otherwise not only are they paying for info they already have, but it would be possible for news of the vulnerability to pass from hunter to hunter) and understandably programmes don’t want to share publicly what vulnerabilities have been found but not yet fixed, so inevitably you end up with hunters using their time and talents to hunt down valid vulnerabilities, only to be told that they’ll get no reward as it’s already been reported.

In many cases internal programmes can lessen the pain, either by allowing staff to their internal bug trackers or encouraging communication between hunters and triagers can at least feedback if there have been reports for classes of vulnerabilities in the areas hunters are hunting.

Better Signal to Noise Ratio

Internal schemes can be much easier for internal triagers as the quality of the reports can be much higher and obtaining elaboration on unclear points can be so much easier to obtain.

Never underestimate the time that can be saved when the reporter has a decent understanding of your estate, the technology you use, your internal processes and your organisations general risk appetite. You tend to find that not only will you get high quality reports of the actual vulnerabilities, but often pointers to the source of the vulnerability.

Of course there will always be exceptions, but as a general rule internal reports are generally much quicker and easier to triage, contain much more info helpful to the team doing the remediation and contain far fewer false positives and incorrect assumptions.

Subterfuge and collusion

Whenever I talk about internal programmes the first question is “How do you stop people adding bugs so they can report them?”.

Whilst a rule that nobody can report any problem in a part of the code base they’ve worked on (or in some cases they were part of a team that worked on that area) is common, the real answer is much more cultural. Do you have a hiring policy that would attract people who would risk their employment and their reputation for a few thousand dollars in bounty payments?

The cons

So far I feel I’ve put forward a fairly compelling case that if you are in a position to launch a programme, start it as internal only, but what about the cons?

Over burdening hunters. Bug Bounty Hunting is fun, but it can also be infuriating, stressful and time consuming. Do you want your engineers spending all their free time thinking about finding vulnerabilities in your codebase? Or do you want them to switch off and think about something else entirely? Are you confident they’ll limit their hunting to just their free time, or will they actually be picking up a second job that impacts their first once. However, as with “Subterfuge and Collusion” if you feel this might be a problem, are you hiring the right people?

Rapidly Declining Interest. Whilst a new internal programme may introduce many new people to the world of bug bounty hunting, for the majority of people interest will wane over time and the flow of new vulnerability info will reduce. Things like short term incentives and events like charity hack days can garner increased participation but at some point it’ll stop being the cool and new thing and you’ll have to start looking externally for new participants.

Disillusioning external hunters. Bug Bounty hunters hanker for new programmes where all the low hanging fruit has been hoovered up and everything the find isn’t already reported and therefore a dupe. So, when you’re ready to move from internal to external, having it known amongst external hunters that your programme has already been ravaged by bug hunters and there the effort/reward balance will be more favourable elsewhere

Final Tips

Don’t launch an Internal Bug Bounty Programme, build an internal Bug Hunting Community, with the programme at the heart of it.

Plan for the future, always ask yourself “Would this also be Ok if the programme was a public one?”

Fix problems at source. It’s easy to use the Bug Bounty Programme as a ticket creation factory, but ensure you’re collecting data on the root cause of vulnerabilities and feeding that data back into the SDLC

Define your metrics, goal and reporting from the start. Even internal Programmes are expensive to run, so ensure you can prove the value of the scheme. Remember knowing about 1000 unmitigated make you at no less risk the knowing about 10 unmitigated ones, be able to prove programme is actually lowering risk.

Make the journey from your Bug Bounty Platform as frictionless as possible, so people choose to report things via the bug bounty platform, even if they are not sure if they actually qualify for bounties, rather have them sat on bugs where they can see something isn’t right, but can’t exploit it in a way that would guarantee a bounty

Don’t be afraid to change things. Launching as an Internal Programme gives you more freedom to change things that aren’t working as well as you’d hoped, use that to your advantage.


* You’ll notice that when referring to Bug Bounty Programmes I use the older English (British) spelling of the word Programme, but when referring to computer Programs I use the newer (US) one. This is intentional, though I can’t really justify it, other than I’ve always felt it was correct

** If you’re interested in Bug Bounties and you don’t know who Katie is, stop reading now and go read everything she’s written on the subject first.

*** Whilst it doesn’t really help on-going programmes for their events (where certain sites are put in in-scope to a limited period to an invited set of hunters) HackerOne have an initial window (normally 1hr) where hackers can report all the low-hanging-fruit they found during reconnaissance before the event and they all the share the bounty payment, which is an interesting way to tackle the inevitable dupes where when of the best bounty hunters in the world are given the same target and some advanced warning.

An early New Year Resolution

Previously, I had a job whose Social Media policy pretty much prevented me from talking about Security related stuff on Social Media / Blogs / Cons etc as people might believe it was pertaining to their specific environment and concerns (even if it wasn’t). So I cleared down my blog of anything technical and let this site go fallow.

I’m now very grateful I work somewhere much much cooler about such things, in fact we’re positively encouraged to blog and speak at events etc, but I’ve never really taken advantage of it, until now. So my New Year Resolution (one of several) is to blog much more.

However, one thing it’s very important to keep in mind is that whilst some writing may be based from my current role, much of it isn’t. Much of it is learnt from third parties who I’ve gone to, to understand a subject more completely, so don’t assume anything best practice or worst practice, witty anecdote or cautionary tale is actually indicative of any employer past or present.

WannaCry – When the press pick the wrong boogeyman and nobody listens

It was weird watching the events around WannaCry unfold on Friday in the context of my current job role as for the first time since, the ILOVEYOU worm (which turned 17 years old earlier this month) my role meant that I didn’t have any responsibility for any potentially impacted hosts. After a lifetime of working in small companies in either a security, IT, incident response or management, there was now a major bit of malware propagating that people were unprepared for and I now worked somewhere big enough that we had other people to manage such things.

So, rather than being in the middle of the fire-fight, I sat back, glued to twitter and watched things unfold. It was great to watch some of the worlds (or at least in the early hours, the UKs) best security researchers dig into whats happening and share their findings in real time on twitter.

It became apparent that the world’s tech journalists were also following them and a lot of non-technical ones too. But as we saw a few weeks ago when the latest ShadowBrokers cache was dropped, stuff goes from 140 chars of “current thinking” to being reported worldwide as fact, pretty quickly.

The same seems to have happened again with this, only this time the misapprehension of some early tweets is leading to the blame for the rapid propagation being laid fairly and squarely at the feet of Windows XP and other EoL software. Whilst they are undoubtedly contributing, they are far from the only culprits and come Monday morning there are going to be a lot of people who thought “We don’t have any XP machines, nothing to worry about” who’ll be facing a huge infection.

Now, XP didn’t become the boogey man by accident, there are some reasons why XP was mentioned a lot in those early tweets,

1. Unlike newer versions of Windows, at the time of the the outbreak there was no patch to prevent infection for XP (Microsoft released a patch for newer OSs in March, but has also now released an XP patch)

2. The infection requires SMB v1, a protocol that can happily be disabled in a Windows environment if you don’t have XP/Server 2003 machines on your network.

It’s also noted that NHS is notorious for running legacy software like XP.

However, what the journalists on the whole have failed to take into account is

1. Just because you can (and should) disable SMB v1, it doesn’t mean people have done. Many people won’t know to, others will have non-windows devices using SMB v1 and for some, it’s just too much of a risk to change anything unnecessarily on a production network.

2. Just because Windows OSs newer than XP/Server 2003 have had a patch available since March, it doesn’t mean people have applied it.

3. Most big corps will have perimeter firewalls that prevent direct infections, but how many people have their work laptops getting infect on public wifi this weekend, only to plug that laptop onto the corporate LAN on Monday morning.

So, this ISN’T about XP,  people could have removed their last XP machine decades ago, but unless they’re getting patch management right and are on top of their network configuration (disabling unnecessary features and segmenting as much as they practically can) then come Monday morning the could be met with a rather unexpected headache.

Pentestit.ru noobs guide

Pentestit.ru noob guide

Whilst I’m a “blue teamer” (I specialise in the defensive side of InfoSec), I do enjoy doing Pentesting challenges both for fun and because “know your enemy” & “always think like an attacker” are invaluable bits of advice for any defender.

One of my favourites is Pentestit.ru Labs which closely mimic real environments you may come across in actual pentests and are designed in such as way that whilst they have a gentle learning curve, they normally require you to have decent IT and security fundamentals, rather than being aimed at people who have installed Kali and are watching “How to be a hacker” YouTube videos.

However, for the last year or so I’ve been sat in the pentestit.ru Telegram channel and whilst there are and awful lot of very knowledgeable people in the channel (far more knowledgeable than me, many are professional pentesters or prepping for things like OSCP exams) there are also a lot of people who join the channel that are really struggling with the basics. So I thought I’d put together a quick FAQ for those guys.

Do I need to use Kali Linux?

No. In fact, I intend to do the next lab entirely from a Windows Box to prove it’s possible (and because I love Powershell). However, if you’re asking that question, I strongly recommend you do as it’s probably the easiest starting platform to attack from.

Do I need to use PentestIt.ru’s downloadable Kali VM?

Again, no.  I’d wager almost everyone completing the labs is doing so using a vanilla Kali build. The only difference with the one on the Pentestit.ru website is it come pre-configured with everything you need to connect to their vpn.

If you’re going to struggle connect to a vpn from Kali when then instructions are on their website, you’d probably be better spending your time reading some linux vpn guides first.

The vpn is connected and I can ping their gateway machine(s). Now what?

Start your pentest! Normally these labs only start with one (or possibly two) gateways machines exposed, so don’t expect to be able to access the servers behind the gateway directly. However, usually these labs do have port forwarding set up for some services, so for example hitting port 25 on the gateway machine is likely to be forwarded to port 25 on the “email host” on the internal network.

You’re normally looking for some way to compromise the gateway machine (or some machine port forwarded to from it) and then pivot to the internal hosts.

I’m on the vpn, but I get disconnected constantly. Why?

Check you don’t have more than one vpn connection active to their labs as this will normally cause them to disconnect. Failing that, check out the Pentestit.ru Service channel for service outage notifications.

I’m on the vpn, but I get disconnected hourly. Why?

Many of the hosts reset on the hour. so may disconnect you and remove any changes you have made. This in intentional. If you have something running on a host that takes over an hour (i.e. some kind of brute force attack) you are probably approaching the problem in the wrong way.

I used “<insert tool name>”  and it found nothing. Why?

One of the great things about these labs is that they are often engineered to make life harder for people using automated tools (especially with the default options) and easier on those actually doing the attacks manually. So, just because sqlmap fails to find an sql injection point, a given password isn’t in the default john the ripper list, nmap doesn’t find an open port on a default scan or a folder isn’t found by dirb doesn’t mean that that approach isn’t going to work. The lab designers know these tools well too and want to give you more of a challenge than “can you run the right tool with the default options”.

How do I get admin/root?

I’m sure they’ll prove me wrong at some point, but it’s not likely you ever need admin or root on a host to get the token. This actually makes sense, as with root access you could easily screw over the challenge for other people. However traversing between users with different privs is quite common, often using techniques more commonly associated with escalating to root.

I’m stuck, now what?

Try Harder!

Seriously, that’s probably the first response you’re likely to get in the Telegram channel. Possibly with a link to this

It’s good advice. Go away, make a coffee, have a smoke, play Hello Kitty Island Adventure … whatever works for you. We’ve all got stuck on a challenge, then come back later with a fresh bunch of ideas.

But keep trying. These challenges are usually pretty logical and based on real world exploits, so take what you know about the situation and go hit the books (or google) and see if there is something more to learn.

Seriously, I’ve been trying for days, now what?

Well, the telegram channel is always there, but most of the people in it try to keep it spoiler free, so the usual etiquette is to ask for somebody to DM you about whatever you are struggling with.

Also, once the winners of a challenge are announced, people start publishing their solutions, these are great for getting you past you’re current hurdle, however, be very cautious as once you’ve cheated and taken a peek that first time, it becomes much easier to cheat every other time.

I’ve finished this lab, now what?

Try this list, or come hang out in the telegram channel and see what others are currently working on.

Obviously Disclaimer: I’m not part of the pentestit.ru team, just a fan of their work and this does not constitute official pentestit.ru documentation or is in anyway endorsed by them