Tuesday, June 24, 2014

El Jefe v2.1 Release

It has been a couple of weeks since our last communication but the team is working hard to make El Jefe your first and best choice for threat analysis.

Today's release, while technically a minor release, has many new features.

Cuckoo configuration is not only fully integrated into our WebUI, but we also modified the implementation to allow you to use remote Virtual Machines! El Jefe now lets you configure where and how you want to run the sample inside the sandbox, allowing you to properly use it to look at more complex malware problems.
Configure a remote Sandbox Virtual Machine

For example, assume that you are ahead of the curve and you are using El Jefe to monitor your C-level executives. Each of these will probably have their own software set-up, operating system, etc. Once you find a potential threat (manually, or using the built in El Jefe heuristics), you will want to understand the penetration behavior of the sample in the exact same environment it was run in on the executive's computer. That's why you set up a different Virtual Machine to mimic each executive's personal environment and you can have the whole process run through El Jefe.

Select the right Sandbox Virtual Machine to run your sample

We want to be as open as possible and give you the freedom to interface El Jefe in new and powerful ways that we have not even thought of. That's why we built a plugin to allow you to use COSEINC's  CAMAL instead of Cuckoo to get even more accurate results from your sandbox. (Kudos to Thomas and the COSEINC team for their fast response and an excellent product!)
Integration with CAMAL

El Jefe clients are now fully configured via the El Jefe UI, which will certainly make your life easier when deploying across your user-base.
Set-up your client completely from the WebUI

El Jefe clients now support proxies - which is obviously important for those of you in large enterprises.

People seem to love the Event filter, so we've started adding more features to it. You can now trigger an email warning and send suspicious binaries directly to sandbox analysis. This is just an example, of course. Our new API is easy enough to learn that you can create your own event filter in less than 20 minutes - which is important if you are in the middle of an Incident. Fast reaction times when facing custom malware and an advanced penetration team are often the only hope of your IR team for containing an ongoing compromise.

In case you didn't notice, El Jefe is released by Immunity under the GPL v3 license. But, just in case, we now let you know in every file. Source code to El Jefe (for both the client and the server) is provided on the website and will soon be moved to a GitHub repository.

We also put together an installer for RHEL (widely used in enterprises) along with the Ubuntu one to make everything install automatically and we have a beautifully crafted PDF with a step by step guide on how to install El Jefe.

Whether you're currently responding to an incident or you think you may someday want to respond to an incident, we think installing and getting to know El Jefe is a great first step for you. It's Free, and it works, and it's only getting better.

Of course, we welcome any of your feedback - send it to support@immunityinc.com and we'll be happy to respond!

Download it here
sha1sum: b8ee361ecf67e76ec0888e570153f76b15dfcea5  eljefe2.1.release.tar.gz

Thursday, May 8, 2014

Connecting El Jefe 2.0 with the Cuckoo malware sandbox

One of the great new features in ElJefe May release is the integration of the Cuckoo malware analysis system as part of our interface. Cuckoo runs the malware executable in a sandboxed environment inside a virtual machine and produces amazing data, which we display in the El Jefe interface for you. However, configuring it is not a trivial task. That's why we put together a little blogpost to make our users happier. Always thinking about our El Jefe users, is our motto!

We are going to setup the host machine and the guest machine. The last one is where we are going to run the malicious files.
The following commands will install the necessary files:
  $ sudo apt-get install python python-sqlalchemy python-bson python-pip libcap2-bin
  $ sudo pip install sqlalchemy bson Django
You also need to install mongodb. You can download the necessary files from http://www.mongodb.org/downloads.
When Cuckoo is analyzing a submitted files there are some modules and libraries we can install to get better and more complete analysis and reports. These modules are optional but their installation is highly recommended.
We are using setcap to give privileges to tcpdump to run as root without having to run Cuckoo as root.
  $ sudo setcap cap_net_raw,cap_net_admin=eip /usr/sbin/tcpdump
We can check the results with:

  $ getcap /usr/sbin/tcpdump
  /usr/sbin/tcpdump = cap_net_admin,cap_net_raw+eip

You will need to create a new user, you can do that with the following command:
  $ sudo adduser cuckoo
You can download Cuckoo from http://www.cuckoosandbox.org/download.html. You will need to copy the Cuckoo folder into the El Jefe root folder. You should see a directory listing like this:
client cuckoo dependencies installer webapp

You will also need to uncomment some lines of code to make cuckoo work:
webapp/settings.py lines 105 to 124
webapp/xmlserver/ElJefeXmlServer.py lines from 55 to 58.
webapp/home/views.py lines 44, 45, 1055, 1316 to 1334.
webapp/analysis/views.py lines 20 to 25.
webapp/templates/base_.html remove the {% comment %} and {% endcomment %} tags on lines 121 and 142.

Now it's time to setup the virtual machine where we are going to run and analyze our binaries. Cuckoo supports VirtualBox, KVM and VMWare for virtualization. We choose VMWare as our virtualization software, so the following steps will show you how to configure Cuckoo to work with VMWare. If you wish to use other virtualization software, follow the guidelines in the following URL: http://docs.cuckoosandbox.org/en/latest/installation/guest/.

First, we'll need to create a new virtual machine. The preferred OS is Windows XP, but you can use the OS of your preference. Obviously for future versions of El Jefe we will support automatically choosing the right target VM based on the El Jefe client's OS.

We need to install Python on the virtual machine and PIL (http://www.pythonware.com/products/pil/) if we want Cuckoo to be able to take screenshots of the binary as it runs.

Now it's time to configure the virtual machine's networking. We are going to disable the Windows Firewall first, and then create a virtual host only network for the guest and the host.

We are going to use the network, the guest configuration will be:
IP Address:
Cuckoo supports DNS resolution, so you can use as your DNS server. In our experience, we get better analysis results by using a public DNS server.

And the host configuration (for the vmnet adapter) will be:
Ip Address:
You can choose whatever network and addreses you like.

Now we need to configure packet forwarding on the host machine to give the guest machine Internet access. We can do this with the following commands ( replacing eth0 with your outgoing interface and vboxnet0 with your virtual interface).
iptables -A FORWARD -o eth0 -i vboxnet0 -s -m conntrack --ctstate NEW -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A POSTROUTING -t nat -j MASQUERADE
sysctl -w net.ipv4.ip_forward=1
This concludes the networking setup. Now we need to install the cuckoo agent on the guest, by copying agent.py, located in cuckoo/agent in the El Jefe root folder, to the Windows Startup folder.

Now it's time to make a snapshot. Before executing a malware binary the snapshot is reverted and then the binary is executed and analyzed.

Next, we will see how to setup the configuration files on Cuckoo.

Change the interface to the one you are using.

Set your db connection on the [database] section and the result server IP address on the
[resultserver] section (we are using

Set the mongodb settings on the [mongodb] section.

If you are going to use only one VM, you will need to modify the stuff inside the [cuckoo1] section only, otherwise you will have to create one additional section for every extra VM you wish to use.
Inside these sections you will have to set the VM path, snapshot name and IP address of the machine (we are using

We are done setting Cuckoo for El Jefe. Before starting El Jefe remember to start the mongodb and postgresql services. You will also need to start the cuckoo.py service from cuckoo/cuckoo.py in the El Jefe root folder.

So in conclusion, you should now have Cuckoo set up for yourself, integrated into the world's most powerful open source host monitoring system, and ready for your incident response team to use. We welcome any comments or questions, and of course are ready to help you if you have problems.

David Arch

Friday, April 11, 2014

Revamping El Jefe

El Jefe 2.0 - Process Chain Visualization and Heuristics

One improvement in modern Anti-Virus is the move away from signatures to heuristics and program behavior analysis on any one process. This turns out to be a decent protection at first, but as attackers evolve it will rapidly become less effective. El Jefe offers a radically different leap-frog in the level of analysis done with its protective heuristics by looking at the entire chain of process creation, rather than each process creation event alone.

Walking through the events on multiple stations

Intuitively, a human being will ask questions based on process chain anomalies:
  • Why is IExplorer.exe popping up commands like crazy? 
  • Why does Adobe PDF process spawn another process as LOCAL/SYSTEM?
  • Why did this user suddenly start using commands like wmic, which only a system administrator normally would use?
A big part of doing analysis is looking at pages and pages of text looking for patterns and ways to correlate stuff to find anomalies. This takes a tremendous amount of work, and has the risk of becoming a stultifying routine for the analyst. In other words, many companies are drowning in their own Big Security Data. 
As part of our Digital Executive Protection program, we spend hours looking at our client's high process creation data looking for potential signs of attack. One way we found to make this more effective is to add alternative ways to inspect and visualize this data.
The D3 JavaScript library provides us with  great resource for doing that. The first visualization tool we built show us easily the amount of usage of any given process. Instead of going the traditional way with big colorful circles for the most used processes, we invert it so with one fast view you can look at the processes that are hardly ever used - the exceptions which are most likely to be worth looking at further.
Process Usage: An easy way to identify process used only a small amount of time

The second graph we are experimenting with examines the relationship between all the processes executed on the system. This provides us with an easy way to move around different processes that have been run on any system and understand how they were triggered, by who and which activities were done.

Event Relationsip: Move around events and find the relationship with each other.

You might notice events are a key feature in El Jefe. One thing we want to explore is the correlation of an event's properties within multiple instances of that event. For example, when people use IE to host their trojans that IE has a very different memory and thread-count from a normal IE. This can be visualized in El Jefe and will stand out even though IE itself is not malicious.

Analyzing triggered events over time 
The second big feature we will introduce in this new version is the integration with the famous Cuckoo sandbox. We love the work that team has done over the years and it matches up perfectly with the procedure our analysts were using for El Jefe. We connect to Cuckoo within El Jefe, so now every time you find a suspicious binary, you can seamlessly ask El Jefe to grab the binary from the target machine and run it on the isolated Cuckoo sandbox. The result, for those unfamiliar with Cuckoo, is a beautiful report on what the binary has done, including files dropped, registries touched and a PCAP with everything the binary has sent over the network.
Event Inspection
Based on these new cool features we think El Jefe is a good addition to a company's security stance - given that it is both free, extensible, and more effective than traditional AV tools.

Keep tuned, because this new release is around the corner!

Monday, November 4, 2013

Exploiting CVE-2013-3881: A Win32k NULL Page Vulnerability

Microsoft Security Bulletin MS13-081 announced an elevation of privilege vulnerability [http://technet.microsoft.com/en-us/security/bulletin/ms13-081]. Several days later Endgame published [http://endgame.com/news/microsoft-win32k-null-page-vulnerability-technical-analysis.html] some further details on the vulnerability in question but did not provide full exploitation details. In this post we will discuss how to successfully exploit CVE-2013-3881.

The Vulnerability

The vulnerability resides in xxxTrackPopupMenuEx, this function is responsible for displaying shortcut menus and tracking user selections. During this process it will try to get a reference to the GlobalMenuState object via a call to xxxMNAllocMenuState, if the object is in use, for example: when another pop-up menu is already active, this function will try to create a new instance.

If xxxMNAllocMenuState fails it will return False but it will also set the pGlobalMenuState thread global variable to NULL. The caller verifies the return value, and in case of failure it will try to do some cleanup in order to fail gracefully.

During this cleanup the xxxEndMenuState procedure is called. This function's main responsibility is to free and unlock all the resources acquired and saved for the MenuState object, but it does not check that the pGlobalMenuState variable is not NULL before using it. As a result a bunch of kernel operations are performed on a kernel object whose address is zero and thus potentially controlled from userland memory on platforms that allow it.

Triggering the vulnerability is relatively easy by just creating and displaying two popup instances and exhausting GDI objects for the current session, as explained by Endgame. However, actually getting code execution is not trivial.


Usually a NULL dereference vulnerability in the kernel can be exploited by mapping memory at address zero in userland memory (when allowed by the OS), creating a fake object inside of this null page and then triggering the vulnerability in the kernel from the current process context of your exploit which has the null page mapped with attacker controlled data. With some luck we get a function pointer of some sort called from our controlled object data and we achieve code execution with Kernel privileges (e.g. this was the case of MS11-054). As such, NULL dereference vulnerabilities have for many years provided a simple and straightforward route to kernel exploitation and privilege escalation in scenarios where you are allowed to map at zero.

Unfortunately  in the case of CVE-2013-3881 life is not that simple, even on platforms that allow the null page to be allocated.

When xxxTrackPopupMenuEx calls xxxMNAllocMenuState and fails, it will directly jump to destroy the (non-existant) MenuState object, and after some function calls, it will inevitably try to free the memory. This means that it does not matter if we create a perfectly valid object at region zero. At some point before xxxEndMenuState returns, a call to ExFreePoolWithTag(0x0, tag) will be made. This call will produce a system crash as it tries to access the pool headers which are normally located just before the poolAddress which in this case is at address 0. Thus the kernel tries to fetch at 0-minus something which is unallocated and/or uncontrolled memory and we trigger a BSOD.

This means the only viable exploitation option is to try and get code execution before this Free occurs.

Situational Awareness

At this point we try to understand the entire behavior of xxxEndMenuState, and all of the structures and objects being manipulated before we trigger any fatal crash. The main structure we have to deal with is the one that is being read from address zero, which is referenced from the pGlobalMenuState variable:

+0x000 pGlobalPopupMenu : Ptr32 tagPOPUPMENU
+0x004 fMenuStarted : Pos 0, 1 Bit
+0x004 fIsSysMenu : Pos 1, 1 Bit
+0x004 fInsideMenuLoop : Pos 2, 1 Bit
+0x004 fButtonDown : Pos 3, 1 Bit
+0x004 fInEndMenu : Pos 4, 1 Bit
+0x004 fUnderline : Pos 5, 1 Bit
+0x004 fButtonAlwaysDown : Pos 6, 1 Bit
+0x004 fDragging : Pos 7, 1 Bit
+0x004 fModelessMenu : Pos 8, 1 Bit
+0x004 fInCallHandleMenuMessages : Pos 9, 1 Bit
+0x004 fDragAndDrop : Pos 10, 1 Bit
+0x004 fAutoDismiss : Pos 11, 1 Bit
+0x004 fAboutToAutoDismiss : Pos 12, 1 Bit
+0x004 fIgnoreButtonUp : Pos 13, 1 Bit
+0x004 fMouseOffMenu : Pos 14, 1 Bit
+0x004 fInDoDragDrop : Pos 15, 1 Bit
+0x004 fActiveNoForeground : Pos 16, 1 Bit
+0x004 fNotifyByPos : Pos 17, 1 Bit
+0x004 fSetCapture : Pos 18, 1 Bit
+0x004 iAniDropDir : Pos 19, 5 Bits
+0x008 ptMouseLast : tagPOINT
+0x010 mnFocus : Int4B
+0x014 cmdLast : Int4B
+0x018 ptiMenuStateOwner : Ptr32 tagTHREADINFO
+0x01c dwLockCount : Uint4B
+0x020 pmnsPrev : Ptr32 tagMENUSTATE
+0x024 ptButtonDown : tagPOINT
+0x02c uButtonDownHitArea : Uint4B
+0x030 uButtonDownIndex : Uint4B
+0x034 vkButtonDown : Int4B
+0x038 uDraggingHitArea : Uint4B
+0x03c uDraggingIndex : Uint4B
+0x040 uDraggingFlags : Uint4B
+0x044 hdcWndAni : Ptr32 HDC__
+0x048 dwAniStartTime : Uint4B
+0x04c ixAni : Int4B
+0x050 iyAni : Int4B
+0x054 cxAni : Int4B
+0x058 cyAni : Int4B
+0x05c hbmAni : Ptr32 HBITMAP__
+0x060 hdcAni : Ptr32 HDC__

This is the main object which xxxEndMenuState will deal with, it will perform a couple of actions using the object and finally attempts to free it with the call to ExFreePoolWithTag. The interaction with the object that occurs prior to the free are the ones we have to analyze deeply as they are our only hope in getting code execution before the imminent crash.

xxxEndMenuState is a destructor, and as such it will first call the destructor of all the objects contained inside the main object before actually freeing their associated allocated memory, for example:


The _MNFreePopup call is very interesting, as PopupMenu objects contain several WND objects and these have Handle references. This is relevant because if this WND object has its lock count equal to one when MNFreePopup is called, at some point it will try to destroy the object that the Handle is referencing. These objects are global to a user session. This means that we can force the deletion of any object within the current windows session, or at the very least decrement its reference count.

+0x000 fIsMenuBar : Pos 0, 1 Bit
+0x000 fHasMenuBar : Pos 1, 1 Bit
+0x000 fIsSysMenu : Pos 2, 1 Bit
+0x000 fIsTrackPopup : Pos 3, 1 Bit
+0x000 fDroppedLeft : Pos 4, 1 Bit
+0x000 fHierarchyDropped : Pos 5, 1 Bit
+0x000 fRightButton : Pos 6, 1 Bit
+0x000 fToggle : Pos 7, 1 Bit
+0x000 fSynchronous : Pos 8, 1 Bit
+0x000 fFirstClick : Pos 9, 1 Bit
+0x000 fDropNextPopup : Pos 10, 1 Bit
+0x000 fNoNotify : Pos 11, 1 Bit
+0x000 fAboutToHide : Pos 12, 1 Bit
+0x000 fShowTimer : Pos 13, 1 Bit
+0x000 fHideTimer : Pos 14, 1 Bit
+0x000 fDestroyed : Pos 15, 1 Bit
+0x000 fDelayedFree : Pos 16, 1 Bit
+0x000 fFlushDelayedFree : Pos 17, 1 Bit
+0x000 fFreed : Pos 18, 1 Bit
+0x000 fInCancel : Pos 19, 1 Bit
+0x000 fTrackMouseEvent : Pos 20, 1 Bit
+0x000 fSendUninit : Pos 21, 1 Bit
+0x000 fRtoL : Pos 22, 1 Bit
+0x000 iDropDir : Pos 23, 5 Bits
+0x000 fUseMonitorRect : Pos 28, 1 Bit
+0x004 spwndNotify : Ptr32 tagWND
+0x008 spwndPopupMenu : Ptr32 tagWND
+0x00c spwndNextPopup : Ptr32 tagWND
+0x010 spwndPrevPopup : Ptr32 tagWND
+0x014 spmenu : Ptr32 tagMENU
+0x018 spmenuAlternate : Ptr32 tagMENU
+0x01c spwndActivePopup : Ptr32 tagWND
+0x020 ppopupmenuRoot : Ptr32 tagPOPUPMENU
+0x024 ppmDelayedFree : Ptr32 tagPOPUPMENU
+0x028 posSelectedItem : Uint4B
+0x02c posDropped : Uint4B

In order to understand why this is so useful, let's analyze what happens when a WND object is destroyed:

pWND __stdcall HMUnlockObject(pWND pWndObject)
pWND result = pWndObject;


if (!pWndObject->cLockObj)
result = HMUnlockObjectInternal(pWndObject);
return result;

The first thing done is a decrement of the cLockObj counter, and if the counter is then zero the function HMUnlockObjectInternal is called.

pWND __stdcall HMUnlockObjectInternal( pWND pWndObject)
pWND result;
char v2;

result = pWndObject;

unsigned int entryIndex;
pHandleEntry entry;

entryIndex = pWndObject->handle & 0xFFFF;
entry = gSharedInfo.aheList + gSharedInfo.HeEntrySize * entryIndex

if ( entry->bFlags & HANDLEF_DESTROY )
if ( !(entry->bFlags & HANDLEF_INDESTROY) )
result = 0;
return result;

Once it knows that the reference count has reached zero, it has to actually destroy the object. For this task it gets the handle value and applies a mask in order to get the index of the HandleEntry into the handle table.
Then it validates some state flags, and calls HMDestroyUnlockedObject.
The HandleEntry contains information about the object type and state. This information will be used to select between the different destructor functions.

int __stdcall HMDestroyUnlockedObject(pHandleEntry handleEntry)
int index;
index = 0xC * handleEntry->bType
handleEntry->bFlags |= HANDLEF_INDESTROY;

return (gahti[v2])(handleEntry->phead);


The handle type information table (gahti) holds properties specific to each object type, as well as their Destroy functions. So this function will use the bType value from the handleEntry in order to determine which Destroy function to call.

At this point it is important to remember that we have full control over the MenuState object, and that means we can create and fully control its inner PopupMenu object, and in turn the WND objects inside this PopupMenu. This implies that we have control over the handle value in the WND object.

Another important fact is that entry zero on the gahti table is always zero, and it represents the FREE object type.

So our strategy in order to get code execution here is to, by some means, create an object whose HandleEntry in the HandleEntry table has a bType = 0x0, and bFlags = 0x1. If we can manage to do so we can then create a fake WND object with a handle that makes reference to this object of bType=0x0. When the HMDestroyUnlockedObject is called it will end up in a call gahti[0x0]. As the first element in gahti table is zero, this ends up as a "call 0". In other words we can force a path that will execute our controlled data at address zero.

What we need

We need to create a user object of bType=FREE (0x0) and bFlags= HANDLEF_DESTROY (0x1).
This is not possible directly, so we first focus on getting an object with the bFlag value equal to 0x1. For this purpose we create a Menu object, set it to a window, and then Destroy it. The internal reference count for the object did not reach zero because it is still being accessed by the window object, so it is not actually deleted but instead flagged as HANDLEF_DESTROY on the HandleEntry. This means the bFlag will equal to 0x1.

The bType value is directly associated to the Object Type. In the case of a menu object the value is 0x2 and there is no way of creating an object of type 0x0. So we focus on what ways we have to alter this value using some of the functions being called before destroying the WND object.

As you can probably remember from the PopupMenu structure shown before, it contains several WND objects, and one of the first actions performed when HMUnlockObject(pWnd) is called is to decrement the lockCount. So we simply set-up two fake WND objects in such a way that the lockCount field will be pointing to the HandleEntry->bType field. When each of those fake WND objects is destroyed it will actually perform a “dec” operation over the bType of our menu object, thus decrementing it from 0x2 to 0x0. We now have a bFlag of 0x1 and a bType of 0x0.

Using this little trick we are able to create a User object with the needed values on the HandleEntry.


First we will create a MenuObject and force it to be flagged as HANDLEF_DESTROY.

Then we will trigger the vulnerability, where xxxEndMenuState will get a reference to the menuState structure from a global thread pointer, and its value will be zero. So we map this address and create a fake MenuState structure at zero.

XxxEndMenuState will call FreePopup(..) on a popup object instance we created, and will in turn try to destroy its internal objects. Three of these objects will be fake WND objects which we also create. The first two will serve the purpose of decrementing the bType value of our menu object, and the third one will trigger a HMDestroyUnlockedObject on this same object. This will result on code execution being redirected to address 0x0 as previously discussed.

We have to remember that when we redirect execution to address 0, this memory also servers as a MenuState object. In particular the first field is a pointer to the PopupMenu object that we need to use. So what we do is to choose the address of this popup menu object in such a way that the least significant bytes of the address also represent a valid X86 jump opcode (e.g. 0x04eb translates to eb 04 in little endian memory ordering which represents a jump 4).

Finish him!

Once we achieve execution at ring 0 we patch the Enabled field on the _SEP_TOKEN_PRIVILEGES structure from the MOSDEF callback process in order to enable all the privileges for the process. We fix up the HandleEntry we modified before, and restore the stack in order to return after the PoolFree thus skipping the BSOD.

Once all of this is done we return to user-land, but now our MOSDEF process has all the privileges, this allows us to for example migrate to LSASS and get System privileges.

-- Matias Soler

Tuesday, August 6, 2013

Blackhat 2013 -- A Vendor's Perspective

Immunity was a Blackhat sponsor again this year, potentially our last outing for a while. Thanks to everyone who came by our booth! It was fun to meet customers face to face and friends we don't get to see that often.

Things I observed

- Let me define booth babes as someone you short-term hire specifically to work your booth to attract people's attention based on their looks. I only saw one vendor, ironically an educational vendor, who had staff that fit this description.

- I made it a point to talk to some women who came through our booth about booth babes and I found some very different definitions as to what would qualify someone. The most liberal definition was the babe in question could be a full time employee but if they got especially "tarted up" for their booth time then they qualified. By this definition there appeared to be significantly more booth babes in attendance.

- One vendor who put up an enormous booth near the front had, and I'm not kidding, a grandpa doing a magic show. Later their PR person came over and introduced himself scouting for business. I wish I had the presence of mind to ask how that decision happened.

- Did Randy Couture count a male booth babe or as a celebrity endorsement? If he is a booth babe he's the only one who can easily get me in a rear naked choke, so he's whatever he wants to be.

Things we Learned

- The big buzzword this year was "managed", manage your VPNs, manage your logs, manage your certificates, manage your ssh keys (?!), manage your life!

- Nico and I both walked around and didn't see any new products that blew our minds.

- Immunity went with no dedicated sales staff and I think it worked out well. People were pretty surprised when they talked to someone who knew what was going on with their product. Is it worth taking technical people off of other projects to staff a booth? I think regarding reputation it probably is, regarding revenue still remains to be seen.

- I saw a bunch of vendors with six figure booths setting up seating and making people watch movies. I didn't see a lot of butts in seats. What did work surprisingly well was a trivia game the Venafi folks set up where you could win an Apple TV. Every time they did this they had a pretty sizable crowd and they were nice guys to boot.

- When Nico approaches your booth where you're advertising a product to implement "zero day protection" to ask some very pointed questions, that's an intimidating situation. But these folks weren't intimidated. Why? Because they were marketing and sales engineering people who had no idea how their product actually worked to survive any level of professional scrutiny.

- Almost all of the material I demoed for SWARM was stuff I found the day before the sponsor hall opened. David A. and I put in a crap ton of work getting the SWARM set up working in a laptop powered VM but not so much on what we were going to show. It created the opportunity to find something new in our dataset and get excited about it which made a really effective demo.

- We had a bunch of grumpy old men approach our booth this year. They all seemed to respond well to me giving it right back to them. Perhaps a winning strategy?

- I saw folks throwing out some guesses about the number of women present. I saw 1:15 through 1:30, I wasn't keeping count (that would be creepy) but it seemed like more than last year. I chatted with @Tardissauce a bit about this at hackcup. Her thought was that Blackhat tends to attract attendees higher up the corporate ladder than DefCon, there are more women in these positions now and therefore that ratio is going to start to even out. It's odd since the talks are normally highly technical. It is the rare manager who can appreciate a talk on double-fetch bugs in the Windows Kernel.

Booth stuff

- Investing in carpet and padding underneath is completely worth it, my knees and feet were saved

- If you're going to buy labor, buy tear down labor rather than setup labor. You'll want to get your booth set up just the way you want it initially but by the end of the conference you're so tired you just want someone else to pack everything up. We waited 3.5 hours for our pallet and supplies to come to our booth at the end of the conference. Things got weird.

- In our 3.5 hours of time I did a lot of walking around the vendor hall as it was being packed up. I counted about 5 servers or devices I could've made off with without anyone being the wiser. If you're bringing that type of gear secure it yourself before tear down.

- I think our booth looked pretty good but we did have a lot of people asking us "so what do you guys do?" If we were going to do something like this again we'd want to put some kind of sign up like: "Pen-Testing Tools for Professionals". It was pretty liberating to repeatedly tell people that I didn't give a toss about configuring a firewall though.

- Invest in shirts that are not black. Everyone wore black shirts.

- I can almost guarantee your sales slicks are too wordy. I ain't reading a white paper here.

- There needs to be a medical reason for you to wear sunglasses at your booth, which is inside.

Vendor Freebies

- Best Overall: Again Qualys wins with their red freebie bag. As soon as you walked in the vendor hall you saw Qualys' booth and had the opportunity to get a reasonable quality bag for all your freebies. Everyone had one and everyone put all the other vendor freebies into their Qualys bag, reducing the exposure of other vendors and limiting the impact of their marketing investment. WELL PLAYED QUALYS >:[

- Best Shirt: Spider Labs' mall-airbrush-kiosk style graffiti on a bright orange shirt
- Shirt Runner-Up: Splunk "Taking the sh out of it"
- Shirt honorable mention: Core Security, faux-tux shirt

- Worst Overall: I didn't like the light saber thingies at all and no one I talked to about it did either. I guess the hook was that if you took this training it turned you into some kind of hacking Jedi? Brotip: if you're turning people into Jedi's you should at least be able to talk about your syllabus without referring people directly to your website :P

Shameless plugs

You can read my 2012 vendor perspective blog post here.