I love JobChangeNotifier. It’s a simple web application that sends me an email each week showing which of my LinkedIn connections have new jobs, and where they are now. It’s fun to watch my friends’ careers evolve.

But more importantly, when a customer changes jobs, I suddenly have a sales opportunity.  If they liked Scripted at their old company, they’ll probably promote us at their new one. We have a dozen new deals this year alone from customer job changes.

However, JobChangeNotifier can’t help me if I’m not connected to my customer on LinkedIn. Some friends and I decided to hack a fix.

Step 1. Start with  a CSV.

Every legit CRM will have an export tool. I’m going to start with a CSV rather than explain how to use the Salesforce API to pull contacts. If you’ve read my previous posts on using APIs, you should be able to figure that out. Getting your contacts into a spreadsheet is a trivial step.

All we need is first name, last name, and company name in this three-column CSV. It should look something like this:

Easy enough, right?

Step 2. Get the LinkedIn profile.

There are at least two ways to get your contact’s LinkedIn profile. The first and most direct (and perhaps terms compliant) approach is to use LinkedIn’s People Search API. There is complex authentication and required, and even when you’re authenticated, building the right query is actually not easy.

I decided to skip API approach in favor of a less direct but still very effective approach: the Internet search.

The code that follows is admittedly on the sketchier side of hacky. It uses a Python library to emulate a browser, so search engines treat your page loads the same as if you were doing it with a mouse. If you’re not comfortable with this approach, you can use urllib or urllib2 instead which adds no disguise about this being a Python script. But for those who want to see this hack in action, here you go.

from bs4 import BeautifulSoup
import urllib2, re, csv, sys, urllib, json, mechanize, os, time, random
import mechanize
import cookielib
#http://stockrt.github.com/p/emulating-a-browser-in-python-with-mechanize/
# Browser
br = mechanize.Browser()
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)

# Browser options
br.set_handle_equiv(True)
br.set_handle_gzip(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)

# Follows refresh 0 but not hangs on refresh > 0
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)

# User-Agent (this is cheating, ok?)
br.addheaders = [('User-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1309.0 Safari/537.17')]

The Python library I use here is called Mechanize, and once the options are set, it’s very easy to use. I create a browser instance called br with this line:

br = mechanize.Browser()

And you’ll see it’s used later in the script to load Bing queries and open public LinkedIn profile pages.

Now, let’s build a Bing query to find the LinkedIn profile.

burl = 'http://www.bing.com/search?q=site:linkedin.com %s %s %s' % (first, last, company)

I found that querying for “site:linkedin.com ryan buckley scripted” yields the target LinkedIn profile as the top search result about 80% of the time. That’s good enough for me, and it makes it easy to parse the results page and grab that first result.

Step 3. Parse the Bing results page.

Here’s some code you should understand from my previous posts.

r = br.open(burl.replace(' ', '%20'))
html = r.read()
soup = BeautifulSoup(html)
result = soup.find('div', { "class" : "sb_tlst" } )
if not hasattr(result, 'a'):
	print "No link?", result
	continue
if not result.a.has_key('href'):
	print "No href?", result
	continue
link = result.a['href']

The first line replaces all spaces with their HTML character, “%20″. This helps mechanize open the page. With the page loaded, I read it into a variable called html, which passes to BeautifulSoup for parsing. Again, all of this should be familiar (well, except the %20 thing, but you’d have discovered this need if you ran the code without it.)

The “soup.find” command looks for the first div with class=”sb_tlst”. If I did find_all, it wouldhave returned an array, but find returns only the first one.

The next lines are some error testing. I need to make sure of two things:

1. The result object has an “a” attribute (meaning, it find the link within this div), and
2. The anchor tag has an “href” attribute (this has the LinkedIn URL that I’m looking for).

If these tests pass, then I store the link into a variable. And I’m off to the next step.

Step 4. Parse the LinkedIn profile.

Alright! We’re almost there, but LinkedIn makes this last part is tricky.

There are two types of public profiles, shown here:

Does not have a headline

Has a headline

One has a profile and the other doesn’t. We have to account for this difference in our code:

r = br.open(link)
html = r.read()
soup = BeautifulSoup(html)
result = soup.find('p', { "class" : "headline-title title" }) if soup.find('p', { "class" : "headline-title title" }) else soup.find('ul', { "class" : "current" })
if not result:
	continue
if result.string:
	if ' at ' in result.string:
		new_title, new_company = result.string.strip().split(' at ')
elif result.li:
	if ' at ' in result.find("li").get_text():
		new_title, new_company = result.find("li").get_text().strip().split(' at ')
if not new_company:
	continue

This little bit of code opens the LinkedIn profile URL we just saved from Bing, finds the profile headline using BeautifulSoup, and then parses the headline based on the word “at” into the title and company and stores them into two new variables. With some error handling, all that is just 15 lines of code. Python is amazing!

Step 5. Print changes to screen and/or write the CSV.

To recap, we read the contents of a three-column CSV including first name, last name, and company name. We used Bing to get the URL for the public LinkedIn profile. We loaded that page, parsed out the current job info and stored it.

Now, we compare the LinkedIn data to our spreadsheet and print people whose company names don’t match.

if company not in new_company.strip():
	ins['First'], ins['Last'], ins['Company'], ins['New Title'], ins['New Company'], ins['Link'] = first, last, company, new_title.strip(), new_company.strip(), link
	f = open(outpath,'a')
	dw = csv.DictWriter(f, fieldnames=fields)
	dw.writerow(ins)
	f.close()
	print "%s - %s %s: was at %s, now at %s (%s)" % (i, ins['First'], ins['Last'], ins['Company'], ins['New Company'], ins['Link'])

You won’t be able to run this, so let’s just read it. The dictionary “ins” is not defined yet (I did that earlier in the script but haven’t shown it yet in this post). Before I can insert these data to the dictionary, I have to define it (which also empties it at the top of each loop) like this:

ins = {}

The lines that follow the creation of the dictionary are the standard process you’ll copy every time you need to write a CSV. I’m a huge fan of the DictWriter now. The code is cleaner, it makes sense, and it’s quick. The print command at the end shows me the data I’m writing to the CSV.

Here’s the full script in all of its glory. Note that you should set inpath and outpath to match your file structure, and you’ll need that three column CSV to start with. For your convenience, I’m sharing one here.

from bs4 import BeautifulSoup
import urllib2, re, csv, sys, urllib, json, mechanize, os, time, random
import mechanize
import cookielib
#http://stockrt.github.com/p/emulating-a-browser-in-python-with-mechanize/
# Browser
br = mechanize.Browser()
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)

# Browser options
br.set_handle_equiv(True)
br.set_handle_gzip(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)

# Follows refresh 0 but not hangs on refresh > 0
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)

# User-Agent (this is cheating, ok?)
br.addheaders = [('User-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1309.0 Safari/537.17')]

# Assumes list is in ~/data/starting/list.csv
inpath = os.getenv("HOME")+"/data/starting/linkedin_test.csv"
outpath = os.getenv("HOME")+"/data/finished/linkedin_test.csv"
#field for outpath
fields = ['First', 'Last', 'Company', 'New Title', 'New Company', 'Link']

cos = csv.reader(open(inpath, 'rU'))
i = 0
for co in cos:
	try: 
		i += 1
		if i < 6: 
			continue
		ins, new_title, new_company = {}, '', ''
		first, last, company = co[0].strip(), co[1].strip(), co[2].strip()
		burl = 'http://www.bing.com/search?q=%s %s %s site:linkedin.com' % (first, last, company)
		r = br.open(burl.replace(' ', '%20'))
		html = r.read()
		soup = BeautifulSoup(html)
		result = soup.find('div', { "class" : "sb_tlst" } )
		if not hasattr(result, 'a'):
			print "No link?", result
			continue
		if not result.a.has_key('href'):
			print "No href?", result
			continue
		link = result.a['href']
		r = br.open(link)
		html = r.read()
		soup = BeautifulSoup(html)
		result = soup.find('p', { "class" : "headline-title title" }) if soup.find('p', { "class" : "headline-title title" }) else soup.find('ul', { "class" : "current" })
		if not result:
			continue
		if result.string:
			if ' at ' in result.string:
				new_title, new_company = result.string.strip().split(' at ')
		elif result.li:
			if ' at ' in result.find("li").get_text():
				new_title, new_company = result.find("li").get_text().strip().split(' at ')
		if not new_company:
			continue
		if company not in new_company.strip():
			ins['First'], ins['Last'], ins['Company'], ins['New Title'], ins['New Company'], ins['Link'] = first, last, company, new_title.strip(), new_company.strip(), link
			f = open(outpath,'a')
			dw = csv.DictWriter(f, fieldnames=fields)
			dw.writerow(ins)
			f.close()
			print "%s - %s %s: was at %s, now at %s (%s)" % (i, ins['First'], ins['Last'], ins['Company'], ins['New Company'], ins['Link'])
	except Exception as detail:
		print "Failed on record %s with error %s" % (i, detail)
		#c.close()

First, let’s start with the imports and data sources.

import json, urllib, os, csv
inpath = os.getenv("HOME")+"/data/starting/permalinks.csv"
outpath = os.getenv("HOME")+"/data/finished/funding_data.csv"

I like to define the source of data and its resting place upfront. You’ll see if you keep following me on these, that I’ll always call them inpath and outpath. I also usually run them on my Mac, but sometimes if it’s a big file I’ll shoot it up to my server and let it run there. The os library allows me to define a home path without providing absolute folders, so paths work the same on my Mac as they do on the server.

To make this clearer, my home directory on my MacBook is /Users/rbucks. I keep all of my scripts in /Users/rbucks/scripts/, and my data is all in /Users/rbucks/data/starting/ and /Users/rbucks/data/finished/.

I keep this same file structure on an Ubuntu server, but of course the home directory is different. On Ubuntu, it’s /home/rbucks/. In order to save myself time, and make the same script run locally and on my server, I like to stick os.getenv(“HOME”) in all of my script file paths.

Next, let’s make a function to write a csv.

def write(sales_data):
	f = open(outpath,'a')
	fields = ['permalink', 'total_money_raised', 'dates_raised']
	dw = csv.DictWriter(f, fieldnames=fields)
	dw.writerow(sales_data)
	f.close()

We covered functions in an earlier post. This one takes a list of dictionaries and writes it beautifully into a csv. All you need to do is tell it which keys to look for (that’s the fields list).

There’s another function called csv.writer but I’m sold now on dictionary writing. Without the keys, you need to offset empty fields when you create the list to write the csv, and the code is much messier. But csv.writer works well enough, and it was my first csv writing tool.

But don’t bother with it. DictWriter is way cooler.

Finally, loop and write.

Here’s the code for the rest of it.

key = "get your key from crunchbase"
cos = csv.reader(open(inpath, 'rU'))
for co in cos:
	try:
		sales_data = {}
		sales_data['permalink'] = co[0]
		qry_url = 'http://api.crunchbase.com/v/1/company/%s.js?api_key=%s' % (co[0], key)
		qry_response = urllib.urlopen(qry_url).read()
		qry_result = json.loads(qry_response)
		if qry_result.has_key('error'):
			continue
		if qry_result.has_key('total_money_raised'):
			sales_data['total_money_raised'] = qry_result['total_money_raised']
		if qry_result.has_key('funding_rounds'):
			dates = []
			for r in qry_result['funding_rounds']:
				funded_date = "%s/%s/%s" % (r['funded_month'], r['funded_day'], r['funded_year'])				
				dates.append(funded_date)
			sales_data['dates_raised'] = ", ".join(dates)
		write(sales_data)
		print "Ran "+co[0]
	except Exception as detail:
		print "Gah", detail
		print qry_url
		continue

I’m going to breeze through this, and I’m not trying to be lazy. If you followed the first three of these, the Python code here should make sense, and if it doesn’t, you should know how to use the Python console to make it make sense.

I load a list of permalinks into a variable called cos via the csv.reader function that we imported in the first step. This csv is little more than a single field with a bunch of permalinks I pulled in Step 3, like this:

55
158
lover-ly
netconstructor-com
212-media
sales-marketing
netconstructor
1800pharmacy
1-800-therapist-llc
10-20-media
1000jobboersen-de
10th-degree
10th-degree
128b
140-proof
140fire

If you want to play along, take that list and save it into a file called “permalinks.csv” in your home’s data/starting folder. The csv.reader function loads this into an iterable csv object that I’m calling cos. At each iteration of the for loop, it’s pulling a row into a variable I’m calling “co,” which is a list of row values. To get the first value, I use co[0] (also covered earlier).

Now watch this closely. sales_data is a dictionary, defined by the empty curly brackets. Dictionaries take keys and values, unlike lists which just take values. So in the third line of the loop, I’m adding a key-value pair for ‘permalink’ in my dictionary. I’m defining permalink as that row’s value in the csv. I want to record this permalink in my new csv.

Also note that all of the fields in my write function get defined in this loop. So I create a key-value pair also for ‘total_money_raised’ and ‘dates_raised.’ This is very important! The write function will break if it sees a key in sales_data that’s not defined in the fields list.

The rest of this is same as last time. To learn how this works, take it line by line in the Python console. Go ahead and start with “scripted-com” as your permalink, and play with the JSON response.

qry_url = 'http://api.crunchbase.com/v/1/company/%s.js?api_key=%s' % ('scripted-com', key)
qry_response = urllib.urlopen(qry_url).read()
qry_result = json.loads(qry_response)

(Note: you need to have key defined with your own Crunchbase API key.)

Type qry_result in the Python console and look at the data. There’s a lot. Try iterating through the first level like this:

for r in qry_result:
	print r

Then try printing out the keys in r. Type r['total_money_raised']. Or r['funding_rounds']. You should start to see how this script works, and how awesome JSON data is for salespeople to work with.

Scripted recently released a new feature called Teams, which allows us to efficiently and confidently select expert writers in a given subject to be on a team, the idea being that a business looking for experts in that field can easily find writers who are highly qualified (as both a writer and a domain expert) to write about it. Part of what determines whether a writer is a good fit for a team is knowing how many pieces they’ve written about that team’s subject matter. This entailed an interesting machine learning problem…

The Problem

How can I get a computer to tell me what an article is about (provided methods such as bribery and asking politely do not work)?

To formally explain the problem as well as the proposed solution to it, I’m going to stay fairly high-level and use a toy example, with links to resources for further reading and a disclaimer that in reality you would need a dataset consisting of more than four samples to actually make any of this work (like a lot more…).

For the sake of this example, imagine we are engineers at a much tinier version of Scripted, with only four pieces of writing in our system, the titles of which are:

“The Perfect Panini”
“Sandwich Sorcery”
“Boston Terriers: Friend or Foe?”
“How to Tell if your Dog is a Dog and not a Cat”

Note: These four documents, in the language of natural language processing, are known as a ‘corpus’. You can think of this as a collection of written documents that we want a computer to learn from. 

As imaginary employees of tiny Scripted, we’ve noticed that many clients are looking for writers who are experts in the fields of dogs and sandwiches to produce content for them. Accordingly, we make a “Dogs” team and a “Sandwiches” team. The question now is, which documents belong on which team?

Describing Documents

Imagine you’re playing a guessing game with someone. You provide your partner with a list of animals, and a list of attributes about each animal. You describe a domesticated animal that is furry, says ‘meow,’ and is prominently featured on the internet. Based on these clues, and the information you initially supplied, the person you’re playing with guesses “cat.”

This is pretty much the same process our algorithm will use (this task is actually a very familiar one to anyone who’s dabbled in machine learning, and is known formally as classification). To put it a bit more formally and relate it to our specific problem, we want the process of putting a document on a team to eventually look like:

1. Extracting features from the document which are known to us
2. Using these features to describe the document to the system
3. Having the system guess which team that document belongs to based on that description.

First thing’s first. We need a way to numerically describe documents to a computer so that we can tell how similar two documents are to each other. You can think of this as a mapping from a document to a point in space such that the closer two of these points are to each other the more similar their documents corresponding to those points are. Thinking of documents as vectors in this way is formally known as the Vector Space Model.

Fig. 1. A two-dimensional vector space illustrating the concept of considering document similarity in terms of Euclidean Distance. All the dots represent documents. The small red dots represent the documents most similar to the large red dot in the middle.

A common way to represent a written document as a vector is to think about it in terms of “Bag of Words” (BoW) vectors. These are vectors of word counts where each slot in the vector represents the number of times a certain word was used. The list of all the words we keep track of in these vectors is known as our vocabulary.

Let’s say that our entire vocabulary consists of only the words “dog” and “cat.” Then a BoW vector of a document with this vocabulary would be:

<# Times ‘dog’ was used in doc, #Times ‘cat’ was used in doc>

For example, the text "dog cat cat dog dog" would be represented as the vector <3,2>. This pattern holds for larger documents and vocabularies. Now that we have some understanding of the vector space model, we can get cracking on some code.

Setup

The entirety of the code covered in this tutorial can be found here. We’re going to be working in Python, and will use the following modules to make our lives way easier:

• Numpy - To goto Python library for fast numerical computing.
• Scipy - A scientific computing module with loads of functionality built on top of Numpy.
• Gensim - “Topic Modeling for Humans”
• Scikit-Learn - Machine learning library for Python also built on top of Numpy.

I’m assuming you have pip and git installed. If this isn’t the case, you should definitely start there, as these tools are super useful for developers. First thing we want to do with those installed is grab a local copy of the tutorial’s source code:

git clone https://github.com/Scripted/NLP-Tutorial

Now, we need to install the dependencies. In a perfect world, running pip install -r requirements.txt in the NLP-Tutorial directory would take care of all of this for you. Unfortunately, Numpy and Scipy can be tricky to install. You might want to try installing each individually in the following order:

pip install numpy
pip install scipy
pip install gensim
pip install scikit-learn

To test whether your installation was successful, run python classifier.py. If it doesn’t give you any errors, then you’re all set! If not, just follow the links I provided above next to each dependency and follow their installation instructions.

The Code

Now that you’re all set up, let’s switch gears back to the algorithm. Recall that the first step of the process we described was to turn documents into vectors. Let’s look at how to actually do that in code. We’ll start by pulling in all our dependencies.

import warnings
warnings.filterwarnings('ignore', category=DeprecationWarning)
from math import sqrt
import gensim
from sklearn.svm import SVC
import os

Now that we’ve got all the resources we need, we start by loading in our corpus. If you recall, a corpus is a collection of documents we wish to learn from and work with in the context of an NLP algorithm. I’ve provided a toy corpus in the repository which is aptly named… “corpus.“ This directory contains the following four documents:

dog1.txt - "dog runs and barks at dog"
dog2.txt - "the dog runs and barks at the apatosaurus"
sandwich1.txt - "a sandwich of cheese and meat and bread and cheese is 
                   supercalifragilisticexpialidocious"
sandwich2.txt - "the sandwich of meat and meat and cheese and meat"

You might astutely observe that these documents are unrealistically simple, contain a very limited vocabulary, and no punctuation. Yup, that’s by design. This toy corpus was meticulously designed by yours truly to demonstrate certain aspects of the process, while not getting mired in the gritty details of practical application.

While things like cleaning punctuation, HTML, and URL’s out of your text and lemmatization are absolutely things you should look into for a “real world” application, they are definitely low-level details that distract us from the high level process. What we ultimately want all of these things to do is split a block of text into words.

if __name__ == '__main__':
    #Load in corpus, remove newlines, make strings lower-case
    docs = {}
    corpus_dir = 'corpus'
    for filename in os.listdir(corpus_dir):
        path = os.path.join(corpus_dir, filename)
        doc = open(path).read().strip().lower()
        docs[filename] = doc
    names = docs.keys()

    #Remove stopwords and split on spaces
    print "\n---Corpus with Stopwords Removed---"
    stop = ['the', 'of', 'a', 'at', 'is']
    preprocessed_docs = {}
    for name in names:
        text = docs[name].split()
        preprocessed = [word for word in text if word not in stop]
        preprocessed_docs[name] = preprocessed
        print name, ":", preprocessed

The output so far…

---Corpus with Stopwords Removed---
sandwich2.txt : ['sandwich', 'meat', 'and', 'meat', 'and', 'cheese', 'and', 'meat']
dog2.txt : ['dog', 'runs', 'and', 'barks', 'apatosaurus']
dog1.txt : ['dog', 'runs', 'and', 'barks', 'dog']
sandwich1.txt : ['sandwich', 'cheese', 'and', 'meat', 'and', 'bread', 'and', 'cheese', 'supercalifragilisticexpialidocious']

What we’ve done so far is just some pretty basic Python to read in the documents in our corpus and store them in a dict. All we did to split the blocks of text into lists of words was call the string’s builtin split method, which for our purposes pretty much just splits on spaces.

Preprocessing

The code also hints at another aspect of NLP called preprocessing. You can think of preprocessing as filtering out words which ultimately don’t tell us a whole lot about the text. Words such as ['the', 'of', 'a', 'at', 'is'] which appear with great frequency in the vast majority of documents we’ll ever analyze are called stop words and should definitely be filtered out. Similarly, you might also want to filter out words which appear either extremely frequently, or extremely rarely. This is where gensim comes in handy.

#Build the dictionary and filter out common/rare terms
    dct = gensim.corpora.Dictionary(preprocessed_docs.values())
    unfiltered = dct.token2id.keys()
    dct.filter_extremes(no_below=2)
    filtered = dct.token2id.keys()
    filtered_out = set(unfiltered) - set(filtered)
    print "\nThe following super common/rare words were filtered out..."
    print list(filtered_out), '\n'
    print "Vocabulary after filtering..."
    print dct.token2id.keys(), '\n'

Which outputs…

The following super common/rare words were filtered out...
['and', 'apatosaurus', 'bread', 'supercalifragilisticexpialidocious'] 

Vocabulary after filtering...
['cheese', 'runs', 'sandwich', 'meat', 'barks', 'dog']

To recap, the above code filters out words from our corpus which are too common or too rare to be useful. It does so by leveraging gensim’s Dictionary class which stores word counts for each term encountered in the corpus. After we feed it all the documents, we tell it to filter out words which are very common (defined by whatever gensim’s default threshold is) as well as words which occur only once. This leaves us with only six words in our vocabulary: ['cheese', 'runs', 'sandwich', 'meat', 'barks', 'dog'] . All words not in the vocabulary are ignored from here on out.

At this point, we’ve got all we need to start bringing bag of words vectors into the picture.

From Texts to Vectors

From this point on, we’ll be thinking about documents as vectors. That handy Dictionary class we used above for preprocessing also contains functionality to take a list of words, compute word counts for words in our vocabulary, and return the corresponding bag of words vectors.

#Build Bag of Words Vectors out of preprocessed corpus
    print "---Bag of Words Corpus---"

    bow_docs = {}
    for name in names:

        sparse = dct.doc2bow(preprocessed_docs[name])
        bow_docs[name] = sparse
        dense = vec2dense(sparse, num_terms=len(dct))
        print name, ":", dense

Note: vec2dense is just a helper function I wrote to convert vectors to a more familiar format for display purposes

Here are the resulting bag of words vectors

---Bag of Words Corpus---
sandwich2.txt : [1.0, 0.0, 1.0, 3.0, 0.0, 0.0]
dog2.txt : [0.0, 1.0, 0.0, 0.0, 1.0, 1.0]
dog1.txt : [0.0, 1.0, 0.0, 0.0, 1.0, 2.0]
sandwich1.txt : [2.0, 0.0, 1.0, 1.0, 0.0, 0.0]

There we go — our documents expressed as points in 6-dimensional space. For a practical application, working with 6-dimensional data is pretty damn reasonable. However, one can imagine a scenario where our corpus is composed of hundreds of thousands of documents which sample from a much more extensive vocabulary (like say the entire English language). Even after removing stop words, and exceedingly common/rare terms we are left with bag of words vectors with upwards of 50,000 unique word-counts/dimensions to keep track of. This should make the data nerd in you uncomfortable.

The Curse of Dimensionality

There are all kinds of terrible things that happen as the dimensionality of your descriptor vectors rises. One obvious one is that as the dimensionality rises, both the time and space complexity of dealing with these vectors rises, often exponentially.

Another issue is that as dimensionality rises, the amount of samples needed to draw useful conclusions from that data also rises steeply. Another way of phrasing that is with a fixed number of samples, the usefulness of each dimension diminishes. Finally, as the dimensionality rises, your points all tend to start becoming equidistant to each other, making it difficult to draw solid conclusions from them. The umbrella term that covers all these adverse effects of high dimensionality is “the curse of dimensionality.”

The Remedy

Fortunately, there’s a whole family of techniques called dimensionality reduction techniques which are entirely geared toward bringing down the number of dimensions in our descriptor vectors to something more reasonable. While the low level details often entail fairly advanced mathematics, the high level ideas behind the techniques are quite intuitive (also, the low-level details are often implemented for you, like in gensim).

Let’s look at those bag of words vectors again.

---Bag of Words Corpus---
sandwich2.txt : [1.0, 0.0, 1.0, 3.0, 0.0, 0.0]
dog2.txt : [0.0, 1.0, 0.0, 0.0, 1.0, 1.0]
dog1.txt : [0.0, 1.0, 0.0, 0.0, 1.0, 2.0]
sandwich1.txt : [2.0, 0.0, 1.0, 1.0, 0.0, 0.0]

Notice that certain dimensions are highly correlated with one another. For example, in both dog1.txt and dog2.txt, the first, third, and fourth values all seem to occur together. In terms of the text, this means that the terms ‘dog,’ ‘runs,’ and ‘barks’ frequently seem to occur together. In that case, maybe rather than thinking of our documents in terms of individual word counts, we should be thinking about them in terms of topics (or groups of words that occur together).

Dimensionality reduction techniques help us do exactly that. They math-magically (it’s a technical term, I promise) express our high dimensional data in a lower dimensional space. The only manual bit that many of them feature is that you need to specify how many dimensions the lower-dimensional space should feature. Unfortunately, there’s no algorithm that I know of that can look at your data and auto-detect the perfect dimensionality to reduce to. In most cases, there’s not even a clearcut solution to that problem were you to ask a human. Fortunately, this silly toy corpus isn’t most cases — it’s an especially trivial case! Arguably the best kind of case.

We look at the corpus, and we intuitively say that these documents are about either ‘dogs’ or ‘sandwiches,’ and thus, the number of dimensions in our lower dimensional space should be two. The algorithm that we use to do the dimensionality reduction in this case is called “Latent Semantic Indexing,” generally abbreviated to LSI.

Going into the math that makes LSI work is way beyond the scope of this article, so I’ll just summarize by saying it uses a technique from linear algebra called singular value decomposition to reduce the input matrix (your document vectors stacked on top of each other) to one of a lower rank (the number of dimensions you specified). If you didn’t understand any of what just happened in that last sentence, it’s totally fine, because gensim‘s done the heavy lifting for us.

#Dimensionality reduction using LSI. Go from 6D to 2D.
    print "\n---LSI Model---"
    lsi_docs = {}
    num_topics = 2
    lsi_model = gensim.models.LsiModel(bow_docs.values(),
                                       num_topics=num_topics)
    for name in names:
        vec = bow_docs[name]
        sparse = lsi_model[vec]
        dense = vec2dense(sparse, num_topics)
        lsi_docs[name] = sparse
        print name, ':', dense

And here are our simplified 2d vectors…

---LSI Model---
sandwich2.txt : [3.222517, 0.0]
dog2.txt : [0.0, 1.6870012]
dog1.txt : [0.0, 2.4343436]
sandwich1.txt : [2.1483445, 0.0]

Look at that. Two-dimensional vectors which represent all the information we care about without having to consider our data in terms of high-dimensional word counts. The actual numbers in the vectors don’t matter so much as the fact that the first slot is clearly our sandwich topic while the second is our dog topic.

This is probably a good time to reiterate how much training on four samples of data would never, ever, ever work to generate a topic model like LSI for non-trivial data. You’re also not likely to have outcomes as cut and dry as these, where one topic contributes 100% to a given vector’s magnitude, while the other contributes nothing.

Document Similarity

I’ve alluded above to the fact that once documents are represented as points in space, we can tell how similar they are by how close they are to each other. Now that we’re at that point in the process, let’s go over what it means for points to be “close” to each other.

It turns out that there are many different ways of considering how close two points in space are to each other. The most natural one to us is called Euclidean Distance. If I were to draw a straight line between two points, and measure that line, that is the Euclidean distance. To see an example of Euclidean Distance in action refer to Fig. 1. The big red dot represents the document whose closest matches we’re trying to find. The smaller red dots around it are the most similar documents by the Euclidean distance metric.

There are, however, issues with Euclidean distance. In many text processing applications, we care more about the direction of the vector (or the angle from the origin to the point) than its actual location. To demonstrate why we might want to consider the vector in terms of direction rather than magnitude, let’s consider another toy bag of words example. We consider the following three documents.

doc1 - ['dog','dog','dog','dog','dog','dog']
doc2 - ['dog','dog']
doc3 - ['cat']

In their bag of words form:

doc1 - [6,0]
doc2 - [2,0]
doc3 - [0,1]

Intuitively, we know that in terms of subject matter, doc1 and doc2 should be more similar to each other than they are to doc3. They are both clearly 100% about dogs, while, doc3 is 100% about cats. However, when we take the Euclidean distances between them, we find that the distance between doc2 and doc1 is 4, while the distance between doc2 and doc3 is ~2.24. The length of the first document led us to believe that doc2 was more like doc3 than it was to doc1.

The fact that document length has nothing to do with what the document is actually about is exactly why we want to downplay the importance of vector magnitude and instead focus on direction. There are a few ways we can accomplish this.

Firstly, we can modify the vectors themselves by dividing each number in each vector by that vector’s magnitude. In doing so, all our vectors have a magnitude of 1. This process is called unit vectorization because the output vectors are units vectors.

The unit vectors make it so that the dog documents are now closest to each other:

doc1 - [1,0]
doc2 - [1,0]
doc3 - [0,1]

Another technique is to leave the vectors alone and just take the angle between them. Measuring similarity based on angle between vectors is know as cosine distance, or cosine similarity. In our example up there, you can see the angle between doc1 and doc2 is 0° because they are pointed in exactly the same direction. On the either hand, the angle between either dog document and doc3 is 90°. 0° is less than 90°, therefore, the dog articles are once again more similar to each other than they are to the one about cats.

Both Euclidean and cosine distance are called distance metrics. A distance metric is simply the formula we provide to an algorithm that will dictate how close we consider two vectors to be to each other. With that bit of theory under our belts, let’s go back to the code.

Cosine Distance in Action

We last left off at transforming our bag of words corpus into two-dimensional topic vectors. At this point, we want to unit vectorize those topic vectors because, again, we care more about the angle of the vector than the magnitude. The reason I’m using both unit vectorization and cosine distance in this code is twofold. Firstly, it couldn’t hurt. Second, when we move on to classification, we will be using a model that can only use Euclidean distance (as far as I know).

By unit vectorizing our corpus, we make Euclidean and Cosine distance equivalent in terms of ordering. They will not return the same exact distance for instance (as one is measuring magnitude while the other is measuring angle). However, what they will do is say that if cosine_distance(A,B) < cosine_distance(B,C) then euclidean_distance(A,B) < euclidean_distance(B,C) for all points A,B,C in our corpus. Intuitively, you can think of this as saying “if two points are closer to each other on the unit circle/sphere/hypersphere, then the angle between them is smaller.”

Here’s where the magic happens:

#Normalize LSI vectors by setting each vector to unit length
    print "\n---Unit Vectorization---"

    unit_vecs = {}
    for name in names:

        vec = vec2dense(lsi_docs[name], num_topics)
        norm = sqrt(sum(num ** 2 for num in vec))
        unit_vec = [num / norm for num in vec]
        unit_vecs[name] = unit_vec
        print name, ':', unit_vec

and here’s the output:

---Unit Vectorization---
sandwich2.txt : [1.0, 0.0]
dog2.txt : [0.0, 1.0]
dog1.txt : [0.0, 1.0]
sandwich1.txt : [1.0, 0.0]

Now what we want to do is illustrate cosine distance correctly matching up documents that should be similar. Without further delay, here’s the code to do it.

#Take cosine distances between docs and show best matches
    print "\n---Document Similarities---"

    index = gensim.similarities.MatrixSimilarity(lsi_docs.values())
    for i, name in enumerate(names):

        vec = lsi_docs[name]
        sims = index[vec]
        sims = sorted(enumerate(sims), key=lambda item: -item[1])

        #Similarities are a list of tuples of the form (doc #, score)
        #In order to extract the doc # we take first value in the tuple
        #Doc # is stored in tuple as numpy format, must cast to int

        if int(sims[0][0]) != i:
            match = int(sims[0][0])
        else:
            match = int(sims[1][0])

        match = names[match]
        print name, "is most similar to...", match

You might be asking yourself where the cosine distance computation happened in that code. gensim features a class called MatrixSimilarity which is a type of index, or a data structure used to efficiently store vectors and data so that when it comes time to make a similarity query, the search to find vectors closest to our query point is much faster than brute force.

The actual query is made when we pick a pick from our corpus (called vec in the code)  and say sims = index[vec]. sims is a list of all points and their distance to the query point (vec). We proceed to sort that list, pull out the closest match, and print it out which yields the following:

---Document Similarities---
sandwich2.txt is most similar to... sandwich1.txt
dog2.txt is most similar to... dog1.txt
dog1.txt is most similar to... dog2.txt
sandwich1.txt is most similar to... sandwich2.txt

Seems reasonable to me. At this point we’re very close to solving the problem we set out to solve.

Classification

Our progress so far answers the question “which document is this document most similar to?” The question we ultimately want to answer is “what is this document about?” There are different ways to answer that question, including keyword extraction, clustering, and all sorts of other techniques. The one that I like best involves supervised learning, where you train the algorithm on samples which have the “correct” answer provided with them.

The specific supervised learning problem we’re addressing here is called classification. You train an algorithm on labelled descriptor vectors, then ask it to label a previously unseen descriptor vector based on conclusions drawn from the training set. The way we are going to accomplish this in our case is you make use of support vector machines, a family of algorithms which define decision boundaries between classes based on labelled training data.

To give a high-level view of what exactly support vector machines (or SVM’s for short) do, I’ll refer back to our points in space. For our ‘dog’ vs. ‘sandwich’ classification problem, we provide the algorithm with some training samples. These samples are documents which have gone through our whole process (BoW vector -> topic vector -> unit vector) and carry with them either a ‘dog’ label or a ‘sandwich’ label. As you provide the SVM model with these samples, it looks at these points in space and essentially draws a line between the ‘sandwich’ documents and the ‘dog’ documents. This border between “dog”-land and “sandwich”-land is known as a decision boundary. Whichever side of the line the query point falls on determines what the algorithm labels it.

The Final Step

Time to build our SVM, train it, and test it.

#We add classes to the mix by labelling dog1.txt and sandwich1.txt
    #We use these as our training set, and test on all documents.
    print "\n---Classification---"

    dog1 = unit_vecs['dog1.txt']
    sandwich1 = unit_vecs['sandwich1.txt']

    train = [dog1, sandwich1]

    # The label '1' represents the 'dog' category
    # The label '2' represents the 'sandwich' category

    label_to_name = dict([(1, 'dogs'), (2, 'sandwiches')])
    labels = [1, 2]
    classifier = SVC()
    classifier.fit(train, labels)

    for name in names:

        vec = unit_vecs[name]
        label = classifier.predict([vec])[0]
        cls = label_to_name[label]
        print name, 'is a document about', cls

    print '\n'

And the end result of all of this code is…

---Classification---
sandwich2.txt is a document about sandwiches
dog2.txt is a document about dogs
dog1.txt is a document about dogs
sandwich1.txt is a document about sandwiches

Voila, we’ve successfully answered the question we set out to solve! This part of the code is where we take advantage of scikit-learn to do all the SVM-related heavy lifting for us. I constructed a training set out of two of the documents by manually labeling them ‘dog,’ and ‘sandwich,’ then correctly classified all four documents.

Cross-Validation

As a result of the simplicity of our example, we have done a big no-no in “testing” our algorithm, which is to train and test on what essentially ended up becoming the same data. In reality, your dataset would be much more varied and numerous than our toy example and you would test your data by partitioning it into a training set and a test set.

All samples in both training and test sets are labeled. However, in practice, you would build the model on the labeled training set, ignore the labels on the test set, feed them into the model, have the model guess what those labels are, and finally check whether or not the algorithm guessed correctly. This process of testing out your supervised learning algorithm with a training and test set is called cross-validation.

A Quick Reality Check

So you’ve read through the article, understand everything that’s going on, but sense there’s something missing. I’ve pointed out several times that this example has been simplified to the point of triviality for the sake of demonstrating the most important aspects of the algorithm which might otherwise get lost in the details. How do you take this framework we’ve run through and apply it to something practical?

As usual, the devil’s in the details. I can’t spell out the entire process out for you (otherwise this article would be the size of a textbook), but I can list for you the questions you’ll need to answer in order to make any such machine learning application work. Hopefully research, experimentation, and intuition will take you the rest of the way.

First questions you should ask yourself

• What is the problem I’m trying to solve?
• What do I want to learn from my data? How will my findings be actionable?
• What tools exist that can help me solve this problem?

I can’t stress enough how important these first few questions are. The answers to these questions should drive every design decision you make. It’s all too easy to get caught up in all of the awesome algorithms and techniques out there in this field, and get completely distracted from your end goal. That being said, there is obviously a place for playing around, exploring, and experimenting. Just be sure you always keep the problem you’re trying to solve in mind. Anyhow… back to the list!

Preprocessing

• Where do I find a corpus?
• How do I extract text from web pages in a scalable/generalizable way (if using a corpus from the web)
• How do I split my text into individual words and extract the roots of those words?
• How do I build my vocabulary?
• Which words do I filter out?

Vector Space Model

• What sort of normalization does the problem call for?
• Which distance metric makes sense for the problem at hand?

Dimensionality Reduction

• How do I pick how many dimensions I ultimately want to be working with?

Machine learning

• Am I solving a supervised or an unsupervised learning problem?
• Which model/algorithm makes sense for the problem at hand?

Testing

• How do I gauge how well my algorithm is doing?
• Is it enough just to look at accuracy?
• Should I consider precision/recall?
• Are my classes of equal sizes, or does one dominate the other? How do I remedy this?
• Are my training/test datasets actually representative of what my algorithm will encounter in every day use?

If you take anything out of this article, it should be that useful applications of machine learning are all about the decisions you make. There are countless algorithms and techniques to choose from, but they are only as useful as your application of them. Identify the problem you’re solving, experiment with solutions, understand where there might be shortcomings in those solutions and remedy them when possible.

Plug and Play with Pathogen

For Python Programmers

When she’s voted employee of the month and gets to give a speech to the shareholders, the other members of the department are furious. Long story short, they end up giving her a box with a red light on it and tell her it’s the Internet. The whoooole Internet. They recommend that she should present it to the shareholders, and that it would get quite the reaction.

Her coworkers’ attempt to humiliate her backfires, however, when the shareholders also believe it is the Internet. Like Jen herself, the shareholders have no idea what she does for a living.

In a way, I can sympathize with Jen (although I know quite well what I’m doing for a living). When I first moved to the Silicon Valley, my tech knowledge ended at some basic HTML. When the head of the engineering department told us about the unicorn sending out workers to keep the system running, I accepted it without question (as I should have – he is wise in the ways of science. Er, coding). But the phrase “a unicorn sends out workers to retrieve it” struck chords of “this, Jen, is the Internet” in me. I wanted to learn more about what happened on the back end, both so I had a better understanding of how our system works and so I could help keep work flow smooth between the departments.

In particular, there were three quotes from The I.T. Crowd that actually helped me accomplish this, by showing me the opposite of how to act:

1. “I like you, Jen. You don’t ask questions.” First, I started keeping lists of the questions that came up for me during the week. This helped me find common themes in what I didn’t understand. Knowing where to start was a challenge, and keeping track of questions that I (or others in my office) had helped me to identify the best place to begin.
2. “I have it on good authority that if you type ‘Google’ into Google, you’ll break the Internet.” Instead of listening to random hearsay, I went to Meet Up events where people were discussing some of these same themes – search engine functionality became a popular topic, and it actually led me to groups and events about SEO.
3. “I have a lot of experience with the whole computer…thing. You know, emails, sending emails, receiving emails, deleting emails. I could go on.” I brought some friends with me to these panels, and together we made sure to take a more hands-on approach to getting better acquainted with code by signing up for Codecademy, Team Treehouse, or a few other free online intro classes through several universities.

As painful as it is to admit, this was also a great example of the benefits (albeit small) of T.V. in everyday life. Who knows, maybe next Liz Lemon will teach me the best way to leave a meeting.

The day before I arrived in New York, Mayor Bloomberg cancelled the race, reversing his vocal support of the marathon only two days earlier. I was relieved. I did not train nearly as much as I did last year, and I knew that I wasn’t in the best shape to wake up at 5am and suffer through six hours of road running.

When I arrived in New York on Saturday afternoon, my friend and fellow guide casually asked me how I felt about running the marathon anyway. “Sure,” I replied, not taking him seriously. When he paid for lunch and said, “Cool, now I can guilt-trip you even more,” a brief moment of dread passed over me. Oh man, I thought. He wasn’t joking.

The next day at 10am I was standing at the original marathon’s finish line in Central Park, where several thousand marathon refugees decided to run 26.2 miles anyway in 4.25 loops around Central Park. I met my friend’s athlete there, a tall man named EJ Scott, who suffers from a degenerative eye disease. He was going to run the course blind, guided only by a small rolled-up towel about the size you’d pick up at a gym, held between EJ and his guide.

I was to flank on his right; the towel and guide were on his left. Over the next 6 hours, in order to help the time pass and preserve my sanity (it’s a real mental challenge to run a 6-mile loop four times), I thought about how this experience applies to startups. Here’s my list.

WHAT STARTUPS CAN LEARN FROM THE 2012 NYC MARATHON

1. The people in charge can screw up. Whether it’s your investors, advisors, or largest clients, just because they have money and/or influence doesn’t mean they’re always right. Mayor Bloomberg screwed up by saying the marathon was still on two days before he canceled it.
2. Consumers will find a way to get what they want. There are about 50,000 customers of the New York City marathon every year, and you can’t tell that many people yes and then no when thousands of dollars and hundreds of hours were invested in the product. It took only a few hours for the alternate route around Central Park to get distributed on Facebook and Twitter.
3. Breaking up the monotony makes it easier. Finally, a note about running. Our athlete preferred to run a 10:1 split, meaning for every 10 minutes running, take a minute to walk. When he said this, I did the math in my head: 6 hours, 6 times per hour, that means starting and stopping a timer 36 times. More repetition! Ugh! In fact, this technique made the marathon bearable. Instead of thinking about how we have two laps to go, I thought, alright, only six more splits until our last lap. Thinking in ten minute chunks actually made the six hours go by faster.

They say that building a company is a marathon and not a sprint. And in running they say a marathon is 6 mile race with a 20 mile warm-up. Combine the two and I think you have a very compelling picture of what it’s like to build a company.

It starts slow. You may be working like a dog, but user growth is slower than your projections, customers don’t come as fast as you’d hoped, and fundraising’s not easy either. But you slog through the lows and relish the highs. You’re joined by competitors and partners, some of whom you like and others you don’t. You keep running. Finally you reach a critical point right around mile 20. Runners call it “the wall.” Here you have a choice, either summon all your guts and break through it, or crumble and sit on the sidelines.

HootSuite is a social media management system that allows businesses and organizations to manage their social media outlets in bulk from one simple dashboard. You can view your scheduled posts for the day or month, make amendments to the posting schedule, and even re-tweet, message, and respond to posts directly from the HootSuite dashboard.

With the Scripted app for HootSuite, you can now view and publish Scripted content seamlessly from the HootSuite dashboard. Posts are visible in batches all on one page, allowing for automated streaming and simple distribution. No more hassling with excel sheets and altering formatting.

For real-time posts, click “Send Now”, and your post will be published immediately to one or more of your social media profiles. Or, select the calendar button to schedule posts in advance.

Additional Features in HootSuite:

• Receive custom analytics on the performance of your posts. See how Scripted content stacks up to your other content.
• Manage multiple Twitter and Facebook accounts. Use Scripted content to promote your day business, volunteer organizations, or for personal branding.
• View your mentions, inbox, and Tweets/Facebook posts from one tool, designed for simple social media management.

The Scripted app fuses the high quality content-production capacity of Scripted with the content distribution and scheduling capacity of HootSuite. To install the Scripted app and see first-hand how it works, login to your HootSuite account today and visit the App Directory.

What’s an API? It stands for Application Programming Interface, but you should think of it more as a drive-thru window. If the inside of a McDonalds is the website or web app itself, then the drive-thru with its (sometimes) limited menu and streamlined experience  is the API. Since we’re bypassing the user interface altogether, everything becomes much easier to program.

Step 1.

Register for the API. If you don’t already have a Mashery account, you may need to cerate one. This is easy and free.

Step 2.

Save your key into a local .py file. Again, you can use any text editor, but there are many free text editors for writing code. Save this file (call it crunchbase_api.py or something like that) in a folder on your computer. We’ll write the rest of the code in this file too. And while you’re at it, let’s import the libraries you’ll need: json and urllib.

import json, urllib
key = 'not_gonna_show_you_my_key'

 Step 3.

Prepare to be amazed! This is going to be so much easier than the last two.

First, let’s noodle through the documentation a little bit. You should actually read through this because it’s written in English and will give you a sense of how APIs work. You might notice that working with the API still involves URLs. Indeed, most APIs are little more than http calls to websites, which return data in plain structured text.

You might also have noticed a new acronym: JSON. This sounds scarier than it is (for it’s length, not the hockey-masked killer). You should rejoice whenever your API returns JSON data, because it’s super easy to use and has a native library to interpret it in Python. I’ll explain more when we get our first response.

Let’s start by listing some advertising companies.

url = 'http://api.crunchbase.com/v/1/search.js?query=advertising'
response = urllib.urlopen(url).read()
result = json.loads(response)

If you view the result, you’ll just see a lot of text. But buried within here are signs that the text is structured and iterable. You should note the locations of the commas, colons, square brackets, and curly brackets. They all play an important role in parsing this data. I’ll show you what I mean:

for r in result:
	print r

You should see this output:

total
crunchbase_url
page
results

Now go back and print the result (type “result” and hit enter). Scroll to the top of that block of text. Do you see “{u’total’: 7646, u’crunchbase_url’: u’http://www…”? JSON is basically a python dictionary, or a hash of key-value pairs. Here, ‘total’ is a key and 7646 is the value. To get only the total out of result, simply type result['total'] and press enter. Pretty intuitive, right? You get the Crunchbase URL the same way.

JSON results can be deeply nested, as seen in result['results']. The value of the key ‘results’ is a list of exactly 10 companies and their data (I calculated this with len(result['results']). This is, of course, iterable:

for r in result['results']:
	print r

Let’s just take one of them and store the name so we can look up the juicy data we really want.

company = result['results'][0]
name = company['name']

Here I took the first company in the list of results and then stored the value of ‘name’ in a variable called name. If you print “company”, you’ll see that company is itself a JSON-type dictionary object. That’s why I can use the ['name'] index to pull the name data I need for the next step.

Step 4.

Finally, the good stuff.  You might have noticed we haven’t used the API key yet. Now we will, and I’m going to show you another cool thing about Python.

qry_url = 'http://api.crunchbase.com/v/1/company/%s.js?key=%s' % (urllib.quote(name), key)
qry_response = urllib.urlopen(qry_url).read()
qry_result = json.loads(response)

Alright. What’s up with the % signs? These are placeholders for inserting variables into a string. The first %s is where our URL-friendly company name goes, and the second %s takes the key. This structure is common in modern languages, so take note. The % between the variables and the string with the %s’s is just convention that tells Python we’re using this technique.

The urllib.quote function replaces the space in company['name'] with safe %20 characters. Try typing urllib.quote(name) in the Python prompt and you’ll see what I mean.

And now.. drumroll… go ahead and print qry_result. There’s our data! More specifically:

qry_result['email_address']
qry_result['blog_url']
qry_result['phone_number']

Just like that, juicy, actionable sales data.

Step 5.

Here’s how we’d iterate through all the advertising companies and store the data we want into a list of our own.

my_results = []
for r in result['results']:
	sales_data = {}
	name = r['name']
	qry_url = 'http://api.crunchbase.com/v/1/company/%s.js?key=%s' % (urllib.quote(name), key)
	qry_response = urllib.urlopen(qry_url).read()
	qry_result = json.loads(qry_response)
	sales_data['email'] = qry_result['email_address']
	sales_data['blog'] = qry_result['blog_url']
	sales_data['phone'] = qry_result['phone_number']
	my_results.append(sales_data)

First, create the empty list we’ll fill up with data. Then, when we iterate through the results, we’re going to create a small dictionary for each company. sales_data starts each loop empty and fills up with the email, blog, and phone number of each business in result['results'].

But darn, if you run this, you’ll get this error: KeyError: ‘email_address’. That means one of my results had no email_address key. So, we’ll have to check for it before we pull it into our sales_data dictionary.

for r in result['results']:
	sales_data = {}
	name = r['name']
	print "Running", name
	qry_url = 'http://api.crunchbase.com/v/1/company/%s.js?key=%s' % (urllib.quote(name), key)
	qry_response = urllib.urlopen(qry_url).read()
	qry_result = json.loads(qry_response)
	sales_data['company'] = name
	if qry_result.has_key('email_address'): 
		sales_data['email'] = qry_result['email_address']
	if qry_result.has_key('blog_url'):
		sales_data['blog'] = qry_result['blog_url']
	if qry_result.has_key('phone_number'):
		sales_data['phone'] = qry_result['phone_number']
	my_results.append(sales_data)

There we go. Instead of having to scrape using BeautifulSoup, we can use urllib and json to make API calls and interpret the responses. If all goes well, you should see something like this:

[{'blog': '', 'phone': u'937.531.6631', 'company': u'Commuter Advertising', 'email': u'sparker@commuter-advertising.com'}, {'blog': u'http://www.qubed.us', 'phone': '', 'company': u'qubed advertising', 'email': u'oz@qubed.ro'}, {'blog': u'http://blog.mpression.net/', 'phone': u'+44 (0) 870 235 4042 ', 'company': u'4th Screen Advertising', 'email': u'info@4th-screen.com'}, {'blog': u'http://prova.com/blog/', 'phone': '', 'company': u'Prova | Advertising', 'email': u'support@prova.com'}, {'blog': u'http://hiliteadvertising.com/blog/', 'phone': u'877-457-5837', 'company': u'HiLite Advertising', 'email': u'info@hiliteadvertising.com'}, {'blog': '', 'phone': u'+91-011-26197623', 'company': u'WorldWide Advertising Network Private Ltd', 'email': u'info@worldwideadvertisingnetwork.com'}, {'blog': '', 'phone': u'401-272-1122', 'company': u'Creative Circle Advertising Solutions', 'email': u'bill@creativecirclemedia.com'}, {'company': u'VLG Advertising'}, {'blog': u'http://www.17stories.com/', 'phone': u'(512) 532-2907', 'company': u'Tocquigny Advertising & Interactive', 'email': u'awinsett@tocquigny.com'}, {'blog': u'http://www.blackdogadvertising.com/miami-advertising-blog/', 'phone': '', 'company': u'BlackDog Advertising', 'email': ''}]

There are hundreds of great APIs to explore for sales, including Yelp, LinkedIn, and Twitter. I’ll follow up with additional posts on each of these!

Blogging is Still Core

It can be very difficult for a company to have a social media strategy without having a blog. After all, social media posts are too short form and don’t provide enough content to keep visitors engaged. With a blog, you can create a base camp for all of your content that is always connected to the core of your content strategy.  It’s the perfect place to demonstrate your company’s thought leadership while allowing for readers to register for white papers or download ebooks. Then, by utilizing Facebook, Twitter, and LinkedIn you can easily leverage your content by promoting new content.

Don’t Be So Self-Serving on Twitter 

Once companies have a steady amount of content they assume that they only need to use their social media channels to announce new blog posts, promos, or services. While this is a good strategy, if social media followers only see self-promotion posts they will get bored and not click on your links. A great solution to this is to use a variation of the 4-1-1 strategy where 4 posts are tweets on other original content, 1 post is a re-tweet, and 1 post is self-serving.

Post Content Outside Your Niche on Facebook

With Facebook, the same strategy applies, however it’s important to realize that not every post needs to be relevant to your company’s niche. Post photos, current events, funny videos, and other intriguing posts that people want to click on and view. This is important because Facebook’s algorithm in showing a company’s posts to their followers has to do with the CTR of that company’s posts. With this in mind, a company does not need to only post serious subject matter or call to action promos. A balance of each will lead to more views of posts that actually need to be seen.

Publish Consistently

Lastly, it’s important to publish consistently. Some experts may even say that publishing consistently is more important than volume. By posting a new blog post once a day or even once a week, loyal visitors can depend on viewing new content at the same time. This improves repeat visitors and grows organic traffic and pageviews. With twitter, companies can schedule their tweets to be published at a certain time everyday.

A company’s blog is the hub for all content and information while a company’s social media channels are the hub for promotional announcements and brand engagement. By balancing self-serving posts with the posting of other relevant and entertaining posts a company’s messages will have a higher chance of being seen, read, and clicked on. Furthermore, by publishing content consistently, a brand will gain loyal repeat visitors and increase organic traffic.

1. Commit to Meeting Deadlines: Writing deadlines should be your priority, and you should always work towards delivering your work on time. The commitment to meet deadlines requires early preparation and starting to work on the project early enough. As a writer, get your fingers on the keyboard and start typing immediately after you understand the work content. The more time you waste before starting to work, the more workload piles up and may eventually seem like a monster you want to avoid.
2. Show Discipline: Without discipline, focus on work and erasing all the intrusions that come with working from home, meeting deadlines is impossible. Practicing rigorous self-discipline helps you plan for your work and other things later. For instance, you can set specific hours for strict work whereby you avoid interruptions such as phone calls or eating snacks. This will improve your concentration levels and help you maximize your writing potential.
3. Clear Deadline Communication and Outcomes: Being in agreement with the client on the set deadline is significantly useful for the writer. It can be done by confirming the correct deadline time zone and possible extensions. There are emergency cases when you will be expected to communicate with your client to request for a deadline extension, such as when power goes off or you have become ill.
4. Break Down Large Projects: Some projects are large and require a break down. As a result, you have to break this project into subsections and allocate certain amounts of time for each. The breakdown of a project quickly facilitates working. As you piece up the subsections, you will realize that you have worked faster than working on the whole project unbroken.
5. Research: Accurate and reliable works are credited to thorough research and writing skills. Though creative in your writing, it is required that you deliver your work on time and make readers feel confident with the work. Research offers many resource materials and facts, giving the writer confidence and accurate information. This significantly improves the writing speed.
6. Be Realistic: Be as realistic as you can to meet writing deadlines. Avoid over commitment as it delays your ability to meet deadlines. Do not repeat mistakes such as late work submission or grammatical errors. Do not stay up till late while you need to work the next day. Make sure you get a good night sleep so that you are fresh and alert at all working times.

Meeting deadlines in your work can be very rewarding. By continually meeting deadlines it’s much easier to acquire future work from past clients, and maximize your work time.