{"id":515,"date":"2015-04-03T17:00:04","date_gmt":"2015-04-03T23:00:04","guid":{"rendered":"http:\/\/mattdturner.com\/wordpress\/?p=515"},"modified":"2018-08-03T12:17:17","modified_gmt":"2018-08-03T18:17:17","slug":"scrape-keywords-from-indeed-com-job-postings-2","status":"publish","type":"post","link":"http:\/\/mattdturner.com\/wordpress\/2015\/04\/scrape-keywords-from-indeed-com-job-postings-2\/","title":{"rendered":"Scrape Keywords from Indeed.com Job Postings"},"content":{"rendered":"
\n
\n
<\/div>\n
\n
\n
\n

This is code that will pull each job posting for a specific job title in a specific location (or Nationally) and return \/ plot the percentage of the postings that have certain keywords. The code is set up to search for all words except stopwords, and other user-defined words (there is probably a much more efficient way of doing this, but I had no need to change this once I had the code running). This allows the user to see common technical skills, as well as common soft skills that should be included on a resume.<\/p>\n

NOTE: I got this idea from https:\/\/jessesw.com\/Data-Science-Skills\/<\/a>. Obviously, just using his code would be of no real benefit to me, as I wanted to use the idea to help better my skills with scraping data from HTML files. So, I used his idea and developed my own code from scratch. I also modified the overall process a bit to better fit my needs.<\/p>\n

NOTE2: This code will not be able to identify multiple-word skills. So, for example, ‘machine learning’ will show up as either ‘machine’ or ‘learning’. However, ‘machine’ could show up for other phrases than ‘machine learning’.<\/p>\n

To run the code, change the city, state, and job title to whichever you wish. After generating the plot, you might need to add ‘keywords’ to the attitional_stop_words list if you do not want them to be included.
\n<\/p>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
In\u00a0[114]:<\/div>\n
\n
\n
\n
from<\/span> bs4<\/span> import<\/span> BeautifulSoup<\/span>\r\nimport<\/span> urllib<\/span>\r\nimport<\/span> re<\/span>\r\nfrom<\/span> time<\/span> import<\/span> sleep<\/span>\r\nfrom<\/span> collections<\/span> import<\/span> Counter<\/span>\r\nfrom<\/span> nltk.corpus<\/span> import<\/span> stopwords<\/span>\r\nimport<\/span> pandas<\/span> as<\/span> pd<\/span>\r\n%<\/span>matplotlib<\/span> inline\r\nimport<\/span> matplotlib.pylab<\/span> as<\/span> plt<\/span>\r\nfrom<\/span> matplotlib.backends.backend_pdf<\/span> import<\/span> PdfPages<\/span>\r\nplt<\/span>.<\/span>rcParams<\/span>[<\/span>'figure.figsize'<\/span>]<\/span> =<\/span> (<\/span>10.0<\/span>,<\/span> 8.0<\/span>)<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
<\/div>\n
\n
\n
Define the city, state, and job title.<\/p>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
In\u00a0[115]:<\/div>\n
\n
\n
\n
city<\/span> =<\/span> 'Seattle'<\/span>\r\nstate<\/span> =<\/span> 'WA'<\/span>\r\njob_title<\/span> =<\/span> 'Data Scientist'<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
<\/div>\n
\n
\n
Define a function that will take the url and pull out the text of the main body as a list of strings. Remove common words such as ‘the’, ‘or’, ‘and’, etc.<\/p>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
In\u00a0[168]:<\/div>\n
\n
\n
\n
def<\/span> clean_the_html<\/span>(<\/span>url<\/span>):<\/span>\r\n    # First try to download the html file<\/span>\r\n    try<\/span>:<\/span>\r\n        html<\/span> =<\/span> urllib<\/span>.<\/span>urlopen<\/span>(<\/span>url<\/span>)<\/span>\r\n    except<\/span>:<\/span>\r\n        return<\/span>\r\n    \r\n    #print url<\/span>\r\n    \r\n    # Open html in BeautifulSoup<\/span>\r\n    soup<\/span> =<\/span> BeautifulSoup<\/span>(<\/span>html<\/span>)<\/span>\r\n        \r\n    # Extract everything within the <p> tags<\/span>\r\n    text<\/span> =<\/span> soup<\/span>.<\/span>findAll<\/span>(<\/span>'body'<\/span>)<\/span>\r\n    word_list<\/span> =<\/span> ''<\/span>\r\n    for<\/span> line<\/span> in<\/span> text<\/span>:<\/span>\r\n        word_list<\/span> =<\/span> ' '<\/span>.<\/span>join<\/span>([<\/span>word_list<\/span>,<\/span>line<\/span>.<\/span>get_text<\/span>(<\/span>' '<\/span>,<\/span>strip<\/span>=<\/span>True<\/span>)<\/span>.<\/span>lower<\/span>()])<\/span>\r\n    \r\n    # Remove non text characters from list<\/span>\r\n    word_list<\/span> =<\/span> re<\/span>.<\/span>sub<\/span>(<\/span>'[^a-zA-Z+3]'<\/span>,<\/span>' '<\/span>,<\/span> word_list<\/span>)<\/span>\r\n\r\n    list_of_words<\/span> =<\/span> word_list<\/span>.<\/span>encode<\/span>(<\/span>'ascii'<\/span>,<\/span>'ignore'<\/span>)<\/span>.<\/span>split<\/span>()<\/span>\r\n                   \r\n    stop_words<\/span> =<\/span> set<\/span>(<\/span>stopwords<\/span>.<\/span>words<\/span>(<\/span>\"english\"<\/span>))<\/span>\r\n    additional_stop_words<\/span> =<\/span> [<\/span>'webfont'<\/span>,<\/span>'limited'<\/span>,<\/span>'saved'<\/span>,<\/span>'disability'<\/span>,<\/span>\\\r\n                             'desirable'<\/span>,<\/span>'nreum'<\/span>,<\/span>'skills'<\/span>,<\/span>'net'<\/span>,<\/span>'+'<\/span>,<\/span>'k'<\/span>,<\/span>\\\r\n                            'above'<\/span>,<\/span>'it'<\/span>,<\/span>'end'<\/span>,<\/span>'excellent'<\/span>,<\/span>'join'<\/span>,<\/span>'want'<\/span>,<\/span>\\\r\n                            'how'<\/span>,<\/span>'well'<\/span>,<\/span>'sets'<\/span>,<\/span>'like'<\/span>,<\/span>'page'<\/span>,<\/span>'home'<\/span>,<\/span>'demonstrated'<\/span>,<\/span>\\\r\n                            'other'<\/span>,<\/span>'re'<\/span>,<\/span>'size'<\/span>,<\/span>'etc'<\/span>,<\/span>'gettime'<\/span>,<\/span>'work'<\/span>,<\/span>'ms'<\/span>,<\/span>\\\r\n                            'zqdxyrmad'<\/span>,<\/span>'description'<\/span>,<\/span>'value'<\/span>,<\/span>'re'<\/span>,<\/span>'transactionname'<\/span>,<\/span>\\\r\n                            'education'<\/span>,<\/span>'daylight'<\/span>,<\/span>'highly'<\/span>,<\/span>'bodyrendered'<\/span>,<\/span>\\\r\n                            'amazon'<\/span>,<\/span>'new'<\/span>,<\/span>'bam'<\/span>,<\/span>'techniques'<\/span>,<\/span>'com'<\/span>,<\/span>city<\/span>.<\/span>lower<\/span>(),<\/span>\\\r\n                            state<\/span>.<\/span>lower<\/span>(),<\/span>'min'<\/span>,<\/span>'need'<\/span>,<\/span>'email'<\/span>,<\/span>'job'<\/span>,<\/span>'content'<\/span>,<\/span>'features'<\/span>,<\/span>\\\r\n                            'service'<\/span>,<\/span>'wa'<\/span>,<\/span>'id'<\/span>,<\/span>'modern'<\/span>,<\/span>'looking'<\/span>,<\/span>'eastern'<\/span>,<\/span>\\\r\n                            'qualifications'<\/span>,<\/span>'teams'<\/span>,<\/span>'based'<\/span>,<\/span>'false'<\/span>,<\/span>'times'<\/span>,<\/span>\\\r\n                            'software'<\/span>,<\/span>'career'<\/span>,<\/span>'ability'<\/span>,<\/span>'platform'<\/span>,<\/span>'years'<\/span>,<\/span>'data'<\/span>,<\/span>\\\r\n                            'date'<\/span>,<\/span>'product'<\/span>,<\/span>'team'<\/span>,<\/span>'time'<\/span>,<\/span>'agent'<\/span>,<\/span>'information'<\/span>,<\/span>\\\r\n                            'methods'<\/span>,<\/span>'candidate'<\/span>,<\/span>'customers'<\/span>,<\/span>'back'<\/span>,<\/span>'info'<\/span>,<\/span>'scientist'<\/span>,<\/span>\\\r\n                            'experience'<\/span>,<\/span>'apply'<\/span>,<\/span>'us'<\/span>,<\/span>'engineering'<\/span>,<\/span>'learning'<\/span>,<\/span>\\\r\n                            'strong'<\/span>,<\/span>'business'<\/span>,<\/span>'design'<\/span>,<\/span>'title'<\/span>,<\/span>'large'<\/span>,<\/span>'e'<\/span>,<\/span>'document'<\/span>,<\/span>\\\r\n                            'science'<\/span>,<\/span>'company'<\/span>,<\/span>'location'<\/span>,<\/span>'field'<\/span>,<\/span>'communication'<\/span>,<\/span>\\\r\n                            'customer'<\/span>,<\/span>'tools'<\/span>,<\/span>'used'<\/span>,<\/span>'research'<\/span>,<\/span>'model'<\/span>,<\/span>\\\r\n                            'opportunity'<\/span>,<\/span>'online'<\/span>,<\/span>'including'<\/span>,<\/span>'degree'<\/span>,<\/span>\\\r\n                            'preferred'<\/span>,<\/span>'across'<\/span>,<\/span>'beacon'<\/span>,<\/span>'using'<\/span>,<\/span>'friend'<\/span>,<\/span>'function'<\/span>,<\/span>\\\r\n                            'position'<\/span>,<\/span>'window'<\/span>,<\/span>'role'<\/span>,<\/span>'3'<\/span>,<\/span>'written'<\/span>,<\/span>'build'<\/span>,<\/span>\\\r\n                            'presentation'<\/span>,<\/span>'getelementbyid'<\/span>,<\/span>'technical'<\/span>,<\/span>'posted'<\/span>,<\/span>\\\r\n                            'newrelic'<\/span>,<\/span>'decision'<\/span>,<\/span>'log'<\/span>,<\/span>'errorbeacon'<\/span>,<\/span>'solutions'<\/span>,<\/span>\\\r\n                            'applicationtime'<\/span>,<\/span>'enable'<\/span>,<\/span>'responsibilities'<\/span>,<\/span>\\\r\n                            'models'<\/span>,<\/span>'applicationid'<\/span>,<\/span>'complex'<\/span>,<\/span>'licensekey'<\/span>,<\/span>\\\r\n                            'high'<\/span>,<\/span>'browser'<\/span>,<\/span>'d'<\/span>,<\/span>'nr'<\/span>,<\/span>'develop'<\/span>,<\/span>'please'<\/span>,<\/span>\\\r\n                            'selection'<\/span>,<\/span>'queuetime'<\/span>,<\/span>'cookies'<\/span>,<\/span>'icimsaddonload'<\/span>,<\/span>\\\r\n                            'computer'<\/span>,<\/span>'icims'<\/span>,<\/span>'scientists'<\/span>,<\/span>'great'<\/span>,<\/span>'returning'<\/span>,<\/span>\\\r\n                            'systems'<\/span>,<\/span>'writing'<\/span>,<\/span>'united'<\/span>,<\/span>'working'<\/span>,<\/span>'iframe'<\/span>,<\/span>\\\r\n                            'analyses'<\/span>,<\/span>'applications'<\/span>,<\/span>'try'<\/span>,<\/span>'related'<\/span>,<\/span>\\\r\n                            'states'<\/span>,<\/span>'languages'<\/span>,<\/span>'yghvbe'<\/span>,<\/span>'language'<\/span>,<\/span>'one'<\/span>,<\/span>\\\r\n                            'site'<\/span>,<\/span>'llc,'<\/span>,<\/span>'category'<\/span>,<\/span>'personalized'<\/span>,<\/span>'knowledge'<\/span>]<\/span>\r\n    \r\n    # Remove words from list<\/span>\r\n    truncated_list<\/span> =<\/span> [<\/span>w<\/span> for<\/span> w<\/span> in<\/span> list_of_words<\/span> if<\/span> not<\/span> (<\/span>w<\/span> in<\/span> stop_words<\/span> or<\/span> \\\r\n                      w<\/span> in<\/span> additional_stop_words<\/span>)]<\/span>\r\n    \r\n    truncated_set<\/span> =<\/span> set<\/span>(<\/span>truncated_list<\/span>)<\/span>\r\n    truncated_list<\/span> =<\/span> list<\/span>(<\/span>truncated_set<\/span>)<\/span>\r\n        \r\n    return<\/span> truncated_list<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
<\/div>\n
\n
\n
Define a function to generate a list of urls for a given search (i.e., ‘Data Scientist’). Each search result page has 10 non-sponsored links. Search the first url for ‘Jobs # to # of ###’ in order to determine how many iterations to perform.<\/p>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
In\u00a0[167]:<\/div>\n
\n
\n
\n
def<\/span> gen_url_list<\/span>(<\/span>city<\/span>,<\/span>state<\/span>,<\/span>job_name<\/span>):<\/span>\r\n    base_url<\/span> =<\/span> 'http:\/\/www.indeed.com\/'<\/span>\r\n    \r\n    job_term<\/span> =<\/span> re<\/span>.<\/span>sub<\/span>(<\/span>' '<\/span>,<\/span>'+'<\/span>,<\/span>job_name<\/span>.<\/span>lower<\/span>())<\/span>\r\n    \r\n    search_url<\/span> =<\/span> ''<\/span>.<\/span>join<\/span>([<\/span>base_url<\/span>,<\/span>'jobs?q='<\/span>,<\/span>job_term<\/span>,<\/span>'&l='<\/span>,<\/span>city<\/span>,<\/span>'%2C+'<\/span>,<\/span>state<\/span>])<\/span>\r\n    \r\n    try<\/span>:<\/span>\r\n        html<\/span> =<\/span> urllib<\/span>.<\/span>urlopen<\/span>(<\/span>search_url<\/span>)<\/span>\r\n    except<\/span>:<\/span>\r\n        return<\/span>\r\n    \r\n    soup<\/span> =<\/span> BeautifulSoup<\/span>(<\/span>html<\/span>)<\/span>\r\n    \r\n    total_jobs<\/span> =<\/span> soup<\/span>.<\/span>find<\/span>(<\/span>id<\/span> =<\/span> 'searchCount'<\/span>)<\/span>.<\/span>string<\/span>.<\/span>encode<\/span>(<\/span>'utf-8'<\/span>)<\/span>\r\n    job_nums<\/span> =<\/span> int<\/span>([<\/span>int<\/span>(<\/span>s<\/span>)<\/span> for<\/span> s<\/span> in<\/span> total_jobs<\/span>.<\/span>split<\/span>()<\/span> if<\/span> s<\/span>.<\/span>isdigit<\/span>()][<\/span>-<\/span>1<\/span>]<\/span>\/<\/span>10<\/span>)<\/span>\r\n    print<\/span> total_jobs<\/span>\r\n\r\n    job_URLS<\/span> =<\/span> []<\/span>\r\n    for<\/span> i<\/span> in<\/span> range<\/span>(<\/span>job_nums<\/span>+<\/span>1<\/span>):<\/span>\r\n        if<\/span> i<\/span> %<\/span> 10<\/span> ==<\/span> 0<\/span>:<\/span>\r\n            print<\/span> i<\/span>\r\n        page_url<\/span> =<\/span> ''<\/span>.<\/span>join<\/span>([<\/span>base_url<\/span>,<\/span>job_term<\/span>,<\/span>'&1='<\/span>,<\/span>city<\/span>,<\/span>'%2C+'<\/span>,<\/span>state<\/span>,<\/span>\\\r\n                            '&start='<\/span>,<\/span>str<\/span>((<\/span>i<\/span>+<\/span>1<\/span>)<\/span>*<\/span>10<\/span>)])<\/span>\r\n        html<\/span> =<\/span> urllib<\/span>.<\/span>urlopen<\/span>(<\/span>search_url<\/span>)<\/span>\r\n        \r\n        soup<\/span> =<\/span> BeautifulSoup<\/span>(<\/span>html<\/span>)<\/span>\r\n        \r\n        job_link_area<\/span> =<\/span> soup<\/span>.<\/span>findAll<\/span>(<\/span>'h2'<\/span>,{<\/span>'class'<\/span>:<\/span>'jobtitle'<\/span>})<\/span>\r\n\r\n        for<\/span> link<\/span> in<\/span> job_link_area<\/span>:<\/span>\r\n            match_href<\/span> =<\/span> re<\/span>.<\/span>search<\/span>(<\/span>'<a\\shref=\"(.+?)\"'<\/span>,<\/span>str<\/span>(<\/span>link<\/span>))<\/span>\r\n            if<\/span> match_href<\/span>:<\/span>\r\n                job_URLS<\/span>.<\/span>append<\/span>([<\/span>base_url<\/span> +<\/span> match_href<\/span>.<\/span>group<\/span>(<\/span>1<\/span>)])<\/span>\r\n\r\n    return<\/span> job_URLS<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
<\/div>\n
\n
\n
Now that we have a list of all of the URLs of job postings, pull the information from each site, clean the data, and populate the keyword list.<\/p>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
In\u00a0[164]:<\/div>\n
\n
\n
\n
def<\/span> job_posting_analysis<\/span>(<\/span>url_list<\/span>):<\/span>\r\n    job_skills<\/span> =<\/span> []<\/span>\r\n    count<\/span> =<\/span> 0<\/span>\r\n    for<\/span> url<\/span> in<\/span> url_list<\/span>:<\/span>\r\n        count<\/span> +=<\/span> 1<\/span>\r\n        if<\/span> count<\/span> %<\/span> 10<\/span> ==<\/span> 1<\/span>:<\/span>\r\n            print<\/span> count<\/span>\r\n        \r\n        posting_keywords<\/span> =<\/span> clean_the_html<\/span>(<\/span>url<\/span>[<\/span>0<\/span>])<\/span>\r\n        if<\/span> posting_keywords<\/span>:<\/span>\r\n            job_skills<\/span>.<\/span>append<\/span>(<\/span>posting_keywords<\/span>)<\/span>\r\n        sleep<\/span>(<\/span>0.5<\/span>)<\/span>\r\n        \r\n    return<\/span> job_skills<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
<\/div>\n
\n
\n
Now that the various functions are defined, run the code.<\/p>\n
First: run gen_url_list for the specified city, state, and jobtitle in order to generate
\nthe list of job posting links<\/p>\n
Second: run job_posting_analysis to pull out the job_skills listed for each job posting.<\/p>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
In\u00a0[169]:<\/div>\n
\n
\n
\n
print<\/span> 'Crawl indeed.com for '<\/span> +<\/span> city<\/span> +<\/span> ', '<\/span> +<\/span> state<\/span> +<\/span> ' '<\/span> +<\/span> job_title<\/span> +<\/span> \\\r\n' postings and generate a list of all of the job posting links'<\/span>\r\n\r\nurl_list<\/span> =<\/span> gen_url_list<\/span>(<\/span>city<\/span>,<\/span>state<\/span>,<\/span>job_title<\/span>)<\/span>\r\n\r\nprint<\/span> \"Given the job posting links, pull out the keywords for each posting\"<\/span>\r\n\r\njob_skills<\/span> =<\/span> job_posting_analysis<\/span>(<\/span>url_list<\/span>)<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
\n
<\/div>\n
\n
Crawl indeed.com for Seattle, WA Data Scientist postings and generate a list of all of the job posting links\r\nJobs 1 to 10 of 725\r\n0\r\n10\r\n20\r\n30\r\n40\r\n50\r\n60\r\n70\r\nGiven the job posting links, pull out the keywords for each posting that is found in the provided keywords_input variable\r\n1\r\n11\r\ntruncated for readability<\/strong>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
<\/div>\n
\n
\n
Now that we have the list of keywords in the job postings, calculate the number of postings in which each keyword appears. Then plot the data on a bar graph<\/p>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
In\u00a0[171]:<\/div>\n
\n
\n
\n
skill_frequency<\/span> =<\/span> Counter<\/span>()<\/span> # This will create a full counter of our terms. <\/span>\r\n[<\/span>skill_frequency<\/span>.<\/span>update<\/span>(<\/span>item<\/span>)<\/span> for<\/span> item<\/span> in<\/span> job_skills<\/span>]<\/span> # List comp<\/span>\r\nprint<\/span> skill_frequency<\/span>.<\/span>items<\/span>()<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
\n
<\/div>\n
\n
[('addedtojobcart', 73), ('applicationstatusdetail', 73), ('auc', 73), ('matlab', 365), ('worth', 73), ('merchant', 73), ('collaborate', 219), ('every', 146), ('tagging', 73), ('skillz', 73), ('companies', 219), ('vector', 73), ('clicktracks', 73), ('enhance', 146), ('enjoy', 73), ('leaders', 146), ('direct', 73), ('rigorous', 73), ('machines', 73), ('even', 73), ('hide', 73), ('selected', 73), ('children', 73), ('designing', 73), ('supplies', 73), ('centric', 73), ('behavior', 73), ('men', 73), ('createde', 73), ('hundreds', 73), ('employees', 146), ('economics', 146), ('reports', 73), truncated for readability <\/strong>]\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
In\u00a0[172]:<\/div>\n
\n
\n
\n
data_to_plot<\/span> =<\/span> pd<\/span>.<\/span>DataFrame<\/span>(<\/span>skill_frequency<\/span>.<\/span>items<\/span>(),<\/span>columns<\/span> =<\/span> [<\/span>'Skill'<\/span>,<\/span>'Occurances'<\/span>])<\/span>\r\n\r\ndata_to_plot<\/span>.<\/span>Occurances<\/span> =<\/span> (<\/span>data_to_plot<\/span>.<\/span>Occurances<\/span>)<\/span>*<\/span>100<\/span>\/<\/span>len<\/span>(<\/span>job_skills<\/span>)<\/span>\r\n\r\ndata_to_plot<\/span>.<\/span>sort<\/span>(<\/span>columns<\/span> =<\/span> 'Occurances'<\/span>,<\/span>ascending<\/span> =<\/span> False<\/span>,<\/span>inplace<\/span> =<\/span> True<\/span>)<\/span>\r\n\r\ntest_data<\/span> =<\/span> data_to_plot<\/span>.<\/span>head<\/span>(<\/span>15<\/span>)<\/span>  # plot only top 15 skills<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
In\u00a0[173]:<\/div>\n
\n
\n
\n
print<\/span> data_to_plot<\/span>.<\/span>head<\/span>(<\/span>20<\/span>)<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
\n
<\/div>\n
\n
            Skill  Occurances\r\n1148       python   80.109739\r\n66        machine   80.109739\r\n500             r   80.109739\r\n329           sql   80.109739\r\n468    statistics   70.096022\r\n431   statistical   70.096022\r\n544            js   60.082305\r\n476   programming   60.082305\r\n1382          pig   60.082305\r\n1066         hive   60.082305\r\n934        hadoop   60.082305\r\n77     algorithms   50.068587\r\n187     analytics   50.068587\r\n55      scripting   50.068587\r\n1242   predictive   50.068587\r\n161            d3   50.068587\r\n3          matlab   50.068587\r\n943      analysis   50.068587\r\n610         world   40.054870\r\n341           llc   40.054870\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
In\u00a0[174]:<\/div>\n
\n
\n
\n
frame<\/span> =<\/span> test_data<\/span>.<\/span>plot<\/span>(<\/span>x<\/span>=<\/span>'Skill'<\/span>,<\/span>kind<\/span>=<\/span>'bar'<\/span>,<\/span>legend<\/span>=<\/span>None<\/span>,<\/span>\\\r\n                  title<\/span>=<\/span>'Percentage of Data Scientist Job Postings with each Skill, ('<\/span>\\\r\n                  +<\/span> city<\/span> +<\/span> ', '<\/span> +<\/span> state<\/span> +<\/span> ')'<\/span>)<\/span>\r\n\r\n#plt.ylim([40,90])<\/span>\r\n\r\nfig<\/span> =<\/span> frame<\/span>.<\/span>get_figure<\/span>()<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
\n
<\/div>\n
<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
In\u00a0[175]:<\/div>\n
\n
\n
\n
filename<\/span> =<\/span> '_'<\/span>.<\/span>join<\/span>([<\/span>city<\/span>,<\/span>state<\/span>,<\/span>'skills'<\/span>])<\/span>\r\nfilename<\/span> =<\/span> ''<\/span>.<\/span>join<\/span>([<\/span>filename<\/span>,<\/span>'.pdf'<\/span>])<\/span>\r\npp<\/span> =<\/span> PdfPages<\/span>(<\/span>filename<\/span>)<\/span>\r\npp<\/span>.<\/span>savefig<\/span>(<\/span>fig<\/span>)<\/span>\r\npp<\/span>.<\/span>close<\/span>()<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
In\u00a0[\u00a0]:<\/div>\n
\n
\n
\n
 \r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
This is code that will pull each job posting for a specific job title in a specific location (or Nationally) and return \/ plot the percentage of the postings that have certain keywords. The code is set up to search for all words except stopwords, and other user-defined words (there is probably a much more […]<\/p>\n","protected":false},"author":1,"featured_media":519,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[3,235,234,64,11,233,20],"tags":[247,30,241,240,236,246,239,238,80,237,245,244],"class_list":["post-515","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-command-line","category-data-science","category-html-scraping","category-linux-2","category-mac","category-python","category-scripting","tag-beautifulsoup","tag-command-line-2","tag-data","tag-data-science","tag-html","tag-indeed-com","tag-matplotlib","tag-pandas","tag-python","tag-scraping","tag-stopwords","tag-urllib"],"_links":{"self":[{"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/posts\/515"}],"collection":[{"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/comments?post=515"}],"version-history":[{"count":4,"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/posts\/515\/revisions"}],"predecessor-version":[{"id":6145,"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/posts\/515\/revisions\/6145"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/media\/519"}],"wp:attachment":[{"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/media?parent=515"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/categories?post=515"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/tags?post=515"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}