Reflections on the CS academic and industry job markets (part 1)

I went on the Computer Science academic and industry job market last year (2021-2022), and am writing this post to share a bit about my experience to other applicants. I received so much help along the way, so I wanted to pay some of that forward.

First, a bit about me – I recently finished up my PhD from the University of Washington in June 2022, so I was applying as a still-PhD student. My research interests lie in the intersection of natural language processing and computer vision, towards machines that understand the world beyond those modalities.

In this post (part 1) I'll be talking about my experience interviewing for primarily academic jobs, and in the next (part 2) I'll talk about why I changed my mind and decided to join OpenAI instead.

Disclaimer: these opinions are my own, so take everything I say with a grain of salt. Many other applicants in my cohort were far more successful at the computer science academic job search than me. Moreover, you should listen to professors who have been on the other end of things (evaluating people, not just being evaluated)!

I'm discussing my own experience here, trying to be as candid as possible.

Background: Going on the job market

"You wouldn't be able to do the research you want to do here at {industrial research lab}," one of my mentors (a research manager at said lab) told me.

That was the main factor behind wanting to go on the academic job market. I really enjoyed the work that I was doing. My PhD was going pretty well despite a lot of bumps along the way. I had some benchmarks that were widely used, and a new class of models that learned from YouTube videos, among other things.

The other two main factors to me: I enjoy mentoring and find that work valuable, and it felt like many of my most ambitious friends and colleagues were also going on the job market.

"Your work striked me as being more academic," another mentor told me.

Writing applications – October/November 2021

By Fall 2021, I was actively getting ready to go on the academic job market.

After a pretty lucky PhD, I thought that now was the time to go on the market. I was worried that, if I were to do a postdoc, some of the early work I did during my PhD would get stale. I just felt like I needed one more paper to round out my story – that ended up being MERLOT Reserve, learning from YouTube while combining vision, sound, and language.

"I feel like I'm going crazy with work" read a page in my journal from October 2021. I was running experiments for MERLOT reserve, advising mentees on their research projects, and then all the job market stuff: drafting a list of schools, writing the application materials, and contacting my letter writers.

Probably not as bad as the workload that many junior professors have, though!

My first "job search" task was to make a list of schools, which was kind of daunting. I ended up combing through the CRA jobs board, which has job ads for schools. Many of the schools are always hiring. I ended up using this list; feel free to build off it.

A few random tidbits:

Choosing schools: I mostly applied to US schools, plus a few in Canada and Europe with more US-like application processes. I didn't feel like I had much of an understanding about how the system outside the US works. I had to submit letters on time which was important, and many of those places had vastly earlier deadlines. The US schools felt more in-sync. Effectively my first deadline would be November 30th.

Statements: I spent a lot of time from August to November 2021 on my statements: research, teaching, and diversity. I found the teaching and diversity statements to be not so difficult – mostly I was just writing down what I had done in those areas.

The research statement was hardest. It took a few tries to lay out all that I had worked on in different areas (from AI for fake news detection, to benchmarks, to models) into a coherent document. Per the advice I was given, I tried to narrow it down to three interesting directions, and to emphasize impact. You can read what I wrote here.

Letter-writers: I needed four letters. My advisors, Yejin and Ali, were 2. I asked for external letters from Luke (at UW) and Oren (from AI2). I can't thank my letter-writers enough: they were all super helpful at not just writing me a letter but providing me excellent career and life advice throughout the process.

Finishing up research - mid-November 2021

I was still actively doing research up until the CVPR deadline (November 16 2021), when we submitted MERLOT Reserve. Honestly, it felt like the chillest deadline I had had in a while. I was working with an amazing group of collaborators, and I felt like I had chosen a good scope for the project (and was kind to us all in pre-training big models with enough time to spend). Everything else job-search wise was a lot more stressful!

We waited until January 2022 to arxiv it. Some designers at AI2 were helping us with a demo, and we wanted to release that together with the paper.

Submitting applications - Early December 2021

I had everything written, now I was just filling out a ton of forms (and checking on the status of my letters of recommendation).

Interfolio helped me a bunch during this time. I signed up for Interfolio Pro, which costed like $50 but saved me a lot of stress. This let me re-send submitted letters of recommendation to a bunch of schools.

After a stressful fall, Winter break was great! My partner and I drove all the way from Seattle to Salt Lake City, where we went skiing! We drove in part because we were hauling gear, in part because we wanted to go on an adventure, and in part because of concern about COVID (the Omicron variant was on the rise). This ended up being a bit ironic, though – come 2022, I was about to do a lot of flying.

Screening chats – December 2021 to February 2022.

I got my first email response while in Salt Lake City, from Cornell. It was to set up a screening interview with the search committee. (Over time, I'd get a bunch more of these over the course of January and February, too.)

The screening interviews were generally set up as 30 minute zoom calls, where I chatted with anywhere between one and six faculty members in the department.

Half of these conversations mostly seemed to be scripted, which in my experience made things a bit awkward. Namely: even when I knew the professors pretty well, I often was asked introductory questions like “what conferences do you publish in?”. (The scriptedness seemed to generally be required by the university, to standardize things).

The other focus of these interviews was to gauge my interest in these schools. I tried to be as honest as possible – disclosing the fact that I was open to moving across the country or even internationally, but that I wanted my partner to be happy with such a move. (She works as a software engineer, and was super supportive throughout the whole process -- even though me pursuing the academic path meant we'd have to leave Seattle, where we both had set up roots and made friends over the years.)

Scheduling interviews - January 2022 to March 2022

While bouldering one morning in early January, I checked my email and found I had my first interview invitation! It was from UChicago CS.

One thing that this (and other) interview invitations had in common was assigning me to a faculty host at that school. In most cases, the faculty host seemed to be the professor deemed closest to my research area, or perhaps the person who vouched for me the loudest. It's hard to say for sure, but it's something I tried to read into (for university X, who might already be on my side, and who do I need to work harder to win over?).

I scheduled the UChicago interview for February 2022, in person. I wanted to have a bit of time to finish up my job talk, but also wanted to be able to make improvements to it after the interview. I chose in-person to indicate more interest.

Scheduling strategy. Over the coming months – including after the UChicago interview passed, I got other interview invitations that I tried to schedule around. I tried to schedule interviews at the most competitive schools for March – so after I had some practice, but not too near the end of the cycle (when maybe they wouldn't have any slots left). It's hard to say whether this strategy helped, though.

The best advice I can give is to respond to these scheduling interviews quickly – though this is also unfortunate, because it forces candidates to be checking email constantly. For a different school, I ticked off my host because I didn't respond to a scheduling email sent Friday evening until the next monday – and as a result, got a later and perhaps less desirable slot than others.

Crafting my Job Talk - January 2022

I spent most of January working on my job talk -- see here for a recording.

It took a ton of revision, and workshopping it with different research groups. The hardest part for me was the intro. I'm someone who likes design and presentation, so having a theme was important for me. The theme I ended up going with was a forest of trees -- one for language, one for computer vision, with "grounding" as bringing them together. Cheesy.

The job talk that worked for me ended up being a completely different structure than my research statement; that's totally normal. I went with three sections - for benchmarks, one section for PIGLeT, and a final section for video models (MERLOT).

Per the advice I got, the job talk is a game of pragmatics. What work can you present to get people on your side? How can you get the crowd to think that you're both great at explaining stuff, while also establishing that you have "technical depth"? I tried to tailor the three sections of my talk for different crowds. VCR was for people who might think commonsense reasoning was something made up. PIGLeT was built around classic AI people who like the idea of "neuro-symbolic" models (even though that's not my primary takeaway of that). MERLOT was to show a connection between vision, language and speech; to connect to classic AI ideas of "script knowledge" and to have bar graphs with state of the art.

I prepared the job talk to be around 40 to 45 minutes – the standard talk length. What I didn't account for, though, was delays due to interruptions. Questions can take up a bunch of time, especially when people shout out them. I got delays of around 5 or 10 minutes due to questions, and giving satisfiable responses while not alienating the rest of the crowd was hard. I think in hindsight, I might have done better focusing on one or two directions - probably dropping the PIGLeT stuff because I didn't think it was as exciting as the rest of my story.

The talk title and bio also took a fair amount of iteration, too – the same pragmatics rules apply.

Academic Interviewing – February to April 2022

Interviewing was challenging and super unfamiliar to me at first, but it was something that I improved with practice – I think.

Each onsite interview generally consists of one-on-ones with professors, and a job talk. Having the job talk go first can help you contextualize your research before those later one-on-ones. (That's assuming those professors have the time to attend your job talk, which often isn't the case). Otherwise, you have to start by giving an elevator pitch about you and your research.

I'll start by summarizing my first job talk, and then some takeaways from one-on-ones –

Giving my first job talk – at UChicago

As mentioned earlier, I scheduled my interviews for in-person if possible, and my first one was UChicago. A sidenote – because all the academic interviews are during the first few months of the (calendar) year, they tend to coincide with cold weather and flight delays. Sure enough, when I was flying to UChicago, they were having a blizzard!

It was my first time giving my job talk publicly, and it didn't go as well as hoped. For starters: I have an example that I always give in my talks – about parsing a complex scene, that just so happens to be from a movie from the 90's that not many people have watched. Sure enough – when I asked the audience “Why do you think this person is pointing?” someone who was familiar with the movie responded with a movie-specific answer! Which made everyone else probably even more lost.

The hybrid setting was also tough to handle. I had questions both from in-person as well as those staying home (most people – since it was blizzarding that day). A tough thing about questions from remote attendees was that you have almost no warning as to when a remote question will pop up.

I had one question from the department chair, which was to the tune of "you haven't defined commonsense!" Which sure enough, I hadn't (and I updated my slides afterwards to have such a definition). But on a more broader note, communicating research ideas that appeal to a broad slice of computer science researchers is difficult. Many more traditional old-school professors, in my field but particularly in others like systems, tend to work on problems that are much more directly measurable than AI. In fact – one of the main problems that I study is to try to benchmark the hard-to-define. But it's still a challenge to communicate that to a broad audience.

I didn't get an offer here, but I'm happy I got it out of the way first as a practice run.

One-on-one and small-group interviews

Time-wise, the one-on-one and small-group interviews made up the majority of each academic interview. A few random clusters emerged –

Professor interviews. Professors in my area (closer to NLP and vision) were generally super supportive, and occasionally we talked shop about more detailed topics. For professors in other areas, it was important to contextualize my research more broadly, while also demonstrating interest in theirs – particularly if there might be opportunities for collaboration.

Most professors I chatted with were super fun to talk to. The wisdom I got from mentors – and something I'm sympathetic to – are that professor interviews are perhaps one of the highlights of going on the academic job market. You get a chance to refine your 'pitch' to a bunch of experts – spanning your field and others as well. It definitely taught me a bit more about 'what's out there,' including context about research topics outside of NLP/Vision.

That's the ideal, though. Though I met a bunch of amazing people and had a bunch of great conversations, I also had a bunch of frustrating conversations with people who seemed hostile. Some schools were FAR worse than others here, but I don't know whether those schools were hostile to me, or to everyone. A takeaway I would later reflect quite a lot on: if I noticed a school seemed hostile to me – a cisgendered white guy – how might it be for others from marginalized identities? How might it be for others who more directly challenge entrenched power structures in academia, through research, activism, or otherwise?

Timing was another difficult aspect. There were a few times when a meeting ran over - but at the same time, I didn't want to be rude to the person interviewing me. In hindsight, I should have worn a watch, maybe. Conversely, I felt like I had to kill time for longer interviews or with less-talkative faculty.

Overall – the whole process is exhausting. At the beginning I was taking notes, but after 160 one-on-one interviews I got generally burnt out (more on that later) and stopped.

Department chair/dean interviews. At first, I felt super intimidated by meeting with department chairs and deans – but in hindsight, they seem not as bad. Department chairs and dean-types seemed to all be great conversationalists – I never had to kill time with them. Much of it was them telling me random information to sell me on the school (even before I got an offer), like how the tenure process works.

Student meetings. These were generally in small groups of anywhere between two and ten. I felt like the student interviews were generally awkward. As a student at UW, I had a familiar script of how I thought these should go – we generally focused on topics like mentorship and diversity, equity, and inclusion. Not only because as students we have a unique perspective on these things, but also because it doesn't alienate others in the room who might be out-of-area.

Instead, I got asked a range of questions. Some were a bit narrow and technical (did you experiment with X loss function on Y dataset?). Some students asked for life/career advice, which made me feel a bit sheepish as I don't feel qualified to give it – I was a student myself while answering those questions! Many other places, though, had student interviews that were more similar to UW's – a range of students with different backgrounds, asking predominantly mentorship/advising questions.

Diversity, equity, and inclusion. I met a bunch of professors who really cared about their students, who were fighting for things like DEI at a school/department level. I felt like these made for some really valuable conversations.

On the other hand, I feel like I didn't get enough of these questions. Most professors I met seemed to want to talk mostly about research, which I also sympathize with – DEI is hard, and most institutions under-invest here.

Industry interviews – February 2022 to May 2022

Around the same time, I interviewed on the industry job markets. At first, it was primarily for one-year postdocs, but as my job market progressed, I became more and more interested in taking an industry job instead of an academic one. More on that later.

I'm really happy with how the industry job search turned out, but I don't have much specific advice here. I applied at the usual suspects, and tried to leverage connections I had when possible. Industry job talks are generally shorter, so I condensed my job talk a bit to two sections – focusing on VCR and MERLOT, which I was most excited about. Many places asked a bunch of Leetcode-style questions, so I had to practice those. All my interviews were virtual.

I suspect the job search will look a lot different for folks applying now, and perhaps later. The advice I got was to apply broadly, and to schedule my interviews with the bigger (slower) companies like Google first. I think those principles are still true – startups move faster than big companies at hiring.

Yet, changing macroeconomic conditions might also change these hiring dynamics. (They also change hiring dynamics at universities, but at a slower timescale, so I'm told.) From my own experience – during my search, Facebook went on a hiring freeze. At the time I'm writing this – it seems like many other ‘usual suspects' are slowing or freezing hiring too, or are laying off researchers.

As I said before and will elaborate on later – I'm really enjoying working at OpenAI, and we're hiring :)

The waiting game + conclusion

Perhaps the most stressful part about all this was waiting for decisions. Would I recieve an email from school X? Would that email be positive or negative? Beyond emails, I also spent way too much time on a slack channel created for applicants this cycle. I think solidarity between job-market applicants is important -- we would all (presumably) be fellow academic workers someday; but there was also a lot of anonymous bullying on the channel. I hope this culture changes in future years.

There's a whole lot more I could talk about – both highs and lows – but I think those might be better for an in-person conversation. If you made it to this point, I'd be happy to chat more about any of these things in person -- maybe over a coffee?

I'll pause here though, and discuss more about my thought process – why I decided to go to OpenAI – in the next post.