Notes from the Other Side of the Application Table

I’ve been on the applicant side of the table many times over the past couple of years, applying for MS and PhD programs, scholarships, and jobs. This, along with seeing different components of other successful and unsuccessful applications here and there, had given me a sense of what a good application might look like. However, this sense changed quite a bit recently, when I had to serve as part of the reading committee for the Stanford EE MS program. While I had similar experiences before, this year I had to read ~50 applications and score them, and it was different from what I expected: I originally had a sense of what I expect from a good application, such as good grades, relevant experience, a well-thought-out SOP, and strong recommendation letters. However, when I had to read 50 applications and somehow rate/rank them, I realized that almost all of them were “very good” based on my conventional standards, and that I had to somehow distinguish between a group of very good applications. I did not really have a plan as to how I wanted to do this, but once I went through them, I think my mind kind of automatically started to observe some patterns and extract some features that could actually distinguish between the applications. Here, I’m listing some of the most interesting observations, which may help someone prepare an application in the future. I’ve organized this into a list of observations that I made, and for each observation I have a recommendation that is of course a completely personal opinion, but could increase an applicant’s chances if I were a member of their reading committee. So here it goes:

Reference Letters

1. The part you control the least may matter the most: reference letters!

I found the reference letters to play the most important part in my decisions, not because they are inherently more important, but because they offered the most variance. There were many candidates whose profiles were very similar, and the only place I could meaningfully distinguish them was when reading the letters.

Recommendation: Take your reference letters very seriously. Long before you apply, try to build relationships that can result in good reference letters in the future!

2. I’ve seen this letter before. And before that. And before that: AI-generated letters!

Many letters were AI-generated or AI-assisted. Being a very heavy AI user myself, I think I’ve developed quite a sharp eye for catching AI. When it comes to letters especially, since the template is very standard, I kept seeing the exact same template, language, and writing style over and over again. Again, I’m not saying this makes a bad letter, but after you see a phrase like “It is without reservation that I recommend X in the strongest possible terms. I am confident they will be an asset to your program and to the field,” 10 times, it starts to become a baseline, and will essentially mean very little in my brain’s process of trying to find candidates who stand out.

Recommendation: I’m not exactly sure here, because it might not be nice to ask your reference to not use AI, and it may not even be a good idea at all. But it would be nice if you can get a letter from someone who you know will take some time preparing your letter, and would not just give your CV and a summary of your work to ChatGPT and ask it to generate (a draft of) the letter. Also, see the final remark about AI-generated vs. AI-assisted at the end of this page for some relevant comments.

3. The good, the bad, and the useless: three types of letters!

After reading a lot of reference letters, I (to some extent subconsciously) created a number of mental categories, each of which I think captures the letter’s contents in a single word or phrase:

• Did-well-in-class (DWIC): Letters where the only interaction between the candidate and the letter writer is an instructor-student relationship, mostly in a class where the student got a good grade. Depending on how much the letter can tell stories about you and how you handled the challenges in the class, it usually ranges from a useless letter to a mediocre one. Such letters can’t usually do the heavy lifting of your application, unless they come with a very special tag like “s/he was the smartest student I’ve seen in this class in 10 years!” Also, did-well-in-class letters, especially if more than one, give me another negative signal that says the candidate has not built enough meaningful relationships, and had to resort to multiple DWIC letters.
• CV-summary (CV-S): This was my general mental category for letters (or parts of a letter) where the letter writer starts telling stories about the candidate that are not directly relevant to the interactions between the two of them. There were many cases where a reference would write paragraphs on the candidate’s work in other classes, other research projects, internships, competitions, etc. When I read something like this, unless it is somehow relevant to the point the letter is trying to make, I read “I don’t have much to say about this person myself, so I’m giving you a summary of the impressive things I found on their CV,” hence the title “CV-summary!”
• Interaction story (IS): This is generally the best one, where the letter writer has had direct interactions with the candidate and has material to explain how those interactions went and what aspects of the candidate’s personality and qualities they got to observe. These letters usually involve first-hand observations and actual short stories or anecdotes, and often come from research advisors, mentors, or managers and supervisors. One bad variant of this category, which I observed a lot, was when the letter gets too focused on the technical aspects of the project itself, instead of focusing on the candidate’s performance. I remember reading letters that spent paragraphs explaining exactly what the project the candidate worked on was, with details I really did not need to know or would not really understand.

Recommendation: I would say you definitely need at least one IS letter. Not having one, in my opinion, can kill your application. Having two IS letters is pretty standard; statistically speaking, most of the applications that I saw had at least 2, so if you only have one, you might still be a little behind other applicants. It goes without saying that these categories can be combined, and it would be great if you have, for example, a DWIC/IS combination in one letter, from someone who was your class professor and later became a research advisor. Note that this advice is very standard; all applicants know that they need IS letters, and all letter writers know that they have to provide some stories and examples from their interactions. As a result, I think we end up with many such letters that look pretty much the same, such that when you metaphorically squint your eyes, you’ll find all those letters indistinguishable. To stand out, try to build meaningful interactions with at least one of your references over time, and maybe send them a list of interesting observations they might’ve made about you before they write the letter, so that they can come up with better examples that make you stand out.

4. The numbers that don’t quite add up: qualitative scores!

I think most universities have some form of collecting initial feedback from a reference, in the form of qualitative Likert-scale scores, asking them to indicate how strongly they recommend the candidate, possibly on a scale of 1 to 5, and also in what top percentage of the students they have worked with they would classify this candidate, often with the best choice being the top 1-2%, then the top 5%, and so on. First, I have to say that I find this piece of information extremely noisy, because letter writers seem to have very different thought processes when providing it, and I found very strong letters with low qualitative scores and quite weak letters with the highest possible qualitative scores. That said, if I see that all three references of a candidate decided to give them not-so-high scores, this probably carries some information, especially when you consider that these are probably the people the candidate thought had the best impression of them, which is likely higher than the average impression they actually make.

Recommendation: Try to have at least one reference who has a high regard for you and can write a very positive letter. Sometimes, often as a tie-breaker, you might want to go with a reference who has a more supportive style. Similarly, you might want to choose someone as your reference if you think they would consider you a top 1-2% student based on their impression of you.

5. Find your champion: the power of a star reference!

This is quite similar to the previous one, but since it’s important, I’m mentioning it independently: When evaluating the applications, there were a few where the references said something like “s/he is the best student researcher I have worked with in the past few years,” or “…one of the best in a very long time…,” and whenever I read something like this, it would probably become the one sentence in the entire application with the strongest impact on my evaluation. To make this a little more nerdy: if every reference letter has an overall impact score on a scale of 1 to 5 (with 5 being the strongest letter — one that can have the highest impact and impression on whoever reads your application), I would argue that if your reference letters make a (5,2,2) combination, it is probably better than a (3,3,3), even though they both average to 3. That five can often do something that 10 threes may not be able to accomplish.

Recommendation: Try to have a star reference. This requires investment, as they need to get to know you over time. Also, not everyone can become your star. They might have very high standards, or just not be the type to write very glowing letters. But if you are lucky enough to have someone who has the potential to become your star reference, it might be worth making some effort.

Short personal story: When I was applying for PhD programs, I had 3 senior references, all of whom had quite some history with me and would write good letters. There was also a junior professor whom I had taken two classes with and TAed for once. I reached out to that professor (as a backup) and asked if they could write me a letter, and if so, how strong that letter would be, given that I didn’t have any direct research or collaboration experience with them. Their response was so positive that it almost made me cry! This person had only been a professor for a few years at the time, and even though I wasn’t really that exceptional of a student, their response made me feel like they would very easily refer to me as the best student they’d ever worked with. I ended up using their letter instead of one of the senior ones, and I think it was the right decision!

Statement of Purpose

6. Your SOP is not a Wikipedia page: tell a coherent story!

I think the most important role of an SOP is to give a narrative about you, your work and background, what makes you think this is the right next step for you, and what comes next after that. I saw many SOPs that read like a collection of bullet points. It seemed like many candidates thought they needed to start with a nice “when-I-was-a-kid” paragraph, then list the things they did during and after their undergrad, and then wrap up with a “Stanford-is-my-perfect-next-step-because” paragraph. There is a very subtle point I’m trying to make here: a perfect SOP might have the exact same structure I just described, but having this structure doesn’t make your SOP a good one, nor does a good SOP need to follow this flow.

Recommendation: One might disagree with me, but in my opinion, when I’m reading an SOP, just like a novel, I should have a very good sense of what the storyline is and why one paragraph came after another. There should be a natural flow, not a list of every impressive thing you’ve done in your life. Do mention those impressive things, of course, but don’t make it a soup of random impressive facts and accomplishments that you included just because you felt you had to.

7. The paragraph everyone includes and very few get right: why you want to be here!

It is pretty standard to include a paragraph or two in the SOP explaining why you think this particular school or position is the perfect next step for you, and why you’re excited about it. In the ~50 SOPs that I read, every single one of them technically checked this box. But many of them seemed to only be trying to check the box! I’m pretty sure I read about “Stanford’s collaborative and interdisciplinary environment” more than 10 times. Yes, Stanford does have a collaborative and interdisciplinary environment, but unless you get more specific, this phrase is just a very templated cliché, which in my opinion adds no real value to your SOP. In my opinion, many of the SOPs that I read had their entire Stanford-is-my-perfect-next-step-because paragraphs full of these devoid-of-actual-information phrases and sentences. Another adjacent note is where candidates talk about the professors and advisors they want to work with. Again, I saw a lot of SOPs throwing in classic one-liners about professors, like “Prof. X’s work on multi-agent systems and transformer-based control problems is very much aligned with my background and interests.” Again, not that this is bad, but throwing one-liners about a number of professors is, in my opinion, much less valuable than elaborating on how one potential advisor’s work and research might actually be a good fit for what you want to do. It also signals that you have done more meaningful research than quickly asking ChatGPT for a list of potential advisors and a short summary of their work.

One funny example before I wrap up this observation: I often see people mentioning the program’s “emphasis on research and faculty mentorship” as one of the reasons they want to be at Stanford EE MS, while this might actually be one of the few strong EE MS programs that can be completed in a fully course-based manner, meaning you can do it with absolutely zero “emphasis on research and faculty mentorship!” If I read that in an SOP, I’ll be like “well, this may not be the right place for this candidate after all…”

Recommendation: Do the actual hard work. Read about the details of the program. Read about the details of a potential supervisor’s research. Then reflect your understanding in the SOP. I found it very impressive when a candidate mentioned a very specific property of the Stanford EE MS program, a specific aspect of a professor’s research, or a particular Stanford resource that could, for example, help them with their entrepreneurial ambitions. I would even find something like “I want to be at the heart of Silicon Valley and use its industry connections to get a job at Apple” more meaningful than “Stanford’s collaborative and interdisciplinary environment,” because the former is quite Stanford-specific while the latter probably applies to hundreds of great schools across the world.

8. Don’t let your SOP show up to work in pajamas: care about aesthetics!

This is not really that big of a deal, but when I see an SOP that has no title, no letterhead, no date or signature, and essentially nothing beyond content typed in Word using the default font, I just get a bad feeling. I don’t judge it based on these observations, and I actively try not to get biased by this, but it’s just a bad look, and I can’t help noticing it. There are other similar things as well. For example, if you are asked to adhere to a 2-page limit and you don’t, there might be a few reasons: 1. you didn’t properly read the requirements and didn’t know there was a limit, or 2. you noticed it, but didn’t care because you assumed they probably put it there to stop people from sending very long letters, and figured they won’t actually discard your application because of it. Well, both of these reasons, along with any other reason I can think of, are a bad look for the candidate.

Recommendation: This is an easy one! Care about how any document you produce looks! I’m pretty sure you will benefit from this at some point in your life, and it won’t waste too much of your time!

9. ChatGPT says I’m exceptional: AI-generated SOPs!

As opposed to recommendation letters, which can be AI-generated and you may not have any control over, you have full control over whether or not your SOP is AI-generated! AI-generated SOPs are not necessarily bad, but they tend to be quite easy to identify, and in my experience as a very heavy AI user, they tend to generate a lot of “nice” content that doesn’t really provide any real value to your SOP, just like the “Stanford’s collaborative and interdisciplinary environment” example I mentioned in Observation 7. Based on my observations, there are a lot of AI-generated SOPs out there, and they all look very similar. So, even though it’s not necessarily bad, it may just push your SOP into the looks-like-all-the-others pile.

Recommendation: Either write your first version yourself and then polish it using AI (my preferred method), or first get a draft from AI and then read every single word and sentence very carefully. This is something I recommend in all writing, especially academic writing: look at every single sentence in your text and ask yourself “What is this sentence doing here? How does it exactly contribute to the message I’m trying to convey? What would happen if I removed or changed it? Can I replace it with something better?” You’ll be surprised by how much you might end up changing your initial AI-generated draft. I’ll summarize and conclude this by teaching you a Persian idiom: look at every sentence with a buyer’s eye — the way you would look at something before buying it; you’re not going to spend your money until you make sure there is not a single scratch or flaw.

Also, see the final remark about AI-generated vs. AI-assisted at the end of this page for some relevant comments.

CV / List of Experiences

10. Jack of all trades, master of none: the hidden cost of listing everything you’ve ever done!

You will most probably upload a CV or fill out some forms listing your research and work experience, honors and awards, etc. There were cases where I would see a very long list of such items. While it can be impressive, for example, for an undergrad to have done 5 different research projects, it is also, quite frankly, impossible! Or, in more accurate terms, it is impossible to do 5 research projects while also being a full-time undergrad student, unless they are shallow and easy projects, or you are not that involved and most of the work is being done by others. Now, what can happen is that there might be one or two very important and impressive projects, and the rest are either lightweight or primarily led by others. This is very reasonable. However, keep in mind that throwing a long list of every single thing you’ve done in your CV or application form might serve as a double-edged sword. While it can look impressive, a reader might quickly glance at the list and assume that none of them is really that big of a deal.

Recommendation: Either keep it short and effective, or somehow make sure your most important works and awards don’t get lost in a crowd of less significant ones!

A Final Remark on the Use of AI

Like I mentioned earlier, I’m personally a really heavy AI user. It’s like a second brain, to which I often outsource thinking, remembering, and decision making! I don’t want to get into all the different aspects of this here, but as far as this article is concerned, let’s make a distinction between AI-generated and AI-assisted texts and essays. Consider the example of writing an SOP: I can collect my CV, a summary of my past work, my transcript, and of course some information about the place or position I’m applying to, and then give it all to an AI and ask it to give me a draft of my SOP. What you get at the end is what I refer to as “AI-generated.” Now, if you process this very meticulously like I explained in Recommendation 9, the result may look less AI-generated, something that I might start to call “AI-assisted.” But my favorite way of producing AI-assisted work is when you first think, come up with a plan for how you want to tell your story and construct the narrative, then write the SOP or an outline of it yourself, and finally get help from AI to make it perfect. You may not fully write it from the start. Maybe you leave some placeholders like [explain how being physically present at Stanford can increase my chances of getting a job at Apple] and let the AI actually write that segment for you later. But what happens is that you are the one who created and owns the narrative. If you use AI on a regular basis, it is often extremely easy to tell the difference between AI-generated and AI-assisted. The latter has the benefits of AI: no typos or grammatical errors, perfect punctuation, good word choice, etc., while not having the famous AI tone that so many texts have these days. That way, your SOP or essay will be less likely to end up in the looks-like-all-the-others pile!