I've tried the deep dive thing a few times and always come out frustrated. It seems like the important stuff that differentiates the expert from someone with casual knowledge isn't documented. Sometimes it's explained by the knowledge being locked behind a paywall or IP protection of some sort. Sometimes its not, but the knowledge seems to be locked in the brains of a few individuals or organizations and can only be unlocked if your'e lucky enough for them to take you in (employment at a company that specializes in something, or having a mentor type relationship with another expert). On the other hand, a minority of the experts I know are nearly completely self taught and I'm struggling to figure out how to manage. Generally as a hobbysit Like I said every time I've tried to deep dive I reach a point where I'm not sure what I'm missing, where to find it.
Anyone who tries to convince you that your should pay for advice on "how to become an expert" has expertise at only one thing: lying.
PS: I can teach you how to be an expert at spotting liars. For 5 dollars I'll tell you the secret.
Behind paywall 1: First lesson - don't pay for lessons on expertise. For 5 more dollars, I'll let you read chapter 2
Behind paywall 2: You are almost there! So close! You should be getting familiar with the trick by now. There was a secret lesson in part one. I'll tell the secret behind the trick for just 5 more dollars.
Behind paywall 3: Fool me once, shame on you. Fool me twice, shame on me. Fool me three times, and now we are here! Wanna know how I fooled you three times? Only 5 more dollars!
Now you are 15 dollars into spending 5 dollars on how to spot a liar. Just 5 more dollars!
Once you figure out the trick, you can consider yourself an expert.
A thousand years ago, anyone could have invented relatively basic math - but they didn't. The reason is that they didn't have other people to inspire and motivate and teach them.
Very little knowledge comes ab initio or from a vacuum. Expertise is not created by working in isolation for decades. To build expertise, you need to be interacting with other experts. This doesn't mean they need to teach you, nor does it mean they need to be be in person. But you have to be at least reading what they write.
It's much easier to reach the clouds if you stand on the shoulders of giants.
It isn't documented because it can't. The difference between an expert and a novice is that an expert has been exposed to and working with a particular subject for so long that there are higher level abstractions that have become "chunked" or "codified" in their mind such that they look at the problem differently than novice people.
Example
`for(int i = 0; i <= list.length; i++) { ... }`
A novice will see this and after parsing the pieces can tell you that this takes a counter i and does an action until i is the length of the list.
A more experienced person will see this as a for loop iterating through a list.
An even more experienced person may intuitively map this to the need of the program to operate on a list data structure and begin to immediately start asking why we didn't use a more functional approach, what effect will this have on the program when my list has 0 items, is there a way I can simplfy this code? Etc
The expert knows this not because he has read it in a book or because he followed design docs, he knows because he has practiced 10,000 hours, which includes doing something evaluating what went well, what did not, what could be improved, looking for new knowledge and new ways to do things.
In contrast too the man who has practiced 1 hours 10,000 times that has mindlessly written this block of code and continues to repeat this abstraction because this is his go to and familiar tool.
I would use the analogy that just because a woman can have a baby in 9 months, 9 women cannot have a baby in 1 month. You must spend 9 months to have a baby. So to be an expert at anything, it takes years of practice, errors, lessons learned, experiences to fully grasp a topic as an expert.
What the experts know is the things that aren't documented. Some of those are things that are hard to document, because the right thing to do requires situational knowledge, others are things that experts don't realize aren't obvious.
The way to become an expert isn't solely by reading documentation, it's by trying to do the thing, and figuring out what works (and hopefully why). For software, that often means diving into the source code, as well as the source code of the environment the code runs in (and sometimes that environment, etc --- prefer less layers and simpler systems, because it's easy to dive down to the bottom)
0: learn the rules
1: what do I have
2: what might the others have
3: what might the others think I have
4: what might the others think that I think they have...
You can find more info on a similar school of thought by googling for "dreyfus levels of skill acquisition", which explains the 5 levels you have to go through to master something:
0: What are the rules
1: I know the rules
2: I understand how the rules work
3: I understand how the rules work together
4: I see patterns and develop an instinct
5: I understand why the rules are there, and would know when to deviate
I actually had a talk a decade ago about this: https://www.slideshare.net/mobile/TomJanssens1/getting-bette...
Edit: updated my last poker rule; it has been a while, and I am typing this on my mobile..
1) You have a few successes when a few people (who know each other) ask you to solve a problem in a particular area. You may have to study and stretch in order to accomplish those successes, but if the people asking you have far less previous knowledge of the problems that you solved than you do, solving the types of questions that they could formulate might be easy.
2) They start calling you an expert between each other, and referring other people who are solving the same types of problems to you. You study hard to protect this reputation, to avoid failures that will reflect on the people talking you up.
3) They make up myths about your genius and spread them, which also fluffs up their own reputation for knowing you and referring others to you.
4) You start to believe the myths.
So extensive practice motivated by fear of failures which would result in rejection and looking like a fraud (and possible commensurate loss of income.) I think you can only get this as a hobbyist if you belong to a hobbyist community that could also provide that social pressure. That community would also have to have an element of material output; i.e. something to differentiate talk from skill, and something that allows you to collaborate with and advise others.
I think you can become pretty adequate at something by yourself, without social elements, but not an expert. I don't even think there's such a thing as an unrecognized expert; that's like an unrecognized genius. Being called an expert or a genius is just code for being respected.
The sharply refined ability to learn, internalize and execute knowledge work looks like expertise to everyone.
> always come out frustrated
You can't stop there.
Yes, a lot of expert knowledge is locked up in the heads of experts. It is very hard (if not impossible) to write down all the implicit and explicit knowledge that experts have, so it doesn't always happen. It's very hard to become an expert alone. I think this also says something about the nature of expertise: it is something that is constructed by experts themselves in their minds. There was a story that a famous mathematician would tell is grad students, holding up an important book, "You should know everything in this book ... but don't read it!"
In my mind there are two types of expertise: learning something that exists or making something new.
I've always done the second: find an opportunity to do something new, deep dive on it and figure it out, and then you are the expert by default. Then it's just a matter of keeping up on the state of the art while everyone else starts doing what you're doing but does it better than you.
> Generally as a [hobbyist] ...
It sounds as though you expect that reading or doing something once will confer expert status. Instead, try to think of it more like a lifestyle.
If you study something every day for ten years — constantly trying out new ideas to test the limits of the field and your understanding, and teaching others how to get started to develop a deeper intuition and understanding of the basics — you will most definitely be an expert at the end of those ten years.
At no point will you necessarily _feel_ like an expert; that's just a label people give to those who live/breathe/dream a topic deeply for many years.
At least one person on my work team thinks I'm really smart and amazing, but I don't feel like I'm an expert in anything I do.
I would say that an expert knows how to put together knowledge in a certain area of expertise and can quickly judge whether something is plausible and implausible. An expert can identify problems and potential solutions.
I don't think being frustrated or feeling some level of imposter syndrome makes you not an expert.
You need to do the thing to gain expertise but also learn it inside out. Academic + school of life = expert.
Academic can mean formal education, can also mean self taught from books, papers or talking to other people.
I considered myself competent in some stuff, but never an 'expert'. About 10 years past, peoples in various IEEE societies prodded me to publish stuff and do talks at some of the respective annual symposiums. I still do not consider myself an expert - because I do know some actual experts in some of my professional fields and endeavors, but there are other engineers that do rank me as such, which I have accepted because it gets me more work.
My wife became an expert by solving some difficult problems. She solved these seemingly intractable problems by deep observations, knowing what questions to ask, and knowing whom to ask questions.
I have increased my professional competence by education (both formal and self-taught) and carefully choosing companies and problems to solve. But I am too general of a practitioner to be considered an 'expert'.
After 4 years of doing it, I became probably one of the 10 or even less best experts in the world with this product but I was really tired of it. The technology was already outdated but still alive, so before giving up for more interesting things I became freelance during 2 years and got paid 4 times my previous salary. It was painful and boring to keep doing the same but it helped me to pay my house.
I could have extended it for 2 or 3 more years before the technology died for good, but I was so tired of it that I decided the money was not worth it anymore and I started again to be an apprentice of the technologies I missed during the 6 years period of being an expert in a very single thing.
Now I still struggling every day because I've been learning too many things but I still didn't have the chance of a long term project that helps me become an expert.
Experts historically tended to write prolifically. They’re known as experts because others are familiar with their work. The modern equivalent of writing is more broad but the part about communication of ideas broadly still applies.
A parole officer will not learn much if 4 years after their decision (one of thousands of decisions they made) they are told it was wrong.
One of my favorite papers: In Praise Of Epistemic Irresponsibility: How Lazy And Ignorant Can You Be? by Michael A. Bishop
https://philosophy.fsu.edu/sites/g/files/imported/storage/or...
Marketing. Most "experts" are also very good at marketing themselves as such. They may not know much more than their peers in that area - but they are the go-to because they are visible: either on social media, through their writings, etc...
I recognize this is a somewhat alternative view point from many of the responses provided here. I am certainly not downplaying the time and dedication required to a particular subject matter as a pre-requisite to becoming an expert. An additional viewpoint that positioning yourself as an expert in the public eye may also improve your communication and teaching skills when conveying ideas in that subject matter, thus further reinforcing your expertise.
https://cryptome.org/2013/01/aaron-swartz/On-the-Internet.pd... (PDF)
The later chapters on virtual worlds in the second edition have dated a bit, but before that, it's almost entirely gold, even though it was written so long ago.
When you do something a lot and you also study the subject (whether at university or self-taught) you'll really understand why how you do stuff matters. Whether it's picking the correct data structure for your program or deciding what material to use to repair a damaged wall. You'll be a true expert that people rely on and trust. And, they'll pay for that expertise.
Find Q&A sites that talk about the think you're trying to become expert in and try to answer as many questions as you can. You don't have to actually answer the questions -- those sites are often hellscapes.
> It seems like the important stuff that differentiates the expert from someone with casual knowledge isn't documented.
Experience is the biggest driver of expertise, and most people just don't document the things they find out unless they have to. But often the information does exist, if it's been around for a while. Ask yourself how the expert became an expert. What other fields might they have studied? What jobs might they have had? What similar subjects might reveal some of the details?
> Sometimes its not, but the knowledge seems to be locked in the brains of a few individuals or organizations and can only be unlocked if your'e lucky enough for them to take you in
Yep. Working with others is an important part of building expertise. Building relationships opens up new pathways to find information.
> On the other hand, a minority of the experts I know are nearly completely self taught and I'm struggling to figure out how to manage.
Relax. Expertise is not something you can force, it comes with time. Some things are easier than others, but if you continue to slowly work at it, you'll get better.
> Generally as a hobbysit Like I said every time I've tried to deep dive I reach a point where I'm not sure what I'm missing, where to find it.
Not every fish can be found by diving deep. Searching for knowledge is a normal part of building expertise; don't stress if you're not immediately sure where to look. If you get stuck, start again from the basics and question your assumptions. Try thinking about it from both a low-level and high-level view. Try to change your perspective. You will invariably come up with new places to search.
As you search, write down questions. If you get stuck on one, start looking into another. Remember to take breaks and get enough sleep, exercise, good diet. We learn best when we are stress-free, healthy and happy.
Becoming an expert is not a one-time effort of studying. It takes time. I like the cliche above because it not only explains a trap many people fall into of thinking that doing the same thing multiple times is equivalent to learning, it also sets an expectation that true expertise takes years, not months.
Though I do not have the citations, I am sure others here could relate many examples.
Everyone I've known who has mastered something simply has done it a million times, observed what they did right and wrong, corrected it, and leveled up each time.
I don’t think anyone can read their way to expertise. There’s no secret technique or knowledge hidden behind paywalls.
Rather than thinking about “becoming an expert” reframe it as “mastering a skill or domain.” No on is an expert, that’s a shorthand to describe someone who achieved mastery or deep understanding through deliberate practice.
By leveraging your academic credentials, legal authority and social connections to promote an agenda (e.g. corporate agenda) through carefully crafted lies which intelligent people in your field cannot easily disprove but which the masses will openly embrace because it aligns with their primal incentives.
It's about being able to bypass people's logic and reasoning skills and appealing to their underlying primal instincts; namely their need to feel safe, to feel accepted, to acquire resources, etc...
The main role of experts is to give people a false sense of certainty and safety.
2. Start learning about it and doing stuff with this knowledge. Intermittently look for new information about this thing.
3. Keep on working with this thing. Find ways to use it whenever possible, even if it is not perhaps the most efficient tool for the job at hand.
4. Make mistakes with this thing. Make some little ones, make big ones. Whenever you make mistakes try to ask yourself how to make sure you never make this mistake again. ("Making mistakes" is not a thing you try to do; it is a thing you will do. They can be small. Or they can be stressful and complicated and expensive and scarring. An expert will probably have a few of the latter under their belt, unless they were very lucky and only had a few really, really close calls.)
5. Keep working with the thing. Can you figure out a way to get a multi-year project with the thing going? Do you still fundamentally enjoy working with the thing when you are at the stage when you can do this - and when you are finished with that multi-year project, as well? Can you find a way to work under or with someone who is great with the thing? They'll teach you a lot of stuff that doesn't make it into books, as well as call your attention to really fundamental stuff that you've been blowing off like "kid, the way you hold your tool is gonna fuck your hands up in five years, start doing it this way if you want to be able to keep doing this for the rest of your life, here's some exercises to get used to it". (Seriously: I am an artist and I did not get instructed in How To Properly Hold A Drawing Tool until I was in the animation industry.)
6. Hang out somewhere people ask questions about the thing. Answer them. Answer the heck out of the questions you see allllll the time and are tired of answering and make key expansion shortcuts for them. Play around with the rare, interesting questions and answer them in the most novice-friendly way you can. If you know five ways to solve a novice's question about the thing, then go into some detail on all of these ways and talk about which one you would probably reach for if it came up in practice, and about why that's your choice.
7. Keep on doing the thing. Lots of little projects? Big long projects? Maybe both. It's easier to experiment in the little projects but the big ones are important too.
----
There is a character in Zelazny's Chronicles of Amber who is The Absolute Best General And Swordsman In The Entirety Of Reality. He wasn't born that way. Nobody is. But over many centuries of life, he studied and practiced every day. Every day, for hundreds of years, he spent some time studying the great battles of old, practicing his swordsmanship, playing wargames, engaging in actual military campaigns from every level from top to bottom. He has been doing this longer than everyone else alive, and thus, he is an expert.
You do not need to go to this extreme. You probably can't unless you turn out to be an immortal who can hop between alternate realities to see how various changes effect the ultimate outcome of a huge project over multiple trials. But you need to keep coming back to the thing you will become an expert in.
You can't just do one "deep dive" and become an expert. You have to live with the thing, you have to spend time working with it, time just kicking back thinking about the thing, you have to spend time catching up on the latest news about the thing, you have to internalize some of the thing so that the back of your mind just prompts you with little warnings based on something you fucked up ten years ago. You have to dive to the bottom and stay there.
And you will need to sacrifice some other shit on the altar of this expertise. There's only so many hours in the day, and only so many days in your life.
It also depends on who you ask. Some people have a different mental model of the meaning of words or the values attached to them. It can be as simple as: "the person that I ask stuff from and always has the answer" or as complex as "this person doesn't just read, repeat or know the theory or know the abstract theory but is even beyond that".
How do _you_ become an expert? It depends. What do you think an expert should be? Find that, then find where you are and the gap between that and where you need to be and start working on it. An expert at walking your own dog has a different route than say, an expert at silicon substrate doping.
Since we are on Hacker News, we could take being an expert at Linux Device Drivers as a subject. You'd be able to read and test and expand as much as you'd like until you become the expert you desire to be. Examples, documents and sources are all available. It does require base knowledge to build on; i.e. building software, writing and reading C, the specifics of the kernel, the specifics of the kernel build system. At some point if you know how to create and build a module that is a driver, you'd be a beginner and you can start becoming advanced by perhaps expanding on your knowledge by nibbling at the borders of the subject; the build system itself, the configuration system, the hardware underpinnings for drivers. Next you might become interested in memory maps and CPU architectures. At some point you can understand end-to-end how a click of a button translates into an event. You'd be advanced. Now take all of those concepts and things you're reading and the parts you've designed. That is a single subjects within borders. When you go beyond that, i.e. design the theories that define it you're an expert. Not only can you build and work with it, understand its context and modify it at will, but you can re-define and create it based on your own expert knowledge and abstract theory behind the concepts and implementations.
You can also make it much smaller than that. I.e. become an expert at folding t-shirts. Still means you need to know about folding, fabrics, speed vs. quality, quantity vs. quality (are you folding t-shirts for yourself or for a warehouse? is it manual or automated?). Once you know how to define what folding means and how it is affected and how it affects its context (i.e. what happens to the fibers? and what if it's dirty, or new? or when it has been washed 10 times? what about 100?) you could pretty much call yourself a t-shirt folding expert. This is still big, but a lot smaller than a software example.