A few possible definitions and job descriptions of a prompt engineer. A technical look at what the role might mean today.
Job Description
Research, design, and evaluate task-based prompts across a variety of models. Your output will be proposing effective prompts for a given task, supported by experimentation data.
Requirements and Skills
Proficient in Python. What level of proficiency? Somewhere in-between a data scientist and a data analyst. Needs to be able to quickly put together a Python project (setting up notebooks, managing dependencies, and understanding the basics of debugging, APIs, and object-oriented programming). However, it is not expected to write production code or deploy applications.
In the future, better tools might be available for prompt engineers (e.g., low-code UI workbenches or even libraries in other languages like TypeScript). Still, today’s prompt engineers will have to build those themselves.
Able to design and evaluate experiments. Prompts will work differently on different models. Model evaluation is more art than science (today). While task-specific evaluation is easier than general knowledge evaluation, there’s still much work in developing and running these experiments.
Today, it means running models and presenting results that mix prototype code, markdown, and graphs in Jupyter notebooks.
Great candidates will be able to:
- Build reusable experimentation platforms around prompts.
- Develop efficient prompts and understand the mechanics of the underlying tokenizers.
- Design and prototype workflows that involve multiple models (or even multiple modalities).
- Work with data engineers to integrate internal data into AI workflows.
- Evaluate models across various criteria like latency, perplexity, fine-tuning, size, training data, cost, and more (see choosing the right model).
- Propose opportunities for fine-tuning after their prompts have collected data in production.