I've always found training models on code to be interesting – there's large amounts of structured data and text is a fairly simple input/output interface that can be automatically be checked for syntax correctness.
Copilot has been a boost to developer productivity. But what else could ML models solve in the developer workflow?
- Merge conflict resolution. Most merges go smoothly – changes can be cleanly rebased on top of the main branch. But other times, two developers have made irreconcilable changes to the same file. It's difficult to automatically resolve changes like two developers updating packages or even renaming a file. Some of these changes are context specific (where the context isn't recorded in version control), while others require knowledge of languages, frameworks, or some higher level concept (e.g., a package file format). ML models might plausibly learn these patterns and perform higher-order semantic merges.
- Automatic linting and style checking – Developers often keep their code consistent with chosen linting rules – tabs instead of spaces, no unused variables, or other best practices that aren't enforced by the language itself. In addition to linting, there are stylistic suggestions that are often caught at code review time – how logic is split, documented, or called. Could we automatically encode some of the style of a particular project or company to reusable rules?
- Placeholder documentation – Documentation is often an after thought on tight deadlines and understaffed teams. Some engineering cultures mistakenly value shipping code much higher than lowering the cost of maintenance. LLMs can stub documentation for functions that's understandable and mostly correct. While it will likely never reach the quality of hand-written documentation, it could serve as a placeholder for documentation that can greatly speed up the time-to-document code.
- Project scaffolding – Starting a new project requires boilerplate folder and file creation. Even if you use a template (like create-react-app), there's a lot of customization you might need. Could LLMs provide suggestions on folder and file structure as well? Perhaps understanding import cycles and suggesting package boundaries or file boundaries? This is a task that is difficult for even the best developers – applications are constantly changing and shifting their scope of responsibility and thus where the logical boundary should be.
- Test suites and fuzzing – Copilot can already generate test cases, but what would a LLM-first fuzzing library look like?
- Infrastructure as code – Copilot already does this to some degree, code is code. But maybe there's a chance to automatically infer this from the project code. "Self-provisioning" infrastructure. This has been attempted many times before and always failed – usually at the language level.