When the initial Stable Diffusion models first came out, I fine-tuned them to add myself as a new concept. I did this by using the DreamBooth algorithm via fine-tuning. It only took about 5-10 images of myself. The results were pretty good (learning a new token, “mattrickard” on Stable Diffusion v1.5)
![07681c2e-0212-40d7-b80e-521c9cf24f51.jpeg 07681c2e-0212-40d7-b80e-521c9cf24f51.jpeg](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc479e034-c88e-4260-96f0-bd66d629b69d_512x512.jpeg)
![609EC921-998C-4165-8F6C-844D13CBB601.jpeg 609EC921-998C-4165-8F6C-844D13CBB601.jpeg](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3417dd59-e89b-4a8b-ae2e-557b2824e9dd_512x512.jpeg)
This time, I upgraded the model (SDXL) and upgraded the method (LoRA). Instead of fine-tuning, I was left with a relatively small (about 20 MB) of model weight deltas. I ended up doing LoRA applied to Dreambooth. There’s two methods I haven’t tried yet — textual inversion and pivotal tuning. The former adds a new token embedding and learns it via gradient descent. Pivotal tuning combines both textual inversion (training a new token) with Dreambooth LoRa (training a concept).
You can see how the models have improved in just a few months.
![custom 103.png custom 103.png](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91147d58-cb67-4519-9684-938f8a66628c_1024x1024.png)
![Wadim Kashin 001.png Wadim Kashin 001.png](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F35aae613-34de-4d1e-b857-81b0e6231716_1024x1024.png)
![Even Mehl Amundsen 001.png Even Mehl Amundsen 001.png](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd25f4ec-fba4-48f3-a62d-f1c8094a4eb3_1024x1024.png)
![_Style__Cyberpunk 001.png _Style__Cyberpunk 001.png](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab5cdb0d-c937-4f2b-8e44-946364713f68_1024x1024.png)