I let AI take over the editing: I developed an editing workflow of “material in, finished film out” in half a day

As a one-man business owner, my video footage is always accumulated faster than I can edit it. Lecture recordings, screen recordings, product demos… there are a bunch of audio and video files lying on the hard drive, waiting for me to cut them when I have time, but that “free time” never appears.

In the past two days, I decided to ask another question: Instead of squeezing out time to learn editing software, can AI directly take over the editing? This idea actually comes from the same origin. I have always believed that the starting point for making full use of AI is not to pick tools, but to dismantle the workflow. Film editing happens to be a good exercise.

After half a day, I had a working system. This article will talk to you about how it grew, what features it has, what pitfalls we have stepped on, and how this set of things can be used in the future.

Let’s look at the results first: these two are cut by AI.

The two videos below are the actual results of this workflow. From excerpts, camera movements, word cards, subtitles to soundtracks, everything was automatically assembled by the system. I only took action in two key areas. One is a horizontal quick clip for YouTube and Facebook, and the other is a vertical short video for the community.

If you want to see how this system can grow several promotional videos of different styles from a batch of materials in one go, I wrote that practical process as “From a pile of materials to eight promotional videos: It only took me half a day to turn the clips into an AI system”. This article wants to take you under the hood to see how it is designed and what pitfalls it has encountered.

Five features of this system

1. The editing decision table is the only source of truth. Where each cut is made, where the B-roll is stacked, and how the volume changes are all recorded in a versioned JSON file. If you want to change the cutting method, just change the decision table and recalculate the graphics. The original material will never be affected. This also means that every finished film can be completely reproduced and traced completely.

2. People only appear in two levels. The system compressed the professional post-production process into seven stages, but only asked me at two points: once for review when the editing plan was roughed out, and again before delivery of the finished film. The rest of the transcription, excerpt selection, breath cutting, mixing, subtitles, and multi-platform production are all automated.

3. Turn the sense of rhythm into rules that can be checked. The rhythm of a good film was originally a metaphysics, but now the system will automatically check: Are there exciting points arranged at 0%, 25%, 50%, and 75% of the entire film? Did the ending drag on too long? Will B-roll take over the spotlight? The machine first guards the bottom line, and humans only need to make the final judgment on aesthetics.

4. Screen lock is a hard rule. There is an iron rule in the post-production process in the industry: audio and color correction are done only after the picture is finalized (Picture Lock). This system has programmed it: after locking, any tool that wants to move the splicing point will be directly blocked and an error will be reported. Process discipline does not rely on consciousness, but on program code.

5. The title and chapter cards are designed using web technology. The card layout in the video is not drawn using editing software. Instead, HTML and CSS are written, screenshots are taken using a browser, and then combined into the video. This means that the degree of freedom in card design is equivalent to that in web design: four style directions, changing fonts, and changing colors are all a matter of changing a few lines of style.

The pitfalls you have stepped on are more valuable than the feature list

During the production of my first real video, AI helped me catch four problems that synthetic testing would never detect.

Development environment mines. The ffmpeg installed in the package manager is a streamlined version that lacks components needed to burn subtitles, and the error message is completely misleading. I spend more time proving that the problem is not what I thought it was than fixing it.

Blind spots in testing. All unit tests used synthetic materials of the same size. As a result, for the first time, when encountering a real situation where the resolution of the spoken word footage and the screen recording were different, the rendering engine crashed directly. And this is exactly the norm for this type of film. Lesson: The diversity of test materials determines how much reality you can capture in advance.

A contract that was announced but not implemented. There is a field in the editing decision table for ending audio to fade out, but the rendering engine never implements it. The announcement is silently ignored, and the end of the film is cut. This problem was not discovered by me, but was caught by an independent AI acceptance agent: it did not trust any self-declaration, verified it item by item with waveform data, and finally ruled it failed. Since then, this project has an additional rule: each field in the specification must have a corresponding verification test.

AI can see the picture, but not the time. The system uses frame extraction to understand the content of the material, but the four static images cannot tell that the camera was still shaking eight seconds ago. So the opening of the first version of the finished film was a six-second ceiling. The solution is very interesting: instead of re-editing, the opening scene is covered with a slideshow of the theme, and the sound of the golden sentence is completely retained. Constraints lead to better designs.

It can be done in half a day, relying on methods rather than miracles

Looking back on this half-day work process, what is really worth recording is not the speed, but several working methods. These methods were not invented for the purpose of editing the film, but are the assets I accumulated from working with the AI ​​army throughout the year. I talked about how they were developed step by step in “Three months, one person, an army called Claude Code”.

• Parallel division of labor: Fifteen development tasks are divided into three waves. Each wave is implemented by multiple AI agents at the same time. The interface is locked in advance so that no one will step on anyone else. This is what struck me most when I read “Inverse Division of Labor”: when a person starts to own an AI army, the logic of division of labor is completely reversed.
• Verification is not self-inspection: The agent who writes the program does not verify his own work. An independent agent who has not participated in the development always performs adversarial acceptance. Its task is to find the problem, not to confirm that there is no problem. This mechanism caught two real bugs that would leak out in one day.
• Design of artificial judgment points: Instead of giving everything that AI can do to AI, it is clear which judgments essentially belong to humans: the factual correctness of copywriting, the final choice of aesthetics, and the decision-making power of publication. The higher the degree of automation of the system, the clearer these boundaries must be.

What will it look like next?

This system currently fully supports oral broadcasting, screen recording and course videos. Listed on the roadmap are: color grading pipeline, music lag mixing and editing, intelligent reconstruction of straight short videos, and automatic switching of multiple cameras.

But more important than functionality is that it changed my content production structure: where before there was a mountain of edits between recording and release, now there are only two reviews. Lecture recordings can be turned into course videos on the same day, and the accumulated material library finally has an outlet.

In fact, this logic of “build the system first, then iterate production” still holds true in another scenario. When you replace materials with documents and finished films with papers, you are doing the same thing: handing over repetitive processes to an AI system that you can run over and over again. I have organized this road more completely in “Don’t Just Learn AI Tools: How Researchers Create a Repeatable Academic Co-Pilot”. This entire set of editing workflow is actually what I led everyone to practice in the AI content production system workshop:

The experience I most want to share

If you are also using AI for collaborative development, the experience I most want to share is: use AI as a team, not as a tool. Give it clear acceptance conditions, let different roles check and balance each other, leave human judgment where people are really needed, and leave the rest to the process.

This is why I keep saying, The prompt words will expire, but you will not. When execution becomes cheaper and cheaper, what is really valuable is your ability to clearly explain the problems and premises, and to design who will check whom. This feeling of incorporating AI into the workflow and treating it as a long-term partner is actually the same thing as what I wrote before about “When AI Becomes My Editor”, except that this time the stage is changed from text to video.

---

If you also want to turn this method of “material in, film out” into a system that you can run over and over again, welcome to the AI Content Production System Workshop I opened in my one-person company solo.tw. I usually put more ideas for creators and independent workers at content.tw (opens in a new tab), and you are welcome to visit vista.tw often.