Author Archives: jonahjameson

Definitely getting there, in the quest to use Stable Diffusion 1.5 like a Photoshop filter. This is another follow-on from my two previous tutorials for Poser with SD.

The final result

To obtain this final result, for the Img2Img source I started again with exactly the same Poser scene, camera and light as before, but this time I dropped AS’s Hanyma Platform for Poser (now no longer sold) to the scene as a background prop. Poser’s Comic Book inking applied to both.

Raw scene in the Poser UI.

Then I made a quick Preview render of this scene at 768 (remembering to boost the texture sizes from 512), and then in Photoshop used the Glitterato plugin to add a quick starry sky.

In InvokeAI this replaced the previous figure-only Img2Img image, but the Firefly lineart Controlnet image stayed the same, thus giving a fixed figure outline that matches that of the Poser render — possibly important for later consistent colouring.

In this experiment I also added a Moebius LORA, which knows what our hero Lovecraft looks like. No additional prompting is needed to account for the backdrop, since the CFG is so low.

The final result at 1024px

It’s all going a bit ‘black on black’ (arrgh!) for this quick demo, but a non-background SD generation of the figure alone can then be used in Photoshop to mask (Crtl + click on layer, invert) and then fade or lighten the background a touch (as I believe Brian Haberlin does) so as to make the character stand out a little more. And ideally your comic script would try to avoid ‘black cat in a coal cellar’ settings, for this reason.

Simply using the new Img2Img as the Controlnet, replacing the Firefly outlines render? Nope, that doesn’t keep the detailing on the character or keep him consistent across a quad of generations. The Canny Controlnet needs to be focused just on the character, in the same way the comic reader’s eye is. Going from 768px to 1024x in the Img2Img seems to give SD some creative wiggle-room, despite the low CFG. Since it’s a low CFG for the Img2Img, there’s not much shifting in the details of the background. This seems to be the sweet spot: good model, Img2Img with a low CFG but slight upscale, and use the Controlnet with pure Poser Firefly lineart to keep the figure stable and in lockstep with your Poser renders. Presumably all this would also work for two characters interacting.

By having both character and backdrop generated by SD, there’s some SD gloop and later a loss of flexibility when putting the frame together in Photoshop. But one also avoids the need to mask, extract, defringe, colour-balance etc. It may be possible to prompt for lighting and get it, but I haven’t tried that yet.

Obviously for a comic you’d also start breaking free of the stock camera lens and use foreshortening etc for a more dynamic look. Poser has a special camera dial that makes that very easy.

Here’s a follow on from last night’s Successful test – Poser to Stable Diffusion enhancement, a proof-of-concept (or perhaps more accurately proof-of-workflow for the compositing. It’s crude, and he’s meant to be on an airship which isn’t shown, but it shows it can all work.

1. Rerender the Poser Firefly line-art at 4 x the original 768px (i.e. 3027px square) as a .PNG file.

2. In Photoshop I then run this render through GMIC and a custom filter for line-art (very similar to Dynamic AutoPainter’s Comic filter, but free). This turns the thin lines into chunkier ‘inked’ lines. Takes about 40 seconds, but does the job.

3. Size the GMIC result back to 1024px and place it over our final 1024px outcome from the successful test. Set the new layer to Multiply blending mode, and adjust opacity to suit. Flatten, and then select and cut out the figure from the white background. Defringe. Then have Photoshop add a thick ‘holding-line’ around the figure’s outline (Stroke 5px, black). This latter item helps subtly isolate the figure from the background it’s going to be pasted onto.

4. Reselect the resulting cutout figure and drop over a suitable backdrop. Here for speed I’ve merely selected a SD landscape experiment and given it a very crude tooning effect via a Photoshop filter — a final frame would have a much better lineart background. But it serves for now. Make a white layer behind, and fade the background a little by simply opacity-blending it with the white. This is why, ideally, you want figures and backgrounds as separate elements.

5. Colour balance the figure with the background. Flatten layers, tweak the Curves in Photoshop to add contrast and ‘pop’ (without things getting all ‘black on black’). Slot into the page-layout’s frame and add a text box. You’re done.

The demo is very basic and crude, especially the pose and background. But it proves that one can go from a Poser render to a finished comics panel and have it look somewhat acceptable for storytelling purposes. Especially considering the start point was this…

The workflow may sound fiddly, but once nailed down a Poser Python script could handle the renders in one click. Then a set of custom Photoshop Actions would handle much of the rest. Regrettably Stable Diffusion software, despite being built on Python, omitted to add Python scripting for UI and rendering automation.

Another experiment in using Stable Diffusion on a Poser render, as if it were a Photoshop filter.

1. Load and pose the figure and any figure-props in Poser. Here the standard Meshbox H.P. Lovecraft figure is being used, with a brass telescope from Steampunk Balloon. The M4 pose applied is meant to be gripping some rigging rope (not visible here) on a steampunk airship.

2. In Poser, use the Materials python script that ships with every copy of Poser, to lift the scene’s Gamma to 1. For comics you might also apply, as I did here, a good bright even light that tends to flatten things out (a ‘flat light’ as I call them). These measures means the dark suit can now be seen properly — one of the most fatal problems in making comics from 3D is the unfathomable tendency for makers to accept lots of ‘black on black’. Add a Comic Book outline via the Comic Book Preview panel, and render in Preview. This will help the Canny edge detection later on. Output to .PNG format at 768px square.

3. In Poser’s good old Firefly renderer, use my lineart-only render preset to just get the lines. This type of render gives you all the lines, not just the ones Comic Book Preview chooses to show. Render to 768px square, .PNG file. We do lose the hair, which is just an image texture. But the next step will bring it back.

4. Combine the two .PNGs in Photoshop. Do this by dropping the Firefly lineart on top of the Gamma-lifted Preview render, set the layer to Multiply, and adjust to taste. Here I also set a white backdrop layer, since the PNGs otherwise have embedded transparency. To lighten things up a bit more, I also blended a little into the white background.

5. Now start the free InvokeAI. Import your final Step 4 .PNG and use it for both Img2Img and also in the Canny Controlnet. Use the settings seen in the screenshot-combo above, making sure to get them all. You may of course need to juggle the prompt and negative prompt, if using your own test render. The Stable Diffusion model being used is the free Another Damn Art Model (ADAM) 4.5, available at CivitAI.

That’s it. Upscale the best 1024px result 2x so you can mask, cutout and defringe cleanly in Photoshop, if planning to composite the character onto a background. The intended destination is as part of one frame in a comic-book page, thus the roughness and a few imperfections (visible when the image is scrutinised at a large size) don’t really matter. The lack of contrast and colour vibrancy is also a good thing, as it can be tweaked up later on — it would be trickier to try to subdue garish colours / high contrast.

Should also work nicely (not yet tested) if you start with a figure + lighting-matched backdrop render. But obviously having the figure and backdrop separate could make adjustments on the comics page easier in Photoshop (slightly blur or lighten the background, to make the characters stand out etc). You may also want two very different characters interacting, and thus would likely want to deal with them separately and then bring them together in Photoshop.

The Stable Diffusion result is of course not perfect, but you can pick the best from 4 or even 8 image-generations. Here he’s acquired a ring on his finger, and the jawline is too ‘1930s heroic’ and not really ‘Lovecraft deformed’ enough. But the silhouette of the figure and prop match perfectly with the Poser renders, which means you can get consistent colour throughout a comic-book page (here’s how: greyscale, get a full colour render from Poser, size it to fit and lay on top, then set Photoshop’s blending mode to Colour).

One thing I tried along the way was prompting for Cary Grant (the 1930s movie actor). It does pretty well, and SD must have been trained well on his images. Consider using an M4 with a ‘somewhat-Cary morph’ and just prompting for the old movie star, for a more or less consistent head. Or try some other big movie star of the 1920s and 30s. I think the difficulty I had with getting an exact (Lovecraft) was that the ADAM model doesn’t really ‘know’ him well. But it’s the best I’ve yet found for this sort of ‘SD as Photoshop filter’ workflow, being very strong and thus working well at low Img2Img settings.

Part two: Successful test – the ‘proof of workflow’.