There is no precise way to control the auto-generated edges in Blender, thus reducing the edge generation feature's efficiency. You may, however, create and control your own edges, using multiple materials, and advanced UV mapping techniques. In this tutorial we will thoroughly explore this technique.
Final Effect Preview
Step 1
First, you need to create a set of shadeless materials, each with different color, making sure you have enough materials to cover the model, without same colors overlapping each other. Then, switch to the “Shading” (F5) panel, select “Material buttons”, and click the “Add New” button to add a new material.
Step 2
Select the “Material” subpanel. Click the “Shadeless” button. Use the “Col” button, to change the color from the default grey, or pick a color from the color chooser. Your whole model will be re-painted in this color. Note: On the “Links and Pipeline” sub-panel, you may use the small button with a car depicted on it to auto-generate a name for this color. It is quite a handy tool in this case, as you will be generating a number of different colors, and trying to remember the indexes is probably not the best idea.
Step 3
Now add another color to your collection. Click the “<>” button, below the “Link to Object” text, on the “Links and Pipeline” sub-panel, and select the “ADD NEW” option. A new color will be created with the attributes copied from the previous one. For this reason, you do not have to click “Shadeless” all the time.
Step 4
Do the same as with the previous color. Use the “Col” button to change the color from the default grey, or pick a color from the color chooser.Note: Regardless of the fact that the new color completely replaces the previous one, all the previous colors still reside in the memory. You can view them by clicking the same “<>” button. Just remember not to save/ open the model untill all the created colors are assigned to the model parts, as Blender will erase all unassigned colors (they will have '0' prefixing their names). If you absolutely have to stop midway, with some of the colors not yet assigned, just press the “F” button next to the material name, after you are done with that material. This will tell Blender to save the color, even though it has no users.
Execute steps 2 - 4 until you have enough colors for your model. For a medium complexity model, 10 should be enough. As the last step add one generic, grey material, we will need it later on.
Step 5
Now you need to divide your model into areas using the materials. You will use the color change to mark the edge you want to control. Switch to the edit mode editing panel (F9).
Step 6
In the 3D View window, select the area that you want to be in one color, effectively, and without any edges.
Step 7
Move to the “Link and Materials” sub-panel, select a color using the material selector, and click the “Assign” button.
Note: Use the color picker, near the material index selector, to adjust the color if necessary.
Step 8
Select another area that you want to color, and repeat steps 6 and 7. Note: For each iteration of this step, be sure to have a different color selected, other than the grey color.
When you are finished, you should have a model completely painted in different colors, like the one shown here.
Step 9
Now you will use 'Composite Nodes' to filter out all of the information, except for the edges. First, tell the rendering engine to use the composition nodes. Select the “Scene” panel (F10), then “Render buttons”, and click the “Do Composite” button.
Step 10
In the main window, switch views to the “Node Editor”.
Step 11
Click the “Composite Nodes” and “Use Nodes” buttons at the bottom of the “Node Editor” window. Blender will create two initial nodes for you: the input node “Render Layer”, and the output node “Composite”.
Step 12
Erase the node links by pressing the left mouse button, and dragging the outline over the links.
Step 13
To filter out the edge, use a node called “Filter” (who knew?). Select the “Add-> Filter-> Filter” menu item, and make sure its “Image” input is linked to the “Image” output of the “Render Layer” node.
Step 14
Change the filter type from the default “Soften” to “Sobel”. Now, if you link the filter node output with the “Composite” node input, “Image” to “Image”, and run the render, you will see your model in black, with the edges outlined in different colors.
Step 15
You do, however, need the edges to be black. Do do this you must invert the render, by selecting the “Add-> Color-> Invert” menu item, and linking the added node to the “Image” output of the filter node. Then convert the render from 'RGB' to 'black and white' by selecting the “Add-> Convertor-> RGB to BW” menu item. Do not forget to link it to the output of the previous node. The “Val” output of the converter needs to be linked to the “Image” input of the “Composite”. Note: You may use the “Backdrop” option with the “Viewer” output node to display the intermediate results behind the nodes schema.
Step 16
Run the render to check the results. Note:If you do not plan to change to camera placement and model while building the test flow, you may save the output of this step in a file, and use it in further steps. This can greatly improve the rendering time. You can do this by using the “Add-> Output-> File Output” menu option, specifying the file name, and for file type, use one that will provide a lossless compression, or no compression at all.
Step 17
Now you need to unwrap the model, and create a mask texture to cover your edges. Put aside the edges you've created, and return back to your model, and the grey material you left behind. This part can be tricky. Start with the script generated UV map, and adjust it where necessary. Move back to the “3D View” display, and switch to the edit mode (“Tab” key). Split the view, and in another window, select the “UV/Image Editor” option as the window type.
Step 18
Back in the “3D View”, select your whole model, press the “U” key to display the “UV Calculation” menu, and select the “Unwrap (smart projections)” menu item.
Step 19
Click the“OK” button to unwrap the model, and check the result in the UV\Image Editor window. If you are lucky, or the model is quite simple, you will have a continuous UV map from the first attempt. If you like the rest of humanity, you are just going to have to do the unwrapping manually, step by step.
Step 20
Select the portion of your model you will concentrate on, and hide the rest by pressing “Ctrl+I” to invert the selection, and then “H” to hide the selection.
Step 21
Move to the Edge Select Mode, by pressing “Ctrl+Tab” and then “2”, and select the edges (seams) along which you would like the model be unwrapped.
Step 22
Press “Ctrl+E” to display the 'Edge Specials' menu, with the list of edge commands, and select “Mark Seam”. The selected edges will be outlined in orange, indicating that they are marked as seams.
Step 23
Now try the unwrapping process once again. Select the entire portion of the model you were working with, press the “U” key to display the “UV Calculation” menu, and then select the “Unwrap” menu item. Now the UV map should be much more logical. If not, you need to choose better seams, and then repeat the unwrapping process.
Step 24
Unhide the rest of the model, and repeat steps 20 - 23 until the entire model is unwrapped in an easy to paint map.
Note: Feel free to use the scaling and rotation (“S” and “R”) keys in the UV editor, to organize your uv shells in a compact and efficient way.
Step 25
Now you need to create a picture for the UV map you created. In the "UV\Image Editor" window select “Image-> New”.
Step 26
Specify an image name instead of 'untitled', change the image default color to white, and click “OK” to create the image. To save the image, click “Alt+S” or select the “Image-> Save” menu item, and save the image. Note: You still need to specify that the image should be used as a texture, assign this texture to a material, and ensure that this texture affects the color of the rendered model.
Step 27
Add this texture to the grey material you left behind. To do this, move to the “Shading” panel (F5), select “Material buttons”, and select the grey material, using the material selector on the “Links and Pipeline” tab. On the “Texture” sub-panel, click the “Add New” button, to add a texture draft. As a result, the “Map Input” and “Map To” tabs should appear.
Step 28
Now, in order to tell texture to use the UV coordinates, select the “Map Input” tab, click the “UV” button and specify the UV texture (not image) name in the “UV:” field. Note: To find out what is the UV texture name, switch to the “Editing” (F9) panel, and “Mech” sub-panel. The name is specified under the “UV Texture” field.
Step 29
After that, you need to tell the texture what image to use. Go to the “Shading” panel (F5), and then “Texture buttons” (F6). In the “Texture Type” field you should see “none”. Change it to “Image”, and on the “Image” sub-panel, click the “Load” button to load the image you created before.
Step 30
Now, you need to use Composition Nodes to mix the edges generated with the controlling mask. For this you have two choices on how to proceed: 1) With the edges rendering you created initially, or 2) with the map you have now. The problem that arises is that the initial set of materials overrides the texture mapping you just created. So you need to split the rendering into two portions: either in time (using the saved image), or using render layers.
For initial modeling and tuning, the first way is preferable, as you can see the rendering results almost immediately after pressing the render button (made possible by skipping the edge generation phase). For animation or real production, the latter method is preferable, as it will react to the model outline and position changes.
For the purposes of this tutorial, you will be using the first approach (the second will be outlined at the end of the tutorial).
Tell the rendering engine to use the textured material for the model, instead of your colored set. Select the “Scene” panel (F10), then “Render buttons”, and then the “Render Layers” tab. In the “Mat:” field, enter the name of the grey material you textured previously.
Step 31
Now modify the composite nodes to use the saved image. In the main window switch to the “Node Editor”, click the “Composite Nodes” button, and erase all nodes except for the “Render Layer” and “Composite” nodes. Select the “Add-> Input-> Image” menu option to add the "image input" node to our schema, and load the image into it. Now, the only thing left is to mix the mask and the generated edges. To do this, use the “Mix” node, by using the “Add-> Color-> Mix” menu item. Use the mask as a factor for the “Mix” node’s white input, to specify where to paint over the edge render. The result should be something like this. Note: if you run the render, nothing will change when compared with the initial edge selection schema. The default image is white, thus all edges are displayed.
Step 32
Next, you need to paint the areas you do not need on the mask black. To do this, you need to preserve the two window view you used before (the 3D view, and the “UV\Image Editor”). In the “3D View“ window, select the “Texture Paint“ mode, and set “Viewport Shading” to “Textured”. Then, to see the painting tools that are available, click “Editing" (F9), and check the “Paint” sub-panel.
Step 33
To be able to paint in the "UV\Image Editor", click the pencil button to bring you into paint mode, and then press the “C” key to display the tools that are available.
Step 34
In the “Editing-> Paint” panel, change the color to black, set opacity to 1, and size to 25, then in the “3D View”, paint out the edges that you want to hide.
To limit the painting to certain faces only, switch back to the edit mode, select those faces, move back to the texture mode, and press the “F” button. Now when you paint, only the faces that were selected will be affected.
Note: You can also use the "UV\Image Editor" window for minor corrections, while checking the UV texture outlines. The process is exactly the same as with the “3D View”, only the tools selection is available upon pressing the “C” key. Also, be sure that the “Enable Image Painting” button (pencil) is turned on in the UV window.
Step 35
After you are done, save the UV texture by pressing "Alt + S", in the "UV\Image Editor" window, and reload it into the material texture by pressing “F6” to move to “Texture Buttons”, and clicking “Reload” on the “Image” panel.
Step 36
Render and repeat the two previous steps to correct the mapping. Continue repeating until you are happy with the results.
Step 37
Now we can discuss the second method I mentioned earlier. For this you will use the saved edges image used in the composite nodes. After you’ve tuned the flow, you may want to combine the two processes to eliminate the need for the saved image. To do so, add a copy of the render layer without the materials override, and combine the composite maps in the way depicted below. Now, all that's left to do is hit render, and enjoy the results.
