Throughout the course of this two part rendering tutorial you will learn how to use the powerful render layers inside Maya 2010, and also how to output them and prepare them for compositing. In the second part, you will see how you can use After Effects to composite all of those rendered layers to achieve more realistic look with custom depth of field.
Step 1
Open your scene file, or use the one supplied with the project files (found at the top of the post). Be sure that you have Mental Ray turned on in Maya (Windows -> Settings/Preferences -> Plug-in Manager’)
Step 2
In the Plug-in Manager window, scroll and find ‘Mayatomr.mll’ and tick on the ‘Loaded’ option.
Step 3
Then click on the ‘Channel Box / Layer Editor’ (in the top right corner) to start adding the render passes.
Step 4
At the bottom part of the panel you’ll see the ‘Layer Editor’.
Step 5
Click on the ‘Render’ tab, next to the ‘Display’ tab.
Step 6
You’ll see one pre-made layer. Now, the first layer that you’re going to make is called the Ambient Occlusion layer. To make it, first select all of your objects and cameras (except the lights). If you’re using the Project Files, open the ‘Outliner’ and select the objects that are in the ‘objects_group’ and also the ‘Camera’.
Step 7
In the ‘Layer Editor’ click on the button show in the image below. This will create a new render layer and assign the select objects to it.
Step 8
Double click on the new layer and rename it ‘Ambient_Occlusion’.
Step 9
Before you continue you need to set up the basic render settings for the scene, so click on the ‘Render Settings’ button for the ‘masterLayer’.
Step 10
First, make sure that your 'Default Renderer' is set to 'Mental Ray'.
Step 11
In the ‘Common’ tab, set the ‘Image format’ to be ‘TIFF’, the ‘Frame/Animation ext’ to ‘name.#.ext’, the ‘Frame padding’ to 3, the ‘Start frame’ to '1', and the ‘End frame’ to '150'.
Step 12
Scroll down further and set the ‘Renderable Camera’ to ‘Camera’ and the ‘Width’ and ‘Height’ to '700' by '405'.
Step 13
Click on the ‘Quality’ tab, and turn the ‘Max Sample Level’ to '2'. This will smooth out the result of the final render.
Step 14
Scroll some more and turn the ‘Filter’ to ‘Gauss’ then expand the ‘Raytracing’ sub menu.
Step 15
Then under ‘Raytracing’ set the ‘Reflections’ and ‘Refractions’ to '4', and the ‘Max Trace Depth’ to '8', to allow the light to fully pass through any transparent objects.
Step 16
Under ‘Indirect Lighting’, turn on ‘Global Illumination’ and set the ‘Accuracy’ to '1000' (to prevent any artifacts in the render). Also set the ‘Radius’ to '5' and the ‘Merge Distance’ to '0.2' (to smooth it out further). The ‘Radius’ and the ‘Merge Distance’ will depend on the scale of your scene in Maya units, so if your scene is a different size, then change the properties accordingly.
Step 17
If you’re using a custom scene for the tutorial, you can scale your scene so it will match the grid size from the scene in the project files, and then you can use the same numbers for the properties as I have used in the tutorial.
Step 18
Go back to the ‘Layer Editor’, and select the ‘Ambient_Occlusion’ layer.
Step 19
Select all of the objects.
Step 20
Alternately, you can right click on the ‘Ambient_Occlusion’ layer, and from the menu select ‘Select Objects in Layer’.
Step 21
Then right click on the ‘Ambient_Occlusion’ layer again, and go to ‘Overrides -> Create New Materials Override -> Surface Shader’ (see next step for a picture).
Step 22
...which will create an override material for the objects on the current render layer.
Step 23
All of the objects assigned to this layer should not be visible now because of the surface shader. If you press the wireframe button, however, you will be able to see them again in wireframe mode.
Step 24
Go to ‘Windows -> Rendering Editors -> Hypershade’.
Step 25
The ‘Hypershade’ window will open with all the available materials.
Step 26
Scroll down and you should be able to see the surface shader you just created.
Step 27
Right click on the surface shader, and select its name in the menu that shows up. Alternately, you can double click it for the same result.
Step 28
The attributes of the surface shader should now appear. Click on the name of the shader and rename it to ‘AmbientOcclusion_shader’.
Step 29
Click on the checkered square next to the ‘Out Color’.
Step 30
In the ‘Create Render Node’ window that appears, click on the ‘mental ray’ tab.
Step 31
Scroll to the ‘Texture’ sub menu, and select the ‘Mib_amb_occlusion’ node.
Step 32
In the attributes window (for the ambient occlusion node), set the ‘Samples’ to '64' and the ‘Max Distance’ to '2.5' to smooth the result. The ‘Max Distance’ is also relative to the scale of your scene.
Step 33
If you’re not using the scene from the Project Files you will need to tweak this attribute to match your scene size.
Step 34
To make sure that your viewport is using the 'renderable camera' from the ‘Render Settings’, go to ‘Panels -> Perspective ->Camera’ or your main 'renderable camera' if you’re using a custom scene.
Step 35
Select ‘Camera attributes’ in the viewport, or click the camera in the ‘Outliner’.
Step 36
In the camera attributes, scroll to the ‘Environment’ sub menu, expand it, and right click on the ‘Background Color’. Then choose ‘Create Layer Override’, so when you change the attribute they will only change for the current render layer.
Step 37
Double click the color picker, choose a pure white color, and then click ‘Accept’. This is needed in order to see the result properly. Otherwise you would get a completely black image.
Step 38
Now go to the ‘Layer Editor’ again, and click on the ‘Render Settings’ button for the ‘Ambient_Occlusion’ layer.
Step 39
Under the ‘Indirect Lighting’ tab, right click on ‘Global Illumination’ and choose ‘Create Layer Override’.
Step 40
Turn ‘Global Illumination’ off, because ambient occlusion doesn’t use the lights from the scene.
Step 41
To do a test render. Click on the ‘Open Render View’ button.
Step 42
Now in the ‘Render View’ window, select ‘Render -> Render -> Camera’, or your main renderable camera if you’re using a custom scene.
Step 43
Your ambient occlusion layer is now finished. You should have something like this:
Step 44
Next up is the Zdepth layer. Again, select only the 'renderable camera' and all of the objects.
Step 45
Click the button in the ‘Layer Editor’ to assign the selection to a new render layer.
Step 46
Double click on the new layer and rename it to ‘zdepth’.
Step 47
Go to the the command line (bottom left) with the 'zdepth' layer still selected in the ‘Layer Editor’.
Step 48
Now paste this code into the command line: "renderLayerBuiltinPreset linearDepth zdepth". This will attach nodes from the zdepth preset to your layer. Note: The word 'zdepth' from the code refers to the 'zdepth layer'. If your layer is called something else, then you will need to change the code accordingly.
Step 49
If everything is right, when you input the command there should be no warnings in the command response field.
Step 50
The ‘zdepth’ layer should be still selected.
Step 51
Click on the ‘Render Settings’ button for the 'zdepth layer'.
Step 52
Select ‘mental ray’ as the 'default renderer'
Step 53
In the ‘Render Settings’, under the ‘Common’ tab, right click on the ‘Image format’ and then select ‘Create Layer Override’.
Step 54
Set the ‘Image format’ to be ‘TIFF uncompressed’, because you need the maximum image information from the render
Step 55
Scroll down and right click on the ‘Depth channel’, then select ‘Create Layer Override’.
Step 56
Enable the ‘Depth channel’ so it can output the zdepth information.
Step 57
Click on the ‘Indirect Lighting’ tab, and then right click on ‘Global Illumination’ and choose ‘Create Layer Override’.
Step 58
Un-check ‘Global Illumination’.
Step 59
Select all of the objects, lights, and the 'renderable camera' in the ‘Outliner’.
Step 60
Press the button to create a new layer with the selected objects.
Step 61
Double click the new layer and rename it 'Beauty_pass'. This will be the base for your other passes that will be composited together later.
Step 62
Open the ‘Render Settings’ for the new layer.
Step 63
In the ‘File name prefix’ field write ‘scene’, and set the ‘Frame padding’ to '3'.
Step 64
Click the ‘Smooth shade all’ button, so all of your objects are visible again.
Step 65
In the ‘Layer Editor’, make sure that for all of the layers that you have created, the enable button is not crossed, except for the one for the ‘masterLayer’.
Step 66
Choose the ‘Rendering’ option from the shelf selection drop-down menu.
Step 67
Go to ‘File -> Project -> Edit Current…’.
Step 68
In the window that pops up, the ‘Location’ is where your project is situated, and the ‘Images’ folder is where your render passes will be rendered into separate folders. So in this example the scene is going to be rendered at: ‘C\Users\User\My Projects\images’.
Step 69
To render the layers, go to ‘Render -> Batch Render’. It will depend on the power of your computer as to how long it will take to render the layers. The 'ambient occlusion' and the 'zdepth' layer will render much faster than the 'beauty' layer because they don’t use 'global illumination'.
Step 70
When the renders are complete, your render folder should look something like this. If you followed the tutorial exactly, you should have 150 frames in both the 'ambient occlusion' and 'beauty layer' folders, and 300 frames in the 'zdepth' folder
