As the ToolWindow is created dynamically you cannot set the style for it declaratively. The easiest way to do this is to create a theme for your RadDocking control, add the created style for the ToolWindow in it and apply the theme to the control. To learn how to do it read this topic.
To create the needed style for the Theme you should use a dummy control. This will allow you to copy the default style for the ToolWindow and modify it to your liking. To do so open the UserControl that hosts your RadDocking in Expression Blend.
From the Assets tab select Controls -> All -> ToolWindow.
Draw one dummy control of this type somewhere on the scene.
Select it and from the menu choose *Object -> Edit Style -> Edit a Copy. *You will be prompted for the name of the style and where to be placed within your application. After clicking OK, a style for your ToolWindow control will be created.
Go to the 'Resources' pane and see all of the resources generated for your style.
ToolWindow_Background - is the background brush used for the tool window control.
ToolWindow_OuterBorderBrush - is the outer border brush used for the tool window control.
ToolWindow_InnerBorderBrush - is the inner border brush used for the tool window control.
ToolWindow_CloseButtonStyle - defines the style for the window's close button.
Here is an example of the above resources modified:
After finishing with the modifications you have to add the created style to the theme for your RadDocking. To learn how take a look at the Theming the ToolWindow topic.
After moving the style to the theme you can remove it from the resources of your UserControl. Also don't forget to delete the dummy control that you have created.
Here is a snapshot of the final result.