As the ToolWindow is created dynamically you cannot set the style for it declaratively. The easiest way to do this is to create a theme for your RadDocking control, add the created style for the ToolWindow in it and apply the theme to the control. To learn how to do it read this topic.
To create the needed style for the Theme you should use a dummy control. This will allow you to copy the default style for the ToolWindow and modify it to your liking. To do so open the UserControl that hosts your RadDocking in Expression Blend.
From the Assets tab select Controls -> All -> ToolWindow.
Draw one dummy control of this type somewhere on the scene.
Select it and from the menu choose *Object -> Edit Style -> Edit a Copy. *You will be prompted for the name of the style and where to be placed within your application. After clicking OK, a style for your ToolWindow control will be created.
To change the visual appearance of the ToolWindow control you have to modify its template. To do that click on the palette-like breadcrumb icon and select Edit Template -> Edit Current.
From the 'Objects and Timeline' pane select the Border control responsible for the border of the ToolWindow.
Modify the value of its BorderBrush property.
To change the Background of the ToolWindow select the Border control responsible for that.
Modify its Background property.
After finishing with the modifications you have to add the created style to the theme for your RadDocking. To learn how take a look at the Theming the ToolWindow topic.
After moving the style to the theme you can remove it from the resources of your UserControl. Also don't forget to delete the dummy control that you have created.
Here is a snapshot of the final result.