From FCKeditor Docs

SmileyWindowHeight

Sets the height of the "Insert a Smiley" window. The default value is 240 (in pixels). Code:

FCKConfig.SmileyWindowHeight = 240 ;