Steps taken:

1) installed latest code from trunk (using magnolia-module-fckeditor-3.6-20080620.153626-1.jar) with samples
2) changed the enterMode to 'br' and shiftEnterMode to 'p' on /modules/samples/dialogs/samples/samples-editor/tabContent/content
3) restarted server and browser
4) opened up sample page and created new paragraph, selected "Samples: Editor"
5) enter line of text press Enter, creates a <p> tag around text
6) enter line of text press Shift+Enter, creates a <br/> at end of text