The VersionCommand creates a version for all pages and passes the versionMap to the PublishCommand. The method collectFromVersionMap checks for the parameter modifiedOnly and the activiation status of the node. But all nodes have a modified state because of creating a new version in VersionCommand. The following check in the method collectFromVersionMap is always true.
See PublicationCommand

if (!isModifiedOnly() || NodeTypes.Activatable.getActivationStatus(node) != NodeTypes.Activatable.ACTIVATION_STATUS_ACTIVATED) {
    nodes.add(node);
} 

Reproduce
Use the recursive publish action from the versioned catalog in the pages-app with the option modifiedOnly.

publishRecursive:
  icon: icon-publish-incl-sub
  $type: jcrCommandAction
  command: publish
  catalog: versioned
  asynchronous: true
  params:
    modifiedOnly: true
    recursive: true
  availability:
    rules:
      notDeleted: *notDeleted
      hasSubPages:
        $type: jcrHasChildrenRule
      isPublishable: *isPublishable

Note
Maybe this kind of configuration is not possible? Versioning + modifiedOnly. In such a case maybe this becomes a docu ticket.