Optimal and simultaneous designs of Hermitian transforms and masks for reducing intraclass separations of feature vectors for anomaly detection of diabetic retinopathy images.