Responding to Peter Yan and Adam Bradley:
Yes, Frye certainly did know about the Greek modes. In “Modal Harmony in Music” he writes:
In the sixteenth century much greater freedom of tonality was available. The major and minor modes were then celled Ionian and Aeolian respectively, but four others were used. Arranged in order of sharpness, they are: Lydian (F to F on white notes: present major with raised fourth); Ionian (C to C: present major); Mixolydian (G to G: present major with lowered seventh); Dorian (D to D: present natural minor with raised sixth); Aeolian (A to A: present minor); Phrygian (E to E: present natural minor with lowered seventh). A seventh mode, the Locrian, B to B or Phrygian with lowered fifth, had probably only a theoretical existence. These four additional modes, like the two we now have, ended on the tonic chord. Thus, if all modes were impartially used today, a piece ending on G would have a key signature of two sharps in the Lydian modes, one in the major, none in Mixolydian, one flat in Dorian, two in minor, three in Phrygian. Or a piece with a key signature of one sharp could be C Lydian, G major, D Mixolydian, A Dorian, E minor, or B Phrygian. (Northrop Frye’s Fiction and Miscellaneous Writings, 185)
And in “Baroque and Classical Composers” Frye writes:
When rhythm changes from 4/4 to 3/2 the minim of the latter = crochet of former. Key signatures only either none or one flat, & occasionally two flats: no sharps. Fellowes finally, bless his heart, coughs up some dope on the modes. If the piece has no flat in the signature, look at the last bass note and that will give you the mode. A = Aeolian (minor scale), B = Locrian (theoretically: it’s never used), C = Ionian (major scale), D = Dorian, E = Phrygian, F = Lydian, G = Mixolydian. That’s if the melody is authentic: if it’s plagal then prefix hypo to the mode. If there is a flat, transpose a fourth down or fifth up (G with a flat = D without one); if two, tr. [transpose] a tone up. Hence many key signatures until the 18th c. were a flat or a sharp short. Modulation & equal temperament go together. (ibid., 175)
As for the circle of fifths, sometime in the late 1950s or early 1960s Frye provided a schematic for the circle as a way of outlining the twenty‑four parts in the first three units of his ogdoad: Liberal, Tragicomedy, and Anticlimax. The twenty‑four letters of the Greek alphabet provided Frye a convenient name for each of the twelve major and the twelve minor keys. C = alpha, A = beta, G = gamma, etc. Frye didn’t actually draw a diagram, but in his Notebooks for “Anatomy of Criticism” (paragraphs 57, 58, 63, and 73 of Notebook 18), he set down the constituents of a diagram and gave a brief description of the thematic contents of each of the twenty‑four parts, illustrating what he means by saying that the circle of fifths provides a “symmetrical grammar” (Spiritus Mundi, 118).