John Bateman On The Need For Inter-Rater Reliability In SFL Research

John Bateman wrote to Sysfling 21 July 2016 at 19:46:

Low interrater reliability scores can be indicative of several things: most of which relate to problems that should be considered. One cause of low reliability would be that the coders don't fully understand, or agree on, the way categories should be applied; another cause is that the categories are intrinsically poorly defined, and so agreement is unlikely.

Thus, when categories are meant to be applicable according to some theory (e.g., SFL), checking whether coders can actually apply the categories is not a bad step. The paper:

O'Donnell, M.; Zappavigna, M. & Whitelaw, C. (2008) 'A survey of process type classification over difficult cases'

In:

Jones, C. & Ventola, E. (Eds.) From Language to Multimodality: new developments in the study of ideational meaning, Equinox Publishing Ltd., 47-64.

provides some sobering data on how reliably some categories in transitivity are being applied.

and Margaret Berry added on 21 July 2016 at 20:52:

I too would recommend the O'Donnell et al paper. And another paper that is relevant is

Laura Gwilliams and Lise Fontaine. 2015. Indeterminacy in process type classification. Functional Linguistics, 2:8, pages 1-19.

Blogger Comments:

[1] Lack of theoretical understanding is clearly the major reason why Systemic Functional linguists disagree on analyses, as demonstrated by any public discussion — e.g. Sysfling, Sysfunc, Systemic Functional Linguistics Interest Group — in which instances are analysed.

[2] To be clear, it is not so much that "categories are intrinsically poorly defined" but that, on SFL model, language itself is said to be an indeterminate system.  For the types of indeterminacy, and the reasons for it, see the views of Halliday & Matthiessen here.

[3] Neither of these papers includes, in its experimental design, the most fundamental principle of grammatical analysis: taking a trinocular perspective.  Halliday & Matthiessen (1999: 504):

A stratified semiotic defines three perspectives, which (following the most familiar metaphor) we refer to as ‘from above’, ‘from roundabout’, and ‘from below’: looking at a given stratum from above means treating it as the expression of some content, looking at it from below means treating it as the content of some expression, while looking at it from roundabout means treating it in the context of (i.e. in relation to other features of) its own stratum.

Halliday & Matthiessen (2004: 31):

We cannot expect to understand the grammar just by looking at it from its own level; we also look into it ‘from above’ and ‘from below’, taking a trinocular perspective. But since the view from these different angles is often conflicting, the description will inevitably be a form of compromise.

In contrast, O'Donnell, in O'Donnell et al. (2008: 63), demonstrates no knowledge of this fundamental principle, framing the problem in terms of a lack of explicit coding criteria:

Both our analysis of individual clauses (Section 3) and of the grouping of coders (Section 4) show that the divide between using conceptual vs. syntactic criteria is widespread throughout the community as a whole, and each individual chooses which path they follow. This is, we believe, the result of the lack of explicit coding criteria in general, and argue that what the community needs is explicitly stated sets of criteria for coding practices, and perhaps distinct criteria descriptions for particular applications.

Gwilliams & Fontaine (2015: 17) are similarly oblivious, recommending a more "delicate" two level grammatical analysis — one semantic, one syntactic:

Although the motivation for a single-level analysis of experiential meaning is desirable, it does not appear that a one-dimensional classification is always sufficient to account for both syntactic and semantic realisation. If a representative analysis is to be maintained within the SFL framework, it appears that a more delicate analysis of the experiential meta-function is required, in order to provide the individual with all the relevant tools to conduct a fully representative analysis. Specifically the option to annotate syntactic and semantic interpretations separately would alleviate problems associated with the lack of correspondence between these levels.

Lack of theoretical understanding is the pervasive problem in the SFL community, whether it be in analysing language, or in analysing analyses of language — or, indeed, in workbooks designed to teach the theory (evidence here and here).

Inter-rater reliability is merely a statistical measure of the degree of agreement among raters.  In a community where lack of theoretical understanding is demonstrably widespread, at all levels, agreement is not a measure of theoretical competence.

The distinction between interpersonal agreement and experiential consistency is comically enshrined in this drawing by B Kliban:

David Rose Misrepresenting Michæl Halliday

David Rose wrote to Sysfling on 3 July 2016 at 19:59:

Discourse patterns as low-lying fruit is good metaphor (if we invert stratal model). Grammar is further out of reach, demanding more semiotic labour to learn and apply to text analysis. Halliday’s view…

…lexical meaning is located much nearer the surface of language… Grammar is much more hidden from view (Halliday 2008:75). 

…the patterns we treat as grammatical are those which are buried much deeper below the level of people's consciousness… Lexical patterns are nearer the surface of consciousness (Halliday 2003:120).

Blogger Comment:

Here Rose misleads by strategically misrepresenting the view of Halliday.  The Halliday quotes do not refer to the "lexical" relations of Martin's discourse semantics.  For Halliday, the lexical vs grammatical distinction is one of delicacy, not stratification.

David Rose Analysing A Text "We Don't Understand"

Responding to a post by Jim Donohue, David Rose wrote to sysfling on 2 July 2016 at 09:47:

Fantastic effort to combine grammar and discourse, to analyse genre and field. Critical focus is on lexical, conjunction and reference items, for which answer 1 is great text to analyse. You could go a little further and simplify. Here with lexical items in yellow and conjunction, reference, appraisal items in blue.

S1. A system call is a request made by a process to the operating system in order to perform tasks only the operating system can complete. 

Retrieving a file from a disk, or retrieving/displaying inputs/outputs, are examples of this

nuclear relations, activity sequence and conjunction - what’s going on
line 1 defines system call as type of request from process to operating system
line 2 explains what operating system does then (in order to, only)
line 3 exemplifies types of tasks requested by system call (examples of this)

(nuclear relations are lexical relations within clauses, configured by grammar)

taxonomic relations - why it makes sense
line 1 system call is a type of call, call and request are synonyms, process is technical part of computing field,
lines 1-2 operating and perform tasks are synonymous, operating system is repeated, complete is part of task
lines 2-3 retrieving/displaying are types of task, file/disk/inputs/outputs are parts of operating system

Blogger Comments:

[1] The "fantastic effort" that Rose applauds misconstrues these two clauses of a student's exam answer as attributive rather than identifying.  See clause analysis here.

As the clause analysis shows — but which Rose's discourse semantic analysis doesn't — in the first clause, the student decodes a system call and, in the second clause, the student then encodes examples of this.

[2] To be clear, in Martin's discourse semantics:

• 'nuclear relations' are expansion relations — mostly misapplied — between experiential elements of clause and group structure;
• 'activity sequence' is misconstrued as an aspect of field, not discourse semantics;
• 'conjunction' is the logical system of discourse semantics — which confuses structural (logical) deployments of expansion with cohesive (textual) deployments;
• 'what's going on' could be either field ('activity sequence') or genre ('social process').

[3] This is only part of the definition, as the grammatical analysis makes clear.  Rose's analysis makes no use of any of the discourse semantic resources: nuclear relations, activity sequence or conjunction.

[4] This part of the definition construes purpose, not time.  Here Rose has applied the discourse semantic resource of conjunction to an embedded clause complex, but misidentified the expansion relation.

[5] As the clause analysis shows, this clause encodes examples of a system call.  In the absence of a clause analysis, Rose misconstrues the meaning realised by the clause, as evinced by his "tasks requested by a system call"; cf. 'a system call is a request made by a process…'.  Again, Rose's analysis makes no use of any of the discourse semantic resources: nuclear relations, activity sequence or conjunction.

[6] Martin's nuclear relations (1992: 309-21) are not lexical relations, but (misapplied) expansion relations between elements of the experiential function structure of clauses (Process, Medium, Range, circumstance), nominal groups (Thing, Classifier, Epithet, Qualifier) and verbal groups (Event, Particle, Quality).  Rose has used none of these relations in his analysis.

[7] As demonstrated here, Martin's experiential discourse semantic system, termed ideation, is a confusion of lexical cohesion (textual metafunction), lexis as most delicate grammar and expansion relations between elements of function structures (logical metafuction).  The taxonomic relations Rose refers to here is the 'lexical cohesion' component.  That is, these relations help to make the text cohesive.

[8] This is not a taxonomic relation in the text — despite the use of 'part' to suggest meronymy.

[9] The suggestion here is that there is a meronymic relation between 'complete' (part) and 'task' (whole).  This can be examined through the lens of expansion relations.  If 'complete' is regarded as a phase of a process, then the type of expansion is elaboration (Halliday & Matthiessen 2014: 569ff), whereas a meronymic relation is one of extension (Halliday & Matthiessen 1999: 89).  Halliday & Matthiessen (1999: 91) provide a different argument:

We can certainly recognise that processes have phases — 'begin to do, keep doing, stop doing'; but it is not immediately clear that these form a process meronymy analogous to the parts of a participant. Although we might reconstrue he began to dance metaphorically as the beginning of his dance on the model of 'the beginning of the book', this is a metaphorical reification of the process 'dance' and we have to be cautious in interpreting the implications for the congruent process 'dance'. If we probe a little further, we can see that process phase is concerned with the occurrence of a process in time — its temporal unfolding: 'begin to do' means 'begin to be actualised (to occur) as doing in time'. In contrast, participant meronymy is not tied to the existence of a participant in referential space.

Conclusion:

Despite claiming to provide a discourse semantic analysis of these clauses, Rose makes no use of the system of nuclear relations (discourse semantic ideation) or activity sequences (Martin's register: field), and the one use he makes of discourse semantic conjunction, he gets wrong.  Further, in his use of Martin's lexical relations (discourse semantic ideation) — a relabelling of Halliday's lexical cohesion — he misrepresents meronymic relations.  Moreover, there was no analysis of appraisal, genre or field.

Most importantly, the fact that the student provided two identifying clauses, with the first decoding a technical term, and the second encoding examples, was not recognised.

Given all of the above, it is fair to say that the aim of the exercise was not pedagogy.

FYI:

In computing, a system call is the programmatic way in which a computer program requests a service from the kernel of the operating system it is executed on.

David Rose Promoting Jim Martin's Ideation

David Rose wrote to sysfling on 30 June 2016 at 22:06:

To interpret relations between a technical field and its realisation as text, the challenge for linguists, as ever, is to escape the limits of grammar. Fields are realised as patterns of ideation - taxonomic relations, nuclear relations and activity sequences, organised in hierarchies of periodicity.

Blogger Comments:

[1] This blurs the distinction between stratification and instantiation.  In terms of stratification, field, the ideational dimension of context, is realised in semantics.  In terms of instantiation, text is an instance of potential.  As an instance of potential, a text realises an instance of context, namely: a context of situation.

[2] This is merely an unsolicited plug for Martin's discourse semantics.  (For a very thorough critique of this theory, see here.)  The misunderstanding here is stratificational.  Grammar and semantics are two levels of symbolic abstraction, two angles on the same phenomenon, namely: the content plane of language.  As Halliday points out, the semantics that language has is only made possible by the grammar.  The grammar doesn't just realise the semantics, it construes it.

[3] This again blurs the distinction between stratification (realisation) and instantiation (patterns).  Field is realised by semantics (stratification), system is realised by structure (axis), on each stratum.  Patterns are formed by the selection of features during the instantiation process, whereby systemic potential is actualised as text during logogenesis.

[4] As the critiques on Discourse Semantic Theory demonstrate, Martin's experiential discourse semantic system of ideation is a confusion of lexical cohesion (textual metafunction), lexis as most delicate grammar (delicacy) and logical relations between elements of clause structure (mostly misapplied).  Of his model, Martin (1992: 325) writes:

The level of discourse semantics is the least differentiated as far as ideational meaning is concerned. This is mainly due to the fact that the description developed here has focussed on relationships between experiential meanings, rather than the experiential meanings themselves.

[5] This is even inconsistent with Martin's model.  Martin (1992: 321ff, 517) regards activity sequences as an aspect of field, not ideation (experiential discourse semantics).  See a brief critique here, or other arguments critiquing this chapter at Discourse Semantic Theory.