I'm wondering in how far the introduction of the off-topic close reason pointing to the ID guide has 'helped'.
'Helped' can be interpreted in different ways, and I find it hard to describe measurable criteria, so maybe these need to be defined first.
I'm especially thinking about the value of the site for its (first time) users.
Things that come to mind:
- How much faster are they put on hold than before the introduction?
- Is the percentage of ID questions put on hold decreasing?
(That seems like the ultimate goal to me: ID question being well written so that they stay open and get answers)
- How many are edited, reopened, then answered? Again, in relation to 'before'.
(I see very little posts in the reopen review queue, actually)
- Do first time users with ID questions that are put on hold ever come back (edits, new questions, comment responses)?
- (How fast) Do questions put on hold get deleted so that they disappear out of view?
(I'm not sure if this is actually a good thing, a certain amount of [on hold] questions may put out a signal to ask good questions).
I'm not proficient in SEDE or the site data structures.
I suggest one answer each for the points in this question, with the query, the interpretation of its current result, and maybe a graph.
And please feel free to edit other points into this question or put them in answers as well.