[OH-Dev] Project Dashboard / Monitoring : Training Missions

[Asheesh Laroia]

On Tue, 8 Jan 2013, konark modi wrote:

> Hi Asheesh,

Hi Konark! Sorry about the delay in responding.

You are very very welcome to join us, and so please really do accept my 
apology with the delay.

> Thank you very much for letting me an opportunity to contribute in the OpenHatch project. I would
> like to divide this e-mail into the following parts :
> A. What have I understood about Training Missions since we first discussed.
> B. What are the requirement and how do we go about them .
> 
> A. My Understanding of Training missions :
> I assume that there are 'n' number of training missions , and each training mission has 'n' of
> steps. Now given that we would like to have a dashboard layer, which helps us monitor our training
> missions and how our users are performing.

Yup. This understand is accurate, if vague, so let me see if I can clarify 
it.

The missions are listed here (I'm listing github.com URLs, but you can 
follow along in your own "git clone" of the repository on your own 
computer as well):

https://github.com/openhatch/oh-mainline/blob/master/mysite/missions/fixtures/initial_data.json

I guess that's just the step identifiers. But if you take the strings 
there and split on "_" and take only the first part, you end up with a 
name you can use for the mission.

So the "tar" mission has two steps: "tar" and "tar_extract".

Similarly, the most recent commit to that file added the "git_config" 
step.

These are listed in a basically arbitrary order; git_config is first in 
the UI, but last in the list.

> I was able to find two tables 'mission_steps' and 'mission_steps_completion'. I am not able to
> link the steps within each training mission.

The above should let you link steps.

In general, mission_step AKA mysite.missions.models.Step lists the steps 
available to complete, and only really exists so that StepCompletion (AKA 
steps_completion) can use a foreign key rather than a string.

Does that help with linking the steps?

> B. Requirements :
> 1. What are the desired data points : Example :
> a. Average number of steps completed per user.
> b. Per training mission  average %age of steps completed.
> c. Average time taken to move from one step to other within a training mission.
> d. Quickest mission completed by average users.
> e. User can see how many other users are on the same step / mission.
> 
> There can be lot of more...

So I'm somewhat torn! Should the dashboard let us see users' usernames, 
or not?

I also somewhat suspect we will be somewhat disappointed when we see the 
stats, that fewer people use *lots and lots* of missions than we were 
hoping.

Regardless, I think it's extremely worth having these figures.

Anyway, so I think for an initial dashboard, here's what I think would be 
great:

Example things to show:

(these numbers are totally made-up)

For each mission, show, for each step, # of people who have done that step

Tar:

* Total people who have solved any steps: 1250
* tar: 1200 (96%)
* tar_extract: 80 (6.4%)

It's 1250 because (for example) of the 80 people who did tar_extract, only 
30 actually overlap with people who did the "tar" step.

The above can calculated based on StepCompletion, ignoring the date value.

It'd be superb to also show % of steps, like you said, for each mission!

As for how to display it... we could use a simple, boring text report, 
like http://openhatch.org/+meta/ but if you have any other ideas that 
could be splendid. I'd say a text version would be fine for an initial go.

Later, we could increase how nice it looks by using (say) a JavaScript 
charting library. We could export data to the charting library by creating 
exporting it as JSON from Django code, either directly from a Django view, 
or (better) by using the "api.py" convention that we already 
have -- look at mysite/customs/api.py for an example.

I really like the idea of users seeing how many other users are on the 
same step! I think my preference for what to show is, Number of people 
who have completed a step (while you're looking at that step page), to 
encourage you to finish it.

Perhaps at the opening of every training mission we can tell you that N 
other people have completed all the steps, to encourage you to complete 
all the steps.

I'd really love to be able to run experiments to e.g. find out if adding 
the # of people who have completed all the steps increases people's 
getting all the way through. I'd be OK with running experiments by just 
changing something, waiting a week, and seeing if the rates are the same 
as for the week before.

For that reason, a future addition might let you configure which dates to 
show in the report.

> 2. Frequency of updating the dashboard.
> 3. Public or private view of dashboard .
> 4. Any example you think can be inspiring.

For (2): I was thinking we can have the dashboard be generated by Django, 
and therefore be up-to-date when you load it.

For (3): I was thinking there should definitely be a public view. For the 
above, we don't display private user information, so I think all the above 
can be public.

For (4): See above. (-:

Let me know what you think!

-- Asheesh.