Excellent Mike and exactly where I'm headed in my thinking. The whole 'distance travelled' concept is precisely what youth work is about. If this model can be developed slightly I plan to try it out on a small test group at a generic youth club. I want to see how this type of evaluation can be adopted by statutory agencies.
Any thoughts on how this could be used to measure impact of whole agencies? Keep up the good work!
mas
· 10 months ago
Hi Jon - the measuring of whole agencies/big projects was what we were getting into with OnTheUp. It became a bit silly really with lots of talk of how data could be captured and then reports tailored to support funders etc. to be able to assess particular aspects of things they'd funded and whether they were meeting certain needs as well as identifying unexpected outcomes. I say silly because it needs to start from a more basic development first so getting a simpler paper version would be really useful. With time and development though yes it could definitely achieve this and a lot more. One of the biggest challenges is deciding on the methods for obtaining the ratings - we looked at timelines, sms responses and so on, but actually in my experience young people generally seem to quite like doing little rating type exercises so it maybe that this is still the best option.
Will get some thoughts up about a paper based 'prototype' in the next few days.
Chris Cook
· 10 months ago
With all these ideas I would add the developers caveat - 80% of the project can be developed in 20% of the time. Starting with a very big basis for the information you want to collect through all the things you want it to do is going to lead to the same problem government departments have with big databases.
I am still thinking through some of my own thoughts about outcomes and I like where you and Jon are going - and I completely agree with what Hilary said when she says "they are built into youth work programmmes from the outset and are a product of the process".
I still keep coming back to the model used by the cubs and scouts. Badges for different skills learnt, perhaps at different levels and earned as and when the young person completes the task. The downside as you have pointed out here is the competition element where young people compare themselves against others. It has got to be about the process / journey for the young person - not a competition - although I would suggest that some element of competing will help motivate young people.
In pondering your description of your ideal application I wonder if there is some scope of developing mini-applications within facebook or bebo that produced badges for the young person. These could be visible to friends or not. The promoting site could of course track progress for all those who have registered. Being within a social media site it potentially leads to a viral adoption by users. Just thinking out loud. I'm tempted to play around with something myself!
mas
· 10 months ago
Hi Chris - I learned the truth of your first statement the hard way - three quarters of the time developing OnTheUp was getting carried away with the possibilities.
I think a 'pathway for progression' is very important and it is another thing that some good learning can be taken from the uniformed youth movements. For volunteers and Trainers we had progression from 'participant', to 'volunteer' (occasional & regular), to 'apprentice trainer', then 'trainer' and finally 'lead trainer'. Alongside this was the idea of 'competencies' - these were different areas and skills related to delivering training that young people needed to become 'competent' in - initially to be able to do, then to be able to train/support others, and then ultimately to be able to 'train trainers'. These were written out in some detail and we did initially begin an assessment method with ratings and personal diaries and each person supported by a peer mentor - but soon realised the key thing with regard to assessing competence was about having relationships and plenty of conversations. This worked well with a fairly close knit group like ours but of course would be more complicated to 'scale' or roll out for bigger organisations. That said my opinion from this is that the value lies in the skills of those leading it and if you can find good people you're better off anyway.
As for competition, I'm in favour of it. Its just that in this particular context its not really possible as the 'ratings' aren't based on any particular standards, yet there is a danger that it could be seen that there is (ie. 10 = perfect (more likely over confident!)) and so needs careful facilitation. You're quite right the perception of competition can be helpful though if managed properly.
We considered SNS applications during the Social Innovation Camp weekend and there probably is some use for them. The question is whether or not young people would use them, plus which network do you choose (are all the young people you work with regularly on one?) and so on. I've just posted up an example of how google spreadsheets could be used for capturing word clouds of skills during projects - something like this could work well if it also allowed young people to submit their responses during projects using a central form, or on a weekly emailed form, or through facebook/bebo applications etc. etc. the more options the better! What I like too about using google spreadsheets is that the data comes in to a format that most people could easily start to manipulate really easily so I'm going to explore this a bit more.
As for playing around yourself please do! I think the more ideas like this and variations/spin offs etc. the better, hopefully then more within the sector will start to see potential for using technology within their work.
Chris Cook
· 10 months ago
I must admit that I also learned the truth of that statement the hard way myself.
Before I went into fulltime youthwork my last IT role was as Project Manager for a team of 7 developers and 2 accountants writing Corporate Tax software for a large accountancy firm. Lots of great ideas but never quite panned out the way we hoped. I know lots more about developing buggy software than I do about stuff that works.
My view these days is to build things incrementally. A bit like your google spreadsheet. Try out some ideas, then knock up a new version, try again, and so on. I recall a former manager saying to me that most IT projects don't work until the third rewrite. The first time you develop something both you and the client are learning about each others business - the second time you get rid of all the mistakes you made the first time and make some new ones - by the third rewrite you actually have something that delivers.
I will have a play with developing an app for Facebook or Bebo. From what I can see the mechanism is fairly similar between the two - just slightly different calls at the front end. I think whatever is developed for young people to use it has to be fun, whizzy and not too time consuming to use.
mas
· 10 months ago
lol :-) Its the same with the offline stuff I've made - most of the resources have been through at least 3 revisions and even now I'm looking foward to the next reprint of one of them so I cam make some updates.
Look forward to your apps - am going to play some more with spreadsheets later in the week and see if I can revise any of the gadgets available - thinking a version of the word cloud as a timeline would be interesting.
mas
· 10 months ago
@Chris & Jon (and anyone else interested) is it worth me setting up a blog to keep sharing ideas & developments on this? If so can get something up & send round a log-in for anyone interested in contributing ideas for approaches to measuring outcomes/development etc.
Hilary
· 10 months ago
Hi! I'm here and listening / reading in the background! (Just really busy!) Really interested in this stuff,. The areas you're suggesting Mike in this and your follow up post are ones that I think I've mentioned before. In my experience getting young people to rate themselves and others helps them to support their thinking and is something they can 'see' more closely. In previous work I have done with young people I designed a system whereby they fed back at different stages on certain skills and competencies. This was right from the beginning to the end of the project. They scored on a graph and we looked at different scores from 1 - 10 or from :( to :). Effectively the 'scores' were just a mechanism to support their reflection and understanding of the skills and didn't really mean anything in themselves. However the key thing was that they provided a tool to work with for both the young person and their worker to build discussions on.
I still (however!) think that this can be built into accreditation schemes and that neither are exclusive (sorry to keep harping back).
I am just looking at how we use this sort of method for the young people involved in the Inspire India programme and one of the areas I'm looking at as well is video interviews and recordings. For me one of the key areas is how to make this self reflection exciting and unthreatening to young people who probably don't reflect on their own skils or development that much.
Anyways, these are evening thoughts after a busy day but I just wanted to say that I'm with you if you're setting up a blog or whatever to keep sharing info and am just off to look at your word clouds and spreadsheets to see if I can try some of it out this weekend! Thanks for pursuing this Mike!!
mas
· 10 months ago
Thanks Hilary. Maybe surprisingly I do agree that this *can* be built into in accreditation schemes. I think I've mentioned before that I would make use of accreditation schemes where appropriate - we integrated first aid courses, supported groups to use the Youth Achievement Awards and I was part of the Training Action Group for the National Association of Club for Young People helping to develop their Keystone Awards programme to gain accredited status (through the Open College Network). So I'm not completely anti-accreditation, I just think the emphasis is currently wrong.
Where a programme offers real development opportunities for young people and its possible to add in the ability to gain a certificate or qualification without having to significantly alter how that programme works I think it makes good sense to provide.
Where a programme is directed by the objectives that somebody else has set for the purposes of providing accreditation but doesn't fit with the principles of allowing young people to find their own paths I'm not in favour of it. My experience is that its very hard to find accreditation with that kind of flexibility - I know that Youth Achievement Awards and ASDAN claim to be somewhere near this but I didn't find either flexible enough for the work I was doing.
My other issue is something I think the approach suggested here tackles much better and that is what do young people actually gain? If they do a series of things they're told to do in order to gain a certificate they may or may not take in what it is they were supposed to achieve or gain. If they're encouraged to reflect and consider their own personal developments I think they come out of an experience with a much better awareness of that development - and with this much more confidence to express what this is, and therefore putting them into a stronger position in things like interview situations.
That said, again you're quite right that this *can* be achieved by using accreditation too - if there's good consideration to the process, but its not achieved if people rely simply on the (usually questionable) value of the certificate. I have similar views about a curriculum too - on the one hand I see it as something to cover up lazy, unimaginative and unskilled work - on the other hand if structured properly it makes good sense for consistency and purpose (contradictory arguments like these keep me amused for ages!)
Chris Cook
· 10 months ago
Yes - I defintely want to keep exploring this. I am very interested in the idea of linking it into timelines and it strikes me that embedding videos of part of the journey could also be useful. I've started reading the api calls for Facebook and Bebo and they don't seem too hard but it will be the time to develop and I'm thinking its going to be at least 8 solid hours of reading and experimenting to produce something as good as your Google spreadsheet and Wordcloud.
I'll have a play a bit later this week when I have some spare time - instead of trying to improve my character in World of Warcraft. I wonder if Ning's support apps as well?
Anyway - when I have something worth playing with I'll let you know.
mas
· 10 months ago
excellent - I'll get a 'play space' up in a bit :-)
Jon Jolly
· 9 months ago
Hi Mike, sorry that I ducked out the conversation for a few days! I would definitely be interested in continuing to explore this area so please get something online!
I still have thoughtful concerns over how the very nature of youth work seems to be against evaluating work, and instead relies on faith and trust in the process. I had some interesting conversations today where workers admitted to "playing the game" of evaluation to satisfy funders, but really felt that the facts and figures they provided didn't reflect the depth of their work.
Can there be a valid model of fully evaluating youth work?
Oh and what's with the switch from Disqus to Intense Debate?
masyomo
· 9 months ago
Disqus was being a bit quirky with bad links from comments (think their database got confused) so having a trial with Intense Debate (although note the Disqus plugin just got updated!)
A thought I'm having is in defining what youth work is. For example I can see that "relationship building" is hard to evaluate - but then I'd question the value of an approach based only on that, yet I think there are many who would fight strongly that that's pretty much all youth work should be about. Whereas I'd advocate a project based approach to working with young people which is much more measurable for both outputs and outcomes. Even with that though its still true that a lot of good stuff does happen beyond the project and certainly the very expensive research commissioned for the Young Movers programme still didn't capture what the programme achieved with/for young people.
Anyway will get a space up & transfer some of these debates over to it.
All Comments
Any thoughts on how this could be used to measure impact of whole agencies? Keep up the good work!
Will get some thoughts up about a paper based 'prototype' in the next few days.
I am still thinking through some of my own thoughts about outcomes and I like where you and Jon are going - and I completely agree with what Hilary said when she says "they are built into youth work programmmes from the outset and are a product of the process".
I still keep coming back to the model used by the cubs and scouts. Badges for different skills learnt, perhaps at different levels and earned as and when the young person completes the task. The downside as you have pointed out here is the competition element where young people compare themselves against others. It has got to be about the process / journey for the young person - not a competition - although I would suggest that some element of competing will help motivate young people.
In pondering your description of your ideal application I wonder if there is some scope of developing mini-applications within facebook or bebo that produced badges for the young person. These could be visible to friends or not. The promoting site could of course track progress for all those who have registered. Being within a social media site it potentially leads to a viral adoption by users. Just thinking out loud. I'm tempted to play around with something myself!
I think a 'pathway for progression' is very important and it is another thing that some good learning can be taken from the uniformed youth movements. For volunteers and Trainers we had progression from 'participant', to 'volunteer' (occasional & regular), to 'apprentice trainer', then 'trainer' and finally 'lead trainer'. Alongside this was the idea of 'competencies' - these were different areas and skills related to delivering training that young people needed to become 'competent' in - initially to be able to do, then to be able to train/support others, and then ultimately to be able to 'train trainers'. These were written out in some detail and we did initially begin an assessment method with ratings and personal diaries and each person supported by a peer mentor - but soon realised the key thing with regard to assessing competence was about having relationships and plenty of conversations. This worked well with a fairly close knit group like ours but of course would be more complicated to 'scale' or roll out for bigger organisations. That said my opinion from this is that the value lies in the skills of those leading it and if you can find good people you're better off anyway.
As for competition, I'm in favour of it. Its just that in this particular context its not really possible as the 'ratings' aren't based on any particular standards, yet there is a danger that it could be seen that there is (ie. 10 = perfect (more likely over confident!)) and so needs careful facilitation. You're quite right the perception of competition can be helpful though if managed properly.
We considered SNS applications during the Social Innovation Camp weekend and there probably is some use for them. The question is whether or not young people would use them, plus which network do you choose (are all the young people you work with regularly on one?) and so on. I've just posted up an example of how google spreadsheets could be used for capturing word clouds of skills during projects - something like this could work well if it also allowed young people to submit their responses during projects using a central form, or on a weekly emailed form, or through facebook/bebo applications etc. etc. the more options the better! What I like too about using google spreadsheets is that the data comes in to a format that most people could easily start to manipulate really easily so I'm going to explore this a bit more.
As for playing around yourself please do! I think the more ideas like this and variations/spin offs etc. the better, hopefully then more within the sector will start to see potential for using technology within their work.
Before I went into fulltime youthwork my last IT role was as Project Manager for a team of 7 developers and 2 accountants writing Corporate Tax software for a large accountancy firm. Lots of great ideas but never quite panned out the way we hoped. I know lots more about developing buggy software than I do about stuff that works.
My view these days is to build things incrementally. A bit like your google spreadsheet. Try out some ideas, then knock up a new version, try again, and so on. I recall a former manager saying to me that most IT projects don't work until the third rewrite. The first time you develop something both you and the client are learning about each others business - the second time you get rid of all the mistakes you made the first time and make some new ones - by the third rewrite you actually have something that delivers.
I will have a play with developing an app for Facebook or Bebo. From what I can see the mechanism is fairly similar between the two - just slightly different calls at the front end. I think whatever is developed for young people to use it has to be fun, whizzy and not too time consuming to use.
Look forward to your apps - am going to play some more with spreadsheets later in the week and see if I can revise any of the gadgets available - thinking a version of the word cloud as a timeline would be interesting.
I still (however!) think that this can be built into accreditation schemes and that neither are exclusive (sorry to keep harping back).
I am just looking at how we use this sort of method for the young people involved in the Inspire India programme and one of the areas I'm looking at as well is video interviews and recordings. For me one of the key areas is how to make this self reflection exciting and unthreatening to young people who probably don't reflect on their own skils or development that much.
Anyways, these are evening thoughts after a busy day but I just wanted to say that I'm with you if you're setting up a blog or whatever to keep sharing info and am just off to look at your word clouds and spreadsheets to see if I can try some of it out this weekend! Thanks for pursuing this Mike!!
Where a programme offers real development opportunities for young people and its possible to add in the ability to gain a certificate or qualification without having to significantly alter how that programme works I think it makes good sense to provide.
Where a programme is directed by the objectives that somebody else has set for the purposes of providing accreditation but doesn't fit with the principles of allowing young people to find their own paths I'm not in favour of it. My experience is that its very hard to find accreditation with that kind of flexibility - I know that Youth Achievement Awards and ASDAN claim to be somewhere near this but I didn't find either flexible enough for the work I was doing.
My other issue is something I think the approach suggested here tackles much better and that is what do young people actually gain? If they do a series of things they're told to do in order to gain a certificate they may or may not take in what it is they were supposed to achieve or gain. If they're encouraged to reflect and consider their own personal developments I think they come out of an experience with a much better awareness of that development - and with this much more confidence to express what this is, and therefore putting them into a stronger position in things like interview situations.
That said, again you're quite right that this *can* be achieved by using accreditation too - if there's good consideration to the process, but its not achieved if people rely simply on the (usually questionable) value of the certificate. I have similar views about a curriculum too - on the one hand I see it as something to cover up lazy, unimaginative and unskilled work - on the other hand if structured properly it makes good sense for consistency and purpose (contradictory arguments like these keep me amused for ages!)
I'll have a play a bit later this week when I have some spare time - instead of trying to improve my character in World of Warcraft. I wonder if Ning's support apps as well?
Anyway - when I have something worth playing with I'll let you know.
I still have thoughtful concerns over how the very nature of youth work seems to be against evaluating work, and instead relies on faith and trust in the process. I had some interesting conversations today where workers admitted to "playing the game" of evaluation to satisfy funders, but really felt that the facts and figures they provided didn't reflect the depth of their work.
Can there be a valid model of fully evaluating youth work?
Oh and what's with the switch from Disqus to Intense Debate?
A thought I'm having is in defining what youth work is. For example I can see that "relationship building" is hard to evaluate - but then I'd question the value of an approach based only on that, yet I think there are many who would fight strongly that that's pretty much all youth work should be about. Whereas I'd advocate a project based approach to working with young people which is much more measurable for both outputs and outcomes. Even with that though its still true that a lot of good stuff does happen beyond the project and certainly the very expensive research commissioned for the Young Movers programme still didn't capture what the programme achieved with/for young people.
Anyway will get a space up & transfer some of these debates over to it.